BioBloom tools: fast, accurate and memory-efficient host species sequence screening using bloom filters

A formula is read-once if each variable appears at most once in it. An arithmetic read-once formula is one in which the operators are addition, subtraction, multiplication, and division. We present polynomial time algorithms for exact learning of arithmetic read-once formulas over a field. We present a membership and equivalence query algorithm that identifies arithmetic read-once formulas over an arbitrary field. We present a randomized membership query algorithm (i.e., a randomized black box interpolation algorithm) that identifies such formulas over finite fields with at least $2n + 5$ elements (where n is the number of variables) and over infinite fields. We also show the existence of nonuniform deterministic membership query algorithms for arbitrary read-once formulas over fields of characteristic 0, and division-free read-once formulas over fields that have at least $2n^{3} + 1$ elements. For our algorithms, we assume we are able to perform efficiently arithmetic operations on field elements and compute square roots in the field. It is shown that the ability to compute square roots is necessary in the sense that the problem of computing $n - 1$ square roots in a field can be reduced to the problem of identifying an arithmetic formula over n variables in that field. Our equivalence queries are of a slightly nonstandard form, in which counterexamples are required not to be inputs on which the formula evaluates to $0/0$. This assumption is shown to be necessary for fields of size $o(n/ \log n)$ in the sense that we prove there exists no polynomial time identification algorithm that uses only membership and standard equivalence queries.

This paper concentrates on the issue of detecting abnormal flows in distributed monitoring systems, which has many network management applications such as anomaly detection and traffic engineering. Collecting massive network traffic in real‐time remains a large challenge due to the limited system resource. Most existing approaches perform abnormal flow detection at one measurement point, while they cause large computation and memory overhead for recovering abnormal flows. In this paper, we propose a novel data streaming method that supports accurate abnormal flow detection with a low memory requirement. The key idea of our method is that each monitor compresses flow information to summary data structure, sends the generated data structure to the controller; then the controller aggregates the received data structures, recovers candidates of abnormal flows and estimates their size and change to find abnormal flows on the basis of the aggregated data structure. The experimental results based on real network traffic show that the proposed approach can detect up to 97% of abnormal flows with low memory and update requirements in comparison with related approaches.

Hierarchical packet classification using a Bloom filter and rule-priority tries

We study the use of sparse structured associative memories as a memory-efficient and computationally-efficient data structure for representing a set of elements when one wishes to perform set-membership queries and some errors (false positives) are tolerable. Associative memories, when viewed as representing a set, enjoy a number of interesting properties, including that set membership queries can be carried out even when the input (query element) is only partially known or is partially corrupted. The associative memories considered here (initially proposed in [Gripon and Berrou, 2011]) encode the set in the edge structure of a graph. In this paper we generalize this construction to encode the set in the edge structure of a hypergraph. We derive bounds on the false positive rates (the probability that the associative memory erroneously declares that an element is in the stored set when it, in fact, was not). Interestingly, the proposed structures enjoy many of the same properties as Bloom filters (e.g., they have zero false negative rate, the time to perform an insert and lookup does not depend on the number of elements stored, and the false positive rate can be reduced by using additional memory for storage), while also offering the properties of associative memories (allowing for queries on partial or corrupted inputs).

Bloom filter is a simple space-efficient randomized data structure for representing a set in order to support membership queries, which uses an m-bit array to represent a data set. Dynamic bloom filter (DBF) can support concisely representation and approximate membership queries of dynamic set instead of static set. It has been proved that DBF not only possess the advantage of standard bloom filter, but also has better features when dealing with dynamic set. But DBF also has a disadvantage: the addition operation which mapped element x into bloom filter s will become no sense, if some of the first s-1 bloom filters have already responded that element x is in set A with some false positive probability. We point out this shortcoming and improve the addition operation with a new algorithm. We call this improved dynamic bloom filter i-DBF. Finally, we prove that this i-DBF has better performance both in the storage space and in the false positive probability

Bloom filter and its variants are elegant space-efficient probabilistic data structures for approximate set membership queries. It has been recently shown that the space cost of Bloom filters can be significantly reduced via a combination with pre-trained machine learning models, named Learned Bloom filters (LBF). LBF eases the space requirement of a Bloom filter by undertaking part of the queries using a classifier. However, current LBF structures generally target a static member set. Their performances would inevitably decay when there is a member update on the set, while this update requirement is not uncommon for real-world data streaming applications such as duplicate item detection, malicious URL checking, and web caching. To adapt LBF to data streams, we propose the Stable Learned Bloom Filters (SLBF) which addresses the performance decay issue on intensive insertion workloads by combining classifier with updatable backup filters. Specifically, we propose two SLBF structures, Single SLBF (s-SLBF) and Grouping SLBF (g-SLBF). The theoretical analysis on these two structures shows that the expected false positive rate (FPR) of SLBF is asymptotically a constant over the insertion of new members. Extensive experiments on real-world datasets show that SLBF introduces a similar level of false negative rate (FNR) but yields a better FPR/storage trade-off compared with the state-of-the-art (non-learned) Bloom filters optimized on data streams.

MANET, a self-organizing, infrastructure-less, wireless network is a fast-growing technology in day-to-day life. There is a rapid growth in the area of mobile computing due to the extent of economical and huge availability of wireless devices which leads to the extensive analysis of the mobile ad-hoc network. It consists of the collection of wireless dynamic nodes. Due to this dynamic nature, the routing of packets in the MANET is a complex one. The integration of distributed hash table (DHT) in MANET is performed to enhance the overlay of routing. The node status updating in the centralized hash table creates the storage overhead. The bloom filter is a data structure that is a space-effective randomized one but it allows the false-positive rates. However, this can be able to compensate for the issue of storage overhead in DHT (Distributed hash table). Hence, to overcome the storage overhead occurring in DHT, and reduce the false positives, the Bloom's filter is integrated with the DHT initially. Furthermore, the link stability is measured by the distance among mobile nodes. The optimal node selection should be done for the transmission of packets which is the lacking factor. If it fails to select the optimal path then the removal of malicious nodes may lead to the unwanted entry of nodes into the other clustering groups. Therefore, to solve this problem, the bloom's filter is modified for enhancing the link stability. The novelty of this proposed work is the integration of Bloom's filter with the Distributed Hash Table which provides good security on transmission data by removing false-positive errors and storage overhead.

Set is widely used as a kind of basic data structure. However, when it is used for large scale data set the cost of storage, search and transport is overhead. The bloom filter uses a fixed size bit string to represent elements in a static set, which can reduce storage space and search cost that is a fixed constant. The time-space efficiency is achieved at the cost of a small probability of false positive in membership query. However, for many applications the space savings and locating time constantly outweigh this drawback. Dynamic bloom filter (DBF) can support concisely representation and approximate membership queries of dynamic set instead of static set. It has been proved that DBF not only possess the advantage of standard bloom filter, but also has better features when dealing with dynamic set. This paper proposes a time-dependent multiple bit-strings bloom filter (TMBF) which roots in the DBF and targets on dynamic incremental set. TMBF uses multiple bit-strings in time order to present a dynamic increasing set and uses backward searching to test whether an element is in a set. Based on the system logs from a real P2P file sharing system, the evaluation shows a 20% reduction in searching cost compared to DBF.

Bloom filters are probabilistic data structures that are popular in networking for set representation; however, they show an inherent inaccuracy due to false positives. One of the potential attacks on Bloom filters is to pollute them with elements that cause the filter to have a larger false positive probability than under normal operation; Pollution is simple when an attacker knows the details of the filter implementation. Recent research has shown that also black-box adversaries can pollute a counting Bloom filter (a common variant of the filter that also supports removals) with no knowledge of its implementation. As over time, many variants and improvements of Bloom filters have been proposed, it is of interest to study whether they can also be polluted and if so also the increase in their false positive probability. This paper first proposes and then evaluates pollution attacks for some of the most common variants including the Block Bloom filters (BBFs), the Variable Increment and Fingerprint Counting Bloom filters (VI-CBFs and FP-CBFs). The results show that with or without knowledge of the implementation, these variants of the Bloom filter are significantly more vulnerable to pollution attacks than the traditional Bloom filter. In particular, BBFs are extremely vulnerable, so providing an insight on their impact and use in practical systems when the number of memory accesses per lookup must be reduced.

Scalable Bloom Filters

As a key component in implementing Named Data Networking (NDN), Pending Interest Table (PIT) requires an efficient exact-matching algorithm for a scalable and fast PIT lookup. A Bloom filter (BF) is a memory-efficient data structure for performing exact matching operations. In this paper, three different BF-based PIT architectures are proposed: PIT using functional Bloom filters (FBF-PIT), PIT using counting Bloom filters with return values (rCBF-PIT), and a refined rCBF-PIT with signatures (R-rCBF-PIT). The proposed BF-based PITs incrementally allocate a new BF for storing multiple incoming faces of Interest packets with the same content name. For a Data packet lookup, the proposed PIT architectures simultaneously access every BF structure to find matching faces and delete the faces (i.e., matching Interest packet information). The functional Bloom filter (FBF) used in an FBF-PIT is a key-value data structure that stores values only without keys. However, because the number of non-reusable conflict cells in the FBF increases as the number of stored packets increases in the FBF-PIT, the indeterminable rate increases. To decrease the indeterminable rate, we propose the rCBF-PIT, which uses counting Bloom filters with return values (rCBFs), allowing reusable conflict cells. False positives for Interest packets lead to incorrect deletions that can cause false negatives for incoming Data packets. Because most of the false positives occur in the first BF structure, we finally propose the R-rCBF-PIT, in which the first rCBF is replaced with an rCBF with a signature field. The proposed PITs also provide an aging mechanism using a valid bit and a hit bit for entry expiration. Simulation results show that rCBF-PIT and R-rCBF-PIT both reduce the indeterminable rate by more than 81% compared with FBF-PIT. The results also show that R-rCBF-PIT resolves false negatives caused by incorrect deletions by including the signature fields in the first rCBF.

Improving retouched Bloom filter for trading off selected false positives against false negatives

Motivation: Approximate membership query (AMQ) structures such as Cuckoo filters or Bloom filters are widely used for representing large sets of elements. Their lightweight space usage explains their success, mainly as they are the only way to scale hundreds of billions or trillions of elements. However, they suffer by nature from non-avoidable false-positive calls that bias downstream analyses of methods using these data structures.Results: In this work we propose a simple strategy and its implementation for reducing the false-positive rate of any AMQ data structure indexing \(k\)-mers (words of length k). The method we propose, called findere, enables to speed-up the queries by a factor two and to decrease the false-positive rate by two order of magnitudes. This achievement is done on the fly at query time, without modifying the original indexing data-structure, without generating false-negative calls and with no memory overhead.This method yields so-called “construction false positives”, but the amount of such false positives is negligible when the method is used within classical parameter ranges. This method, as simple as effective, reduces either the false-positive rate or the space required to represent a set given a user-defined false-positive rate.Availability: https://github.com/lrobidou/findere.

Bloom filters have been employed in various fields because of its simple and effective structure in identifying the membership of an input. Since a Bloom filter can produce false positives, the positive results of a Bloom filter should be identified whether the positives are true or not by accessing the original database. A complement Bloom filter (C-BF) was introduced to identify the true positiveness of a given Bloom filter without accessing the original database. A critical problem of the C-BF is that every element included in the complement set of the given set should be programmed into the C-BF. Since the number of elements included in the complement set can be considerably large, the C-BF would require the significant amount of memory. In this paper, we claim that the elements that produce negative results from the given Bloom filter are not necessarily programmed into the C-BF, since Bloom filters never produce false negatives. In other words, we propose the Petit-BF (P-BF) which programs only the elements that cause false positives from the given Bloom filter. Simulation results and theoretical analysis show that the proposed method can achieve the same performance using a considerably smaller amount of memory.

Bloom filter is a compact and randomized data structure for representation of a set, in order to answer membership queries; Queries such as Is element x in set Y? There is no need to have set Y, for answering these kinds of questions; these questions can be answered by constructed Bloom filter from the set. Therefore, space saving is the main advantage of using Bloom filter. In this paper, first general structure of Bloom filter is explained and then by using the Bloom filter, a new method for error management and detection in computer networks is presented. In this approach, the sender makes a Bloom filter for the flow data that is to be sent and attaches this Bloom filter to data flow and sends them to receiver. The receiver by receiving the data flow and Bloom filter, makes the Bloom filter corresponding to data flow, then by comparing this Bloom filter with the received Bloom filter detects the possible errors. The results show that Bloom filter with the length of x input string is able to detects all errors except for errors with the length of exactly 2x, 4x and... etc.