Efficient join processing on uncertain data streams

Frequent itemsets (FIs) mining from uncertain database is a very popular research area nowadays. Many algorithms have been proposed to mine FI from uncertain database. But in typical FI mining process, all the FIs have to be mined individually, which needs a huge memory. Four trees are proposed in this paper which are: (i) maximal frequent itemset from uncertain database (MFU) tree which contains only the maximal frequent itemsets generated from uncertain database, (ii) closed frequent itemset from uncertain database (CFU) tree which contains only closed frequent itemsets generated from uncertain database, (iii) maximal frequent itemset from uncertain data stream (MFUS) tree which contains maximal frequent itemsets generated from uncertain data stream and (iv) closed frequent itemset from uncertain data stream (CFUS) tree which contains closed frequent itemsets generated from uncertain data stream. Experimental results are also presented which show that maximal and closed frequent itemsets mining requires less time and memory than typical frequent itemsets mining.

Outlier detection on data streams is an important task in data mining. The challenges become even larger when considering uncertain data. This paper studies the problem of outlier detection on uncertain data streams. We propose Continuous Uncertain Outlier Detection (CUOD), which can quickly determine the nature of the uncertain elements by pruning to improve the efficiency. Furthermore, we propose a pruning approach — Probability Pruning for Continuous Uncertain Outlier Detection (PCUOD) to reduce the detection cost. It is an estimated outlier probability method which can effectively reduce the amount of calculations. The cost of PCUOD incremental algorithm can satisfy the demand of uncertain data streams. Finally, a new method for parameter variable queries to CUOD is proposed, enabling the concurrent execution of different queries. To the best of our knowledge, this paper is the first work to perform outlier detection on uncertain data streams which can handle parameter variable queries simultaneously. Our methods are verified using both real data and synthetic data. The results show that they are able to reduce the required storage and running time.

Uncertainty is inherent in data streams, and present new challenges to data streams mining. For continuous arriving and large size of data streams, modeling sequences of uncertain time series data streams require significantly more space. Therefore, it is important to construct compressed representation for storing uncertain time series data. Based on granules, sequential sketches are created to store hash-compressed granules. And based on sliding windows, a sketch update strategy is given to store most resent granules. As the sequential sketches may be saturated with the increasing of data streams, this paper presents an optimization strategy to delete the absolute sparse patterns. Based on the sequential sketches, a sequential pattern mining algorithm is proposed for mining uncertain data streams. The experimental results illustrate the effectiveness of the pattern mining algorithm.

Recently, data mining over uncertain data streams has attracted a lot of attentions because of the widely existed imprecise data generated from a variety of streaming applications. In this paper, we try to resolve the problem of clustering over uncertain data streams. Facing uncertain tuples with different probability distributions, the clustering algorithm should not only consider the tuple value but also emphasis on its uncertainty. To fulfill these dual purposes, a metric named tuple uncertainty will be integrated into the overall procedure of clustering. Firstly, we survey uncertain data model and propose our uncertainty measurement and corresponding properties. Secondly, based on such uncertainty quantification method, we provide a two phase stream clustering algorithm and elaborate implementation detail. Finally, performance experiments over a number of real and synthetic data sets demonstrate the effectiveness and efficiency of our method.

In order to improve the monitoring ability of uncertain large data stream, an uncertain large data flow monitoring algorithm based on artificial intelligence is proposed. The collected uncertain big data flow is constructed by low dimensional feature set, and the rough set model of uncertain large data stream distribution is constructed. The fuzzy C-means clustering method is used to analyze the uncertain big data flow by fusion clustering and adaptive grid partition analysis. All the abnormal samples of large data stream are sampled and trained, and the feature quantities of association rules of uncertain large data stream are extracted. Combined with artificial intelligence method, the monitoring of uncertain large data stream is realized. The simulation results show that the method has high accuracy and good ability to resist abnormal traffic interference, and the traffic security monitoring ability of the network is improved.

Two birds with one stone: Classifying positive and unlabeled examples on uncertain data streams

Probabilistic distance based abnormal pattern detection in uncertain series data

This paper proposes a grid-based clustering algorithm Clu-US which is competent to find clusters of non-convex shapes on uncertain data stream. Clu-US maps the uncertain data tuples to the grid space which could store and update the summary information of stream. The uncertainty of data is taken into account for calculating the probability center of a grid. Then, the distance between the probability centers of two adjacent grids is adopted for measuring whether they are close enough in grids merging process. Furthermore, a dynamic outlier deletion mechanism is developed to improve clustering performance. The experimental results show that Clu-US outperforms other algorithms in terms of clustering quality and speed.

Recently, some approaches of finding probabilistic skylines on uncertain data have been proposed. In these approaches, a data object is composed of instances, each associated with a probability. The probabilistic skyline is then defined as a set of non-dominated objects with probabilities exceeding or equaling a given threshold. In many applications, data are generated as a form of continuous data streams. Accordingly, we make the first attempt to study a problem of continuously returning probabilistic skylines over uncertain data streams in this paper. Moreover, the sliding window model over data streams is considered here. To avoid recomputing the probability of being not dominated for each uncertain object according to the instances contained in the current window, our main idea is to estimate the bounds of these probabilities for early determining which objects can be pruned or returned as results. We first propose a basic algorithm adapted from an existing approach of answering skyline queries on static and certain data, which updates these bounds by repeatedly processing instances of each object. Then, we design a novel data structure to keep dominance relation between some instances for rapidly tightening these bounds, and propose a progressive algorithm based on this new structure. Moreover, these two algorithms are also adapted to solve the problem of continuously maintaining top-k probabilistic skylines. Finally, a set of experiments are performed to evaluate these algorithms, and the experiment results reveal that the progressive algorithm much outperforms the basic one, directly demonstrating the effectiveness of our newly designed structure.

Modeling of time-varying dynamic systems in real time requires the use of streams of sensor data and incremental learning algorithms. This paper introduces an incremental fuzzy modeling approach based on uncertain data streams. By uncertain data we mean data originated from unreliable sensors, imprecise perception, or description of the value of a variable represented as a fuzzy interval. An online incremental learning algorithm is used to develop the antecedent part of functional fuzzy rules and the rule base that assembles the model. A recursive least squares-like algorithm updates the parameters of a discrete state-space representation of the fuzzy rule consequents. Data uncertainty is accounted for using specificity measures of the input data. An illustrative example concerning the Lorenz attractor is given.

This paper proposes a novel naïve Bayesian classifier in categorical uncertain data streams. Uncertainty in categorical data is usually represented by vector valued discrete pdf, which has to be carefully handled to guarantee the underlying performance in data mining applications. In this paper, we map the probabilistic attribute to deterministic points in the Euclidean space and design a distance based and a density based algorithms to measure the correlations between feature vectors and class labels. We also devise a new pre-binning approach to guarantee bounded computation and memory cost in uncertain data streams classification. Experimental results in real uncertain data streams prove that our density-based naive classifier is efficient, accurate, and robust to data uncertainty.

Uncertain data streams can have tuples with both value and existential uncertainty. A tuple has value uncertainty when it can assume multiple possible values. A tuple is existentially uncertain when the sum of the probabilities of its possible values is $$<$$ 1. A situation where existential uncertainty can arise is when applying relational operators to streams with value uncertainty. Several prior works have focused on querying and mining data streams with both value and existential uncertainty. However, none of them have studied, in depth, the implications of existential uncertainty on sliding window processing, even though it naturally arises when processing uncertain data. In this work, we study the challenges arising from existential uncertainty, more specifically the management of count-based sliding windows, which are a basic building block of stream processing applications. We extend the semantics of sliding window to define the novel concept of uncertain sliding windows and provide both exact and approximate algorithms for managing windows under existential uncertainty. We also show how current state-of-the-art techniques for answering similarity join queries can be easily adapted to be used with uncertain sliding windows. We evaluate our proposed techniques under a variety of configurations using real data. The results show that the algorithms used to maintain uncertain sliding windows can efficiently operate while providing a high-quality approximation in query answering. In addition, we show that sort-based similarity join algorithms can perform better than index-based techniques (on 17 real datasets) when the number of possible values per tuple is low, as in many real-world applications.

With the rapid development of network data communication technology, the complexity of the network environment makes the data in the data stream have uncertain characteristics, and a large number of data streams are generated in many fields. Therefore, it is necessary to find an efficient and accurate processing method to process a large amount of data in a computer network. This paper takes the skyline probability calculation and the actual efficiency of the unknown object index structure in the skyline query processing technology on the uncertain data stream as the research object. Based on the skyline query technology, we study an efficient skyline query processing method on the unknown network traffic based on the model SUMG. The method includes two algorithms: dynamic modeling algorithm DMG and skyline query algorithm GST. The DMG algorithm samples the data in the sliding window of the uncertain data stream and establishes a model to convert the data stream into the parameter stream in the uncertain object probability density function, the GST algorithm establishes the R-tree index structure, which is in order to reduce the amount of calculation, it uses the parameter flow of the model. In both methods, the set of local skyline results is first obtained at the distributed node, and the skyline query is performed again on the union of the local skyline results to obtain the global skyline result set. The experimental results show that compared with the skyline query method BNL for unknown network traffic without an index structure, the SUMG method can not only effectively model the link-type unknown object to assist the skyline query, but also effectively prune the uncertain data object and improve the skyline. The distributed skyline query method can cope with the skyline query task on the distributed data stream.

In the modern age of information explosion, everyone can easily obtain all kinds of data, so how to find the most valuable information in massive data has become an important issue. In general, most data collected from the applications of Internet of Things (IoT) become uncertain since there is a probability or part of the data is missing. However, the calculation of the uncertain data will be much more complicated than certain (or deterministic) data. As a result, the performance of uncertain data handling becomes a significant challenge in meeting low latency requirements. In this work, we propose a distributed computing algorithm and apply it to an edge computing to calculate the probabilistic top-k dominating (PTKD) objects of uncertain data. The overall latency of PTKD query processing is significantly reduced. The main idea of this method is to reduce the cost of time without unnecessary calculations of objects. Experiments show that the proposed algorithm can improve 58% latency on average. With a high pruning rate, performance can be reduced by up to 92%.

The skyline query processing over uncertain data streams has received considerable attention, due to its importance in helping users make intelligent decisions over complex data. Nevertheless, existing studies only focus on retrieving the skylines over data streams in a centralized environment typically with one processor, which limits the scalability of algorithms and cannot meet the requirement for massive data analysis. The emerging cloud computing environment provides much more reliable and stable environments than the traditional distributed environments, which can be well adapted to the massive data management and complex queries. Unfortunately, existing parallel frameworks in clouds such as MapReduce and its variants are not suitable for the skyline queries over uncertain data streams. In this paper, we propose a general framework for parallelizing the probabilistic streaming skyline operator with the sliding window partitioning. Particularly, we propose four items mapping strategies CMS, AMS, DMS and APS to optimize the queries based on the proposed parallel framework. Extensive experiments with real deployment are conducted to demonstrate the effectiveness and efficiency of the proposals.