Parallelizing Probabilistic Streaming Skyline Operator in Cloud Computing Environments

Recently, some approaches of finding probabilistic skylines on uncertain data have been proposed. In these approaches, a data object is composed of instances, each associated with a probability. The probabilistic skyline is then defined as a set of non-dominated objects with probabilities exceeding or equaling a given threshold. In many applications, data are generated as a form of continuous data streams. Accordingly, we make the first attempt to study a problem of continuously returning probabilistic skylines over uncertain data streams in this paper. Moreover, the sliding window model over data streams is considered here. To avoid recomputing the probability of being not dominated for each uncertain object according to the instances contained in the current window, our main idea is to estimate the bounds of these probabilities for early determining which objects can be pruned or returned as results. We first propose a basic algorithm adapted from an existing approach of answering skyline queries on static and certain data, which updates these bounds by repeatedly processing instances of each object. Then, we design a novel data structure to keep dominance relation between some instances for rapidly tightening these bounds, and propose a progressive algorithm based on this new structure. Moreover, these two algorithms are also adapted to solve the problem of continuously maintaining top-k probabilistic skylines. Finally, a set of experiments are performed to evaluate these algorithms, and the experiment results reveal that the progressive algorithm much outperforms the basic one, directly demonstrating the effectiveness of our newly designed structure.

With the rapid development of network data communication technology, the complexity of the network environment makes the data in the data stream have uncertain characteristics, and a large number of data streams are generated in many fields. Therefore, it is necessary to find an efficient and accurate processing method to process a large amount of data in a computer network. This paper takes the skyline probability calculation and the actual efficiency of the unknown object index structure in the skyline query processing technology on the uncertain data stream as the research object. Based on the skyline query technology, we study an efficient skyline query processing method on the unknown network traffic based on the model SUMG. The method includes two algorithms: dynamic modeling algorithm DMG and skyline query algorithm GST. The DMG algorithm samples the data in the sliding window of the uncertain data stream and establishes a model to convert the data stream into the parameter stream in the uncertain object probability density function, the GST algorithm establishes the R-tree index structure, which is in order to reduce the amount of calculation, it uses the parameter flow of the model. In both methods, the set of local skyline results is first obtained at the distributed node, and the skyline query is performed again on the union of the local skyline results to obtain the global skyline result set. The experimental results show that compared with the skyline query method BNL for unknown network traffic without an index structure, the SUMG method can not only effectively model the link-type unknown object to assist the skyline query, but also effectively prune the uncertain data object and improve the skyline. The distributed skyline query method can cope with the skyline query task on the distributed data stream.

Frequent itemsets (FIs) mining from uncertain database is a very popular research area nowadays. Many algorithms have been proposed to mine FI from uncertain database. But in typical FI mining process, all the FIs have to be mined individually, which needs a huge memory. Four trees are proposed in this paper which are: (i) maximal frequent itemset from uncertain database (MFU) tree which contains only the maximal frequent itemsets generated from uncertain database, (ii) closed frequent itemset from uncertain database (CFU) tree which contains only closed frequent itemsets generated from uncertain database, (iii) maximal frequent itemset from uncertain data stream (MFUS) tree which contains maximal frequent itemsets generated from uncertain data stream and (iv) closed frequent itemset from uncertain data stream (CFUS) tree which contains closed frequent itemsets generated from uncertain data stream. Experimental results are also presented which show that maximal and closed frequent itemsets mining requires less time and memory than typical frequent itemsets mining.

Uncertainty is inherent in data streams, and present new challenges to data streams mining. For continuous arriving and large size of data streams, modeling sequences of uncertain time series data streams require significantly more space. Therefore, it is important to construct compressed representation for storing uncertain time series data. Based on granules, sequential sketches are created to store hash-compressed granules. And based on sliding windows, a sketch update strategy is given to store most resent granules. As the sequential sketches may be saturated with the increasing of data streams, this paper presents an optimization strategy to delete the absolute sparse patterns. Based on the sequential sketches, a sequential pattern mining algorithm is proposed for mining uncertain data streams. The experimental results illustrate the effectiveness of the pattern mining algorithm.

Two birds with one stone: Classifying positive and unlabeled examples on uncertain data streams

Outlier detection on data streams is an important task in data mining. The challenges become even larger when considering uncertain data. This paper studies the problem of outlier detection on uncertain data streams. We propose Continuous Uncertain Outlier Detection (CUOD), which can quickly determine the nature of the uncertain elements by pruning to improve the efficiency. Furthermore, we propose a pruning approach — Probability Pruning for Continuous Uncertain Outlier Detection (PCUOD) to reduce the detection cost. It is an estimated outlier probability method which can effectively reduce the amount of calculations. The cost of PCUOD incremental algorithm can satisfy the demand of uncertain data streams. Finally, a new method for parameter variable queries to CUOD is proposed, enabling the concurrent execution of different queries. To the best of our knowledge, this paper is the first work to perform outlier detection on uncertain data streams which can handle parameter variable queries simultaneously. Our methods are verified using both real data and synthetic data. The results show that they are able to reduce the required storage and running time.

Recently, data mining over uncertain data streams has attracted a lot of attentions because of the widely existed imprecise data generated from a variety of streaming applications. In this paper, we try to resolve the problem of clustering over uncertain data streams. Facing uncertain tuples with different probability distributions, the clustering algorithm should not only consider the tuple value but also emphasis on its uncertainty. To fulfill these dual purposes, a metric named tuple uncertainty will be integrated into the overall procedure of clustering. Firstly, we survey uncertain data model and propose our uncertainty measurement and corresponding properties. Secondly, based on such uncertainty quantification method, we provide a two phase stream clustering algorithm and elaborate implementation detail. Finally, performance experiments over a number of real and synthetic data sets demonstrate the effectiveness and efficiency of our method.

In the modern age of information explosion, everyone can easily obtain all kinds of data, so how to find the most valuable information in massive data has become an important issue. In general, most data collected from the applications of Internet of Things (IoT) become uncertain since there is a probability or part of the data is missing. However, the calculation of the uncertain data will be much more complicated than certain (or deterministic) data. As a result, the performance of uncertain data handling becomes a significant challenge in meeting low latency requirements. In this work, we propose a distributed computing algorithm and apply it to an edge computing to calculate the probabilistic top-k dominating (PTKD) objects of uncertain data. The overall latency of PTKD query processing is significantly reduced. The main idea of this method is to reduce the cost of time without unnecessary calculations of objects. Experiments show that the proposed algorithm can improve 58% latency on average. With a high pruning rate, performance can be reduced by up to 92%.

Join processing in the streaming environment has many practical applications such as data cleaning and outlier detection. Due to the inherent uncertainty in the real-world data, it has become an increasingly important problem to consider the join processing on uncertain data streams, where the incoming data at each timestamp are uncertain and imprecise. Different from the static databases, processing uncertain data streams has its own requirements such as the limited memory, small response time, and so on. To tackle the challenges with respect to efficiency and effectiveness, in this paper, we formalize the problem of join on uncertain data streams (USJ), which can guarantee the accuracy of USJ answers over uncertain data, and propose effective pruning methods to filter out false alarms. We integrate the pruning methods into an efficient query procedure for incrementally maintaining USJ answers. Extensive experiments have been conducted to demonstrate the efficiency and effectiveness of our approaches.

This paper proposes a novel naïve Bayesian classifier in categorical uncertain data streams. Uncertainty in categorical data is usually represented by vector valued discrete pdf, which has to be carefully handled to guarantee the underlying performance in data mining applications. In this paper, we map the probabilistic attribute to deterministic points in the Euclidean space and design a distance based and a density based algorithms to measure the correlations between feature vectors and class labels. We also devise a new pre-binning approach to guarantee bounded computation and memory cost in uncertain data streams classification. Experimental results in real uncertain data streams prove that our density-based naive classifier is efficient, accurate, and robust to data uncertainty.

An effective query processing plays an important role in the uncertain data streams. Specifically, multiple top-k queries processing on uncertain data streams obtained from large applications of several fields such as sensor network monitoring and internet traffic control requires periodic execution of queries and sharing results among them. The system that monitors uncertain events in such data streams manipulates the top-k queries. Here the problem is that systems were not designed to allow query results sharing which in turn leads to high computation cost and inaccurate response from the system. To overcome these issues (Queries results sharing), the system using a sampling algorithm for sample the top possible worlds from well-known possible worlds based on their high probability. System uses an optimal dynamic programming approach that split the multiple queries into number of groups. Then the query groups are scheduled and planned for sharing results to yield minimum computation cost. A faster greedy algorithm is used to reduce the time and storage space of the top-k queries based on the greedy rule. Thus the proposed approach allows sharing computation among multiple top-k queries and generates best plan of query execution.

Uncertain data streams can have tuples with both value and existential uncertainty. A tuple has value uncertainty when it can assume multiple possible values. A tuple is existentially uncertain when the sum of the probabilities of its possible values is $$<$$ 1. A situation where existential uncertainty can arise is when applying relational operators to streams with value uncertainty. Several prior works have focused on querying and mining data streams with both value and existential uncertainty. However, none of them have studied, in depth, the implications of existential uncertainty on sliding window processing, even though it naturally arises when processing uncertain data. In this work, we study the challenges arising from existential uncertainty, more specifically the management of count-based sliding windows, which are a basic building block of stream processing applications. We extend the semantics of sliding window to define the novel concept of uncertain sliding windows and provide both exact and approximate algorithms for managing windows under existential uncertainty. We also show how current state-of-the-art techniques for answering similarity join queries can be easily adapted to be used with uncertain sliding windows. We evaluate our proposed techniques under a variety of configurations using real data. The results show that the algorithms used to maintain uncertain sliding windows can efficiently operate while providing a high-quality approximation in query answering. In addition, we show that sort-based similarity join algorithms can perform better than index-based techniques (on 17 real datasets) when the number of possible values per tuple is low, as in many real-world applications.

In order to improve the monitoring ability of uncertain large data stream, an uncertain large data flow monitoring algorithm based on artificial intelligence is proposed. The collected uncertain big data flow is constructed by low dimensional feature set, and the rough set model of uncertain large data stream distribution is constructed. The fuzzy C-means clustering method is used to analyze the uncertain big data flow by fusion clustering and adaptive grid partition analysis. All the abnormal samples of large data stream are sampled and trained, and the feature quantities of association rules of uncertain large data stream are extracted. Combined with artificial intelligence method, the monitoring of uncertain large data stream is realized. The simulation results show that the method has high accuracy and good ability to resist abnormal traffic interference, and the traffic security monitoring ability of the network is improved.

The cloud provides a flexible environment for customers to host and process their information. Many applications dealing with massive data and complex queries are routed to the cloud. The response time depends on query optimization and resource scheduling. There are several techniques to optimize the query and resource scheduling but the information about the query is not stored i.e. the frequent queries will recursively do the same set of procedures. Hence, to reduce the time consumed by redundant query processing procedure we introduce agents in Hadoop framework (JobTracker). As JobTracker is a single point of failure i.e. if this goes down, all running jobs are halted. Therefore, we introduce agents in place of JobTracker to intellectually process the query and assign the task to TaskTrackers. Here we use RETSINA toolkit which enables rapid development interacting and intelligent agent based system platform. This will improve the speed and provide timely monitoring of information which is passed across agents.

Lou, H., 2019. Massive ship fault data retrieval algorithm supporting complex query in cloud computing. In: Guido-Aldana, P.A. and Mulahasan, S. (eds.), Advances in Water Resources and Exploration. Journal of Coastal Research, Special Issue No. 93, pp. 1013–1018. Coconut Creek (Florida), ISSN 0749-0208.Aiming at the problems of the current retrieval algorithm for massive ship fault data retrieval in cloud computing, which does not support complex query, large retrieval cost and low precision, a massive ship fault data retrieval algorithm supporting complex query in cloud computing based on query performance prediction is proposed. Introducing the Lagrangian algorithm to preprocess the ship fault data information; combining the attribute values of the ship fault data and the context to calculate the similarity between different attributes, and identifying the same attribute according to the similarity; The attribute recognition result is queried by using multiple retrieval models, and the model complex query request is converted into a one-dimensional query key value, and the query key values obtained by the plurality of models are predicted, and the search result with the optimal prediction performance of the complex query mode is used as the final result. The experimental results show that the proposed algorithm has good retrieval accuracy in complex query mode, and it has less overhead than the current search algorithm.