Enabling Fast Lazy Learning for Data Streams

Streaming Applications usually run in parallel or in series that incrementally transform a stream of input data. It poses a design challenge to break such an application into distinguishable blocks and then to map them into independent hardware processing elements. For this, there is required a generic controller that automatically maps such a stream of data into independent processing elements without any dependencies and manual considerations. In this paper, Kahn Process Networks (KPN) for such streaming applications is designed and developed that will be mapped on MPSoC. This is designed in such a way that there is a generic Cbased compiler that will take the mapping specifications as an input from the user and then it will automate these design constraints and automatically generate the synthesized RTL optimized code for specified application.

Sustainable stream processing algorithms have gained popularity in recent years. Flow control is a way of searching and modifying real-time data streams. Missing values are ubiquitous in real-world data streams, making data stream privacy challenging to safeguard. On the other hand, most privacy preservation methods need not take absent values into account when developed. They can anonymize data in certain study, however this results in data loss. This research proposes a unique parallel distributed approach for protecting privacy while using incomplete data streams. This method uses a production computational system to continually anonymize data streams, using clustering to construct each tuple. It clusters data in partial and complete forms using variable and array dimensions as similarity metrics. In order to prevent values and outliers’ pollution, a generalization approach that is based on more than matches is used. The experiments used real data to compare current systems with varied settings. This research will cover several anonymization mechanisms and their advantages. There are also drawbacks. Finally, we will explore the future of continuous data anonymization research.

Recent research in data mining has focussed on developing new algorithms for mining high-speed data streams. Most real-world data streams have in common that the underlying data generation mechanism changes over time, introducing so-called concept drift into the data. Many current algorithms incorporate a time-based window to be able to cope with drift in order to keep their model up-to-date with the data stream. A major problem with this approach is the potential loss of valuable information as data slides out of the time window. This is particularly a concern in those environments where patterns recur. In this paper, we present a concept-based window approach, which is integrated with a high-speed decision tree learner. Our approach uses the content of the data stream itself in order to decide which information is to be erased. Several methodologies, all based around minimising the overall information loss when pruning the decision tree, are discussed.

With the rapid development of network data communication technology, the complexity of the network environment makes the data in the data stream have uncertain characteristics, and a large number of data streams are generated in many fields. Therefore, it is necessary to find an efficient and accurate processing method to process a large amount of data in a computer network. This paper takes the skyline probability calculation and the actual efficiency of the unknown object index structure in the skyline query processing technology on the uncertain data stream as the research object. Based on the skyline query technology, we study an efficient skyline query processing method on the unknown network traffic based on the model SUMG. The method includes two algorithms: dynamic modeling algorithm DMG and skyline query algorithm GST. The DMG algorithm samples the data in the sliding window of the uncertain data stream and establishes a model to convert the data stream into the parameter stream in the uncertain object probability density function, the GST algorithm establishes the R-tree index structure, which is in order to reduce the amount of calculation, it uses the parameter flow of the model. In both methods, the set of local skyline results is first obtained at the distributed node, and the skyline query is performed again on the union of the local skyline results to obtain the global skyline result set. The experimental results show that compared with the skyline query method BNL for unknown network traffic without an index structure, the SUMG method can not only effectively model the link-type unknown object to assist the skyline query, but also effectively prune the uncertain data object and improve the skyline. The distributed skyline query method can cope with the skyline query task on the distributed data stream.

In order to interpret, enrich, and analyze the streaming data, stream applications often access the data stored in an external database. Although there has been a lot of studies on stream processing, little attention has been paid so far to the join between streaming data and stored data. In this paper, we propose a comprehensive solution called DS-join for distributed processing of the join under the micro-batch model of recently distributed stream processing engines (SPEs), such as spark streaming. The micro-batch model performs stream processing as a series of very small batch jobs and is more fault-tolerant in a distributed environment compared with the record-at-a-time model. The DS-join reduces the number of database accesses by using micro-batching. Furthermore, the DS-join optimizes the join operation by minimizing the data shuffling, managing a cache in a distributed SPE, parallelizing the join processing, and balancing the load between the SPE and the external database system. The experimental results using real and synthetic datasets show that, compared with the state-of-the-art methods, the DS-join significantly improves throughput, especially for large databases.

Novel class detection in data streams using local patterns and neighborhood graph

formula omitted]S: Adaptive anomaly detection on sporadic data streams

The success of data stream mining techniques has allowed decision makers to analyze their data in multiple domains, ranging from monitoring network intrusion to financial markets analysis and online sales transactions exploration. Specifically, online ensembles that construct accurate models against drifting data streams have been developed. Recently, there has been a surge in interest in mobile (or so-called pocket) data stream mining, aiming to construct near real-time models for data stream mining applications that run on mobile devices. In such a setting, it follows that the computational resources are limited and that there is a need to adapt analytics to map the resource usage requirements. Consequently, the resultant models should not only be highly accurate, but they should also adapt swiftly to changes. In addition, the data mining techniques should be fast, scalable, and efficient in terms of resource allocation. It then becomes important to consider Return on Investment (ROI) issues such as storage requirements and memory utilization. This paper introduces the Adaptive Ensemble Size (AES) algorithm, an extension of the Online Bagging method, to address these issues. Our AES method dynamically adapts the sizes of ensembles, based on ROI usage patterns. We illustrate our approach by analyzing the performances against both synthetic and real-world data streams. The results, when comparing our AES algorithm with the state-of-the-art, indicate that we are able to obtain a high Return on Investment (ROI) and to swiftly adapt to change, without compromising on the predictive accuracy.

Massive volumes of data streams can be found in numerous applications such as network intrusion detection, financial transaction flows, telephone call records, sensor streams, and meteorological data. In recent years, there are increasing demands for mining data streams. Unlike the finite, statically stored data sets, stream data are massive, continuous, temporally ordered, dynamically changing, and potentially infinite [5]. For example, Cortes et al. report that AT&T long distance call records consist of 300 million records per day for 100 million customers. For the stream data applications, the volume of data is usually too huge to be stored or to be scanned for more than once. Further, in data streams, the data points can only be sequentially accessed. Random access to data is not allowed. Extensive research has been done for mining data streams, including those on the stream data classification [3, 20], mining frequent patterns [9, 17, 18], and clustering stream data [1, 2, 8, 9, 10, 11, 12, 13, 14, 16, 19]. In this paper, we study the clustering of multiple and parallel data streams. Our study should be differentiated from some previous studies on clustering stream data [19, 1]. Our goal is to group multiple streams with similar behavior and trend together, instead of to cluster the data records within one data stream. There are various applications where it is desirable to cluster the streams themselves rather than the individual data records within them. For example, the price of a stock may rise and fall from time to time. To reduce the financial risk, an investor may prefer to spread his investment over a number of stocks which may exhibit different behaviors. As another application, in meteorological study and disaster prediction, it is useful to cluster meteorological data streams from different geographical regions of similar curvature trends in order to identify regions with similar meteorological behaviors. Yet another example is that a super market may record sales on different merchandizes. There may be some relationship among the sales of different merchandizes and thus the merchant can make use of the correlation to manipulate the prices to maximize the profit. Clustering refers to partition a data set into clusters such that members within the same cluster are similar in a certain sense and members of different clusters are dissimilar. Current clustering techniques can be broadly classified into several categories: partitioning methods (e.g., k-means and k-medoids), hierarchical methods (e.g. BIRCH [22]), densitybased methods (e.g. DBSCAN [15]), and grid-based methods (e.g. CLIQUE [4]). However, these methods are designed only for static data sets and can not be directly applied to data streams. O pe n A cc es s D at ab as e w w w .in te ch w eb .o rg

New challenges have emerged in data mining as the traditional techniques have floundered with real-time data streams. The traditional technique needs refurbishing so as to acclimatize with concept drifting data streams. Thus dealing with the concept changes is the most imperative task of stream data mining. Ensemble classifiers have the ability to automatically adapt with the incoming drifts and, therefore, it is the most interesting research area in data stream mining. Bagging, Boosting and Random forest generation are the common ensemble techniques and are the most popular machine learning approaches in the current scenario for static data (Gomes HM, Bifet A, Read J, Barddal JP, Enembreck F, Pfharinger B, Abdessalem T (2017) Adaptive random forests for evolving data stream classification. Mach Learn 106(9–10):469–1495, [1]). A large number of base classifiers in an ensemble can cause computational overhead. Data mining classifiers for real-time data streams, therefore, need to be updated constantly and retrained with the labeled instances of the newly arrived novel classes in data streams and to cope with concept drift; otherwise, the mining models will become less and less accurate as time passes by. However, for data streams, adaptive random forest algorithms have been widely used for ensemble generation due to its competence to handle different types of drifts. This paper proposes a modified adaptive random forest with meta level learner algorithm and concept adaptive very fast decision tree to overcome the concept drift problem in real-time data streams. The proposed algorithm is experimentally compared with state-of-the-art adaptive random forest algorithm on several real synthetic datasets. Results indicate its efficiency in terms of accuracy and processing time.

Smart cities should be built on harmonious relationships with the environment, with rational measures to use environmental resources in many ways. Resources essential to human life such as water and energy must have special attention. With the advance of technologies use, human being becomes one of the main promoters of information and solutions for smart cities. Stream machine learning is an approach where data becomes available sequentially, called data stream, which adapts the prediction model with fixed window-sized data stream. The objective of this work is to analyze the effectiveness of buildings energy consumption using stream learning. To achieve this goal, MOA tool was used to evaluate the performance of the selected stream learning algorithms on an an open database. Batch algorithms were also used as testbed, to analyze the performance of stream algorithms related to them

Clustering data streams has drawn lots of attention in the last few years due to their ever-growing presence. Data streams put additional challenges on clustering such as limited time and memory and one pass clustering. Furthermore, discovering clusters with arbitrary shapes is very important in data stream applications. Data streams are infinite and evolving over time, and we do not have any knowledge about the number of clusters. In a data stream environment due to various factors, some noise appears occasionally. Density-based method is a remarkable class in clustering data streams, which has the ability to discover arbitrary shape clusters and to detect noise. Furthermore, it does not need the number of clusters in advance. Due to data stream characteristics, the traditional density-based clustering is not applicable. Recently, a lot of density-based clustering algorithms are extended for data streams. The main idea in these algorithms is using density-based methods in the clustering process and at the same time overcoming the constraints, which are put out by data stream’s nature. The purpose of this paper is to shed light on some algorithms in the literature on density-based clustering over data streams. We not only summarize the main density-based clustering algorithms on data streams, discuss their uniqueness and limitations, but also explain how they address the challenges in clustering data streams. Moreover, we investigate the evaluation metrics used in validating cluster quality and measuring algorithms’ performance. It is hoped that this survey will serve as a steppingstone for researchers studying data streams clustering, particularly density-based algorithms.

Data streams may encounter data distribution changes, which can significantly impair the accuracy of models. Concept drift detection tracks data distribution changes and signals when to update models. Many drift detection methods apply thresholds to distinguish between drift or non-drift streams and to claim their method outperforms others with non-aligned drift thresholds. We consider that selecting a proper drift threshold could be more important than developing a new drift detection algorithm, and different drift detection algorithms may end up with very similar performance with aligned drift thresholds. To better understand this process, we propose a novel threshold selection algorithm to align the drift thresholds of a set of algorithms so that they are all at the same sensitivity level. Based on comprehensive experiment evaluations, we observed that several state-of-the-art drift detection algorithms could achieve similar results by aligning their thresholds, providing a novel insight to explain how drift detection algorithms contribute to data stream learning. We noticed that a higher detection sensitivity improves accuracy for data streams with frequent distribution change. The evaluation results are showing that drift thresholds should not be fixed during stream learning. Rather, they should adjust dynamically based on the prevailing conditions of the data stream.

Spatial concentrations (or spatial clusters) of moving objects, such as vehicles and humans, is a mobility pattern that is relevant to many applications. Fast detection of this pattern and its evolution, e.g., if the cluster is shrinking or growing, is useful in numerous scenarios, such as detecting the formation of traffic jams or detecting a fast dispersion of people in a music concert. On-Line detection of this pattern is a challenging task because it requires algorithms that are capable of continuously and efficiently processing the high volume of position updates in a timely manner. Currently, the majority of approaches for spatial cluster detection operate in batch mode, where moving objects location updates are recorded during time periods of a certain length and then batch-processed by an external routine, thus delaying the result of the cluster detection until the end of the time period. Further, they extensively use spatial data structures and operators, which can be troublesome to maintain or parallelize in on-line scenarios. To address these issues, in this paper we propose DG2CEP, a parallel algorithm that combines the well-known density-based clustering algorithm DBSCAN with the data stream processing paradigm Complex Event Processing (CEP) to achieve continuous and timely detection of spatial clusters. Our experiments with real-world data streams indicate that DG2CEP is able to detect the formation and dispersion of clusters with small latency while having higher similarity to DBSCAN than batch-based approaches.

Parsimonious random vector functional link network for data streams