StreamCloud: A Large Scale Data Streaming System

Summary form only given. Data streams are continuous flows of data. Examples of data streams include network traffic, sensor data, call center records and so on. Their sheer volume and speed pose a great challenge for the data mining community to mine them. Data streams demonstrate several unique properties: infinite length, concept-drift, concept-evolution, and feature-evolution. Concept-drift occurs in data streams when the underlying concept of data changes over time. Concept-evolution occurs when new classes evolve in streams. Feature-evolution occurs when feature set varies with time in data streams. Each of these properties adds a challenge to data stream mining. This invited talk will present an organized picture on how to handle various data mining techniques in data streams: in particular, how to handle classification in evolving data streams by addressing these challenges.

The near real-time processing of continuous data flows in large scale sensor networks has many applications in risk-critical domains ranging from emergency management to industrial control systems. The problem is how to ensure end-to-end security (e.g., integrity, and authenticity) of such data stream for risk-critical applications. We refer this as an online security verification problem. Existing security techniques cannot deal with this problem because they were not designed to deal with high volume, high velocity data in real-time. Furthermore, they are inefficient as they introduce a significant buffering delay during security verification, resulting in a requirement of large buffer size for the stream processing server. To address this problem, we propose a Dynamic Key Length Based Security Framework (DLSeF) based on the shared key derived from synchronized prime numbers; the key is dynamically updated in short intervals to thwart Man in the Middle and other Network attacks. Theoretical analyses and experimental results of DLSeF framework show that it can significantly improve the efficiency of processing stream data by reducing the security verification time without compromising the security.

Mobility analytics using data generated from the Internet of Mobile Things (IoMT) is facing many challenges which range from the ingestion of data streams coming from a vast number of fog nodes and IoMT devices to avoiding overflowing the cloud with useless massive data streams that can trigger bottlenecks [1]. Managing data flow is becoming an important part of the IoMT because it will dictate in which platform analytical tasks should run in the future. Data flows are usually a sequence of out-of-order tuples with a high data input rate, and mobility analytics requires a real-time flow of data in both directions, from the edge to the cloud, and vice-versa. Before pulling the data streams to the cloud, edge data stream processing is needed for detecting missing, broken, and duplicated tuples in addition to recognize tuples whose arrival time is out of order. Analytical tasks such as data filtering, data cleaning and low-level data contextualization can be executed at the edge of a network. In contrast, more complex analytical tasks such as graph processing can be deployed in the cloud, and the results of ad-hoc queries and streaming graph analytics can be pushed to the edge as needed by a user application. Graphs are efficient representations used in mobility analytics because they unify knowledge about connectivity, proximity and interaction among moving things. This poster describes the preliminary results from our experimental prototype developed for supporting transit systems, in which edge and cloud computing are combined to process transit data streams forwarded from fog nodes into a cloud. The motivation of this research is to understand how to perform meaningfulness mobility analytics on transit feeds by combining cloud and fog computing architectures in order to improve fleet management, mass transit and remote asset monitoring

This paper is a survey of recent stream processing languages, which are programming languages for writing applications that analyze data streams. Data streams, or continuous data flows, have been around for decades. But with the advent of the big-data era, the size of data streams has increased dramatically. Analyzing big data streams yields immense advantages across all sectors of our society. To analyze streams, one needs to write a stream processing application. This paper showcases several languages designed for this purpose, articulates underlying principles, and outlines open challenges.

Streaming data is the continuous (never-ending) flow of data generated by various sources (such as IoT sensors, social media, security logs, applications, or internal/external systems) in various formats and volumes. While traditional solutions are built to ingest, process, and structure data before it can be acted upon, today's applications require models and solutions that are able to analyze and incorporate newly arrived data in motion. There are so many challenges and problems that arise when dealing with streaming data. In this thesis, we try to address some of these problems. To be more precise, in chapter two, we propose a method for learning a model from the data stream that provides comparable properties to an offline solution with an unlimited computational budget. Chapter three looks at the problem of learning from streaming data when the underlying distribution, of which data is drawn, changes over time. We provide an adaptive learning algorithm in a non-stationary setting. We prove that the output of this learning approach is comparable to a model with oracle knowledge of distribution changes. In chapter four, we turn the whole picture around. We look at the problem of answering a query over a stream of data. To be specific, we study the DDW-MaxRS query. This chapter is an exercise to get prepared for chapter five, in which we address the learned indexes problem when the query workload is skewed and changes over time. We proposed a solution that can incorporate the query distribution in the training phase and adapt efficiently as the query distribution changes over time.

Today, there is an increasing need of data stream mining technology to discover important patterns on the fly. Existing data stream models and algorithms commonly assume that users' records or profiles in data streams will not be updated or revised once they arrive. Nevertheless, in various applications such as Web usage, the records/profiles of the users can evolve along time. This kind of streaming data evolves in two forms, the streaming of tuples or transactions as in the case of traditional data streams, and more importantly, the evolving of user records/profiles inside the streams. Such data streams bring difficulties on modeling and clustering for exploringusers' behaviors. In this paper, we propose three models to summarize this kind of data streams, which are the batch model, the Evolving Objects (EO) model and the Dynamic Data Stream (DDS) model. Through creating, updating and deleting user profiles, these models summarize the behaviors of each user as a profile object. Based upon these models, clustering algorithms are employed to discover interesting user groups from the profile objects. We have evaluated all the proposed models on a large real-world data set, showing that the DDS model summarizes the data streams with evolving tuples more efficiently and effectively, and provides better basis for clustering users than the other two models.

Implementation of a context-oriented approach for intelligent processing of navigation data flows in operation of water vehicles was considered. The initial sample of processing data flows in the conditions of the current pilot navigation method does not correspond to the regularities of functioning of a complex object with the modern instrumental method of navigation. Solution to the problem consists in overcoming difficulties by defining complex problems of the processes that differ from a simple sum of parameters of elements with the same type of multilevel connections.A method for processing flows of navigation data under continuous mode was proposed. In the process of study, combining of information from multiple sources with finding more accurate and reliable data was considered. The method of estimation of problem solutions is the criterion method where each single alternative is estimated by a specific number (criterion, objective function). Comparison of alternatives was reduced to a comparison of the corresponding numbers. Various variants of the choice of alternatives and criteria of optimality were taken into account: the criteria of Bayes, Wald, Jain, Laplace. The method used is characterized by multicriteria conditions. Methods of decision-making in games with environment, normalization, the use of neural networks based on the context were applied. The architecture of an artificial neural network was constructed.The study was conducted with the aim of obtaining a stable structure of the system with certain classes of input signals based on artificial neural networks. It is interesting from the theoretical point of view to obtain a parametric variation of the parameters within the specified limits of the optimality criteria. The conducted experiments confirmed efficiency of the use of the proposed methods. The most constructive direction of intelligent processing of the navigational data flow is the context-oriented approach. Implementation of this approach guarantees high level of observance of accuracy criteria in the conditions of river navigation.An applied aspect of using the results obtained in the process of study is the possibility to abandon the pilotage method of navigation and installation of coastal and floating means of navigation equipment. An important result is derivation of a differentiated mapping of the depth array on an electronic chart. The obtained results give grounds to assert the possibility of introducing the proposed context-oriented approach to real navigation.

Recent advances in storage and processing have provided the possibility of automatic gathering of information, which in turn leads to fast and continuous flows of data. The data which are produced and stored in this way are called data streams. Data streams are produced in large size, and much dynamism and have some unique properties which make them applicable to model many real data mining applications. The main challenge of streaming data is the occurrence of concept drift. In addition, regarding the costs of labeling of instances, it is often assumed that only a small fraction of instances are labeled. In this paper, we propose an ensemble algorithm to classify instances of non-stationary data streams in a semi-supervised environment. Furthermore, this method is intended to recognize recurring concept drifts of data streams. In the proposed algorithm, a pool of classifiers is maintained by the algorithm with each classifier being representative of one single concept. At first, a batch of instances is classified by the algorithm. Thereafter, some of these instances are labeled and this partially labeled batch is used to update the classifiers in the pool. This process repeats for consecutive batches of the streams. The main advantage of the algorithm is that it uses unlabeled instances as well as labeled ones in the learning task. Experimental results show the effectiveness of the proposed algorithm over the state-of-the-art methods, in different aspects.

Pipeline architectures provide a versatile and efficient mechanism for constructing visualizations, and they have been implemented in numerous libraries and applications over the past two decades. In addition to allowing developers and users to freely combine algorithms, visualization pipelines have proven to work well when streaming data and scale well on parallel distributed-memory computers. However, current pipeline visualization frameworks have a critical flaw: they are unable to manage time varying data. As data flows through the pipeline, each algorithm has access to only a single snapshot in time of the data. This prevents the implementation of algorithms that do any temporal processing such as particle tracing; plotting over time; or interpolation, fitting, or smoothing of time series data. As data acquisition technology improves, as simulation time-integration techniques become more complex, and as simulations save less frequently and regularly, the ability to analyze the time-behavior of data becomes more important. This paper describes a modification to the traditional pipeline architecture that allows it to accommodate temporal algorithms. Furthermore, the architecture allows temporal algorithms to be used in conjunction with algorithms expecting a single time snapshot, thus simplifying software design and allowing adoption into existing pipeline frameworks. Our architecture also continues to work well in parallel distributed-memory environments. We demonstrate our architecture by modifying the popular VTK framework and exposing the functionality to the ParaView application. We use this framework to apply time-dependent algorithms on large data with a parallel cluster computer and thereby exercise a functionality that previously did not exist.

Recent research shows that rule based models perform well while classifying large data sets such as data streams with concept drifts. A genetic algorithm is a strong rule based classification algorithm which is used only for mining static small data sets. If the genetic algorithm can be made scalable and adaptable by reducing its I/O intensity, it will become an efficient and effective tool for mining large data sets like data streams. In this paper a scalable and adaptable online genetic algorithm is proposed to mine classification rules for the data streams with concept drifts. Since the data streams are generated continuously in a rapid rate, the proposed method does not use a fixed static data set for fitness calculation. Instead, it extracts a small snapshot of the training example from the current part of data stream whenever data is required for the fitness calculation. The proposed method also builds rules for all the classes separately in a parallel independent iterative manner. This makes the proposed method scalable to the data streams and also adaptable to the concept drifts that occur in the data stream in a fast and more natural way without storing the whole stream or a part of the stream in a compressed form as done by the other rule based algorithms. The results of the proposed method are comparable with the other standard methods which are used for mining the data streams.

The confluence of distributed stream processing and Artificial Intelligence (AI) / Machine Learning (ML) presents significant opportunities for real-time intelligent applications.This research plan outlines an investigation into "Real-time Distributed AI on Streaming Data," focusing on challenges and advancements pertinent to the software engineering community, with a forward-looking perspective towards 2025.It aims to synthesize existing open-access literature to provide a comprehensive overview of the current state, identify key software engineering hurdles, and explore cutting-edge research directions.The emphasis is on systems like Apache Flink, Apache Spark Streaming, and Kafka Streams that process massive, continuous data flows, embedding AI/ML inference and online learning capabilities directly within these pipelines with stringent lowlatency requirements.

Permanent change of methods of analytical processing of digital data for objective analysis and adequate decisions based on available information. In the educational process, analytics of significant amounts of data is used to adequately study and understand the data that are continuously generated in the educational sphere for further in-depth analysis to improve management methods that ensure a sufficient level of qualification of students of higher educational institutions. For this purpose, an automated information and reference system of higher education institution management was developed and tested, and on its basis, analytical analysis of the accumulated data flows was carried out. Since analytics is a painstaking process of determining, systematic analysis and interpreting particularly significant patterns from large continuous streams and volumes of data. Analytical data processing is based on innovative methods of intellectual analysis of large amounts of data, such as association, clustering, classification, categorization, correlation, evaluation, forecasting, and analysis in light of current trends and visualization. The purpose of this work is to use predictive analytics to improve the quality of the educational process by providing objective consolidated information to guide higher education institutions in making informed management decisions. This research work mainly focuses on the urgent need to introduce methods of analytical processing of large data flows into the educational management system of the educational institution, and the following strategies are proposed to meet such needs. When implementing automated information and reference system of educational process management, it is necessary to use various components and their functions. In the information and reference system of educational process management, there are various components, such as electronic learning tools, electronic means of providing access to educational content, the use of information technology, various services, etc. This document also focuses on key issues related to the basic components and their functionality in the studied system for processing both dynamic and analytical data. The effective use and processing of analytical educational data by different methods have a significant potential for identifying and forecasting significant accumulated knowledge from educational data, which provides the education management system to recommend appropriate solutions for a flexible planning process that will ensure the quality of educational services for the future.KeywordsEducationInformation systemInformation technologyAnalytical data processingEducational management

Correlated anomaly detection (CAD) from streaming data is a type of group anomaly detection and an essential task in useful real-time data mining applications like botnet detection, financial event detection, industrial process monitor, etc. The primary approach for this type of detection in previous researches is based on principal score (PS) of divided batches or sliding windows by computing top eigenvalues of the correlation matrix, e.g. the Lanczos algorithm. However, this paper brings up the phenomenon of principal score degeneration for large data set, and then mathematically and practically prove current PS-based methods are likely to fail for CAD on large-scale streaming data even if the number of correlated anomalies grows with the data size at a reasonable rate; in reality, anomalies tend to be the minority of the data, and this issue can be more serious. We propose a framework with two novel randomized algorithms rPS and gPS for better detection of correlated anomalies from large streaming data of various correlation strength. The experiment shows high and balanced recall and estimated accuracy of our framework for anomaly detection from a large server log data set and a U.S. stock daily price data set in comparison to direct principal score evaluation and some other recent group anomaly detection algorithms. Moreover, our techniques significantly improve the computation efficiency and scalability for principal score calculation.

Stream join is known as one of the most important and computationally expensive stream operations in data stream management systems (DSMSs). Parallelization techniques that leverage modern multi-core processor have been proposed for stream join in literature. Equi-join is the most frequent type of join in query workloads, and symmetric hash join (SHJ) is the most effective algorithm to process that in data streams. In this paper, as the first research work, we propose a shared-memory parallel symmetric hash join algorithm on multi-core processors for equi-based stream join. Also, we introduce a novel parallel algorithm called chunk-based pairing hash join that significantly elevates throughput and scalability. We have performed extensive experimental evaluation that demonstrates high scalability and low latency for our proposed algorithms.

A big data stream is described using 5 $\mathrm{V}prime$s (Volume, Variety, Velocity, Variability, and Veracity). These characteristics impose various challenges. In many cases, data streams are unbalanced, making traditional data mining approaches impossible to employ. The standard data mining approaches are not suitable for imbalanced data streams for achieving analytical efficiency because they require periodic analyses, but big data requires real-time analytics. Additionally, the induction model must be re-run and rebuilt each time to add up the most recent data. Mining these unique streams, on the other hand, is one of the most intriguing research areas. Deep learning (DL) algorithms were developed to increase classification performance for issues requiring large data sets with varying types and characteristics. Feature selection (F.S.) is a critical stage in any classification application. F.S. entails eliminating superfluous and redundant characteristics, resulting in a prediction model that is more efficient, interpretable, and fast. While complete solutions are available for F.S., managing massive data streams that need instantaneous processing is challenging by its own nature. This work provides a hybrid metaheuristic strategy with FFSMOTE for oversampling imbalanced data stream and Honey Bee algorithm for feature selection. Hybridization aims to improve feature selection processes by combing the advantage of both the algorithms. Furthermore, the Ensemble Deep classifiers are used for data stream classification.