Topology-aware task allocation for online distributed stream processing applications with latency constraints

The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.

Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to achieve better scalability and quality-of-service (QoS) provision. In this paper we present Synergy, a distributed stream processing middleware that provides sharing-aware component composition. Synergy enables efficient reuse of both data streams and processing components, while composing distributed stream processing applications with QoS demands. Synergy provides a set of fully distributed algorithms to discover and evaluate the reusability of available data streams and processing components when instantiating new stream applications. For QoS provision, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. We have implemented a prototype of the Synergy middleware and evaluated its performance on both PlanetLab and simulation testbeds. The experimental results show that Synergy can achieve much better resource utilization and QoS provision than previously proposed schemes, by judiciously sharing streams and processing components during application composition.

Due to the rising demand for large-scale data processing, there is a growing interest in both batch processing, where large volumes of data are processed offline, and stream processing, where large quantities of streaming data are processed online. The dichotomy between these vastly different computing paradigms has led to the development of substantially different methodologies and systems. As there is an increasing number of applications requiring stream and batch processing, there is a need to bridge this gap and offer support for both paradigms. We introduce a new direction for the unification of stream and batch processing, which, contrary to other proposed approaches, uses a stream processing platform as its foundation and supports batch processing on top. Our proof-of-concept implementation of such a middleware layer, called Cyclone, offers the widely popular MapReduce programming model and translates MapReduce jobs for execution on the underlying streaming platform. Cyclone not only achieves a tight integration of batch and stream processing, our evaluation further shows significant performance gains, in particular for sequential and iterative jobs, which naturally arise in many applications.

Many emerging distributed applications require the realtime processing of large amounts of data that are being updated continuously. Distributed stream processing systems offer a scalable and efficient means of in-network processing of such data streams. However, the large scale and the distributed nature of such systems, as well as the fluctuation of their load render it difficult to ensure that distributed stream processing applications meet their Quality of Service demands. We describe a decentralized framework for proactively predicting and alleviating hot-spots in distributed stream processing applications in real-time. We base our hot-spot prediction techniques on statistical forecasting methods, while for hot-spot alleviation we employ a non-disruptive component migration protocol. The experimental evaluation of our techniques, implemented in our Synergy distributed stream processing middleware over PlanetLab, using a real stream processing application operating on real streaming data, demonstrates high prediction accuracy and substantial performance benefits. 1.

Processing high-volume, high-velocity data streams is an important big data problem in many sciences, engineering, and technology domains. There are many open-source distributed stream processing and cloud platforms that offer low-latency stream processing at scale, but the visualization and user-interaction components of these systems are limited to visualizing the outcome of stream processing results. Visual analysis represents a new form of analysis where the user has more control and interactive capabilities either to dynamically change the visualization, analytics or data management processes. VAStream provides an environment for big data stream processing along with interactive visualization capabilities. The system environment consists of hardware and software modules to optimize streaming data workflow (that includes data ingest, pre-processing, analytics, visualization, and collaboration components). The system environment is evaluated for two real-time streaming applications. The real-time event detection using social media streams uses text data arriving from sources such as Twitter to detect emerging events of interest. The real-time river sensor network analysis project uses unsupervised classification methods to classify sensor network streams arriving from the US river network to detect water quality problems. We discuss implementation details and provide performance comparison results of various individual stream processing operations for both stream processing applications.

The early Internet of Things stream processing platforms were mainly designed to collect and display real‐time raw sensor measurements. Data often has to go through several phases of processing to lead to actionable automatic or human decision making. This chapter discusses five main operations performed on streaming data: compression, dimensionality reduction, summarization, learning, and visualization. It distinguishes two main types of stream data processing systems. The first type, also called a data stream management system, is based on relational database principles and introduced the concept of continuous queries. The second type does not enforce a relational view and enables the creation of custom operators. Based on batch and stream processing paradigms, two data processing architectures emerged: the Lambda architecture, which enables both batch and stream processing, and the simpler Kappa architecture, which enables stream processing.

Stream processing has evolved as a paradigm for efficiently sharing and integrating a massive amount of data into applications. However, the integration of globally dispersed sensor data imposes challenges in the effective utilization of the IT infrastructure that forms the global sensor network. Especially simulations require the integration of sensor streams at widely differing spatial and temporal resolutions. For current stream processing solutions it is necessary to generate a separate data stream for each requested resolution. Therefore, these systems suffer from high redundancy in data streams, wasting a significant amount of bandwidth and limiting their scalability.This paper presents a new approach to scalable distributed stream processing of data which stems from globally dispersed sensor networks. The approach supports applications in establishing continuous queries for sensor data at different resolutions and ensures efficient bandwidth usage of the data distribution network. Unlike existing work in the context of video stream processing which provides multiple resolutions by establishing separate channels for each resolution, this paper presents a stream processing system that can efficiently split/combine data streams in order to decrease/increase their resolution without loss in data precision. In addition the system provides mechanisms for load balancing of sensor data streams that allow efficient utilization of the bandwidth of the global sensor network.

As more aspects of our daily lives are being computerized, ever larger amounts of data are being produced at ever greater speeds. In this data lies great value, and we need technologies that enable us to extract this value. This thesis is concerned with one type of technology that allows us to do this: Distributed Stream Processing Systems (DSPS) are systems consisting of many computers that jointly process, and hence extract value from, large amounts of data at high speeds. This dissertation consists of three research projects that investigate two aspects of DSPS: In two projects, different approaches to increase the efficiency of DSPS were studied and in one project, the value of increased efficiency in stream processing was evaluated. All of these projects have been conducted on real computer systems and they are all of quantitative nature. In the first study, a graph partitioning algorithm was leveraged to schedule the workload within a DSPS. This reduced the communication load between hosts, while maintaining or increasing the throughput of the system. The second study was concerned with the auto-configuration of DSPS. We used a probabilistic black-box optimization strategy called Bayesian Optimization to increase throughput performance of DSPSs through configuration. In the third study, we investigated the value of increased efficiency of a DSPS. This was done by building a DSPS based entity ranking system and by evaluating the effect of timely data processing on the quality of the generated rankings.

Recent advances in mobile technologies have enabled a plethora of new applications. The hardware capabilities of mobile devices, however, are still insufficient for real-time stream data processing (e.g., real-time video stream). In order to process real-time streaming data, most existing applications offload the data and computation to a remote cloud service, such as Apache Storm or Apache Spark Streaming. Offloading streaming data, however, has high costs for users, e.g., significant service fees and battery consumption. To address these challenges, we design, implement and evaluate Mobile Storm, the first stream processing platform for mobile clouds, leveraging clusters of local mobile devices to process real-time stream data. In Mobile Storm, we model the workflow of a real-time stream processing job and decompose it into several tasks so that the job can be executed concurrently and in a distributed manner on multiple mobile devices. Mobile Storm was implemented on Android phones and evaluated extensively through a real-time HD video processing application. The result shows that Mobile Storm effectively processes HD Video Stream in a mobile cloud, which would be impossible on a single mobile device.

In order to interpret, enrich, and analyze the streaming data, stream applications often access the data stored in an external database. Although there has been a lot of studies on stream processing, little attention has been paid so far to the join between streaming data and stored data. In this paper, we propose a comprehensive solution called DS-join for distributed processing of the join under the micro-batch model of recently distributed stream processing engines (SPEs), such as spark streaming. The micro-batch model performs stream processing as a series of very small batch jobs and is more fault-tolerant in a distributed environment compared with the record-at-a-time model. The DS-join reduces the number of database accesses by using micro-batching. Furthermore, the DS-join optimizes the join operation by minimizing the data shuffling, managing a cache in a distributed SPE, parallelizing the join processing, and balancing the load between the SPE and the external database system. The experimental results using real and synthetic datasets show that, compared with the state-of-the-art methods, the DS-join significantly improves throughput, especially for large databases.

With the advent of the Internet of Things (IoT), data stream processing have gained increased attention due to the ever-increasing need to process heterogeneous and voluminous data streams. This work addresses the problem of selecting a correct stream processing framework for a given application to be executed within a specific physical infrastructure. For this purpose, we focus on a thorough comparative analysis of three data stream processing platforms - Apache Flink, Apache Storm, and Twitter Heron (the enhanced version of Apache Storm), that are chosen based on their potential to process both streams and batches in real-time. The goal of the work is to enlighten the cloud-clients and the cloud-providers with the knowledge of the choice of the resource-efficient and requirement-adaptive streaming platform for a given application so that they can plan during allocation or assignment of Virtual Machines for application execution. For the comparative performance analysis of the chosen platforms, we have experimented using 8-node clusters on Grid5000 experimentation testbed and have selected a wide variety of applications ranging from a conventional benchmark to sensor-based IoT application and statistical batch processing application. In addition to the various performance metrics related to the elasticity and resource usage of the platforms, this work presents a comparative study of the green-ness of the streaming platforms by analyzing their power consumption - one of the first attempts of its kind. The obtained results are thoroughly analyzed to illustrate the functional behavior of these platforms under different computing scenarios.

A growing number of applications require continuous processing of high-throughput data streams, e.g., financial analysis, network traffic monitoring, or big data analytics. Performing these analyses by using Distributed Stream Processing Systems (DSPSs) in large clusters is emerging as a promising solution to address the scalability challenges posed by these kind of scenarios. Yet, the high time-variability of stream characteristics makes it very inefficient to statically allocate the data-center resources needed to guarantee application Service Level Agreements (SLAs) and calls for original, dynamic, and adaptive resource allocation strategies. In this paper we analyze the problem of planning adaptive replication strategies for DSPS applications under the challenging assumption of minimal statistical knowledge of input characteristics. We investigate and evaluate how different CP techniques can be employed, and quantitatively show how different alternatives offer different trade-offs between problem solution time and stream processing runtime cost through experimental results over realistic testbeds.KeywordsStream ProcessingLarge Neighborhood SearchVLDB EndowmentReplica ActivationDistribute Stream ProcessingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Traditional load-balancing techniques in distributed stream processing systems face significant challenges when handling fluctuating workloads, often resulting in data skew, inefficient partitioning, and delayed responses to workload shifts. This article presents a comprehensive framework for adaptive load balancing that integrates dynamic key partitioning, operator migration, and feedback-driven scaling mechanisms. The proposed solution incorporates artificial intelligence techniques to predict workload patterns in real-time, enabling preemptive resource redistribution that prevents bottlenecks before they occur. The framework extends to event-driven scaling within serverless architectures, where computational resources automatically adjust based on streaming data intensity. By combining reactive mechanisms with predictive capabilities, the system achieves optimal resource utilization while maintaining high throughput. Experimental evaluations demonstrate significant improvements in processing efficiency and resource allocation compared to conventional approaches. The integration of these techniques contributes to the development of self-optimizing stream processing frameworks that minimize operational overhead while maximizing system performance in dynamic environments.

The extreme volume and staggeringly increasing rate inevitably produce unprecedented pressure on any large scale video sharing and hosting systems. Among the efforts to mitigate this pressure, content-based video similarity search is becoming more and more important with the exponential growth of the data size. Though various approaches have been proposed to address this problem, they are mainly focusing on the retrieval accuracy thus bringing video features with high complexity. Due to the complexity of the feature, these systems are based on the assumption that features representing videos have been obtained offline and stored in the database statically. However, the on-call efforts to move the feature extraction and similarity search from offline to online have been ignored in previous work. In this paper, we propose Marlin, a streaming data processing pipeline that efficiently extracts video features and retrieves video similarity information in a large scale video data system. We design a streaming feature extractor to handle the videos streaming into the system and establish the fined-grained resource allocation with a resource-aware data abstraction layer over streaming data to allocate computing resources among the videos with various resource demands. Besides that, we are pipelining the feature extraction and similarity search process with a distributed feature index, which supports real-time query and incremental index update. The experimental and the extensive real-world workload driven simulation results show that the proposed stream processing architecture achieves 25X speedup against the sequential feature extraction algorithm and 23X speedup against the sequential similarity search with a subsecond similarity query latency for a single request.

A new class of stream processing engines has recently established itself as a platform for applications in numerous domains, such as personalized content- and ad-serving, online recommender systems or social media analytics. These new engines draw inspiration from Big Data batch processing frameworks (Google's MapReduce and its descendants) as well existing stream processing engines (e.g. Borealis and STREAM). They process data on-the-fly without first storing it in a file system and their core programming abstractions hide the complexity of distributed-parallel programming.
\nAlthough stream processing applications commonly need to process ingested data within time bounds, this new class of engines so far computes results "as fast as possible". As variations in workload characteristics are often hard to predict and outside the control of the application, this may quickly lead to a situation where "as fast as possible" becomes "not fast enough" for the application.
\nThis thesis revisits the design of this new class of stream processing engines. The core question addressed by this thesis is how latency requirements can be specified and continuously enforced within these engines in a resource-efficient manner. To this end, this thesis contributes (1) a formalism and associated semantics for expressing latency requirements for stream processing applications, (2) a set of techniques for engines to enforce them and (3) an evaluation of the effectiveness of the presented techniques. The proposed techniques optimize resource efficiency by automatically adjusting the data shipping strategy between data flow tasks and adapting the mapping between tasks and execution threads at runtime. Furthermore, based on workload statistics measured at runtime, they adapt the application's parallelism by exploiting the elasticity of shared, resource-managed compute clusters. To evaluate their effectiveness, they have been implemented in a research prototype and an experimental evaluation with several application workloads has been conducted on a large commodity cluster.