A Holistic Stream Partitioning Algorithm for Distributed Stream Processing Systems

SummaryLoad balancing among the processing elements (PEs) of distributed stream processing system (DSPS) is a key issue in the presence of data skewness. Existing data partitioning schemes for DSPS suffer from the scalability problem and system in‐efficiency. Non‐key based partitioning strategies raise prohibitively high memory overhead for the stateful operations with a large number of keys and high data parallelism, while the key‐based schemes introduce load imbalance for highly skewed data. Predicting the nature of stream data in advance can help to reduce the load imbalance among the PEs of DSPS. For this purpose, the heavy hitter algorithms approximate the hot items of streaming data. However, existing designs suffer from unsatisfied prediction accuracy. In this work, we propose an efficient algorithm to filter hot items in a stream of incoming data. The proposed scheme dynamically monitors the items of a stream and greatly improves the accuracy of estimation by keeping the actual key‐value pair for the frequent items. On one hand, to ensure better load balancing for the skewed data streams, the detected hot keys are directed to more than two PEs randomly from the limited workers. On the other hand, for less frequent keys, the proposed scheme explores the principle of the power of two choices to distribute load. We conduct extensive experiments on both real‐world and synthetic data sets. The results show that the proposed pre‐filtering approach significantly outperforms existing designs in terms of prediction accuracy. The results also show that our design achieves a more balanced load as compared to the existing designs.

Modern distributed stream processing systems can potentially be applied to real time network flow processing. However, differences in performance make some systems more suitable than others for being applied to this domain. We propose a novel performance benchmark, which is based on common security analysis algorithms of NetFlow data to determine the suitability of distributed stream processing systems. Three of the most used distributed stream processing systems are bench-marked and the results are compared with NetFlow data processing challenges and requirements. The benchmark results show that each system reached a sufficient data processing speed using a basic deployment scenario with little to no configuration tuning. Our benchmark, unlike any other, enables the performance of small structured messages to be processed on any stream processing system.

Scalability is one of the common goals addressed by distributed stream processing systems. Distributed stream processing systems execute streaming applications that are segmented and distributed among several nodes across clusters, in order to cater heavy and growing stream processing use cases. Due to the tremendous benefits provided by cloud infrastructures, distributed stream processing systems are often adapted to be deployed on cloud-native environments. Container orchestrators tend to provide modularity and ease of management for containerised applications. Dynamic scalability and dynamic scheduling of streaming applications become the key points, in addressing the efficiency of cloud-native distributed stream processing systems. In this paper, the author attempts to summarise the researches that have been conducted within the domain of dynamic scaling and dynamic scheduling of stream processing systems, related to cloud-nativeness. KeywordsDistributed stream processing systemsContainer orchestratorsSchedulersStreaming application

As more aspects of our daily lives are being computerized, ever larger amounts of data are being produced at ever greater speeds. In this data lies great value, and we need technologies that enable us to extract this value. This thesis is concerned with one type of technology that allows us to do this: Distributed Stream Processing Systems (DSPS) are systems consisting of many computers that jointly process, and hence extract value from, large amounts of data at high speeds. This dissertation consists of three research projects that investigate two aspects of DSPS: In two projects, different approaches to increase the efficiency of DSPS were studied and in one project, the value of increased efficiency in stream processing was evaluated. All of these projects have been conducted on real computer systems and they are all of quantitative nature. In the first study, a graph partitioning algorithm was leveraged to schedule the workload within a DSPS. This reduced the communication load between hosts, while maintaining or increasing the throughput of the system. The second study was concerned with the auto-configuration of DSPS. We used a probabilistic black-box optimization strategy called Bayesian Optimization to increase throughput performance of DSPSs through configuration. In the third study, we investigated the value of increased efficiency of a DSPS. This was done by building a DSPS based entity ranking system and by evaluating the effect of timely data processing on the quality of the generated rankings.

With the advent of data-intensive applications that generate large volumes of real-time data, distributed stream processing systems (DSPS) become increasingly important in domains such as social networking and web analytics. In practice, DSPSs must handle highly variable workloads caused by unpredictable changes in stream rates. Cloud computing offers an elastic infrastructure that DSPSs can use to obtain resources on-demand, but an open problem is to decide on the correct resource allocation when deploying DSPSs in the cloud. This paper proposes an adaptive approach for provisioning virtual machines (VMs) for the use of a DSPS in the cloud. We initially perform a set of benchmarks across performance metrics such as network latency and jitter to explore the feasibility of cloud-based DSPS deployments. Based on these results, we propose an algorithm for VM provisioning for DSPSs that reacts to changes in the stream workload. Through a prototype implementation on Amazon EC2, we show that our approach can achieve low-latency stream processing when VMs are not overloaded, while adjusting resources dynamically with workload changes.

The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.

Stream processing is widely used by today's applications such as financial data analysis and disaster response. In distributed stream processing systems, machine fail-stop events are handled by either active standby or passive standby. However, existing high availability (HA) schemes have not sufficiently addressed the situation when a machine becomes temporarily unavailable due to data rate spikes, intensive analysis or job sharing, which happens frequently but lasts for short time. It is not clear how well active and passive standby fare against such transient unavailability. In this paper, we first critically examine the suitability of active and passive standby against transient unavailability in a real testbed environment. We find that both approaches have advantages and drawbacks, but neither is ideal to provide fast recovery at low overhead as required to handle transient unavailability. Based on the insights gained, we propose a novel hybrid HA method that switches between active and passive standby modes depending on the occurrence of failure events. It presents a desirable tradeoff that is different from existing HA approaches: low overhead during normal conditions and fast recovery upon transient or permanent failure events. We have implemented our hybrid method and compared it with existing HA designs with comprehensive evaluation. The results show that our hybrid method can reduce two-thirds of the recovery time compared to passive standby and 80% message overhead compared to active standby, allowing applications to enjoy uninterrupted processing without paying a high premium.

In a distributed stream processing system, streaming data are continuously disseminated from the sources to the distributed processing servers. To enhance the dissemination efficiency, these servers are typically organized into one or more dissemination trees. In this paper, we focus on the problem of constructing dissemination trees to minimize the average loss of fidelity of the system. We observe that existing heuristic-based approaches can only explore a limited solution space and hence may lead to sub-optimal solutions. On the contrary, we propose an adaptive and cost-based approach. Our cost model takes into account both the processing cost and the communication cost. Furthermore, as a distributed stream processing system is vulnerable to inaccurate statistics, runtime fluctuations of data characteristics, server workloads, and network conditions, we have designed our scheme to be adaptive to these situations: an operational dissemination tree may be incrementally transformed to a more cost-effective one. Our adaptive strategy employs distributed decisions made by the distributed servers independently based on localized statistics collected by each server at runtime. For a relatively static environment, we also propose two static tree construction algorithms relying on apriori system statistics. These static trees can also be used as initial trees in a dynamic environment. We apply our schemes to both single- and multi-object dissemination. Our extensive performance study shows that the adaptive mechanisms are effective in a dynamic context and the proposed static tree construction algorithms perform close to optimal in a static environment.

Skewed data streams are characterized by uneven distributions in which a small fraction of elements occur with much higher frequency than others. The detection of these high-frequency elements presents significant practical challenges, particularly under stringent memory constraints, as existing detection techniques have typically relied on predefined thresholds that require significant memory usage. However, this approach is highly inefficient since not all elements require equal storage space. To address these limitations, we introduce HeavyFinder (HF), a novel lightweight network measurement architecture designed to detect high-frequency elements in skewed data. HF employs a threshold-free update strategy that enables dynamic adaptation to variable data, thereby providing greater flexibility for tracking high-frequency elements without requiring fixed thresholds. Furthermore, an included memory-light strategy enables high accuracy for non-uniform distributions, even with limited memory allocation. Experimental results showed that HF significantly improved performance in four query tasks, producing an accuracy of 99.81% when identifying the top-k elements. The average absolute error (AAE) was also reduced to 10-4 using only 100KB of memory, which was significantly lower than that of conventional methods.

In the distributed stream processing, data skew and dynamics can result in imbalanced load distribution at downstream tasks and affect the throughput of the systems. Efficient data distribution method is urgently needed to solve the problem of imbalanced load distribution to improve system throughput. Existing researches cannot better balance the semantic correctness and the throughput of distributed stream processing systems, and it is hard to apply to distributed clusters with different node performance. In this paper, we propose a performance-sensitive data distribution method. Firstly, we propose a load balancing framework that considers the performance of nodes. Then we present a data redistribution algorithm, which aims to balance the execution delay between parallel nodes and follows the principle of "Less Migration key, Less Partition key"(LMLP). The proposed algorithm can reduce the extra cost caused by the data redistribution process and it can be applied to distributed clusters with different performance. Experimental results show that our method is better than the similar methods in terms of load imbalance degree, complete latency, routing table size, etc. And we also use real data to prove that our method can obtain higher throughput and resource utilization.

Communication-efficient processing of multiple continuous aggregate queries

Stream processing is an emerging paradigm to handle data streams upon arrival, powering latency-critical application such as fraud detection, algorithmic trading, and health surveillance. Though there are a variety of Distributed Stream Processing Systems (DSPSs) that facilitate the development of streaming applications, resource management and task scheduling is not automatically handled by the DSPS middleware and requires a laborious process to tune toward specific deployment targets. As the advent of cloud computing has supported renting resources on-demand, it is of great interest to review the research progress of hosting streaming systems in clouds under certain Service Level Agreements (SLA) and cost constraints. In this article, we introduce the hierarchical structure of streaming systems, define the scope of the resource management problem, and present a comprehensive taxonomy in this context covering critical research topics such as resource provisioning, operator parallelisation, and task scheduling. The literature is then reviewed following the taxonomy structure, facilitating a deeper understanding of the research landscape through classification and comparison of existing works. Finally, we discuss the open issues and future research directions toward realising an automatic, SLA-aware resource management framework.

BBoxDB Streams is a distributed stream processing system, which allows the handling of multi-dimensional data. Multi-dimensional streams consist of n-dimensional elements, such as position data (e.g., two-dimensional positions of cars or three-dimensional positions of aircraft). The software is an enhancement of BBoxDB, a distributed key-bounding-box-value store that allows the handling of n-dimensional big data. BBoxDB Streams supports continuous range queries and continuous spatial joins; n-dimensional point and non-point data are supported. Operations in BBoxDB Streams are performed primarily on the bounding boxes of the data. With user-defined filters (UDFs), custom data formats can be decoded, and the bounding box-based operations are refined (e.g., a UDF decodes and performs intersection tests on the real geometries of WKT encoded stream elements). A unique feature of BBoxDB Streams is the ability to perform continuous spatial joins between stream elements and previously stored multi-dimensional big data. For example, the dynamic position of a car can be efficiently joined with the static spatial data of a street network.

High performance distributed systems such as distributed stream processing systems and message-passing parallel programs are often deployed on platforms that make use of vanilla TCP/IP communication, which in turn uses the conventional Nagle's algorithm for congestion control. Recent research in Reinforcement Learning (RL) techniques to either replace or control the conventional TCP approach shows promise in achieving a greater degree of performance, especially when the demand for network resources in a multi-tenant platform is highly dynamic and infeasible to model. Existing results are, however, focused on RL for general Internet communication, with the learning objective being some combination of throughput, loss, and latency, and predominately use a continuous action space to adjust the packet rate at the sender. In this work, we propose a coefficient-free RL objective that perfectly matches the data transmission rate to the underlying communication system's bottleneck, which naturally deters packet loss and thereby converges to the ideal throughput even in lock-free and latency-constrained Big Data applications where packets are dropped due to load shedding or exceeding latency thresholds. Our results compare favorably to other state-of-the-art objective functions using an RL framework, e.g., providing up to 48% reduction in packet loss while obtaining up to a 4% increase in overall throughput when packet latency is highly constrained.

A growing number of applications require continuous processing of high-throughput data streams, e.g., financial analysis, network traffic monitoring, or big data analytics. Performing these analyses by using Distributed Stream Processing Systems (DSPSs) in large clusters is emerging as a promising solution to address the scalability challenges posed by these kind of scenarios. Yet, the high time-variability of stream characteristics makes it very inefficient to statically allocate the data-center resources needed to guarantee application Service Level Agreements (SLAs) and calls for original, dynamic, and adaptive resource allocation strategies. In this paper we analyze the problem of planning adaptive replication strategies for DSPS applications under the challenging assumption of minimal statistical knowledge of input characteristics. We investigate and evaluate how different CP techniques can be employed, and quantitatively show how different alternatives offer different trade-offs between problem solution time and stream processing runtime cost through experimental results over realistic testbeds.KeywordsStream ProcessingLarge Neighborhood SearchVLDB EndowmentReplica ActivationDistribute Stream ProcessingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.