• Home
  • Search
  • Massively parallel stream processing with latency guarantees
  • https://doi.org/10.14279/depositonce-5132Copy DOI Icon

Massively parallel stream processing with latency guarantees

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

A new class of stream processing engines has recently established itself as a platform for applications in numerous domains, such as personalized content- and ad-serving, online recommender systems or social media analytics. These new engines draw inspiration from Big Data batch processing frameworks (Google's MapReduce and its descendants) as well existing stream processing engines (e.g. Borealis and STREAM). They process data on-the-fly without first storing it in a file system and their core programming abstractions hide the complexity of distributed-parallel programming.
\nAlthough stream processing applications commonly need to process ingested data within time bounds, this new class of engines so far computes results "as fast as possible". As variations in workload characteristics are often hard to predict and outside the control of the application, this may quickly lead to a situation where "as fast as possible" becomes "not fast enough" for the application.
\nThis thesis revisits the design of this new class of stream processing engines. The core question addressed by this thesis is how latency requirements can be specified and continuously enforced within these engines in a resource-efficient manner. To this end, this thesis contributes (1) a formalism and associated semantics for expressing latency requirements for stream processing applications, (2) a set of techniques for engines to enforce them and (3) an evaluation of the effectiveness of the presented techniques. The proposed techniques optimize resource efficiency by automatically adjusting the data shipping strategy between data flow tasks and adapting the mapping between tasks and execution threads at runtime. Furthermore, based on workload statistics measured at runtime, they adapt the application's parallelism by exploiting the elasticity of shared, resource-managed compute clusters. To evaluate their effectiveness, they have been implemented in a research prototype and an experimental evaluation with several application workloads has been conducted on a large commodity cluster.

Similar Papers
  • Supplementary Content

Large-scale data mining analytics based on MapReduce

  • Jan 01, 2014
  • OPUS Publication Server of the University of Stuttgart (University of Stuttgart)
  • Sunny Ranjan
  • Supplementary Content
  • Citations1

Young, Massive Star Clusters in the Antennae

  • Dec 01, 2008
  • M H Christopher
  • Supplementary Content

Improving Energy Efficiency of MapReduce Systems

  • Feb 06, 2017
  • Figshare
  • Nidhi Tiwari
  • Supplementary Content

Extragalactic machine learning : in theory and in practice

  • Feb 01, 2021
  • Liverpool John Moores University
  • Sebastian Turner
  • Research Article

Inverting middleware framework

  • Jan 01, 2002
  • Repository for Publications and Research Data (ETH Zurich)
  • Michela Taufer +2
  • Research Article

Aplication of Validity Index in K Means and Fuzzy C Means

  • Aug 31, 2020
  • Jurnal Mantik
  • Jontinus Manullang +2
  • Research Article
  • Citations10

PENGELOMPOKAN PROVINSI DI INDONESIA BERDASARKAN KARAKTERISTIK KESEJAHTERAAN RAKYAT MENGGUNAKAN METODE K-MEANS CLUSTER

  • Oct 30, 2015
  • Jurnal Gaussian
  • Fitra Ramdhani +2
  • Research Article

Proses Pengelompompokan Saraf Menggunakan Jaringan Saraf Tiruan (JST) dengan Algoritme Self-Organizing Maps (SOM)

  • Aug 11, 2017
  • Tantri Lestari +2
  • Research Article

Rotational actuation of magnetic nanoparticle clusters for solution-based biosensing

  • Jan 01, 2012
  • Data Archiving and Networked Services (DANS)
  • Andrea Ranzoni
  • Supplementary Content

The evolution of massive disc galaxies with environment and redshift

  • May 01, 2006
  • Zenodo (CERN European Organization for Nuclear Research)
  • Steven Bamford
  • Supplementary Content

Reliable and energy efficient cluster-based routing protocols for wireless sensor networks

  • Jan 31, 2017
  • Figshare
  • Anwar Sadat
  • Research Article
  • Citations13

Open clusters

  • Apr 30, 2015
  • Astronomy & Astrophysics
  • Y Aidelman +3
  • Supplementary Content
  • Citations4

Using clustering algorithms to identify subproblems in design processes

  • Jan 01, 2017
  • Digital Repository at the University of Maryland (University of Maryland College Park)
  • Michael Morency +3
  • Research Article
  • Citations28

ANALISIS CLUSTER DENGAN ALGORITMA K-MEANS DAN FUZZY C-MEANS CLUSTERING UNTUK PENGELOMPOKAN DATA OBLIGASI KORPORASI

  • Oct 28, 2016
  • Jurnal Gaussian
  • Desy Rahmawati Ningrat +2
  • Research Article

Genetic divergence among Ethiopian linseed (Linumusitatissimum L ) genotypes

  • Jul 28, 2020
  • Nedi Terfa Gemechu +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.