Elastic Scaling for Data Stream Processing

This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitio...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2014-06, Vol.25 (6), p.1447-1463
Main Authors: Gedik, Bugra, Schneider, Scott, Hirzel, Martin, Kun-Lung Wu
Format: Article
Language:English
Subjects:
Availability
Channels
Data stream processing
Data transmission
Dynamical systems
Dynamics
elasticity
Flow graphs
Indexes
Measurement
Parallel processing
parallelization
Partitioning
Profitability
Run time (computers)
Runtime
Safety
Throughput
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2013.295