Safe Data Parallelism for General Streaming

Streaming applications process possibly infinite streams of data and often have both high throughput and low latency requirements. They are comprised of operator graphs that produce and consume data tuples. General streaming applications use stateful, selective, and user-defined operators. The strea...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on computers 2015-02, Vol.64 (2), p.504-517
Main Authors: Schneider, Scott, Hirzel, Martin, Gedik, Bugra, Wu, Kun-Lung
Format: Article
Language:English
Subjects:
Aggregates
Compilers
Computer simulation
Data mining
Data processing
distributed computing
Graphs
Operators
Optimization
Parallel processing
parallel programming
Partitioning
Pipelines
Run time (computers)
Runtime
Safety
Semantics
Streams
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Streaming applications process possibly infinite streams of data and often have both high throughput and low latency requirements. They are comprised of operator graphs that produce and consume data tuples. General streaming applications use stateful, selective, and user-defined operators. The stream programming model naturally exposes task and pipeline parallelism, enabling it to exploit parallel systems of all kinds, including large clusters. However, data parallelism must either be manually introduced by programmers, or extracted as an optimization by compilers. Previous data parallel optimizations did not apply to selective, stateful and user-defined operators. This article presents a compiler and runtime system that automatically extracts data parallelism for general stream processing. Data-parallelization is safe if the transformed program has the same semantics as the original sequential version. The compiler forms parallel regions while considering operator selectivity, state, partitioning, and graph dependencies. The distributed runtime system ensures that tuples always exit parallel regions in the same order they would without data parallelism, using the most efficient strategy as identified by the compiler. Our experiments using 100 cores across 14 machines show linear scalability for parallel regions that are computation-bound, and near linear scalability when tuples are shuffled across parallel regions.
ISSN:0018-9340
1557-9956
DOI:10.1109/TC.2013.221