Esc: Towards an Elastic Stream Computing Platform for the Cloud

Today, most tools for processing big data are batch-oriented. However, many scenarios require continuous, online processing of data streams and events. We present ESC, a new stream computing engine. It is designed for computations with real-time demands, such as online data mining. It offers a simpl...

Full description

Saved in:
Bibliographic Details
Main Authors: Satzger, B., Hummer, W., Leitner, P., Dustdar, S.
Format: Conference Proceeding
Language:English
Subjects:
adaptability
Biomedical monitoring
Context
Correlation
event processing
Fault tolerance
Fault tolerant systems
Heart beat
Monitoring
stream computing
Citations: Items that cite this one
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Today, most tools for processing big data are batch-oriented. However, many scenarios require continuous, online processing of data streams and events. We present ESC, a new stream computing engine. It is designed for computations with real-time demands, such as online data mining. It offers a simple programming model in which programs are specified by directed acyclic graphs (DAGs). The DAG defines the data flow of a program, vertices represent operations applied to the data. The data which are streaming through the graph are expressed as key/value pairs. ESC allows programmers to focus on the problem at hand and deals with distribution and fault tolerance. Furthermore, it is able to adapt to changing computational demands. In the cloud, ESC can dynamically attach and release machines to adjust the computational capacities to the current needs. This is crucial for stream computing since the amount of data fed into the system is not under the platform's control. We substantiate the concepts we propose in this paper with an evaluation based on a high-frequency trading scenario.
ISSN:2159-6182
2159-6190
DOI:10.1109/CLOUD.2011.27