An elastic and traffic-aware scheduler for distributed data stream processing in heterogeneous clusters

Existing Data Stream Processing (DSP) systems perform poorly while encountering heavy workloads, particularly on clustered set of (heterogeneous) computers. Elasticity and changing application parallelism degree can limit the performance degradation in the face of varying workloads that negatively i...

Full description

Saved in:
Bibliographic Details
Published in:The Journal of supercomputing 2023, Vol.79 (1), p.461-498
Main Authors: Hadian, Hamid, Farrokh, Mohammadreza, Sharifi, Mohsen, Jafari, Ali
Format: Article
Language:English
Subjects:
Compilers
Computer Science
Data transmission
Decisions
Elasticity
Interpreters
Machine learning
Markov processes
Nodes
Operators
Performance degradation
Processor Architectures
Programming Languages
Relocation
Replication
Response time (computers)
Run time (computers)
Scaling
Scheduling
Topology
Workload
Workloads
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Existing Data Stream Processing (DSP) systems perform poorly while encountering heavy workloads, particularly on clustered set of (heterogeneous) computers. Elasticity and changing application parallelism degree can limit the performance degradation in the face of varying workloads that negatively impact the overall application response time. Elasticity can be achieved by operator scaling, i.e., by replication and relocation in operators at runtime. However, scaling decisions at runtime is challenging, since it first increases the overall communication overhead between operators and secondly changes any initial scheduling that could lead to a non-optimal scheduling plan. In this paper, we investigate the problem of elasticity and scaling decisions and propose a DSP system called ER-Storm. To curb communication overhead, we propose a new 3-step mechanism for replication and relocation of operators upon detecting a bottleneck operator that overutilizes a worker node. The other challenge is to select the proper worker nodes to host relocated operators. By discretizing the input workload, we model the relocation of operators between worker nodes at runtime through a scalable Markov Decision Process (MDP) and use a model-free notion of reinforcement learning (Q-Learning) to find optimal solutions. We have implemented our propositions on the Apache Storm version 2.1.0. Our experimental results show that ER-Storm reduces the average topology response time by 20–60 percent based on the rate of input workload (low or high) compared to the R-Storm scheduler and the Online-Scheduler of Storm.
ISSN:0920-8542
1573-0484
DOI:10.1007/s11227-022-04669-z