Joint Reducer Placement and Coflow Bandwidth Scheduling for Computing Clusters

Reducing Coflow Completion Time (CCT) has a significant impact on application performance in data-parallel frameworks. Most existing works assume that the endpoints of constituent flows in each coflow are predetermined. We argue that CCT can be further optimized by treating flows' destinations...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM transactions on networking 2021-02, Vol.29 (1), p.438-451
Main Authors: Zhao, Yangming, Tian, Chen, Fan, Jingyuan, Guan, Tong, Zhang, Xiaoning, Qiao, Chunming
Format: Article
Language:English
Subjects:
Algorithms
Bandwidth
Clusters
coflow scheduling
Completion time
Computing clusters
Data transfer
IEEE transactions
Optimization
Placement
Processor scheduling
reducer placement
Schedules
Scheduling
Task analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reducing Coflow Completion Time (CCT) has a significant impact on application performance in data-parallel frameworks. Most existing works assume that the endpoints of constituent flows in each coflow are predetermined. We argue that CCT can be further optimized by treating flows' destinations as an additional optimization dimension via reducer placement. In this article, we propose and implement RPC, a joint online Reducer Placement and Coflow bandwidth scheduling framework, to minimize the average CCT in cloud clusters. We first develop a 2-approximation algorithm to minimize the CCT of a single coflow, and then schedule all the coflows following the Shortest Remaining Time First (SRTF) principle. We use real testbed experiments and extensive large-scale simulations to demonstrate that RPC can reduce the average CCT by 64.98% compared with the state-of-the-art technologies.
ISSN:1063-6692
1558-2566
DOI:10.1109/TNET.2020.3037064