Dependency-Aware Network Adaptive Scheduling of Data-Intensive Parallel Jobs

Datacenter clusters often run data-intensive jobs in parallel for improving resource utilization and cost efficiency. The performance of parallel jobs is often constrained by the cluster's hard-to-scale network bisection bandwidth. Various solutions have been proposed to address the issue, howe...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2019-03, Vol.30 (3), p.515-529
Main Authors: Wang, Shaoqi, Chen, Wei, Zhou, Xiaobo, Zhang, Liqiang, Wang, Yin
Format: Article
Language:English
Subjects:
Adaptive systems
Adaptive task scheduler
Bandwidth
Clusters
Data aggregation
data-parallel clusters
Dependence
Dynamic scheduling
job dependency
Machine learning
network adaptive
Racks
Schedules
Task analysis
Task scheduling
Yarn
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Datacenter clusters often run data-intensive jobs in parallel for improving resource utilization and cost efficiency. The performance of parallel jobs is often constrained by the cluster's hard-to-scale network bisection bandwidth. Various solutions have been proposed to address the issue, however, most of them do not consider inter-job data dependencies and schedule jobs independently from one another. In this work, we find that aggregating and co-locating the data and tasks of dependent jobs offer an extra opportunity for data locality improvement that can help to greatly enhance the performance of jobs. We propose and design Dawn, a dependency-aware network-adaptive scheduler that includes an online plan and an adaptive task scheduler. The online plan, taking job dependencies into consideration, determines where (i.e., preferred racks) to place tasks in order to proactively aggregate dependent data. The task scheduler, based on the output of online plan and dynamic network status, adaptively schedules tasks to co-locate with the dependent data in order to take advantage of data locality. We implement Dawn on Apache Yarn and evaluate it on physical and virtual clusters using various machine learning and query workloads. Results show that Dawn effectively improves cluster throughput by up to 73 and 38 percent compared to Fair Scheduler and ShuffleWatcher, respectively. Dawn not only significantly enhances the performance of jobs with dependency, but also works well for jobs without dependency.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2018.2866993