Justice: A Deadline-Aware, Fair-Share Resource Allocator for Implementing Multi-Analytics

In this paper, we present Justice, a fair-share deadline-aware resource allocator for big data cluster managers. In resource constrained environments, where resource contention introduces significant execution delays, Justice outperforms the popular existing fair-share allocator that is implemented...

Full description

Saved in:
Bibliographic Details
Main Authors: Dimopoulos, Stratos, Krintz, Chandra, Wolski, Rich
Format: Conference Proceeding
Language:English
Subjects:
Admission control
Big Data
Clustering algorithms
deadlines
Edge computing
resource allocation
Resource management
resource-constrained clusters
Yarn
Citations: Items that cite this one
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we present Justice, a fair-share deadline-aware resource allocator for big data cluster managers. In resource constrained environments, where resource contention introduces significant execution delays, Justice outperforms the popular existing fair-share allocator that is implemented as part of Mesos and YARN. Justice uses deadline information supplied with each job and historical job execution logs to implement admission control. It automatically adapts to changing workload conditions to assign enough resources for each job to meet its deadline "just in time." We use trace-based simulation of production YARN workloads to evaluate Justice under different deadline formulations. We compare Justice to the existing fair-share allocation policy deployed on cluster managers like YARN and Mesos and find that in resource-constrained settings, Justice improves fairness, satisfies significantly more deadlines, and utilizes resources more efficiently.
ISSN:2168-9253
DOI:10.1109/CLUSTER.2017.52