Toward Efficient Online Scheduling for Distributed Machine Learning Systems

Recent years have witnessed a rapid growth of distributed machine learning (ML) frameworks, which exploit the massive parallelism of computing clusters to expedite ML training. However, the proliferation of distributed ML frameworks also introduces many unique technical challenges in computing syste...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on network science and engineering 2022-07, Vol.9 (4), p.1951-1969
Main Authors: Yu, Menglu, Liu, Jia, Wu, Chuan, Ji, Bo, S. Bentley, Elizabeth
Format: Article
Language:English
Subjects:
Algorithms
Approximation
approximation algorithm
Approximation algorithms
Computation
Computer aided scheduling
Design optimization
distributed machine learning
Heuristic algorithms
Machine learning
Online resource scheduling
Optimization
Parameters
Resource allocation
Resource management
Rounding
Scheduling algorithms
Servers
Systems design
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent years have witnessed a rapid growth of distributed machine learning (ML) frameworks, which exploit the massive parallelism of computing clusters to expedite ML training. However, the proliferation of distributed ML frameworks also introduces many unique technical challenges in computing system design and optimization. In a networked computing cluster that supports a large number of training jobs, a key question is how to design efficient scheduling algorithms to allocate workers and parameter servers across different machines to minimize the overall training time. Toward this end, in this paper, we develop an online scheduling algorithm that jointly optimizes resource allocation and locality decisions. Our main contributions are three-fold: i) We develop a new analytical model that considers both resource allocation and locality; ii) Based on an equivalent reformulation and observations on the worker-parameter server locality configurations, we transform the problem into a mixed packing and covering integer program, which enables approximation algorithm design; iii) We propose a meticulously designed approximation algorithm based on randomized rounding and rigorously analyze its performance. Collectively, our results contribute to the state of the art of distributed ML system optimization and algorithm design.
ISSN:2327-4697
2334-329X
DOI:10.1109/TNSE.2021.3104513