Characterization and prediction of deep learning workloads in large-scale GPU datacenters

• Main Authors: Hu, Qinghao, Sun, Peng, Yan, Shengen, Wen, Yonggang, Zhang, Tianwei
• Format: Conference Proceeding
• Language: English
• Subjects: Behavioral sciences; Cluster Management System; Cluster Statistical Analysis; Computing methodologies > Distributed computing methodologies; Computing methodologies > Machine learning > Machine learning approaches; Computing methodologies > Modeling and simulation > Simulation evaluation; Deep learning; Deep Learning Training; Energy Conservation; GPU Datacenter; Graphics processing units; High performance computing; Industries; Job shop scheduling; Power demand; Time-series Prediction; Workload Scheduling
• ISSN: 2167-4337
• DOI: 10.1145/3458817.3476223

Modern GPU datacenters are critical for delivering Deep Learning (DL) models and services in both the research community and industry. When operating a datacenter, optimization of resource scheduling and management can bring significant financial benefits. Achieving this goal requires a deep understanding of the job features and user behaviors. We present a comprehensive study about the characteristics of DL jobs and resource management. First, we perform a large-scale analysis of real-world job traces from SenseTime. We uncover some interesting conclusions from the perspectives of clusters, jobs and users, which can facilitate the cluster system designs. Second, we introduce a general-purpose framework, which manages resources based on historical data. As case studies, we design (1) a Quasi-Shortest-Service-First scheduling service, which can minimize the cluster-wide average job completion time by up to 6.5×; (2) a Cluster Energy Saving service, which improves overall cluster utilization by up to 13%.