Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis

HPC datacenters offer a backbone to the modern digital society. Increasingly, they run Machine Learning (ML) jobs next to generic, compute-intensive workloads, supporting science, business, and other decision-making processes. However, understanding how ML jobs impact the operation of HPC datacenter...

Full description

Saved in:
Bibliographic Details
Main Authors: Chu, Xiaoyu, Hofstatter, Daniel, Ilager, Shashikant, Talluri, Sacheendra, Kampert, Duncan, Podareanu, Damian, Duplyakin, Dmitry, Brandic, Ivona, Iosup, Alexandru
Format: Conference Proceeding
Language:English
Subjects:
Cross Analysis
Datacenters
Energy consumption
Failure Analysis
GPU
Graphics processing units
HPC
Machine learning
Measurement
Multivariate Analysis
Production
Resource management
Runtime
Software
Software development management
System analysis and design
System Modeling
Work-load Characterization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:HPC datacenters offer a backbone to the modern digital society. Increasingly, they run Machine Learning (ML) jobs next to generic, compute-intensive workloads, supporting science, business, and other decision-making processes. However, understanding how ML jobs impact the operation of HPC datacenters, relative to generic jobs, remains desirable but understudied. In this work, we leverage long-term operational data, collected from a national-scale production HPC datacenter, and statistically compare how ML and generic jobs can impact the performance, failures, resource utilization, and energy consumption of HPC datacenters. Our study provides key insights, e.g., ML-related power usage causes GPU nodes to run into temperature limitations, median/mean runtime and failure rates are higher for ML jobs than for generic jobs, both ML and generic jobs exhibit highly variable arrival processes and resource demands, significant amounts of energy are spent on unsuccessfully terminating jobs, and concurrent jobs tend to terminate in the same state. We open-source our cleaned-up data traces on Zenodo (https://doi. org/10.5281/zenodo.13685426), and provide our analysis toolkit as software hosted on GitHub (https://github.com/atlarge-research/2024-icpads-hpc-workload-characterization). This study offers multiple benefits for data center administrators, who can improve operational efficiency, and for researchers, who can further improve system designs, scheduling techniques, etc.
ISSN:2690-5965
DOI:10.1109/ICPADS63350.2024.00097