The Case of Performance Variability on Dragonfly-based Systems

Performance of a parallel code running on a large supercomputer can vary significantly from one run to another even when the executable and its input parameters are left unchanged. Such variability can occur due to perturbation of the computation and/or communication in the code. In this paper, we i...

Full description

Saved in:
Bibliographic Details
Main Authors: Bhatele, Abhinav, Thiagarajan, Jayaraman J., Groves, Taylor, Anirudh, Rushil, Smith, Staci A., Cook, Brandon, Lowenthal, David K.
Format: Conference Proceeding
Language:English
Subjects:
3G mobile communication
data analytics
Data models
dragonfly network
forecasting
machine learning
Network topology
performance models
performance variability
Predictive models
Production
Supercomputers
Topology
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Performance of a parallel code running on a large supercomputer can vary significantly from one run to another even when the executable and its input parameters are left unchanged. Such variability can occur due to perturbation of the computation and/or communication in the code. In this paper, we investigate the case of performance variability arising due to network effects on supercomputers that use a dragonfly topology - specifically, Cray XC systems equipped with the Aries interconnect. We perform post-mortem analysis of network hardware counters, profiling output, job queue logs, and placement information, all gathered from periodic representative application runs. We investigate the causes of performance variability using deviation prediction and recursive feature elimination. Additionally, using time-stepped performance data of individual applications, we train machine learning models that can forecast the execution time of future time steps.
ISSN:1530-2075
DOI:10.1109/IPDPS47924.2020.00096