K-Athena: A Performance Portable Structured Grid Finite Volume Magnetohydrodynamics Code

Large scale simulations are a key pillar of modern research and require ever-increasing computational resources. Different novel manycore architectures have emerged in recent years on the way towards the exascale era. Performance portability is required to prevent repeated non-trivial refactoring of...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2021-01, Vol.32 (1), p.85-97
Main Authors: Grete, Philipp, Glines, Forrest W., O'Shea, Brian W.
Format: Article
Language:English
Subjects:
C++ languages
Computational fluid dynamics
Computer architecture
Computer simulation
D.2.8.b performance measures
D.3.2.d concurrent
distributed and parallel languages
Graphics processing units
I.6.8.h parallel
J.2.c astronomy
J.2.i physics
Kernel
Magnetohydrodynamics
Mathematical analysis
Message systems
Parallel programming
Portability
Programming
Structured grids (mathematics)
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Large scale simulations are a key pillar of modern research and require ever-increasing computational resources. Different novel manycore architectures have emerged in recent years on the way towards the exascale era. Performance portability is required to prevent repeated non-trivial refactoring of a code for different architectures. We combine ATHENA++, an existing magnetohydrodynamics (MHD) CPU code, with KOKKOS, a performance portable on-node parallel programming paradigm, into K-ATHENA to allow efficient simulations on multiple architectures using a single codebase. We present profiling and scaling results for different platforms including Intel Skylake CPUs, Intel Xeon Phis, and NVIDIA GPUs. K-ATHENA achieves > 10 8 cell-updates/s on a single V100 GPU for second-order double precision MHD calculations, and a speedup of 30 on up to 24 576 GPUs on Summit (compared to 172,032 CPU cores), reaching 1:94 × 10 12 total cell-updates/s at 76 percent parallel efficiency. Using a roofline analysis we demonstrate that the overall performance is currently limited by DRAM bandwidth and calculate a performance portability metric of 62.8 percent. Finally, we present the implementation strategies used and the challenges encountered in maximizing performance. This will provide other research groups with a straightforward approach to prepare their own codes for the exascale era. K-ATHENA is available at https://gitlab.com/pgrete/kathena.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2020.3010016