GRAthena++: puncture evolutions on vertex-centered oct-tree AMR

• Published in: arXiv.org 2021-01
• Main Authors: Daszuta, Boris, Zappa, Francesco, Cook, William, Radice, David, Bernuzzi, Sebastiano, Morozova, Viktoriya
• Format: Article
• Language: English
• Subjects: Astronomy; Binary stars; Black holes; Computational fluid dynamics; Finite element method; Gravitational waves; Grid refinement (mathematics); Magnetohydrodynamics; Neutron stars; Numerical relativity; Relativity; Robustness (mathematics); Waveforms
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2101.08289

Numerical relativity is central to the investigation of astrophysical sources in the dynamical and strong-field gravity regime, such as binary black hole and neutron star coalescences. Current challenges set by gravitational-wave and multi-messenger astronomy call for highly performant and scalable codes on modern massively-parallel architectures. We present GR-Athena++, a general-relativistic, high-order, vertex-centered solver that extends the oct-tree, adaptive mesh refinement capabilities of the astrophysical (radiation) magnetohydrodynamics code Athena++. To simulate dynamical space-times GR-Athena++ uses the Z4c evolution scheme of numerical relativity coupled to the moving puncture gauge. We demonstrate stable and accurate binary black hole merger evolutions via extensive convergence testing, cross-code validation, and verification against state-of-the-art effective-one-body waveforms. GR-Athena++ leverages the task-based parallelism paradigm of Athena++ to achieve excellent scalability. We measure strong scaling efficiencies above \(95\%\) for up to \(\sim 1.2\times10^4\) CPUs and excellent weak scaling is shown up to \(\sim 10^5\) CPUs in a production binary black hole setup with adaptive mesh refinement. GR-Athena++ thus allows for the robust simulation of compact binary coalescences and offers a viable path towards numerical relativity at exascale.