Provably optimal parallel transport sweeps on semi-structured grids

• Published in: Journal of computational physics 2020-01, Vol.407 (N/A)
• Main Authors: Adams, Michael P., Adams, Marvin L., Hawkins, W. Daryl, Smith, Timmie, Rauchwerger, Lawrence, Amato, Nancy M., Bailey, Teresa S., Falgout, Robert D., Kunen, Adam, Brown, Peter
• Format: Article
• Language: English
• Subjects: MATHEMATICS AND COMPUTING; NUCLEAR PHYSICS AND RADIATION PHYSICS; parallel algorithms; parallel transport sweeps; performance model; s unstructured mesh; scheduling algorithms; STAPL
• ISSN: 0021-9991; 1090-2716

We have found provably optimal algorithms for full-domain discrete-ordinate transport sweeps on a class of grids in 2D and 3D Cartesian geometry that are regular at a coarse level but arbitrary within the coarse blocks. We describe these algorithms and show that they always execute the full eight-octant (or four-quadrant if 2D) sweep in the minimum possible number of stages for a given Px x Py x Pz partitioning. Computational results confirm that our optimal scheduling algorithms execute sweeps in the minimum possible stage count. Observed parallel efficiencies agree well with our performance model. Our PDT transport code has achieved approximately 68% parallel efficiency with > 1.5M parallel threads, relative to 8 threads, on a simple weak-scaling problem with only three energy groups, 10 directions per octant, and 4096 cells/thread. Our ARDRA code has achieved 71% efficiency with < 1.5M cores, relative to 16 cores, with 36 directions per octant and 48 energy groups. We demonstrate similar efficiencies with PDT on a realistic set of nuclear-reactor test problems, with unstructured meshes that resolve fine geometric details. These results demonstrate that discrete-ordinates transport sweeps can be executed with high efficiency using more than 106 parallel processes.