Detailed Modeling of Heterogeneous and Contention-Constrained Point-to-Point MPI Communication

The network topology of modern parallel computing systems is inherently heterogeneous, with a variety of latency and bandwidth values. Moreover, contention for the bandwidth can exist on different levels when many processes communicate with each other. Many-pair, point-to-point MPI communication is...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2023-05, Vol.34 (5), p.1580-1593
Main Authors: Thune, Andreas, Reinemo, Sven-Arne, Skeie, Tor, Cai, Xing
Format: Article
Language:English
Subjects:
Bandwidth
Bandwidths
Benchmark testing
Communication
Computational modeling
Heterogeneity
Intra-node communication
Messages
Microprocessors
Modelling
Multicore processing
Network latency
Network topologies
Nodes
performance modeling
point-to-point MPI communication
Protocols
Size measurement
Sockets
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The network topology of modern parallel computing systems is inherently heterogeneous, with a variety of latency and bandwidth values. Moreover, contention for the bandwidth can exist on different levels when many processes communicate with each other. Many-pair, point-to-point MPI communication is thus characterized by heterogeneity and contention, even on a cluster of homogeneous multicore CPU nodes. To get a detailed understanding of the individual communication cost per MPI process, we propose a new modeling methodology that incorporates both heterogeneity and contention. First, we improve the standard max-rate model to better quantify the actually achievable bandwidth depending on the number of MPI processes in competition. Then, we make a further extension that more detailedly models the bandwidth contention when the competing MPI processes have different numbers of neighbors, with also non-uniform message sizes. Thereafter, we include more flexibility by considering interactions between intra-socket and inter-socket messaging. Through a series of experiments done on different processor architectures, we show that the new heterogeneous and contention-constrained performance models can adequately explain the individual communication cost associated with each MPI process. The largest test of realistic point-to-point MPI communication involves 8,192 processes and in total 2,744,632 simultaneous messages over 64 dual-socket AMD Epyc Rome compute nodes connected by InfiniBand, for which the overall prediction accuracy achieved is 84%.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2023.3253881