The OSU Micro-Benchmarks are a suite of benchmarks for measuring and evaluating the performance of MPI operations.
I will use one of them "osu_alltoall" - MPI_Alltoall Latency Test.
Compile with GCC and OpenMPI. I'm using a module from EasyBuildfro compilers:
module load foss/2023a
Download and build
curl -O https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.3.tar.gz
tar xf osu-micro-benchmarks-7.3.tar.gz
cd osu-micro-benchmarks-7.3
./configure CC=`which mpicc` CXX=`which mpicxx` --prefix=$HOME/osu
make
cd ..
Run (4 server 1 process per server)
mpirun -n 4 --host server01:1,server02:1,server03:1,server04:1 /lustre/osu-micro-benchmarks-7.3/c/mpi/collective/blocking/osu_alltoall
# OSU MPI All-to-All Personalized Exchange Latency Test v7.3
# Datatype: MPI_CHAR.
# Size Avg Latency(us)
1 1.98
2 2.00
4 2.00
8 1.99
16 1.98
32 2.05
64 2.25
128 2.32
256 2.88
512 3.16
1024 3.68
2048 5.21
4096 8.18
8192 13.57
16384 21.96
32768 39.69
65536 77.38
131072 147.30
262144 335.42
524288 606.57
1048576 1149.29