"ior" and "mdtest" parallel I/O benchmarks. They are coming together. Here is the focus on the IOR benchmark. Compile with GCC and OpenMPI. I'm using a module from EasyBuild for compilers:
module load foss/2023a
Download and building
curl -LJO https://github.com/hpc/ior/releases/download/3.3.0/ior-3.3.0.tar.gz
tar xf ior-3.3.0.tar.gz
cd ior-3.3.0/
./configure
make
cd ..
Run "sequential" locally on single server
To get realistic numbers use size 2-3 times large than RAM.
mpirun -np 10 /lustre/ior-3.3.0/src/ior -a POSIX -v -F -C -e -g -k -b 1m -t 1m -s 100000 -w -r -O stoneWallingWearOut=1 -o /mnt/ior_dir
This command will have an aggregate file size: 976.56 GiB.
Run "sequential" on multiple servers
mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:16,server02:16,server03:16,server04:16 /lustre/ior-3.3.0/src/ior -a POSIX -v -F -C -e -g -k -b 1m -t 1m -s 10000 -i 1 -w -r -O stoneWallingWearOut=1 -o /lustre/ior_dir
. . .
Max Write: 4224.01 MiB/sec (4429.19 MB/sec)
Max Read: 5520.59 MiB/sec (5788.75 MB/sec)
Run "sequential" on multiple servers with a time limit
For time limit it is useful to split READ and WRITE into separate commands. Because in most cases READ will be faster.
Important: The READ stage needs to have completed files after the WRITE benchmark. So WRITE stage should have more time to finish files. This will help to avoid end-of-file errors.
Write
mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:8,...,server08:8 /lustre/ior-3.3.0/src/ior -a POSIX -F -C -e -g -k -b 1m -t 1m -s 100000 -i 1 -w -O stoneWallingWearOut=1 -D 600 -o /lustre/ior_dir
Read:
mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:8,...,server08:8 /lustre/ior-3.3.0/src/ior -a POSIX -F -C -e -g -k -b 1m -t 1m -s 100000 -i 1 -r -D 60 -o /lustre/ior_dir
Random
Option "-z".
NOTE: IOR release candidate 4.0.0rc1 has a new randome method with option: "-z -l random".
mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:8,...,server08:8 /lustre/ior-3.3.0/src/ior -a POSIX -F -C -e -g -k -b 1m -t 1m -s 10000 -i 1 -w -r -O stoneWallingWearOut=1 -z -o /lustre/ior_dir
. . .
Max Write: 492.00 MiB/sec (515.89 MB/sec)
Max Read: 2977.34 MiB/sec (3121.97 MB/sec)
Command line options
mpirun -x PATH -x LD_LIBRARY_PATH # clone environment variable to other servers
-np 64 # run 64 processes
--host server01:16 # run 16 processes on server01
-a POSIX # API for I/O
-vv # verbose (repeating flag increases level)
-F # file per process
-C # shift ranks
-e # include fsync(2) time
-g # use barriers between open, write/read, close
-k # don't delete written files
-b 1m # bytes to write per task
-t 1m # size of transfer in bytes
-s 20000 # number of segments
-i 3 # repeat 3 times
-w # perform write benchmark
-r # perform read benchmark
-z # random access
-D 100 # stop after 100 sec.
-o /lustre/ior_dir # benchmark directory
-O stoneWallingWearOut=1 # every rank writes the same number of bytes