Benchmark IOR

Posted by Pavlo Khmel on Wed 27 December 2023

"ior" and "mdtest" parallel I/O benchmarks. They are coming together. Here is the focus on the IOR benchmark. Compile with GCC and OpenMPI. I'm using a module from EasyBuild for compilers:

module load foss/2023a

Download and building

curl -LJO https://github.com/hpc/ior/releases/download/3.3.0/ior-3.3.0.tar.gz
tar xf ior-3.3.0.tar.gz 
cd ior-3.3.0/
./configure
make
cd ..

Run "sequential" locally on single server

To get realistic numbers use size 2-3 times large than RAM.

mpirun -np 10 /lustre/ior-3.3.0/src/ior -a POSIX -v -F -C -e -g -k -b 1m -t 1m -s 100000 -w -r -O stoneWallingWearOut=1 -o /mnt/ior_dir

This command will have an aggregate file size: 976.56 GiB.

Run "sequential" on multiple servers

mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:16,server02:16,server03:16,server04:16 /lustre/ior-3.3.0/src/ior -a POSIX -v -F -C -e -g -k -b 1m -t 1m -s 10000 -i 1 -w -r -O stoneWallingWearOut=1 -o /lustre/ior_dir
. . .
Max Write: 4224.01 MiB/sec (4429.19 MB/sec)
Max Read:  5520.59 MiB/sec (5788.75 MB/sec)

Run "sequential" on multiple servers with a time limit

For time limit it is useful to split READ and WRITE into separate commands. Because in most cases READ will be faster.

Important: The READ stage needs to have completed files after the WRITE benchmark. So WRITE stage should have more time to finish files. This will help to avoid end-of-file errors.

Write

mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:8,...,server08:8 /lustre/ior-3.3.0/src/ior -a POSIX -F -C -e -g -k -b 1m -t 1m -s 100000 -i 1 -w -O stoneWallingWearOut=1 -D 600 -o /lustre/ior_dir

Read:

mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:8,...,server08:8 /lustre/ior-3.3.0/src/ior -a POSIX -F -C -e -g -k -b 1m -t 1m -s 100000 -i 1 -r -D 60 -o /lustre/ior_dir

Random

Option "-z".

NOTE: IOR release candidate 4.0.0rc1 has a new randome method with option: "-z -l random".

mpirun -x PATH -x LD_LIBRARY_PATH -np 64 --host server01:8,...,server08:8 /lustre/ior-3.3.0/src/ior -a POSIX -F -C -e -g -k -b 1m -t 1m -s 10000 -i 1 -w -r -O stoneWallingWearOut=1 -z -o /lustre/ior_dir
. . .
Max Write: 492.00 MiB/sec (515.89 MB/sec)
Max Read:  2977.34 MiB/sec (3121.97 MB/sec)

Command line options

mpirun -x PATH -x LD_LIBRARY_PATH # clone environment variable to other servers
-np 64                   # run 64 processes
--host server01:16       # run 16 processes on server01
-a POSIX                 # API for I/O
-vv                      # verbose (repeating flag increases level)
-F                       # file per process
-C                       # shift ranks
-e                       # include fsync(2) time
-g                       # use barriers between open, write/read, close
-k                       # don't delete written files
-b 1m                    # bytes to write per task
-t 1m                    # size of transfer in bytes
-s 20000                 # number of segments
-i 3                     # repeat 3 times
-w                       # perform write benchmark
-r                       # perform read benchmark
-z                       # random access
-D 100                   # stop after 100 sec. 
-o /lustre/ior_dir       # benchmark directory
-O stoneWallingWearOut=1 # every rank writes the same number of bytes