Benchmark parallel filesystem - IOR and mdtest.

Posted on Tue 21 April 2015 by Pavlo Khmel

OS: CentOS 6.5

Compile MPICH.

yum install gcc gcc-gfortran gcc-c++
mkdir /lustre/software
cd /lustre/software/
wget http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz
tar xzf mpich-3.1.4.tar.gz
cd mpich-3.1.4
./configure --prefix=/lustre/software/mpich3/
make
make install

Update environment.

export PATH=/lustre/software/mpich3/bin:$PATH
export LD_LIBRARY_PATH=/lustre/software/mpich3/lib:${LD_LIBRARY_PATH}

Compile IOR.

cd /lustre/software/
yum -y install git automake
git clone https://github.com/chaos/ior.git
mv ior ior_src
cd ior_src/
./bootstrap
./configure --prefix=/lustre/software/ior/
make
make install

Compile mdtest.

mkdir /lustre/software/
git clone https://github.com/MDTEST-LANL/mdtest.git
cd mdtest
export MPI_CC=mpicc
make

Example run from 4 nodes.
Install on all nodes:

yum -y install libgfortran

IOR

mkdir /lustre/benchmark_dir

# mpiexec --hosts=compute-01,compute-02,compute-03,compute-04 -np 56 /lustre/software/ior/bin/ior -v -F -t 1m -b 35g -o /lustre/benchmark_dir/test.`date +"%Y-%m-%d_%H-%M-%S"`
IOR-3.0.1: MPI Coordinated Test of Parallel I/O

Began: Mon Apr 20 23:11:51 2015
Command line used: /lustre/software/ior/bin/ior -v -F -t 1m -b 35g -o /panfs/pandora/Archive/benchmark_dir/test.2015-04-20_23-11-50
Machine: Linux compute-01
Start time skew across all tasks: 0.00 sec

Test 0 started: Mon Apr 20 23:11:51 2015
Path: /lustre/benchmark_dir
FS: 381.1 TiB   Used FS: 38.5%   Inodes: 190.5 Mi   Used Inodes: 38.5%
Participating tasks: 56
Summary:
  api                = POSIX
  test filename      = /lustre/benchmark_dir/test.2015-04-20_23-11-50
  access             = file-per-process
  pattern            = segmented (1 segment)
  ordering in a file = sequential offsets
  ordering inter file= no tasks offsets
  clients            = 56 (14 per node)
  repetitions        = 1
  xfersize           = 1 MiB
  blocksize          = 35 GiB
  aggregate filesize = 1960 GiB

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Mon Apr 20 23:11:51 2015
write     2078.40    36700160   1024.00    0.821398   965.51     370.44     965.67     0
Commencing read performance test: Mon Apr 20 23:27:57 2015

read      1437.65    36700160   1024.00    0.001976   1396.06    446.79     1396.06    0
remove    -          -          -          -          -          -          8.13       0

Max Write: 2078.40 MiB/sec (2179.36 MB/sec)
Max Read:  1437.65 MiB/sec (1507.48 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        2078.40    2078.40    2078.40       0.00  965.66520 0 56 14 1 1 0 1 0 0 1 37580963840 1048576 2104533975040 POSIX 0
read         1437.65    1437.65    1437.65       0.00 1396.05644 0 56 14 1 1 0 1 0 0 1 37580963840 1048576 2104533975040 POSIX 0

Example run mdtest.

# mpiexec --hosts=compute-01,compute-02,compute-03,compute-04 -np 64 /lustre/software/mdtest/mdtest -I 10 -i 2 -z 5 -b 5 -d /lustre/benchmark_dir/
-- started at 04/21/2015 07:22:10 --

mdtest-1.9.3 was launched with 64 total task(s) on 4 node(s)
Command line used: /lustre/software/mdtest/mdtest -I 10 -i 2 -z 5 -b 5 -d /lustre/benchmark_dir/
Path: /lustre/benchmark_dir
FS: 381.1 TiB   Used FS: 38.7%   Inodes: 190.5 Mi   Used Inodes: 38.7%

64 tasks, 2499840 files/directories


SUMMARY: (of 2 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:       3033.719       3014.906       3024.313          9.407
   Directory stat    :     233342.500     232784.529     233063.515        278.986
   Directory removal :       3086.478       3025.113       3055.795         30.683
   File creation     :       3247.334       2464.163       2855.748        391.585
   File stat         :      80051.357      74710.044      77380.701       2670.656
   File read         :      38698.254      16789.691      27743.972      10954.281
   File removal      :       3796.661       3713.135       3754.898         41.763
   Tree creation     :        439.512        191.244        315.378        124.134
   Tree removal      :       1028.362        994.478       1011.420         16.942