OS: CentOS 6.5
Compile MPICH.
yum install gcc gcc-gfortran gcc-c++
mkdir /lustre/software
cd /lustre/software/
wget http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz
tar xzf mpich-3.1.4.tar.gz
cd mpich-3.1.4
./configure --prefix=/lustre/software/mpich3/
make
make install
Update environment.
export PATH=/lustre/software/mpich3/bin:$PATH
export LD_LIBRARY_PATH=/lustre/software/mpich3/lib:${LD_LIBRARY_PATH}
Compile IOR.
cd /lustre/software/
yum -y install git automake
git clone https://github.com/chaos/ior.git
mv ior ior_src
cd ior_src/
./bootstrap
./configure --prefix=/lustre/software/ior/
make
make install
Compile mdtest.
mkdir /lustre/software/
git clone https://github.com/MDTEST-LANL/mdtest.git
cd mdtest
export MPI_CC=mpicc
make
Example run from 4 nodes.
Install on all nodes:
yum -y install libgfortran
IOR
mkdir /lustre/benchmark_dir
# mpiexec --hosts=compute-01,compute-02,compute-03,compute-04 -np 56 /lustre/software/ior/bin/ior -v -F -t 1m -b 35g -o /lustre/benchmark_dir/test.`date +"%Y-%m-%d_%H-%M-%S"`
IOR-3.0.1: MPI Coordinated Test of Parallel I/O
Began: Mon Apr 20 23:11:51 2015
Command line used: /lustre/software/ior/bin/ior -v -F -t 1m -b 35g -o /panfs/pandora/Archive/benchmark_dir/test.2015-04-20_23-11-50
Machine: Linux compute-01
Start time skew across all tasks: 0.00 sec
Test 0 started: Mon Apr 20 23:11:51 2015
Path: /lustre/benchmark_dir
FS: 381.1 TiB Used FS: 38.5% Inodes: 190.5 Mi Used Inodes: 38.5%
Participating tasks: 56
Summary:
api = POSIX
test filename = /lustre/benchmark_dir/test.2015-04-20_23-11-50
access = file-per-process
pattern = segmented (1 segment)
ordering in a file = sequential offsets
ordering inter file= no tasks offsets
clients = 56 (14 per node)
repetitions = 1
xfersize = 1 MiB
blocksize = 35 GiB
aggregate filesize = 1960 GiB
access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---------- --------- -------- -------- -------- -------- ----
Commencing write performance test: Mon Apr 20 23:11:51 2015
write 2078.40 36700160 1024.00 0.821398 965.51 370.44 965.67 0
Commencing read performance test: Mon Apr 20 23:27:57 2015
read 1437.65 36700160 1024.00 0.001976 1396.06 446.79 1396.06 0
remove - - - - - - 8.13 0
Max Write: 2078.40 MiB/sec (2179.36 MB/sec)
Max Read: 1437.65 MiB/sec (1507.48 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write 2078.40 2078.40 2078.40 0.00 965.66520 0 56 14 1 1 0 1 0 0 1 37580963840 1048576 2104533975040 POSIX 0
read 1437.65 1437.65 1437.65 0.00 1396.05644 0 56 14 1 1 0 1 0 0 1 37580963840 1048576 2104533975040 POSIX 0
Example run mdtest.
# mpiexec --hosts=compute-01,compute-02,compute-03,compute-04 -np 64 /lustre/software/mdtest/mdtest -I 10 -i 2 -z 5 -b 5 -d /lustre/benchmark_dir/
-- started at 04/21/2015 07:22:10 --
mdtest-1.9.3 was launched with 64 total task(s) on 4 node(s)
Command line used: /lustre/software/mdtest/mdtest -I 10 -i 2 -z 5 -b 5 -d /lustre/benchmark_dir/
Path: /lustre/benchmark_dir
FS: 381.1 TiB Used FS: 38.7% Inodes: 190.5 Mi Used Inodes: 38.7%
64 tasks, 2499840 files/directories
SUMMARY: (of 2 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation: 3033.719 3014.906 3024.313 9.407
Directory stat : 233342.500 232784.529 233063.515 278.986
Directory removal : 3086.478 3025.113 3055.795 30.683
File creation : 3247.334 2464.163 2855.748 391.585
File stat : 80051.357 74710.044 77380.701 2670.656
File read : 38698.254 16789.691 27743.972 10954.281
File removal : 3796.661 3713.135 3754.898 41.763
Tree creation : 439.512 191.244 315.378 124.134
Tree removal : 1028.362 994.478 1011.420 16.942