Lustre client compiled with Mellanox InfinibBand driver

Posted by Pavlo Khmel on Mon 11 January 2021

CentOS 8.3 minimal, Mellanox drivers 5.1, Lustre 2.12.6

yum -y install tar perl kernel-rpm-macros lsof gcc kernel-devel-4.18.0-240.el8.x86_64 rpm-build python36-devel gdb-headless pciutils python36 elfutils-libelf-devel createrepo gcc-gfortran tk tcl tcsh
curl -O https://content.mellanox.com/ofed/MLNX_OFED-5.1-2.5.8.0/MLNX_OFED_LINUX-5.1-2.5.8.0-rhel8.3-x86_64.tgz
tar -xf MLNX_OFED_LINUX-5.1-2.5.8.0-rhel8.3-x86_64.tgz 
vi ./MLNX_OFED_LINUX-5.1-2.5.8.0-rhel8.3-x86_64/mlnxofedinstall

# Workaround for issue: Current operation system is not supported!
# Modify this line::
#    } elsif ($dist_rpm =~ /(redhat|centos)-release-8\.([0-3])-/) {
# New line:
#    } elsif ($dist_rpm =~ /(redhat|centos|centos-linux)-release-8\.([0-3])-/) {

./MLNX_OFED_LINUX-5.1-2.5.8.0-rhel8.3-x86_64/mlnxofedinstall --add-kernel-support

Generate new initramfs and restart openibd:

dracut -f
/etc/init.d/openibd restart

Download and compile Luster client:

curl -O https://downloads.whamcloud.com/public/lustre/lustre-2.12.6/el8.3.2011/client/SRPMS/lustre-2.12.6-1.src.rpm
yum -y install kernel-abi-whitelists libtool libyaml-devel
rpmbuild --rebuild --without servers --without lustre-tests lustre-2.12.6-1.src.rpm

# ls -1 /root/rpmbuild/RPMS/x86_64/ | grep lustre
kmod-lustre-client-2.12.6-1.el8.x86_64.rpm
kmod-lustre-client-debuginfo-2.12.6-1.el8.x86_64.rpm
lustre-client-2.12.6-1.el8.x86_64.rpm
lustre-client-debuginfo-2.12.6-1.el8.x86_64.rpm
lustre-client-debugsource-2.12.6-1.el8.x86_64.rpm
lustre-iokit-2.12.6-1.el8.x86_64.rpm

Install Luster client.

You can ger error message: Failed dependencies: ksym(__ib_alloc_pd) = 0xf41059d5 is needed by kmod-lustre-client-2.12.6-1.el8.x86_64

Solution:

yum -y install python2
rpm -e mlnx-ofa_kernel-modules
yum install /root/MLNX_OFED_LINUX-5.1-2.5.8.0-rhel8.3-x86_64/RPMS/kmod-mlnx-ofa_kernel-5.1-OFED.5.1.2.5.8.1.rhel8u3.x86_64.rpm
cd /root/rpmbuild/RPMS/x86_64/
rpm -ivh kmod-lustre-client-2.12.6-1.el8.x86_64.rpm lustre-client-2.12.6-1.el8.x86_64.rpm lustre-iokit-2.12.6-1.el8.x86_64.rpm