BeeGFS install with multiple metadata targets

Posted on Tue 04 March 2025 by Pavlo Khmel

Installing BeeGFS with 32 metadata services. 4 metadata server, 1 storage server and 1 client:

Hosts and IPs:

mds-1 10.3.10.5
mds-2 10.3.10.6
mds-3 10.3.10.7
mds-4 10.3.10.8
oss-1 10.3.10.9
client-1 10.3.10.10

Install RPM packages

On all servers add BeeGFS repository file:

# cat /etc/yum.repos.d/beegfs.repo
[beegfs]
name=BeeGFS 7.4.5 (rhel9)
baseurl=https://www.beegfs.io/release/beegfs_7.4.5/dists/rhel9
gpgkey=https://www.beegfs.io/release/beegfs_7.4.5/gpg/GPG-KEY-beegfs
gpgcheck=1
enabled=1

Import GPG-KEY:

# rpm --import https://www.beegfs.io/release/beegfs_7.4.5/gpg/GPG-KEY-beegfs
dnf clean all

Install on metadata server with management service (mds-1):

dnf -y install beegfs-common beegfs-mgmtd beegfs-meta libbeegfs-ib beegfs-utils

Install on metadata server without managament service (mds-2, mds-3, mds-4):

dnf -y install beegfs-common beegfs-meta libbeegfs-ib beegfs-utils

Install on storage server (oss-1):

dnf -y install beegfs-common beegfs-storage libbeegfs-ib beegfs-utils

Install on client (client-1):

dnf -y install beegfs-common beegfs-client beegfs-helperd libbeegfs-ib beegfs-utils

Configure management service on mds-1

Generate connauthfile file:

dd if=/dev/random of=/etc/beegfs/connauthfile bs=128 count=1
chown root:root /etc/beegfs/connauthfile
chmod 400 /etc/beegfs/connauthfile

Create filesystem:

mkdir -p /beegfs/mgmtd
mkfs.ext4 -m 0 /dev/sdc
mount /dev/sdc /beegfs/mgmtd
/opt/beegfs/sbin/beegfs-setup-mgmtd -p /beegfs/mgmtd

Edit this line in the file: /etc/beegfs/beegfs-mgmtd.conf

connAuthFile = /etc/beegfs/connauthfile

systemctl restart beegfs-mgmtd.service

Configure metadata service on mds-1

Each metadata server will have 8 metadata services:

mkdir -p /beegfs/pool1
mkdir -p /beegfs/pool2

mkfs.ext4 -m 0 /dev/sda
mkfs.ext4 -m 0 /dev/sdb

mount /dev/sda /beegfs/pool1
mount /dev/sdb /beegfs/pool2

mkdir /beegfs/pool1/meta1
mkdir /beegfs/pool1/meta2
mkdir /beegfs/pool1/meta3
mkdir /beegfs/pool1/meta4
mkdir /beegfs/pool2/meta1
mkdir /beegfs/pool2/meta2
mkdir /beegfs/pool2/meta3
mkdir /beegfs/pool2/meta4

mkdir /etc/beegfs/pool1meta1.d
mkdir /etc/beegfs/pool1meta2.d
mkdir /etc/beegfs/pool1meta3.d
mkdir /etc/beegfs/pool1meta4.d
mkdir /etc/beegfs/pool2meta1.d
mkdir /etc/beegfs/pool2meta2.d
mkdir /etc/beegfs/pool2meta3.d
mkdir /etc/beegfs/pool2meta4.d

/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool1/meta1 -s 111 -S mds-1-p1m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta2.d/beegfs-meta.conf -p /beegfs/pool1/meta2 -s 112 -S mds-1-p1m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta3.d/beegfs-meta.conf -p /beegfs/pool1/meta3 -s 113 -S mds-1-p1m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta4.d/beegfs-meta.conf -p /beegfs/pool1/meta4 -s 114 -S mds-1-p1m4 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool2/meta1 -s 121 -S mds-1-p2m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool2meta2.d/beegfs-meta.conf -p /beegfs/pool2/meta2 -s 122 -S mds-1-p2m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool3meta3.d/beegfs-meta.conf -p /beegfs/pool2/meta3 -s 123 -S mds-1-p2m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool4meta4.d/beegfs-meta.conf -p /beegfs/pool2/meta4 -s 124 -S mds-1-p2m4 -m 10.3.10.5 -C

cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool1meta1.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool1meta2.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool1meta3.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool1meta4.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool2meta1.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool2meta2.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool2meta3.d/beegfs-meta.conf
cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/pool2meta4.d/beegfs-meta.conf

Edit these 5 lines in each file:

# /etc/beegfs/pool1meta1.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool1/meta1
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8115
connMetaPortUDP              = 8115

# /etc/beegfs/pool1meta2.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool1/meta2
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8125
connMetaPortUDP              = 8125

# /etc/beegfs/pool1meta3.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool1/meta3
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8135
connMetaPortUDP              = 8135

# /etc/beegfs/pool1meta4.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool1/meta4
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8145
connMetaPortUDP              = 8145

# /etc/beegfs/pool2meta1.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool2/meta1
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8215
connMetaPortUDP              = 8215

# /etc/beegfs/pool2meta2.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool2/meta2
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8225
connMetaPortUDP              = 8225

# /etc/beegfs/pool2meta3.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool2/meta3
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8235
connMetaPortUDP              = 8235

# /etc/beegfs/pool2meta4.d/beegfs-meta.conf
. . .
sysMgmtdHost                 = 10.3.10.5
storeMetaDirectory           = /beegfs/pool2/meta4
connAuthFile                 = /etc/beegfs/connauthfile
connMetaPortTCP              = 8245
connMetaPortUDP              = 8245

Start services:

systemctl disable beegfs-meta
systemctl restart beegfs-meta@pool1meta1.service
systemctl restart beegfs-meta@pool1meta2.service
systemctl restart beegfs-meta@pool1meta3.service
systemctl restart beegfs-meta@pool1meta4.service
systemctl restart beegfs-meta@pool2meta1.service
systemctl restart beegfs-meta@pool2meta2.service
systemctl restart beegfs-meta@pool2meta3.service
systemctl restart beegfs-meta@pool2meta4.service

Check services:

systemctl | grep beegfs

Create file to make beegfs utils to work:

[root@mds-1 ~]# cat /etc/beegfs/beegfs-client.conf
sysMgmtdHost                  = 10.3.10.5
connAuthFile                  = /etc/beegfs/connauthfile
connMgmtdPortTCP              = 8008
connMgmtdPortUDP              = 8008

Test:

beegfs-ctl --listnodes --nodetype=meta
beegfs-df

Configure metadata service on mds-2

The same steps as on mds-1 except beegfs-mgmtd service (is not needed).

rsync mds-1:/etc/beegfs/connauthfile /etc/beegfs/
rsync mds-1:/etc/beegfs/beegfs-client.conf /etc/beegfs/
rsync -r mds-1:/etc/beegfs/pool* /etc/beegfs/

/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool1/meta1 -s 211 -S mds-2-p1m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta2.d/beegfs-meta.conf -p /beegfs/pool1/meta2 -s 212 -S mds-2-p1m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta3.d/beegfs-meta.conf -p /beegfs/pool1/meta3 -s 213 -S mds-2-p1m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta4.d/beegfs-meta.conf -p /beegfs/pool1/meta4 -s 214 -S mds-2-p1m4 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool2/meta1 -s 221 -S mds-2-p2m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool2meta2.d/beegfs-meta.conf -p /beegfs/pool2/meta2 -s 222 -S mds-2-p2m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool3meta3.d/beegfs-meta.conf -p /beegfs/pool2/meta3 -s 223 -S mds-2-p2m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool4meta4.d/beegfs-meta.conf -p /beegfs/pool2/meta4 -s 224 -S mds-2-p2m4 -m 10.3.10.5 -C

Configure metadata service on mds-3

The same steps as on mds-2:

/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool1/meta1 -s 311 -S mds-3-p1m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta2.d/beegfs-meta.conf -p /beegfs/pool1/meta2 -s 312 -S mds-3-p1m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta3.d/beegfs-meta.conf -p /beegfs/pool1/meta3 -s 313 -S mds-3-p1m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta4.d/beegfs-meta.conf -p /beegfs/pool1/meta4 -s 314 -S mds-3-p1m4 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool2/meta1 -s 321 -S mds-3-p2m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool2meta2.d/beegfs-meta.conf -p /beegfs/pool2/meta2 -s 322 -S mds-3-p2m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool3meta3.d/beegfs-meta.conf -p /beegfs/pool2/meta3 -s 323 -S mds-3-p2m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool4meta4.d/beegfs-meta.conf -p /beegfs/pool2/meta4 -s 324 -S mds-3-p2m4 -m 10.3.10.5 -C

Configure metadata service on mds-4

The same steps as on mds-2:

/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool1/meta1 -s 411 -S mds-4-p1m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta2.d/beegfs-meta.conf -p /beegfs/pool1/meta2 -s 412 -S mds-4-p1m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta3.d/beegfs-meta.conf -p /beegfs/pool1/meta3 -s 413 -S mds-4-p1m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta4.d/beegfs-meta.conf -p /beegfs/pool1/meta4 -s 414 -S mds-4-p1m4 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool1meta1.d/beegfs-meta.conf -p /beegfs/pool2/meta1 -s 421 -S mds-4-p2m1 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool2meta2.d/beegfs-meta.conf -p /beegfs/pool2/meta2 -s 422 -S mds-4-p2m2 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool3meta3.d/beegfs-meta.conf -p /beegfs/pool2/meta3 -s 423 -S mds-4-p2m3 -m 10.3.10.5 -C
/opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/pool4meta4.d/beegfs-meta.conf -p /beegfs/pool2/meta4 -s 424 -S mds-4-p2m4 -m 10.3.10.5 -C

Configure storage service on oss-1

rsync mds-1:/etc/beegfs/connauthfile /etc/beegfs/
rsync mds-1:/etc/beegfs/beegfs-client.conf /etc/beegfs/

mkfs.xfs /dev/sdb
mkdir -p /beegfs/storage1
mount /dev/sdb /beegfs/storage1/

Edit these lines in the configuration file /etc/beegfs/beegfs-storage.conf

sysMgmtdHost                 = 10.3.10.5
storeStorageDirectory        = /beegfs/storage1/
connAuthFile                 = /etc/beegfs/connauthfile

Setup storage:

/opt/beegfs/sbin/beegfs-setup-storage -c /etc/beegfs/beegfs-storage.conf -p /beegfs/storage1 -s 1111 -S oss-1-d1 -i 1111 -m 10.3.10.5 -C

Start storage service:

systemctl restart beegfs-storage.service

Test

beegfs-df

Configure client service on client-1

rsync mds-1:/etc/beegfs/connauthfile /etc/beegfs/

Edit these lines in the /etc/beegfs/beegfs-client.conf

sysMgmtdHost                  = 10.3.10.5
connAuthFile                  = /etc/beegfs/connauthfile

Edit this line in the /etc/beegfs/beegfs-helperd.conf

connAuthFile       = /etc/beegfs/connauthfile

Edit /etc/beegfs/beegfs-mounts.conf if need to change mount point location.

Change this line for Mellanox OFED in the file /etc/beegfs/beegfs-client-autobuild.conf

buildArgs=-j8 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include

Rebuild the kernel module: 

/etc/init.d/beegfs-client rebuild

Start client services:

systemctl restart beegfs-helperd.service
systemctl restart beegfs-client.service