Red Hat Cluster Suit (RHCS) on CentOS

Posted on Sat 01 May 2010 by Pavlo Khmel

OS: CentOS 5.4 as XEN guests
Command line installation and configuration
Disable SELinux abd Firewall:



yum groupinstall Clustering

Update /etc/hosts on all cluster nodes, I have 2. phn-node1 phn-node2

Create cluster configuration template:

$ ccs_tool create KHMEL.ORG

File will be created /etc/cluster/cluster.conf

<?xml version="1.0"?>
<cluster name="KHMEL.ORG" config_version="1">

But it is not ready yet.

We need to add fence device.

ccs_tool addfence -C phn-xen-fence fence_xvm

-ะก # do not run "ccs_tool update" after chenges
phn-xen-fence # random name
fence_xvm # choose supported agent:

APC Power Device - fence_apc
APC Power Device (SNMP) - fence_apc_snmp
IBM Blade Center - fence_bladecenter
Brocade Switch - fence_brocade
Bull PAP - fence_bullpap
DRAC - fence_drac
Egenera SAN Controller - fence_egenera
Global Network Block Device - fence_gnbd
HP ILO Device - fence_ilo
IPMI Lan - fence_ipmilan
Manual Fencing - fence_manual
McDATA SAN Switch - fence_mcdata
RPS10 Serial Switch - fence_rps10
IBM RSA II Device - fence_rsa
QLogic SANBox2 - fence_sanbox2
SCSI Reservation Fencing - fence_scsi
Vixel SAN Switch - fence_vixel
WTI Power Device - fence_wti
Virtual Machine Fencing - fence_xvm

Add nodes to the cluster:

ccs_tool addnode -C phn-node1 -n 1 -f phn-xen-fence
ccs_tool addnode -C phn-node2 -n 2 -f phn-xen-fence

phn-nodeX # node name like in /etc/hosts
-n 1 # node ID
-f phn-xen-fence # our random name for fence device.

Add manually to the file /etc/cluster/cluster.conf

       <apache config_file="" httpd_options="" name="phn-apache" server_root="" shutdown_wait=""/>
    <service autostart="1" name="phn-service">
         <apache ref="phn-apache"/>


$ chkconfig --list | grep -e cman -e rgmanager
cman            0:off   1:off   2:off   3:off   4:off   5:off   6:off
rgmanager       0:off   1:off   2:off   3:off   4:off   5:off   6:off
chkconfig cman on
chkconfig rgmanager on

Run cman on both nodes simultaneously

$ service cman start
Starting cluster:
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing... done
                                                           [ OK  ]

Run rgmanager on both nodes:

$ service rgmanager start
Starting Cluster Service Manager:                          [ OK  ]

Update configuration

ccs_tool update /etc/cluster/cluster.conf
cman_tool version -r 21

Fencing configuration on XEN guests.

XEN host should be a cluster node but not a node in existent cluster. We can create additional cluster with one node.

Possible you will need to install libvirt.

yum install libvirt

Generate key for nodes on XEN host Domain-0.

dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4096 count=1
chmod 600 /etc/cluster/fence_xvm.key
scp /etc/cluster/fence_xvm.key root
scp /etc/cluster/fence_xvm.key root

Run XEN on host:


Autorun in /etc/cluster/cluster.conf add "fence_xvmd"

<?xml version="1.0" ?>
<cluster config_version="9" name="XEN-host">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
                <clusternode name="xen-node" nodeid="1" votes="1">
                   <milticast addr="255.0.012" interface="eth0"/>
                                <method name="1">
                                        <device domain="man-fence" name="xen-node"/>
                <fencedevice agent="fence_manual" name="man-fence"/>

Add at the begining of startup script CMAN /etc/rc.d/init.d/cman


Manual test

fence_xvm -H phn-node1 -ddd

Try to stop NIC on one node

ifdown eth0

Other RHCS notes

Check cluster status

clustat -i 5

Check cluster manager status

cman_tool status

Check cluster services start on boot

$ chkconfig --list | grep -e ccsd -e cman -e fenced -e rgmanager
fenced          0:off   1:off   2:on    3:on    4:on    5:on    6:off
ccsd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
rgmanager       0:off   1:off   2:on    3:on    4:on    5:on    6:off
cman            0:off   1:off   2:on    3:on    4:on    5:on    6:off

Stop service

clusvcadm -d my_service
clusvcadm -e my_service

Start and stop all cluster services

/etc/cluster/ start | stop

Move service

$ clusvcadm -r <service name> -m <destination node>

If problem to stop service

$ service cman stop
Stopping cluster:
   Stopping fencing... done
   Stopping cman... failed
/usr/sbin/cman_tool: Error leaving cluster:

Try force

cman_tool leave force
# or
cman_tool leave force remove

Manual cluster start

service ccsd start
service qdiskd start
service cman start
service fenced start
service rgmanager start

Manual cluster stop

service qdiskd stop
service rgmanager stop
service fenced stop
service cman stop
service ccsd stop

Cluster config update

ccs_tool update /etc/cluster/cluster.conf
cman_tool version -r 21

Work configuration /etc/cluster/cluster.conf

<?xml version="1.0"?>
<cluster name="KHMEL.ORG" config_version="32">
    <clusternode name="phn-node1.mydomain" votes="1" nodeid="1">
           <method name="single">
             <device domain="phn-node1.mydomain" name="phn-xen-fence"/>
    <clusternode name="phn-node2.mydomain" votes="1" nodeid="2">
          <method name="single">
            <device domain="phn-node2.mydomain" name="phn-xen-fence"/>
  <cman two_node="1" expected_votes="1"/>
      <fencedevice name="phn-xen-fence" agent="fence_xvm"/>
       <failoverdomain name="filover-domain" ordered="0" restricted="0">
          <failoverdomainnode name="phn-node1.mydomain" priority="1"/>
          <failoverdomainnode name="phn-node2.mydomain" priority="2"/>
       <apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="phn-apache" server_root="/etc/httpd/" shutdown_wait=""/>
       <ip address="" monitor_link="1"/>
    <service autostart="1" domain="filover-domain" name="phn-service" recovery="relocate">
          <apache ref="phn-apache"/>
          <ip ref=""/>