OS: CentOS 5.4 as XEN guests
Command line installation and configuration
Disable SELinux abd Firewall:
system-config-securitylevel-tui
Install
yum groupinstall Clustering
Update /etc/hosts on all cluster nodes, I have 2.
192.168.0.11 phn-node1
192.168.0.12 phn-node2
Create cluster configuration template:
$ ccs_tool create KHMEL.ORG
File will be created /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="KHMEL.ORG" config_version="1">
<clusternodes/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
But it is not ready yet.
We need to add fence device.
ccs_tool addfence -C phn-xen-fence fence_xvm
Explanation:
-ะก # do not run "ccs_tool update" after chenges
phn-xen-fence # random name
fence_xvm # choose supported agent:
APC Power Device - fence_apc
APC Power Device (SNMP) - fence_apc_snmp
IBM Blade Center - fence_bladecenter
Brocade Switch - fence_brocade
Bull PAP - fence_bullpap
DRAC - fence_drac
Egenera SAN Controller - fence_egenera
Global Network Block Device - fence_gnbd
HP ILO Device - fence_ilo
IPMI Lan - fence_ipmilan
Manual Fencing - fence_manual
McDATA SAN Switch - fence_mcdata
RPS10 Serial Switch - fence_rps10
IBM RSA II Device - fence_rsa
QLogic SANBox2 - fence_sanbox2
SCSI Reservation Fencing - fence_scsi
Vixel SAN Switch - fence_vixel
WTI Power Device - fence_wti
Virtual Machine Fencing - fence_xvm
Add nodes to the cluster:
ccs_tool addnode -C phn-node1 -n 1 -f phn-xen-fence
ccs_tool addnode -C phn-node2 -n 2 -f phn-xen-fence
Explanation:
phn-nodeX # node name like in /etc/hosts
-n 1 # node ID
-f phn-xen-fence # our random name for fence device.
Add manually to the file /etc/cluster/cluster.conf
<rm>
<failoverdomains/>
<resources>
<apache config_file="" httpd_options="" name="phn-apache" server_root="" shutdown_wait=""/>
</resources>
<service autostart="1" name="phn-service">
<apache ref="phn-apache"/>
</service>
</rm>
Autorun
$ chkconfig --list | grep -e cman -e rgmanager
cman 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rgmanager 0:off 1:off 2:off 3:off 4:off 5:off 6:off
chkconfig cman on
chkconfig rgmanager on
Run cman on both nodes simultaneously
$ service cman start
Starting cluster:
Loading modules... done
Mounting configfs... done
Starting ccsd... done
Starting cman... done
Starting daemons... done
Starting fencing... done
[ OK ]
Run rgmanager on both nodes:
$ service rgmanager start
Starting Cluster Service Manager: [ OK ]
Update configuration
ccs_tool update /etc/cluster/cluster.conf
cman_tool version -r 21
Fencing configuration on XEN guests.
XEN host should be a cluster node but not a node in existent cluster. We can create additional cluster with one node.
Possible you will need to install libvirt.
yum install libvirt
Generate key for nodes on XEN host Domain-0.
dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4096 count=1
chmod 600 /etc/cluster/fence_xvm.key
scp /etc/cluster/fence_xvm.key root 192.168.0.11:/etc/cluster
scp /etc/cluster/fence_xvm.key root 192.168.0.12:/etc/cluster
Run XEN on host:
/sbin/fence_xvmd
Autorun in /etc/cluster/cluster.conf add "fence_xvmd"
<?xml version="1.0" ?>
<cluster config_version="9" name="XEN-host">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="xen-node" nodeid="1" votes="1">
<milticast addr="255.0.012" interface="eth0"/>
<fence>
<method name="1">
<device domain="man-fence" name="xen-node"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fence_xvmd/>
<fencedevices>
<fencedevice agent="fence_manual" name="man-fence"/>
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
Add at the begining of startup script CMAN /etc/rc.d/init.d/cman
FENCE_XVMD_OPTS="-I eth0"
Manual test
fence_xvm -H phn-node1 -ddd
Try to stop NIC on one node
ifdown eth0
Other RHCS notes
Check cluster status
clustat -i 5
Check cluster manager status
cman_tool status
Check cluster services start on boot
$ chkconfig --list | grep -e ccsd -e cman -e fenced -e rgmanager
fenced 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ccsd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rgmanager 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cman 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Stop service
clusvcadm -d my_service
clusvcadm -e my_service
Start and stop all cluster services
/etc/cluster/cl_service.sh start | stop
Move service
$ clusvcadm -r <service name> -m <destination node>
If problem to stop service
$ service cman stop
Stopping cluster:
Stopping fencing... done
Stopping cman... failed
/usr/sbin/cman_tool: Error leaving cluster:
Try force
cman_tool leave force
# or
cman_tool leave force remove
Manual cluster start
service ccsd start
service qdiskd start
service cman start
service fenced start
service rgmanager start
Manual cluster stop
service qdiskd stop
service rgmanager stop
service fenced stop
service cman stop
service ccsd stop
Cluster config update
ccs_tool update /etc/cluster/cluster.conf
cman_tool version -r 21
Work configuration /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="KHMEL.ORG" config_version="32">
<clusternodes>
<clusternode name="phn-node1.mydomain" votes="1" nodeid="1">
<fence>
<method name="single">
<device domain="phn-node1.mydomain" name="phn-xen-fence"/>
</method>
</fence>
</clusternode>
<clusternode name="phn-node2.mydomain" votes="1" nodeid="2">
<fence>
<method name="single">
<device domain="phn-node2.mydomain" name="phn-xen-fence"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman two_node="1" expected_votes="1"/>
<fencedevices>
<fencedevice name="phn-xen-fence" agent="fence_xvm"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="filover-domain" ordered="0" restricted="0">
<failoverdomainnode name="phn-node1.mydomain" priority="1"/>
<failoverdomainnode name="phn-node2.mydomain" priority="2"/>
</failoverdomain>
</failoverdomains>
<resources>
<apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="phn-apache" server_root="/etc/httpd/" shutdown_wait=""/>
<ip address="192.168.0.13" monitor_link="1"/>
</resources>
<service autostart="1" domain="filover-domain" name="phn-service" recovery="relocate">
<apache ref="phn-apache"/>
<ip ref="192.168.0.13"/>
</service>
</rm>
</cluster>