High Availabity setup

This cookbook presents the steps to setup an automated OpenSVC service failover using open-ha for the heartbeat. open-ha is our preferred heartbeat daemon because it runs on all Unix operating systems, proposes multiple multicast heartbeat paths using a mix of IP and SCSI, and has a flexible trigger support.

Best practice for heartbeat infrastructure

• Try to configure an even number of heartbeats

• Try to configure one heartbeat on a dedicated LAN

• Don't use the same LAN twice for heartbeat

• Don't use LANs with a common point of failure

Installation

OpenSVC

This cookbook focuses on adding HA to an existing OpenSVC clustered service. Please refer to:

• Nodeware installation

• Service creation

Open-HA

Download open-ha from here.

The default location of all this program's files is /usr/local/cluster. Before using any open-ha commands, you have to setup some environment variables:

# export EZ=/usr/local/cluster
# . /usr/local/cluster/env.sh

Setup

Describe cluster nodes

In file /usr/local/cluster/conf/nodes:

node108
node109

Note:

• No space at EOL

• One node per line

• All members of this cluster must be listed

Describe heartbeat paths

To setup 3 multicast heartbeats, in file /usr/local/cluster/conf/monitor:

node108  net  eth1  239.131.50.10    1780  60
node108  net  eth2  239.131.51.10    1780  60
node108  net  eth3  239.131.52.10    1780  60
node109  net  eth1  239.131.50.10    1781  60
node109  net  eth2  239.131.51.10    1781  60
node109  net  eth3  239.131.52.10    1781  60

Where:

• param#1: node hostname

• param#2: heartbeat type

• param#3: multicast IP

• param#4: port number

• param#5: heartbeat period

Note:

• Each heartbeat end-point has a unique port number

• Each heartbeat path has a unique multicast IP

• The monitor file can be copied as-is to the other, except when Solaris IPMP is active (more listeners than senders in this case).

Add services

$EZ_BIN/service -a svc1 /opt/opensvc/etc/svc_name node108 node109 /bin/true

Where:

• '-a': add open-ha service

• param#1: service name, align to OpenSVC is not required, advised anyway

• param#2: primary node

• param#3: secondary node

• param#4: service init script

Startup

Make sure the heartbeat daemons are setup for startup at boot. If not, depending on your system, apply some commands simalar to:

ln -s /usr/local/cluster/bin/ezha /etc/rc3.d/S99cluster
ln -s /usr/local/cluster/bin/ezha /etc/rc0.d/K01cluster
ln -s /usr/local/cluster/bin/ezha /etc/rcS.d/K01cluster

Then run:

/usr/local/cluster/bin/ezha start

Daemons

Starting the ezha init script will spawn the following daemons:

• nmond: state transition and trigger initiator daemon

• heartc: multicast listener daemons, one per local heartbeat receiving end-point

• heartd: multicast sender daemons, one per local heartbeat sending end-point

Use

Display the service cluster status

# /usr/local/cluster/bin/service -s mysvc
service       owned_by  nodeA        nodeB
svc1          nodeB     FROZEN_STOP  STARTED
svc2          nodeA     STARTED      STOPPED

Service status

• STOPPED: last run of the opensvc service shutdown command returned successfully

• STARTED: last run of the opensvc service startup command returned successfully

• STOP_FAILED: last run of the opensvc service shutdown command returned some error. open-ha restrains from running further service commands.

• START_FAILED: last run of the opensvc service startup command returned some error. open-ha restrains from running further service commands.

• STOPPING: last run of the opensvc service shutdown command is pending completion.

• STARTING: last run of the opensvc service startup command is pending completion.

• START_READY: node is ready to take over the service.

• FROZEN_STOP: state reached after a successful 'freeze-stop' open-ha command. open-ha restrains from running further service commands.

• FROZEN_START: state reached after freezing through open-ha a stopped service. open-ha restrains from running further service commands.

Trigger a failover

When a service is handled to any heartbeat daemon, you must not use stop/start OpenSVC commands to drive this service: the heartbeat daemon being in charge, you must use its command set only. On the node where the service is RUNNING:

# /usr/local/cluster/bin/service -s mysvc -a freeze-stop

# /usr/local/cluster/bin/service -s mysvc -a unfreeze