VM encapsulation in a service

Managing VM as services has many advantages:

Same start/stop/status commands accross non-virtualized services on different operating systems and services encapsulating virtual machines from different technologies (kvm, xen, hpvm, zone, lxc, ...)

Mix virtualized and non-virtualized services on the same hosts without increasing complexity (except xen)

Seemless integration of hypervisors in the disaster recovery plan

Benefit from all the OpenSVC data replication modes

Benefit from the OpenSVC Collector performance monitoring, obsolescence management, alarming, and more.

OpenSVC deployment on an existing hypervisor is a straight-forward process which does not require service disruption. This cookbook details the steps involved:

Install a package

Configure some ssh key trusts

Setup a node configuration file (single word)

Setup a service configuration file (~15 lines)

Nodeware installation

Package

Download the lastest 'opensvc' package available for your operating system of choice from http://repo.opensvc.com and install it. Depending on the operating system, and operating system version, you might need to satisfy dependencies using packages provided on this website.

The package post-installation steps are handled by the /opt/opensvc/bin/postinstall script. This script can be safely executed on a node where opensvc is already installed.

Files

The package installs the following directory tree:

/opt/opensvc/
/opt/opensvc/etc
/opt/opensvc/tmp
/opt/opensvc/bin
/opt/opensvc/bin/pkg
/opt/opensvc/bin/cron
/opt/opensvc/var
/opt/opensvc/var/sync
/opt/opensvc/var/lock
/opt/opensvc/usr
/opt/opensvc/usr/share
/opt/opensvc/usr/share/doc
/opt/opensvc/lib
/opt/opensvc/log

Cron jobs

The package installs the following cron jobs, in (by order of preference), /etc/cron.d/opensvc, /var/spool/cron/crontabs/root, /var/spool/cron/root :

0,10,20,30,40,50 * * * * [ -x /opt/opensvc/bin/svcmon ] && /opt/opensvc/bin/svcmon --updatedb >/dev/null 2>&1
0 4 * * * [ -x /opt/opensvc/bin/cron/opensvc.daily ] && /opt/opensvc/bin/cron/opensvc.daily >/dev/null 2>&1

Keys

If the root account has no ssh key, a 1024 bits dsa key is generated by the package post-install. Production node keys must be trusted on all cluster nodes (PRD and DRP), whereas the keys of disaster recovery servers must not be trusted by production nodes. This setup is used for rsync file transfers and remote command execution.

Set host mode

The valid host mode values are PRD, DEV, TMP. The setting is stored in /opt/opensvc/var/host_mode. No comments are allowed in this file.

The host_mode setting is used to enforce the following policies:

Only PRD services are allowed to start on a PRD node

Only PRD nodes are allowed to push data to a PRD node

Configuration for collector usage

The collector is contacted by the node using the generic name dbopensvc. This name should be known to your prefered resolving mecanism : hosts, dns, ... If you choose to use the internet shared collector, the corresponding ip adress must be set to the adress of collector.opensvc.com.

You can accelerate the node discovery by forcing the execution of both opensvc cronjobs after the package installation.

HP-UX specificities

The python package provided by HP will output garbage on exec because it won't find terminfo at the expected places. To fix that, you have to export TERMINFO=/usr/share/lib/terminfo from /etc/profile

The HP-UX base system does not provide tools to handle scsi persistent reservations. You have to install the scu tool if you want to activate this feature.

Linux LVM2 specificities

Opensvc controls volume group activation and desactivation. Most Linux distributions activate all visible volume groups at boot, some even re-activate them upon de-activation events. These mecanisms can be disabled using the following setup. It also provides another protection against unwanted volume group activation from a secondary cluster node.

This setup tells LVM2 commands to activate only the objects tagged with the hostname. Opensvc makes sure the tags are set on start and unset on stop. Opensvc also purges all tags before adding the one it needs to activate a volume group, so opensvc can satisfy a start request on a service uncleanly shut down.

/etc/lvm/lvm.conf

Add the following root-level configuration node

tags {
    hosttags = 1
    local {}
}

And add the 'local' tag to all local volume groups. For example:

vgchange --addtag local rootvg

Finally you need to rebuild the initrd/initramfs to prevent shared vg activation at boot.

/etc/lvm/lvm_{node}.conf

Create this file, {node} being the output of uname -n and add the following configuration.

activation { volume_list = ["@{node}"] }

Service creation

Choose a service name

Guidelines:

Between 8 to 15 characters

Make application code part of the name (first)

Make service type (PRD, DEV) part of the name (second)

Make the last part descriptive of the application role (web, transco)

In case of multiple instance of similar services, happen a number as a suffix

This guidelines result in names like gieprdtransco01, infraprddns01, ...

Following this naming will help grasp importance, clients and role of service from the output of /opt/opensvc/bin/svcmon

Create configuration files

Service configuration files are in /opt/opensvc/etc. Each service must have these three files present to be fully functional. Services using the internet shared collector must be named using the domainname as a suffix to avoid naming conflicts.

/opt/opensvc/etc/unxdevweb01.mydomain.com -> ../bin/svcmgr
/opt/opensvc/etc/unxdevweb01.mydomain.com.env
/opt/opensvc/etc/unxtstsvc01.mydomain.com.d -> /unxtstscv01/etc/init.d

Configuration files role

/opt/opensvc/etc/unxdevweb01.mydomain.com -> ../bin/svcmgr
This symbolic link is meant to be used as a shortcut to pass commands to a specific service. Like /opt/opensvc/etc/unxdevweb01.mydomain.com start for example

/opt/opensvc/etc/unxdevweb01.mydomain.com.env
This is the configuration file proper, including service description and resource definitions. A fully commented template is available on each node at /opt/opensvc/usr/share/doc/template.env. More on this below.

/opt/opensvc/etc/unxdevweb01.mydomain.com.d -> /unxtstscv01/etc/init.d
This symbolic link points to the directory hosting the service application launchers. The service is not considered active if this link is not present. The directory pointed is best hosted on a service-dedicated filesystem. The service application launchers are expected to be in SysV style: [SK][0-9]*appname. S for starters, K for stoppers, number for ordering. Starters and stoppers can be symlink to a single script. Starter are passed 'start' as first parameter, stoppers are passed 'stop' as first parameter.

Customize the service env file

A typical VM service env file should look like:

[default]
vm_name = vm188
app = ERP
comment = recette gen db #2
mode = hpvm
service_type = DEV
nodes =  node109 node110 node111 node112
autostart_node = node109
;drpnode = vm5
;scsireserv = false

[ip#1]
ipname = vm188
ipdev = lan0

[vmdg]
scsireserv = false

vm_name

If this parameter is not set, the VM name defaults to the service name. When encapsulating an existing VM, chances are that you need to set it to the existing VM hostname. This name is used by the hypervisor to communicate with the VM, so you might need to use a fully qualified name if the hypersisor and the VM do not share the same DNS domain. Communication with zones use the zlogin utility, so fqdn are never needed in this case.

mode

Choose among kvm, xen, hpvm, zone and lxc.

nodes

Set to the list of hypervisors able to run the virtual machine in a normal situation.

autostart_node

Set to the hypervisor where the virtual machine should run in a normal situation.

drpnode

Set to the hypervisor where the virtual machine should run in a disaster recovery situation.

drpnodes

Set to the list of hypervisors where the virtual machine may run in a disaster recovery situation.

scsireserv

Set this parameter to true if:

the disks are shared amongst hypervisors

and the disks are dedicated to the service

and the disks support the scsi3 persistent reservation commands

ip resources

All OpenSVC VM drivers except 'zone' leave the ip plumbing to the guest operating system. The ip resources described in the env file are used to check the resource availability. This checks are:

ping 'ipname'

and execute in the context of the guest operating system a command verifying the plumbing on the 'ipdev' adapter.

vmdg resource

Add this resource configlet if you pass-through block devices not handled by a volume manager to the virtual machine. The 'vmdg' resource is a special kind of 'vg' resource whose disklist is obtained from the virtual machine configuration. 'start' and 'stop' OpenSVC commands are reduced to scsi reservation handling for this resource as the necessary operations are taken care of by the hypervisor software. It is also necessary for disk inventory completeness.

Trust the hypervisor-to-hypervisor ssh root sessions

OpenSVC use ssh as root to execute commands in the other hypervisors context. Such commands are limited to:

remote host mode checking

remote host replication target mount point checking

Trust the hypervisor-to-guest ssh root sessions

This step does not apply to zones, as zlogin is always trusted. Other drivers use ssh as root to execute commands in the guest context. Such commands are limited to:

ip resources health checking

application start-up scripts execution in the context of the guest

Populate the application startup scripts directory

This step is recommended but not mandatory. OpenSVC command set allows to start the virtual machine but not the embedded applications through the 'startapp'/'stopapp' commands. For this feature to work as expected, startup scripts should not reside in the operating system's proposed infrastructure (/etc/rcX.d, /sbin/rcX.d, DMF, ...). OpenSVC expects to find app launchers in /svc/etc/init.d in the guest file hierarchy.

Test

You should now be able to run succesfully

/opt/opensvc/etc/yoursvc print_status
/opt/opensvc/etc/yoursvc push
/opt/opensvc/etc/yoursvc diskupdate
/opt/opensvc/etc/yoursvc stop
/opt/opensvc/etc/yoursvc start

OpenSVC

General

Node

Services

Collector

Storage

Cookbook