VM encapsulation in a service

Managing VM as services has many advantages:

Same start/stop/status commands accross non-virtualized services on different operating systems and services encapsulating virtual machines from different technologies (kvm, xen, esx, hpvm, zone, lxc, vz, ...)

Mix virtualized and non-virtualized services on the same hosts without increasing complexity (except xen, ldom, esx)

Seemless integration of hypervisors in the disaster recovery plan

Benefit from all the OpenSVC data replication modes

Benefit from the OpenSVC Collector performance monitoring, obsolescence management, alarming, and more.

OpenSVC deployment on an existing hypervisor is a straight-forward process which does not require service disruption. This cookbook details the steps involved:

Install a package

Configure some ssh key trusts

Setup a node configuration file (single word)

Setup a service configuration file (~15 lines)

Nodeware installation

Package

Download the lastest 'opensvc' package available for your operating system of choice from http://repo.opensvc.com and install it. Depending on the operating system, and operating system version, you might need to satisfy dependencies using packages provided on this website.

The package post-installation steps are handled by the /opt/opensvc/bin/postinstall script. This script can be safely executed on a node where opensvc is already installed.

Files

The package installs the following directory tree:

/opt/opensvc/
/opt/opensvc/etc
/opt/opensvc/tmp
/opt/opensvc/bin
/opt/opensvc/bin/pkg
/opt/opensvc/bin/cron
/opt/opensvc/var
/opt/opensvc/var/sync
/opt/opensvc/var/lock
/opt/opensvc/usr
/opt/opensvc/usr/share
/opt/opensvc/usr/share/doc
/opt/opensvc/lib
/opt/opensvc/log

Cron jobs

The package installs the following cron jobs, in (by order of preference), /etc/cron.d/opensvc, /var/spool/cron/crontabs/root, /var/spool/cron/root :

0,10,20,30,40,50 * * * * [ -x /opt/opensvc/bin/svcmon ] && /opt/opensvc/bin/svcmon --updatedb --maxdelaydb 120 >/dev/null 2>&1
0,10,20,30,40,50 * * * * [ -x /opt/opensvc/bin/cron/opensvc ] && /opt/opensvc/bin/cron/opensvc >/dev/null 2>&1

Linux and HP-UX also have a helper to gather statistics:

0,10,20,30,40,50 * * * * root [ -x /opt/opensvc/bin/perfagt.Linux ] && /opt/opensvc/bin/perfagt.Linux >/dev/null 2>&1

Keys

If the root account has no ssh key, a 1024 bits dsa key is generated by the package post-install. Production node keys must be trusted on all cluster nodes (PRD and DRP), whereas the keys of disaster recovery servers must not be trusted by production nodes. This setup is used for rsync file transfers and remote command execution.

Set host mode

/opt/opensvc/bin/nodemgr set --param node.host_mode --value PRD

The valid host mode values are PRD, DEV, TMP. The setting is stored in /opt/opensvc/etc/node.conf. On a fresh install this file does not exist. Note that the /opt/opensvc/var/host_mode file is deprecated. Upgrading the new OpenSVC package on a system with a /opt/opensvc/var/host_mode file will move the value to /opt/opensvc/etc/node.conf.

The host_mode setting is used to enforce the following policies:

Only PRD services are allowed to start on a PRD node
Only PRD nodes are allowed to push data to a PRD node

Set schedules

Schedules are defined in /opt/opensvc/etc/node.conf. The OpenSVC nodeware schedules the following actions:

Action	Description	Section name	Default
pushstats	Send to the collector the collected performance metrics since last pushstats	`[stats]`	4am-6am daily
pushasset	Send to the collector the collected node information (cpu, memory, board, ...)	`[asset]`	4am-6am daily
pushpkg	Send to the collector the node installed package database	`[packages]`	4am-6am daily
pushpatch	Send to the collector the node installed patch database	`[patches]`	4am-6am daily
checks	Send to the collector the node checks results (fs_u, fs_i, vg_u, ...)	`[checks]`	4am-6am daily
compliance check	Send to the collector the compliance modules checks results	`[comp]`	5am-6am sundays
pushservices	Send to the collector the service configurations	`[svcconf]`	4am-6am daily
syncservices	Try a sync run on all node services	None	4am-6am daily

A typical schedule definition would look like:

[section name]
interval = 1439
days = ["saturday", "sunday"]
period = [["02:00", "04:00"], ["19:00", "21:00"]]

A reference node.conf file can be found on the node at /opt/opensvc/usr/share/doc/node.conf

The syncservices action schedule is defined by the individual sync resource definitions in service configuration files, where sync_interval, sync_days and sync_period parameters are available.

Configuration for collector usage

By default, the collector is contacted by the node using the generic name dbopensvc on ports 80 and 8000. This name should be known to your prefered resolving mecanism : hosts, dns, ... If you choose to use the internet shared collector, the corresponding ip address must be set to the address of collector.opensvc.com.

To override the default collector's xmlrpc urls, you can set them in node.conf:

/opt/opensvc/bin/nodemgr set --param node.dbopensvc --value https://collector.opensvc.com/feed/default/call/xmlrpc
/opt/opensvc/bin/nodemgr set --param node.dbcompliance --value https://collector.opensvc.com/init/compliance/call/xmlrpc

This override is recommended for xmlrpc encryption.

The collector requires the nodes to provide an authentication token (shared secret) with each request. The token is forged by the collector and stored on the node in node.conf. The token initialization is handled by the command:

/opt/opensvc/bin/nodemgr register

Finally, you can accelerate the node discovery by forcing the execution of opensvc cron jobs after the package installation.

/opt/opensvc/bin/nodemgr --force pushasset
/opt/opensvc/bin/nodemgr --force pushpkg
/opt/opensvc/bin/nodemgr --force pushpatch
/opt/opensvc/bin/nodemgr --force pushstats
/opt/opensvc/bin/nodemgr --force pushservices
/opt/opensvc/bin/nodemgr --force checks

To disable collector communications, use:

/opt/opensvc/bin/nodemgr set --param node.dbopensvc --value None
/opt/opensvc/bin/nodemgr set --param node.dbcompliance --value None

HP-UX specificities

The python package provided by HP will output garbage on exec because it won't find terminfo at the expected places. To fix that, you have to export TERMINFO=/usr/share/lib/terminfo from /etc/profile

The HP-UX base system does not provide tools to handle scsi persistent reservations. You have to install the scu tool if you want to activate this feature.

Linux LVM2 specificities

Opensvc controls volume group activation and desactivation. Most Linux distributions activate all visible volume groups at boot, some even re-activate them upon de-activation events. These mecanisms can be disabled using the following setup. It also provides another protection against unwanted volume group activation from a secondary cluster node.

This setup tells LVM2 commands to activate only the objects tagged with the hostname. Opensvc makes sure the tags are set on start and unset on stop. Opensvc also purges all tags before adding the one it needs to activate a volume group, so opensvc can satisfy a start request on a service uncleanly shut down.

/etc/lvm/lvm.conf

Add the following root-level configuration node

tags {
    hosttags = 1
    local {}
}

And add the 'local' tag to all local volume groups. For example:

vgchange --addtag local rootvg

Finally you need to rebuild the initrd/initramfs to prevent shared vg activation at boot.

/etc/lvm/lvm_{node}.conf

Create this file, {node} being the output of uname -n and add the following configuration.

activation { volume_list = ["@local", "@{node}"] }

Service creation

Choose a service name

Guidelines:

Between 8 to 15 characters

Make application code part of the name (first)

Make service type (PRD, DEV) part of the name (second)

Make the last part descriptive of the application role (web, transco)

In case of multiple instance of similar services, happen a number as a suffix

This guidelines result in names like gieprdtransco01, infraprddns01, ...

Following this naming will help grasp importance, clients and role of service from the output of /opt/opensvc/bin/svcmon

Create configuration files

Those are created automatically by the svcmgr create command. Experienced users will find it easier to start from a copy of the env file of an existing similar service. The following information describes the steps needed to create a service manually.

Service configuration files are in /opt/opensvc/etc. Each service must have these three files present to be fully functional. Services using the internet shared collector must be named using the domainname as a suffix to avoid naming conflicts.

/opt/opensvc/etc/unxdevweb01.mydomain.com -> ../bin/svcmgr
/opt/opensvc/etc/unxdevweb01.mydomain.com.env
/opt/opensvc/etc/unxtstsvc01.mydomain.com.d -> /unxtstscv01/etc/init.d
or
/opt/opensvc/etc/unxtstsvc01.mydomain.com.d -> unxdevweb01.mydomain.com.dir
/opt/opensvc/etc/unxdevweb01.mydomain.com.dir

Configuration files role

/opt/opensvc/etc/unxdevweb01.mydomain.com -> ../bin/svcmgr
This symbolic link is meant to be used as a shortcut to pass commands to a specific service. Like /opt/opensvc/etc/unxdevweb01.mydomain.com start for example

/opt/opensvc/etc/unxdevweb01.mydomain.com.env
This is the configuration file proper, including service description and resource definitions. A fully commented template is available on each node at /opt/opensvc/usr/share/doc/template.env. More on this below.

/opt/opensvc/etc/unxdevweb01.mydomain.com.d -> /unxtstscv01/etc/init.d
This symbolic link points to the directory hosting the service application launchers. The service is not considered active if this link is not present. The directory pointed is best hosted on a service-dedicated filesystem. The service application launchers are expected to be in SysV style: [SK][0-9]*appname. S for starters, K for stoppers, number for ordering. Starters and stoppers can be symlink to a single script. Starter are passed 'start' as first parameter, stoppers are passed 'stop' as first parameter.

/opt/opensvc/etc/unxdevweb01.mydomain.com.dir
This optional directory can be used to store locally the startup scripts. As such, it can be linked from /opt/opensvc/etc/unxdevweb01.mydomain.com.d. OpenSVC synchronize this directory to nodes and drpnodes as part of the sync#i0 internal sync resource. If you placed your startup script on a shared volume, this .dir is not needed but you will still have to create a sync resource to send them to the drpnodes.

Customize the service env file

A typical VM service env file should look like:

[default]
vm_name = vm188
app = ERP
comment = recette gen db #2
mode = hpvm
service_type = DEV
nodes =  node109 node110 node111 node112
autostart_node = node109
;drpnode = vm5
;scsireserv = false

[ip#1]
ipname = vm188
ipdev = lan0

[vmdg]
scsireserv = false

vm_name

If this parameter is not set, the VM name defaults to the service name. When encapsulating an existing VM, chances are that you need to set it to the existing VM hostname. This name is used by the hypervisor to communicate with the VM, so you might need to use a fully qualified name if the hypersisor and the VM do not share the same DNS domain. Communication with zones use the zlogin utility, so fqdn are never needed in this case.

mode

Choose among kvm, xen, esx, hpvm, ldom, vbox, zone, vz and lxc.

nodes

Set to the list of hypervisors able to run the virtual machine in a normal situation.

autostart_node

Set to the hypervisor where the virtual machine should run in a normal situation.

drpnode

Set to the hypervisor where the virtual machine should run in a disaster recovery situation.

drpnodes

Set to the list of hypervisors where the virtual machine may run in a disaster recovery situation.

scsireserv

Set this parameter to true if:

the disks are shared amongst hypervisors

and the disks are dedicated to the service

and the disks support the scsi3 persistent reservation commands

ip resources

All OpenSVC VM drivers except 'zone' leave the ip plumbing to the guest operating system. The ip resources described in the env file are used to check the resource availability. This checks are:

ping 'ipname'

and execute in the context of the guest operating system a command verifying the plumbing on the 'ipdev' adapter.

vmdg resource

Add this resource configlet if you pass-through block devices not handled by a volume manager to the virtual machine. The 'vmdg' resource is a special kind of 'vg' resource whose disklist is obtained from the virtual machine configuration. 'start' and 'stop' OpenSVC commands are reduced to scsi reservation handling for this resource as the necessary operations are taken care of by the hypervisor software. It is also necessary for disk inventory completeness.

Trust the hypervisor-to-hypervisor ssh root sessions

OpenSVC use ssh as root to execute commands in the other hypervisors context. Such commands are limited to:

remote host mode checking

remote host replication target mount point checking

Trust the hypervisor-to-guest ssh root sessions

This step does not apply to zones, as zlogin is always trusted. Other drivers use ssh as root to execute commands in the guest context. Such commands are limited to:

ip resources health checking

application start-up scripts execution in the context of the guest

Populate the application startup scripts directory

This step is recommended but not mandatory. OpenSVC command set allows to start the virtual machine but not the embedded applications through the 'startapp'/'stopapp' commands. For this feature to work as expected, startup scripts should not reside in the operating system's proposed infrastructure (/etc/rcX.d, /sbin/rcX.d, DMF, ...). OpenSVC expects to find app launchers in /svc/etc/init.d in the guest file hierarchy.

Test

You should now be able to run succesfully

/opt/opensvc/etc/yoursvc print_status
/opt/opensvc/etc/yoursvc push
/opt/opensvc/etc/yoursvc diskupdate
/opt/opensvc/etc/yoursvc stop
/opt/opensvc/etc/yoursvc start

OpenSVC

Tools to scale

General

Node

Services

Collector

Virtualization

Storage

Compliance

Cookbook