initial checkin

Signed-off-by: Lon Hohberger <lhh@redhat.com>

Makefile

@@ -0,0 +1,23 @@
+###############################################################################
+###############################################################################
+##
+##  Copyright (C) 2006 Red Hat, Inc.
+##  
+##  This copyrighted material is made available to anyone wishing to use,
+##  modify, copy, or redistribute it subject to the terms and conditions
+##  of the GNU General Public License v.2.
+##
+###############################################################################
+###############################################################################
+
+
+all:
+	make -C common
+	make -C client
+	make -C server
+
+clean:
+	make -C common clean
+	make -C client clean
+	make -C server clean
+

README

@@ -0,0 +1,124 @@
+I. Fence_xvm - the Xen virtual machine fencing agent
+
+Fence_xvm is an agent which establishes a communications link between
+a cluster of virtual machines (VC) and a cluster of domain0/physical
+nodes which are hosting the virtual cluster.  Its operations are
+fairly simple.
+
+  (a) Start a listener service.
+  (b) Send a multicast packet requesting that a VM be fenced.
+  (c) Authenticate client.
+  (e) Read response.
+  (f) Exit with success/failure, depending on the response received.
+
+If any of the above steps fail, the fencing agent exits with a failure
+code and fencing is retried by the virtual cluster at a later time.
+Because of the simplicty of fence_xvm, it is not necessary that
+fence_xvm be run from within a virtualized guest - all it needs is
+libnspr and libnss and a shared private key (for authentication; we
+would hate to receive a false positive response from a node not in the
+cluster!).
+
+
+II. Fence_xvmd - The Xen virtual machine fencing host
+
+Fence_xvmd is a daemon which runs on physical hosts (e.g. in domain0)
+of the cluster hosting the Xen virtual cluster.  It listens on a port
+for multicast traffic from Xen virtual cluster(s), and takes actions.
+Multiple disjoint virtual clusters can coexist on a single physical
+host cluster, but this requires multiple instances of fence_xvmd.
+
+NOTE: fence_xvmd *MUST* be run on ALL nodes in a given cluster which
+will be hosting virtual machines if fence_xvm is to be used for 
+fencing!
+
+There are a couple of ways the multicast packet is handled,
+depending on the state of the host OS.  It might be hosting the VM,
+or it might not.  Furthermore, the VM might "reside" on a host which
+has failed.
+
+In order to be able to guarantee safe fencing of a VM even if the
+last- known host is down, we must store the last-known locations of
+each virtual machine in some sort of cluster-wide way.  For this, we
+use the AIS Checkpointing API, which is provided by OpenAIS.  Every
+few seconds, fence_xvmd queries the Xen Hypervisor via libvirt and
+stores any local VM states in a checkpoint.  In the event of a
+physical node failure (which consequently causes the failure of one
+or more Xen guests), we can then read the checkpoint section
+corresponding to the guest we need to fence to find out the previous
+owner.  With that information, we can then check with CMAN to see if
+the last-known host node has been fenced.  If so, then the VM is
+clean as well.  The physical cluster must, therefore, have fencing
+in order for fence_xvmd to work.
+
+Operation of a node hosting a VM which needs to be fenced:
+  
+  (a) Receive multicast packet
+  (b) Authenticate multicast packet
+  (c) Open connection to host contained within multicast
+      packet.
+  (d) Authenticate server.
+  (e) Carry out fencing operation (e.g. call libvirt to destroy or
+      reboot the VM; there is no "on" method at this point).
+  (f) If operation succeeds, send success response.
+
+Operation of high-node-ID:
+
+  (a) Receive multicast packet
+  (b) Authenticate multicast packet
+  (c) Read VM state from checkpoint
+  (d) Check liveliness of nodeID hosting VM (if alive, do nothing)
+  (e) Open connection to host contained within multicast
+      packet.
+  (f) Check with CMAN to see if last-known host has been fenced.
+  (g) If last-known host has been fenced, send success response.
+  (h) Authenticate server & send response.
+
+NOTE: There is always a possibility that a VM is started again
+before the fencing operation and checkpoint update for that VM
+occurs.  If the VM has booted and rejoined the cluster, fencing will
+not be necessary.  If it is in the process of booting, but has not
+yet joined the cluster, fencing will also not be necessary - because
+it will not be using cluster resources yet.
+
+
+III. Security considerations
+
+While fencing is generally expected to run on a more or less trusted
+network, there are cases where it may not be.
+
+* The multicast packet is subject to replay attacks, but because no
+fencing action is taken based solely on the information contained
+within the packet, this should not allow an attacker to maliciously
+fence a VM from outside the cluster, though it may be possible to
+cause a DoS of fence_xvmd if enough multicast packets are sent.
+
+* The only currently supported authentication mechanisms are simple
+challenge-response based on a shared private key and pseudorandom
+number generation.
+
+* An attacker with access to the shared key(s) can easily fence any
+known VM, even if they are not on a cluster node.
+
+* Different shared keys should be used for different virtual
+clusters on the same subnet (whether in the same physical cluster
+or not).  Additionally, multiple fence_xvmd instances must be run
+(each listening on a different multicast IP + port combination).
+
+IV.  Configuration
+
+Generate a random key file.  An example of how to generate it is:
+
+    dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4096 count=1
+
+Distribute the generated key file to all domUs in a cluster as well
+as all dom0s which will be hosting that particular cluster of domUs.
+The key should not be placed on shared file systems (because shared
+file systems require the cluster, which requires fencing...).
+
+Start fence_xvmd on all dom0s
+
+Configure fence_xvm on the domU cluster...
+
+rest...tbd
+

TODO

@@ -0,0 +1,5 @@
+High Priority / Blockers:
+
+* serial server-side listener/dispatcher
+
+* make server configurable

client/Makefile

@@ -0,0 +1,49 @@
+###############################################################################
+###############################################################################
+##
+##  Copyright (C) 2006 Red Hat, Inc.
+##  
+##  This copyrighted material is made available to anyone wishing to use,
+##  modify, copy, or redistribute it subject to the terms and conditions
+##  of the GNU General Public License v.2.
+##
+###############################################################################
+###############################################################################
+
+
+TARGETS=fence_virt fence_xvm
+
+fence_virt_SOURCES = mcast.c serial.c main.c
+
+INCLUDES=-I../include\
+	-I/usr/include/openais -I/usr/include/libvirt \
+	-I/usr/include/nss3 -I/usr/include/nspr4 \
+	-I../../../cman/lib -I../../../ccs/lib -I/usr/include/libxml2 \
+	-I/usr/include/libvirt
+
+CFLAGS+=-DFENCE_RELEASE_NAME=\"devel\" \
+	-Wall -Werror -Wstrict-prototypes -Wshadow -ggdb -D_GNU_SOURCE
+
+LIBS+=-L../../../cman/lib -L../../../ccs/lib -L${libdir}/openais \
+      -L../../../dlm/lib -lnss3 -lxml2
+
+all: ${TARGETS}
+
+fence_xvm: fence_virt
+	ln -snf fence_virt fence_xvm
+
+fence_virt: ${fence_virt_SOURCES:.c=.o}
+	gcc -o $@ $^ $(LIBS) -L../common -lfence_virt
+
+%.o: %.c
+	gcc $(CFLAGS) -c -o $@ $^ $(INCLUDES)
+
+install: all
+	if [ ! -d ${sbindir} ]; then \
+		install -d ${sbindir}; \
+	fi
+	install -m755 ${TARGETS} ${sbindir}
+
+clean:
+	rm -f ${TARGETS} *.o *.d *~
+

client/main.c

@@ -0,0 +1,118 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+/*
+ * @file fence_virtd.c: Implementation of server daemon for Xen virtual
+ * machine fencing.  This uses SA AIS CKPT b.1.0 checkpointing API to 
+ * store virtual machine states.
+ *
+ * Author: Lon Hohberger <lhh at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <pthread.h>
+#include <libgen.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "options.h"
+#include "debug.h"
+#include <client.h>
+
+
+int
+main(int argc, char **argv)
+{
+	fence_virt_args_t args;
+	char *my_options = "di:a:p:r:C:c:k:M:H:uo:t:?hV";
+
+	args_init(&args);
+	if (!strcmp(basename(argv[0]), "fence_xvm")) {
+		args.mode = MODE_MULTICAST;
+	} else {
+		args.mode = MODE_SERIAL;
+	}
+
+	if (argc == 1) {
+		args_get_stdin(my_options, &args);
+	} else {
+		args_get_getopt(argc, argv, my_options, &args);
+	}
+
+	if (args.flags & F_HELP) {
+		args_usage(argv[0], my_options, 0);
+
+                printf("With no command line argument, arguments are "
+                       "read from standard input.\n");
+                printf("Arguments read from standard input take "
+                       "the form of:\n\n");
+                printf("    arg1=value1\n");
+                printf("    arg2=value2\n\n");
+
+		args_usage(argv[0], my_options, 1);
+		exit(0);
+	}
+
+	if (args.flags & F_VERSION) {
+		printf("%s %s\n", basename(argv[0]), XVM_VERSION);
+#ifdef FENCE_RELEASE_NAME
+		printf("fence release %s\n", FENCE_RELEASE_NAME);
+#endif
+		exit(0);
+	}
+
+	args_finalize(&args);
+	dset(args.debug);
+	
+	if (args.debug > 0) 
+		args_print(&args);
+
+	/* Additional validation here */
+	if (!args.domain) {
+		printf("No domain specified!\n");
+		args.flags |= F_ERR;
+	}
+
+	if (args.flags & F_ERR) {
+		args_usage(argv[0], my_options, (argc == 1));
+		exit(1);
+	}
+
+	switch(args.mode) {
+	case MODE_MULTICAST:
+		return mcast_fence_virt(&args);
+	case MODE_SERIAL:
+		return serial_fence_virt(&args);
+	default:
+		break;
+	}
+
+	return 1;
+}

client/mcast.c

@@ -0,0 +1,320 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+/*
+ * @file fence_virtd.c: Implementation of server daemon for Xen virtual
+ * machine fencing.  This uses SA AIS CKPT b.1.0 checkpointing API to 
+ * store virtual machine states.
+ *
+ * Author: Lon Hohberger <lhh at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <pthread.h>
+#include <libgen.h>
+#include <nss.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "ip_lookup.h"
+#include "simple_auth.h"
+#include "options.h"
+#include "tcp.h"
+#include "mcast.h"
+#include "debug.h"
+
+
+static int
+tcp_wait_connect(int lfd, int retry_tenths)
+{
+	int fd;
+	fd_set rfds;
+	int n;
+	struct timeval tv;
+
+	dbg_printf(3, "Waiting for connection from XVM host daemon.\n");
+	FD_ZERO(&rfds);
+	FD_SET(lfd, &rfds);
+	tv.tv_sec = retry_tenths / 10;
+	tv.tv_usec = (retry_tenths % 10) * 100000;
+
+	n = select(lfd + 1, &rfds, NULL, NULL, &tv);
+	if (n == 0) {
+		errno = ETIMEDOUT;
+		return -1;
+	} else if (n < 0) {
+		return -1;
+	}
+
+	fd = accept(lfd, NULL, 0);
+	if (fd < 0)
+		return -1;
+
+	return fd;
+}
+
+
+static int
+tcp_exchange(int fd, fence_auth_type_t auth, void *key,
+	      size_t key_len, int timeout)
+{
+	char ret;
+	fd_set rfds;
+	struct timeval tv;
+
+	/* Ok, we're connected */
+	dbg_printf(3, "Issuing TCP challenge\n");
+	if (tcp_challenge(fd, auth, key, key_len, timeout) <= 0) {
+		/* Challenge failed */
+		printf("Invalid response to challenge\n");
+		return 0;
+	}
+
+	/* Now they'll send us one, so we need to respond here */
+	dbg_printf(3, "Responding to TCP challenge\n");
+	if (tcp_response(fd, auth, key, key_len, timeout) <= 0) {
+		printf("Invalid response to challenge\n");
+		return 0;
+	}
+
+	dbg_printf(2, "TCP Exchange + Authentication done... \n");
+
+	FD_ZERO(&rfds);
+	FD_SET(fd, &rfds);
+	tv.tv_sec = timeout;
+	tv.tv_usec = 0;
+
+	ret = 1;
+	dbg_printf(3, "Waiting for return value from XVM host\n");
+	if (select(fd + 1, &rfds, NULL, NULL, &tv) <= 0)
+		return -1;
+
+	/* Read return code */
+	read(fd, &ret, 1);
+	close(fd);
+	if (ret == 0)
+		printf("Remote: Operation was successful\n");
+	else
+		printf("Remote: Operation failed\n");
+	return ret;
+}
+
+
+static int
+send_multicast_packets(ip_list_t *ipl, fence_virt_args_t *args, void *key,
+		       size_t key_len)
+{
+	fence_req_t freq;
+	int mc_sock;
+	ip_addr_t *ipa;
+	struct sockaddr_in tgt4;
+	struct sockaddr_in6 tgt6;
+	struct sockaddr *tgt;
+	socklen_t tgt_len;
+
+	for (ipa = ipl->tqh_first; ipa; ipa = ipa->ipa_entries.tqe_next) {
+
+		if (ipa->ipa_family != args->net.family) {
+			dbg_printf(2, "Ignoring %s: wrong family\n", ipa->ipa_address);
+			continue;
+		}
+
+		if (args->net.family == PF_INET) {
+			mc_sock = ipv4_send_sk(ipa->ipa_address, args->net.addr,
+					       args->net.port,
+					       (struct sockaddr *)&tgt4,
+					       sizeof(struct sockaddr_in));
+			tgt = (struct sockaddr *)&tgt4;
+			tgt_len = sizeof(tgt4);
+			
+		} else if (args->net.family == PF_INET6) {
+			mc_sock = ipv6_send_sk(ipa->ipa_address, args->net.addr,
+					       args->net.port,
+					       (struct sockaddr *)&tgt6,
+					       sizeof(struct sockaddr_in6));
+			tgt = (struct sockaddr *)&tgt6;
+			tgt_len = sizeof(tgt6);
+		} else {
+			dbg_printf(2, "Unsupported family %d\n", args->net.family);
+			return -1;
+		}
+
+		if (mc_sock < 0)
+			continue;
+
+		/* Build our packet */
+		memset(&freq, 0, sizeof(freq));
+		strncpy((char *)freq.domain, args->domain,
+			sizeof(freq.domain));
+		freq.request = args->op;
+		freq.hashtype = args->net.hash;
+
+		/* Store source address */
+		if (ipa->ipa_family == PF_INET) {
+			freq.addrlen = sizeof(struct in_addr);
+			/* XXX Swap order for in_addr ? XXX */
+			inet_pton(PF_INET, ipa->ipa_address, freq.address);
+		} else if (ipa->ipa_family == PF_INET6) {
+			freq.addrlen = sizeof(struct in6_addr);
+			inet_pton(PF_INET6, ipa->ipa_address, freq.address);
+		}
+
+		freq.flags = 0;
+		if (args->flags & F_USE_UUID)
+			freq.flags |= RF_UUID;
+		freq.family = ipa->ipa_family;
+		freq.port = args->net.port;
+
+		sign_request(&freq, key, key_len);
+
+		dbg_printf(3, "Sending to %s via %s\n", args->net.addr,
+		        ipa->ipa_address);
+
+		sendto(mc_sock, &freq, sizeof(freq), 0,
+		       (struct sockaddr *)tgt, tgt_len);
+
+		close(mc_sock);
+	}
+
+	return 0;
+}
+
+
+/* TODO: Clean this up!!! */
+int
+mcast_fence_virt(fence_virt_args_t *args)
+{
+	ip_list_t ipl;
+	char key[MAX_KEY_LEN];
+	int lfd, key_len = 0, fd;
+	int attempts = 0;
+	
+	/* Initialize NSS; required to do hashing, as silly as that
+	   sounds... */
+	if (NSS_NoDB_Init(NULL) != SECSuccess) {
+		printf("Could not initialize NSS\n");
+		return 1;
+	}
+
+	if (args->net.auth != AUTH_NONE || args->net.hash != HASH_NONE) {
+		key_len = read_key_file(args->net.key_file, key, sizeof(key));
+		if (key_len < 0) {
+			printf("Could not read %s; trying without "
+			       "authentication\n", args->net.key_file);
+			args->net.auth = AUTH_NONE;
+			args->net.hash = HASH_NONE;
+		}
+	}
+
+	/* Do the real work */
+	if (ip_build_list(&ipl) < 0) {
+		printf("Error building IP address list\n");
+		return 1;
+	}
+
+	switch (args->net.auth) {
+		case AUTH_NONE:
+		case AUTH_SHA1:
+		case AUTH_SHA256:
+		case AUTH_SHA512:
+			if (args->net.family == PF_INET) {
+				lfd = ipv4_listen(args->net.port, 10);
+			} else {
+				lfd = ipv6_listen(args->net.port, 10);
+			}
+			break;
+		/*case AUTH_X509:*/
+			/* XXX Setup SSL listener socket here */
+		default:
+			return 1;
+	}
+
+	if (lfd < 0) {
+		printf("Failed to listen: %s\n", strerror(errno));
+		return 1;
+	}
+
+	attempts = args->timeout * 10 / args->retr_time;
+
+	do {
+		if (send_multicast_packets(&ipl, args, key, key_len)) {
+			return -1;
+		}
+
+		switch (args->net.auth) {
+			case AUTH_NONE:
+			case AUTH_SHA1:
+			case AUTH_SHA256:
+			case AUTH_SHA512:
+				fd = tcp_wait_connect(lfd, args->retr_time);
+				if (fd < 0 && (errno == ETIMEDOUT ||
+					       errno == EINTR))
+					continue;
+				break;
+			/* case AUTH_X509:
+				... = ssl_wait_connect... */
+			break;
+		default:
+			return 1;
+		}
+
+		break;
+	} while (--attempts);
+
+	if (fd < 0) {
+		if (attempts <= 0) {
+			printf("Timed out waiting for response\n");
+			return 1;
+		}
+		printf("Operation failed: %s\n", strerror(errno));
+		return -1;
+	}
+
+	switch (args->net.auth) {
+		case AUTH_NONE:
+		case AUTH_SHA1:
+		case AUTH_SHA256:
+		case AUTH_SHA512:
+			return tcp_exchange(fd, args->net.auth, key, key_len,
+					    args->timeout);
+			break;
+		/* case AUTH_X509: 
+			return ssl_exchange(...); */
+		default:
+			return 1;
+	}
+
+	return 1;
+}

client/serial.c

@@ -0,0 +1,225 @@
+/* 
+ * Copyright (C) 2002-2003, 2009 Red Hat, Inc.
+ *
+ * License: GPL.
+ *
+ * Written by Lon Hohberger <lhh@redhat.com>
+ *
+ * Ubersimpledumbterminal "ser" version 1.0.3
+ */
+
+#include <stdio.h>
+#include <termios.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <sys/select.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <fdops.h>
+#include <xvm.h>
+#include <options.h>
+#include <client.h>
+
+
+static int
+char_to_speed(const char *speed)
+{
+	if (!speed || !strlen(speed))
+		return B9600;
+	if (!strcmp(speed,"2400"))
+		return B2400;
+	if (!strcmp(speed,"9600"))
+		return B9600;
+	if (!strcmp(speed,"19200"))
+		return B19200;
+	if (!strcmp(speed,"38400"))
+		return B38400;
+	if (!strcmp(speed,"57600"))
+		return B57600;
+	if (!strcmp(speed,"115200"))
+		return B115200;
+	return -1;
+}
+
+
+static int
+char_to_flags(const char *param)
+{
+	int db_f = CS8, par_f = 0, sb_f = 0, x;
+
+	if (!param || !strlen(param))
+		return (db_f | par_f | sb_f);
+
+	if (strlen(param) < 3) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	for (x = 0; x < 3; x++) {
+		switch (param[0]) {
+		case '5':
+			db_f = CS5;
+			break;
+		case '6':
+			db_f = CS6;
+			break;
+		case '7':
+			db_f = CS7;
+			break;
+		case '8':
+			db_f = CS8;
+			break;
+		case 'n':
+		case 'N':
+			par_f = 0;
+			break;
+		case 'e':
+		case 'E':
+			par_f = PARENB;
+			break;
+		case 'o':
+		case 'O':
+			par_f = PARENB | PARODD;
+			break;
+		case '1':
+			sb_f = 0;
+			break;
+		case '2':
+			sb_f = CSTOPB;
+			break;
+		default:
+			errno = EINVAL;
+			return -1;
+		}
+	}
+
+	return (db_f | par_f | sb_f);
+}
+
+
+static int
+open_port(char *file, char *cspeed, char *cparam)
+{
+	struct termios  ti;
+	int fd, speed = B115200, flags = 0;
+	struct flock lock;
+
+	if ((speed = char_to_speed(cspeed)) == -1) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	if ((flags = char_to_flags(cparam)) == -1) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	if ((fd = open(file, O_RDWR | O_EXCL)) == -1) {
+		perror("open");
+		return -1;
+	}
+
+	memset(&lock,0,sizeof(lock));
+	lock.l_type = F_WRLCK;
+	if (fcntl(fd, F_SETLK, &lock) == -1) {
+		perror("Failed to lock serial port");
+		close(fd);
+		return -1;
+	}
+
+	memset(&ti, 0, sizeof(ti));
+	ti.c_cflag = (speed | CLOCAL | CRTSCTS | CREAD | flags);
+
+	if (tcsetattr(fd, TCSANOW, &ti) < 0) {
+		perror("tcsetattr");
+		close(fd);
+		return -1;
+	}                             
+
+	(void) tcflush(fd, TCIOFLUSH);
+
+	return fd;
+}
+
+
+static void
+hangup(int fd, int delay)
+{
+	unsigned int bits;
+
+	if (ioctl(fd, TIOCMGET, &bits)) {
+		perror("ioctl1");
+		return;
+	}
+
+	bits &= ~(TIOCM_DTR | TIOCM_CTS | TIOCM_RTS | TIOCM_DSR | TIOCM_CD);
+
+	if (ioctl(fd, TIOCMSET, &bits)) {
+		perror("ioctl2");
+		return;
+	}
+	
+	usleep(delay);
+
+	bits |= (TIOCM_DTR | TIOCM_CTS | TIOCM_RTS | TIOCM_DSR | TIOCM_CD);
+
+	if (ioctl(fd, TIOCMSET, &bits)) {
+		perror("ioctl3");
+		return;
+	}
+}
+
+
+int
+serial_fence_virt(fence_virt_args_t *args)
+{
+	serial_req_t req;
+	int fd, ret;
+	char speed[16], *flags = NULL;
+	struct timeval tv;
+	serial_resp_t resp;
+
+	printf("Port: %s Speed: %s\n", args->serial.device, speed);
+
+	if ((flags = strchr(speed, ','))) {
+		flags++;
+	}
+
+	strncpy(speed, args->serial.speed, sizeof(speed));
+
+	fd = open_port(args->serial.device, speed, flags);
+	if (fd == -1)
+		return -1;
+
+	hangup(fd, 300000);
+
+	memset(&req, 0, sizeof(req));
+	req.magic = SERIAL_MAGIC;
+	req.request = (uint8_t)args->op;
+	if (args->flags & RF_UUID)
+		req.flags |= RF_UUID;
+	strncpy((char *)req.domain, args->domain, sizeof(req.domain));
+
+	tv.tv_sec = 3;
+	tv.tv_usec = 0;
+	ret = _write_retry(fd, &req, sizeof(req), &tv);
+	if (ret < sizeof(req)) {
+		if (ret < 0)
+			return ret;
+		printf("Failed to send request\n");
+	}
+
+	tv.tv_sec = args->timeout;
+	tv.tv_usec = 0;
+	ret = _read_retry(fd, &resp, sizeof(resp), &tv);
+
+	if (resp.magic != SERIAL_MAGIC)
+		return -1;
+	ret = (int)resp.response;
+
+	close(fd);
+
+	return ret;
+}

common/Makefile

@@ -0,0 +1,43 @@
+###############################################################################
+###############################################################################
+##
+##  Copyright (C) 2006 Red Hat, Inc.
+##  
+##  This copyrighted material is made available to anyone wishing to use,
+##  modify, copy, or redistribute it subject to the terms and conditions
+##  of the GNU General Public License v.2.
+##
+###############################################################################
+###############################################################################
+
+
+TARGETS=libfence_virt.a
+
+libfence_virt_a_SOURCE=mcast.c ip_lookup.c simple_auth.c tcp.c \
+		    options.c debug.c simple_auth.c fdops.c
+
+INCLUDES=-I../include \
+	-I/usr/include/openais -I/usr/include/libvirt \
+	-I/usr/include/nss3 -I/usr/include/nspr4 \
+	-I../../../cman/lib -I../../../ccs/lib -I/usr/include/libxml2 \
+	-I/usr/include/libvirt
+
+CFLAGS+=-DFENCE_RELEASE_NAME=\"devel\" \
+	-Wall -Werror -Wstrict-prototypes -Wshadow -ggdb -D_GNU_SOURCE
+
+LIBS+=-L../../../cman/lib -L../../../ccs/lib -L${libdir}/openais \
+      -L../../../dlm/lib -lnss3 -lxml2
+
+all: ${TARGETS}
+
+libfence_virt.a: ${libfence_virt_a_SOURCE:.c=.o}
+	ar rc $@ $^
+
+%.o: %.c
+	gcc $(CFLAGS) -c -o $@ $^ $(INCLUDES)
+
+clean:
+	rm -f ${TARGETS} *~ *.o testprog
+
+install:
+

common/bcast.c

@@ -0,0 +1,344 @@
+/*
+ * Author: Lon Hohberger <lhh at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <pthread.h>
+
+/* Local includes */
+#include "bcast.h"
+#include "debug.h"
+
+LOGSYS_DECLARE_SUBSYS ("XVM", SYSLOGLEVEL);
+
+/** 
+  Sets up a multicast receive socket
+ */
+int
+ipv4_bcast_recv_sk(char *addr, int port)
+{
+	int sock, val;
+	struct sockaddr_in sin;
+
+	/* Store broadcast address */
+	memset(&sin, 0, sizeof(sin));
+	sin.sin_family = PF_INET;
+	sin.sin_port = htons(port);
+	if (inet_pton(PF_INET, addr,
+		      (void *)&sin.sin_addr.s_addr) < 0) {
+		log_printf(LOG_ERR, "Invalid broadcast address: %s\n", addr);
+		return -1;
+	}
+
+	dbg_printf(4, "Setting up ipv4 broadcast receive (%s:%d)\n", addr, port);
+	sock = socket(PF_INET, SOCK_DGRAM, 0);
+	if (sock < 0) {
+		log_printf(LOG_ERR, "socket: %s\n", strerror(errno));
+		close(sock);
+		sock = -1;
+		return 1;
+	}
+
+	val = 1;
+	if (setsockopt(sock, SOL_SOCKET, SO_BROADCAST, 1, sizeof(val)) < 0) {
+		log_printf(LOG_ERR, "setsockopt: %s\n", strerror(errno));
+		close(sock);
+		return 1;
+	}
+
+	/*
+	 * Bind broadcast address
+	 */
+	if (bind(sock, (struct sockaddr *) &sin,
+		 sizeof(struct sockaddr_in)) < 0) {
+		printf("bind failed: %s\n", strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}
+
+
+/**
+  Set up multicast send socket
+ */
+int
+ipv4_bcast_send_sk(char *send_addr, char *addr, int port, struct sockaddr *tgt,
+	     socklen_t tgt_len, int ttl)
+{
+	int val;
+	struct ip_mreq mreq;
+	struct sockaddr_in mcast;
+	struct sockaddr_in src;
+	int sock;
+
+	if (tgt_len < sizeof(struct sockaddr_in)) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	/* Store multicast address */
+	mcast.sin_family = PF_INET;
+	mcast.sin_port = htons(port);
+	if (inet_pton(PF_INET, addr,
+		      (void *)&mcast.sin_addr.s_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+	mreq.imr_multiaddr.s_addr = mcast.sin_addr.s_addr;
+
+	/* Store sending address */
+	src.sin_family = PF_INET;
+	src.sin_port = htons(port);
+	if (inet_pton(PF_INET, send_addr,
+		      (void *)&src.sin_addr.s_addr) < 0) {
+		printf("Invalid source address: %s\n", send_addr);
+		return -1;
+	}
+	mreq.imr_interface.s_addr = src.sin_addr.s_addr;
+
+
+	/*************************
+	 * SET UP MULTICAST SEND *
+	 *************************/
+	dbg_printf(4, "Setting up ipv4 multicast send (%s:%d)\n", addr, port);
+	sock = socket(PF_INET, SOCK_DGRAM, 0);
+	if (sock < 0) {
+		perror("socket");
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group.
+	 */
+	dbg_printf(4, "Joining IP Multicast group (pass 1)\n");
+	if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq,
+		       sizeof(mreq)) == -1) {
+		printf("Failed to add multicast membership to transmit "
+		       "socket %s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group.
+	 */
+	dbg_printf(4, "Joining IP Multicast group (pass 2)\n");
+	if (setsockopt(sock, IPPROTO_IP, IP_MULTICAST_IF, &src.sin_addr,
+		       sizeof(src.sin_addr)) == -1) {
+		printf("Failed to bind multicast transmit socket to "
+		       "%s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * set time to live to 2 hops.
+	 */
+	dbg_printf(4, "Setting TTL to %d for fd%d\n", ttl, sock);
+	val = ttl;
+	if (setsockopt(sock, SOL_IP, IP_MULTICAST_TTL, &val,
+		       sizeof(val)))
+		printf("warning: setting TTL failed %s\n", strerror(errno));
+
+	memcpy((struct sockaddr_in *)tgt, &mcast, sizeof(struct sockaddr_in));
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}
+
+
+
+/** 
+  Sets up a multicast receive (ipv6) socket
+ */
+int
+ipv6_recv_sk(char *addr, int port)
+{
+	int sock, val;
+	struct ipv6_mreq mreq;
+	struct sockaddr_in6 sin;
+
+	memset(&mreq, 0, sizeof(mreq));
+	memset(&sin, 0, sizeof(sin));
+	sin.sin6_family = PF_INET6;
+	sin.sin6_port = htons(port);
+	if (inet_pton(PF_INET6, addr,
+		      (void *)&sin.sin6_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+
+	memcpy(&mreq.ipv6mr_multiaddr, &sin.sin6_addr,
+	       sizeof(struct in6_addr));
+
+
+	/********************************
+	 * SET UP MULTICAST RECV SOCKET *
+	 ********************************/
+	dbg_printf(4, "Setting up ipv6 multicast receive (%s:%d)\n", addr, port);
+	sock = socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP);
+	if (sock < 0) {
+		printf("socket: %s\n", strerror(errno));
+		close(sock);
+		sock = -1;
+		return 1;
+	}
+
+	/*
+	 * When using Multicast, bind to the LOCAL address, not the MULTICAST
+	 * address.
+	 */
+	memset(&sin, 0, sizeof(sin));
+	sin.sin6_family = PF_INET6;
+	sin.sin6_port = htons(port);
+	sin.sin6_addr = in6addr_any;
+	if (bind(sock, (struct sockaddr *) &sin,
+		 sizeof(struct sockaddr_in6)) < 0) {
+		printf("bind failed: %s\n", strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	dbg_printf(4, "Disabling IP Multicast loopback\n");
+	val = 1;
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_LOOP, &val,
+		       sizeof(val)) != 0) {
+		printf("Failed to disable multicast loopback\n");
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join multicast group
+	 */
+	dbg_printf(4, "Joining IP Multicast group\n");
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_ADD_MEMBERSHIP, &mreq,
+		       sizeof(mreq)) == -1) {
+		printf("Failed to add multicast to socket %s: %s\n",
+		       addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}
+
+
+/**
+  Set up ipv6 multicast send socket
+ */
+int
+ipv6_send_sk(char *send_addr, char *addr, int port, struct sockaddr *tgt,
+	     socklen_t tgt_len, int ttl)
+{
+	int val;
+	struct ipv6_mreq mreq;
+	struct sockaddr_in6 mcast;
+	struct sockaddr_in6 src;
+	int sock;
+
+	if (tgt_len < sizeof(struct sockaddr_in6)) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	memset(&mreq, 0, sizeof(mreq));
+
+	/* Store multicast address */
+	mcast.sin6_family = PF_INET6;
+	mcast.sin6_port = htons(port);
+	if (inet_pton(PF_INET6, addr,
+		      (void *)&mcast.sin6_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+
+	memcpy(&mreq.ipv6mr_multiaddr, &mcast.sin6_addr,
+	       sizeof(struct in6_addr));
+
+	/* Store sending address */
+	src.sin6_family = PF_INET6;
+	src.sin6_port = htons(port);
+	if (inet_pton(PF_INET6, send_addr,
+		      (void *)&src.sin6_addr) < 0) {
+		printf("Invalid source address: %s\n", send_addr);
+		return -1;
+	}
+
+	/*************************
+	 * SET UP MULTICAST SEND *
+	 *************************/
+	dbg_printf(4, "Setting up ipv6 multicast send (%s:%d)\n", addr, port);
+	sock = socket(PF_INET6, SOCK_DGRAM, 0);
+	if (sock < 0) {
+		perror("socket");
+		return -1;
+	}
+
+	dbg_printf(4, "Disabling IP Multicast loopback\n");
+	val = 1;
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_LOOP, &val,
+		       sizeof(val)) != 0) {
+		printf("Failed to disable multicast loopback\n");
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group.
+	 */
+	dbg_printf(4, "Joining IP Multicast group\n");
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_ADD_MEMBERSHIP, &mreq,
+		       sizeof(mreq)) == -1) {
+		printf("Failed to add multicast membership to transmit "
+		       "socket %s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group (part 2)
+	 */
+	/*
+	if (setsockopt(sock, IPPROTO_IPV6, IP_MULTICAST_IF, &src.sin6_addr,
+		       sizeof(src.sin6_addr)) == -1) {
+		printf("Failed to bind multicast transmit socket to "
+		       "%s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+	*/
+
+	/*
+	 * set time to live to 2 hops.
+	 */
+	val = ttl;
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_HOPS, &val,
+		       sizeof(val)))
+		printf("warning: setting TTL failed %s\n", strerror(errno));
+
+	memcpy((struct sockaddr_in *)tgt, &mcast, sizeof(struct sockaddr_in6));
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}

common/debug.c

@@ -0,0 +1,35 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#include <stdio.h>
+#include "debug.h"
+
+static int _debug = 0;
+
+inline void
+dset(int threshold)
+{
+	_debug = threshold;
+	dbg_printf(3, "Debugging threshold is now %d\n", threshold);
+}
+
+inline int
+dget(void)
+{
+	return _debug;
+}

common/fdops.c

@@ -0,0 +1,193 @@
+/*
+  Copyright Red Hat, Inc. 2002-2003
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge,
+  MA 02139, USA.
+*/
+/** @file
+ * Wrapper functions around read/write/select to retry in the event
+ * of interrupts.
+ */
+#include <unistd.h>
+#include <sys/types.h>
+#include <fcntl.h>
+#include <errno.h>
+
+/**
+ * This is a wrapper around select which will retry in the case we receive
+ * EINTR.  This is necessary for _read_retry, since it wouldn't make sense
+ * to have _read_retry terminate if and only if two EINTRs were received
+ * in a row - one during the read() call, one during the select call...
+ *
+ * See select(2) for description of parameters.
+ */
+int
+_select_retry(int fdmax, fd_set * rfds, fd_set * wfds, fd_set * xfds,
+	       struct timeval *timeout)
+{
+	int rv;
+
+	while (1) {
+		rv = select(fdmax, rfds, wfds, xfds, timeout);
+		if ((rv == -1) && (errno == EINTR))
+			/* return on EBADF/EINVAL/ENOMEM; continue on EINTR */
+			continue;
+		return rv;
+	}
+}
+
+/**
+ * Retries a write in the event of a non-blocked interrupt signal.
+ *
+ * @param fd		File descriptor to which we are writing.
+ * @param buf		Data buffer to send.
+ * @param count		Number of bytes in buf to send.
+ * @param timeout	(struct timeval) telling us how long we should retry.
+ * @return		The number of bytes written to the file descriptor,
+ * 			or -1 on error (with errno set appropriately).
+ */
+ssize_t
+_write_retry(int fd, void *buf, int count, struct timeval * timeout)
+{
+	int n, total = 0, remain = count, rv = 0;
+	fd_set wfds, xfds;
+
+	while (total < count) {
+
+		/* Create the write FD set of 1... */
+		FD_ZERO(&wfds);
+		FD_SET(fd, &wfds);
+		FD_ZERO(&xfds);
+		FD_SET(fd, &xfds);
+
+		/* wait for the fd to be available for writing */
+		rv = _select_retry(fd + 1, NULL, &wfds, &xfds, timeout);
+		if (rv == -1)
+			return -1;
+		else if (rv == 0) {
+			errno = ETIMEDOUT;
+			return -1;
+		}
+
+		if (FD_ISSET(fd, &xfds)) {
+			errno = EPIPE;
+			return -1;
+		}
+
+		/* 
+		 * Attempt to write to fd
+		 */
+		n = write(fd, buf + (off_t) total, remain);
+
+		/*
+		 * When we know our fd was select()ed and we receive 0 bytes
+		 * when we write, the fd was closed.
+		 */
+		if ((n == 0) && (rv == 1)) {
+			errno = EPIPE;
+			return -1;
+		}
+
+		if (n == -1) {
+			if ((errno == EAGAIN) || (errno == EINTR)) {
+				/* 
+				 * Not ready?
+				 */
+				continue;
+			}
+
+			/* Other errors: EIO, EINVAL, etc */
+			return -1;
+		}
+
+		total += n;
+		remain -= n;
+	}
+
+	return total;
+}
+
+/**
+ * Retry reads until we (a) time out or (b) get our data.  Of course, if
+ * timeout is NULL, it'll wait forever.
+ *
+ * @param sockfd	File descriptor we want to read from.
+ * @param buf		Preallocated buffer into which we will read data.
+ * @param count		Number of bytes to read.
+ * @param timeout	(struct timeval) describing how long we should retry.
+ * @return 		The number of bytes read on success, or -1 on failure.
+ 			Note that we will always return (count) or (-1).
+ */
+ssize_t
+_read_retry(int sockfd, void *buf, int count, struct timeval * timeout)
+{
+	int n, total = 0, remain = count, rv = 0;
+	fd_set rfds, xfds;
+
+	while (total < count) {
+		FD_ZERO(&rfds);
+		FD_SET(sockfd, &rfds);
+		FD_ZERO(&xfds);
+		FD_SET(sockfd, &xfds);
+		
+		/*
+		 * Select on the socket, in case it closes while we're not
+		 * looking...
+		 */
+		rv = _select_retry(sockfd + 1, &rfds, NULL, &xfds, timeout);
+		if (rv == -1)
+			return -1;
+		else if (rv == 0) {
+			errno = ETIMEDOUT;
+			return -1;
+		}
+
+		if (FD_ISSET(sockfd, &xfds)) {
+			errno = EPIPE;
+			return -1;
+		}
+
+		/* 
+		 * Attempt to read off the socket 
+		 */
+		n = read(sockfd, buf + (off_t) total, remain);
+
+		/*
+		 * When we know our socket was select()ed and we receive 0 bytes
+		 * when we read, the socket was closed.
+		 */
+		if ((n == 0) && (rv == 1)) {
+			errno = EPIPE;
+			return -1;
+		}
+
+		if (n == -1) {
+			if ((errno == EAGAIN) || (errno == EINTR)) {
+				/* 
+				 * Not ready? Wait for data to become available
+				 */
+				continue;
+			}
+
+			/* Other errors: EPIPE, EINVAL, etc */
+			return -1;
+		}
+
+		total += n;
+		remain -= n;
+	}
+
+	return total;
+}

common/ip_lookup.c

@@ -0,0 +1,318 @@
+/*
+  Copyright Red Hat, Inc. 2004, 2006
+
+  The Magma Cluster API Library is free software; you can redistribute
+  it and/or modify it under the terms of the GNU Lesser General Public
+  License as published by the Free Software Foundation; either version
+  2.1 of the License, or (at your option) any later version.
+
+  The Magma Cluster API Library is distributed in the hope that it will
+  be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
+  of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  Lesser General Public License for more details.
+
+  You should have received a copy of the GNU Lesser General Public
+  License along with this library; if not, write to the Free Software
+  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+  USA.
+ */
+/** @file
+ * Build lists of IPs on the system, excepting loopback ipv6 link-local
+ */
+#include <asm/types.h>
+#include <sys/types.h>
+#include <arpa/inet.h>
+#include <sys/socket.h>
+#include <linux/netlink.h>
+#include <linux/rtnetlink.h>
+#include <errno.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <netdb.h>
+
+/* Local includes */
+#include "ip_lookup.h"
+#include "debug.h"
+
+static int
+send_addr_dump(int fd, int family)
+{
+	struct nlmsghdr *nh;
+	struct rtgenmsg *g;
+	char buf[256];
+	struct sockaddr_nl addr;
+
+	memset(&addr,0,sizeof(addr));
+	addr.nl_family = PF_NETLINK;
+
+	memset(buf, 0, sizeof(buf));
+	nh = (struct nlmsghdr *)buf;
+	g = (struct rtgenmsg *)(buf + sizeof(struct nlmsghdr));
+
+	nh->nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));
+	nh->nlmsg_flags = NLM_F_REQUEST|NLM_F_DUMP;
+	nh->nlmsg_type = RTM_GETADDR;
+	g->rtgen_family = family;
+
+	return sendto(fd, buf, nh->nlmsg_len, 0, (struct sockaddr *)&addr,
+	   	      sizeof(addr));
+}
+
+
+static int
+add_ip(ip_list_t *ipl, char *ipaddr, char family)
+{
+	ip_addr_t *ipa;
+
+	if (family == PF_INET6) {
+		/* Avoid loopback */
+		if (!strcmp(ipaddr, "::1"))
+			return -1;
+
+		/* Avoid link-local addresses */
+		if (!strncmp(ipaddr, "fe80", 4))
+			return -1;
+		if (!strncmp(ipaddr, "fe90", 4))
+			return -1;
+		if (!strncmp(ipaddr, "fea0", 4))
+			return -1;
+		if (!strncmp(ipaddr, "feb0", 4))
+			return -1;
+	}
+	
+	dbg_printf(4, "Adding IP %s to list (family %d)\n", ipaddr, family);
+
+	ipa = malloc(sizeof(*ipa));
+	memset(ipa, 0, sizeof(*ipa));
+	ipa->ipa_family = family;
+	ipa->ipa_address = strdup(ipaddr);
+
+	TAILQ_INSERT_TAIL(ipl, ipa, ipa_entries);
+
+	return 0;
+}
+
+
+static int
+add_ip_addresses(int family, ip_list_t *ipl)
+{
+	/* List ipv4 addresses */
+	struct nlmsghdr *nh;
+	struct ifaddrmsg *ifa;
+	struct rtattr *rta, *nrta;
+	struct nlmsgerr *err;
+	char buf[10240];
+	char outbuf[256];
+	int x, fd, len;
+
+	dbg_printf(5, "Connecting to Netlink...\n");
+	fd = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE);
+	if (fd < 0) {
+		perror("socket");
+		exit(1);
+	}
+	
+	dbg_printf(5, "Sending address dump request\n");
+	send_addr_dump(fd, family);
+	memset(buf, 0, sizeof(buf));
+	
+	dbg_printf(5, "Waiting for response\n");
+	x = recvfrom(fd, buf, sizeof(buf), 0, NULL, 0);
+	if (x < 0) {
+		perror("recvfrom");
+		return -1;
+	}
+	
+	dbg_printf(5, "Received %d bytes\n", x);
+
+	nh = (struct nlmsghdr *)buf;
+	while (NLMSG_OK(nh, x)) {
+
+		switch(nh->nlmsg_type) {
+		case NLMSG_DONE:
+			close(fd);
+    			return 0;
+
+		case NLMSG_ERROR:
+			err = (struct nlmsgerr*)NLMSG_DATA(nh);
+			if (nh->nlmsg_len <
+			    NLMSG_LENGTH(sizeof(struct nlmsgerr))) {
+				fprintf(stderr, "ERROR truncated");
+			} else {
+				errno = -err->error;
+				perror("RTNETLINK answers");
+			}
+			close(fd);
+			return -1;
+
+		case RTM_NEWADDR:
+			break;
+
+		default:
+			nh = NLMSG_NEXT(nh, x);
+			continue;
+		}
+
+		/* RTM_NEWADDR */
+		len = NLMSG_PAYLOAD(nh,0);
+		ifa = NLMSG_DATA(nh);
+
+		/* Make sure we got the type we expect back */
+		if (ifa->ifa_family != family) {
+			nh = NLMSG_NEXT(nh, x);
+			continue;
+		}
+
+		rta = (struct rtattr *)((void *)ifa + sizeof(*ifa));
+		len -= sizeof(*ifa);
+		do {
+			/* Make sure we've got a valid rtaddr field */
+			if (!RTA_OK(rta, len)) {
+				dbg_printf(5, "!RTA_OK(rta, len)\n");
+				break;
+			}
+
+			if (rta->rta_type == IFA_ADDRESS) {
+				inet_ntop(family, RTA_DATA(rta), outbuf,
+					  sizeof(outbuf) );
+				add_ip(ipl, outbuf, family);
+			}
+
+			if (rta->rta_type == IFA_LABEL) {
+				dbg_printf(5, "Skipping label: %s\n",
+					(char *)RTA_DATA(rta));
+			}
+
+			nrta = RTA_NEXT(rta, len);
+			if (!nrta)
+				break;
+
+			len -= ((void *)nrta - (void *)rta);
+			rta = nrta;
+		} while (RTA_OK(rta, len));
+
+		nh = NLMSG_NEXT(nh, x);
+	}
+
+	dbg_printf(5, "Closing Netlink connection\n");
+	close(fd);
+	return 0;
+}
+
+
+int
+ip_search(ip_list_t *ipl, char *ip_name)
+{
+	ip_addr_t *ipa;
+	
+	dbg_printf(5, "Looking for IP address %s in IP list %p...", ip_name, ipl);
+	ipa = ipl->tqh_first;
+	for (ipa = ipl->tqh_first; ipa; ipa = ipa->ipa_entries.tqe_next) {
+		if (!strcmp(ip_name, ipa->ipa_address)) {
+			dbg_printf(4,"Found\n");
+			return 0;
+		}
+	}
+	dbg_printf(5, "Not found\n");
+	return 1;
+}
+
+
+int
+ip_free_list(ip_list_t *ipl)
+{
+	ip_addr_t *ipa;
+	
+	dbg_printf(5, "Tearing down IP list @ %p\n", ipl);
+	while ((ipa = ipl->tqh_first)) {
+		TAILQ_REMOVE(ipl, ipa, ipa_entries);
+		free(ipa->ipa_address);
+		free(ipa);
+	}
+	return 0;
+}
+
+
+int
+ip_build_list(ip_list_t *ipl)
+{
+	dbg_printf(5, "Build IP address list\n");
+	TAILQ_INIT(ipl);
+	if (add_ip_addresses(PF_INET6, ipl) < 0) {
+		ip_free_list(ipl);
+		return -1;
+	}
+	if (add_ip_addresses(PF_INET, ipl) < 0) {
+		ip_free_list(ipl);
+		return -1;
+	}
+	return 0;
+}
+
+
+/**
+  Look up the interface name which corresponds to the given hostname and
+  return the list of matching attrinfo structures.  We do this by looking
+  up all the possible physical and virtual network interfaces on the machine
+  and checking the hostname/IP mappings for each active IP address incurred.
+
+  @param nodename	Interface name
+  @param ret_ai		Structure pointer to allocate & return.
+  @return		-1 on failure or 0 on success.
+ */
+int
+ip_lookup(char *nodename, struct addrinfo **ret_ai)
+{
+	char ip_name[256];
+	struct addrinfo *ai = NULL;
+	struct addrinfo *n;
+	void *p;
+	ip_list_t ipl;
+	int ret = -1;
+
+	dbg_printf(5, "Looking for IP matching %s\n", nodename);
+	/* Build list of IP addresses configured locally */
+	if (ip_build_list(&ipl) < 0)
+		return -1;
+
+	/* Get list of addresses for the host-name/ip */
+	if (getaddrinfo(nodename, NULL, NULL, &ai) != 0) 
+		return -1;
+	
+
+	/* Traverse list of addresses for given host-name/ip */
+	for (n = ai; n; n = n->ai_next) {
+		if (n->ai_family != PF_INET && n->ai_family != PF_INET6)
+			continue;
+
+		if (n->ai_family == PF_INET)
+			p = &(((struct sockaddr_in *)n->ai_addr)->sin_addr);
+		else
+			p = &(((struct sockaddr_in6 *)n->ai_addr)->sin6_addr);
+
+		if (!inet_ntop(n->ai_family, p, ip_name,
+			       sizeof(ip_name)))
+			continue;
+
+		/* Search local interfaces for this IP address */
+		if (ip_search(&ipl, ip_name) != 0)
+			continue;
+
+		/* Found it */
+		ret = 0;
+		break;
+	}
+
+	/* Clean up */
+	if (!ret_ai)
+		freeaddrinfo(ai);
+	else
+		*ret_ai = ai;
+
+	ip_free_list(&ipl);
+
+	return ret;
+}
+

common/mcast.c

@@ -0,0 +1,374 @@
+/*
+  Copyright Red Hat, Inc. 2003, 2004, 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+/*
+ * Author: Lon Hohberger <lhh at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <pthread.h>
+
+/* Local includes */
+#include "mcast.h"
+#include "debug.h"
+
+/** 
+  Sets up a multicast receive socket
+ */
+int
+ipv4_recv_sk(char *addr, int port, unsigned int ifindex)
+{
+	int sock;
+	struct ip_mreqn mreq;
+	struct sockaddr_in sin;
+
+	/* Store multicast address */
+	if (inet_pton(PF_INET, addr,
+		      (void *)&mreq.imr_multiaddr.s_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+
+	/********************************
+	 * SET UP MULTICAST RECV SOCKET *
+	 ********************************/
+	dbg_printf(4, "Setting up ipv4 multicast receive (%s:%d)\n", addr, port);
+	sock = socket(PF_INET, SOCK_DGRAM, 0);
+	if (sock < 0) {
+		printf("socket: %s\n", strerror(errno));
+		close(sock);
+		sock = -1;
+		return 1;
+	}
+
+	/*
+	 * When using Multicast, bind to the LOCAL address, not the MULTICAST
+	 * address.
+	 */
+	sin.sin_family = PF_INET;
+	sin.sin_port = htons(port);
+	sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	if (bind(sock, (struct sockaddr *) &sin,
+		 sizeof(struct sockaddr_in)) < 0) {
+		printf("bind failed: %s\n", strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join multicast group
+	 */
+	/* mreq.imr_multiaddr.s_addr is set above */
+	mreq.imr_ifindex = ifindex;
+	dbg_printf(4, "Joining multicast group\n");
+	if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP,
+		       &mreq, sizeof(mreq)) == -1) {
+		printf("Failed to bind multicast receive socket to "
+		       "%s: %s\n", addr, strerror(errno));
+		printf("Check network configuration.\n");
+		close(sock);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}
+
+
+/**
+  Set up multicast send socket
+ */
+int
+ipv4_send_sk(char *send_addr, char *addr, int port, struct sockaddr *tgt,
+	     socklen_t tgt_len)
+{
+	int val;
+	struct ip_mreq mreq;
+	struct sockaddr_in mcast;
+	struct sockaddr_in src;
+	int sock;
+
+	if (tgt_len < sizeof(struct sockaddr_in)) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	/* Store multicast address */
+	mcast.sin_family = PF_INET;
+	mcast.sin_port = htons(port);
+	if (inet_pton(PF_INET, addr,
+		      (void *)&mcast.sin_addr.s_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+	mreq.imr_multiaddr.s_addr = mcast.sin_addr.s_addr;
+
+	/* Store sending address */
+	src.sin_family = PF_INET;
+	src.sin_port = htons(port);
+	if (inet_pton(PF_INET, send_addr,
+		      (void *)&src.sin_addr.s_addr) < 0) {
+		printf("Invalid source address: %s\n", send_addr);
+		return -1;
+	}
+	mreq.imr_interface.s_addr = src.sin_addr.s_addr;
+
+
+	/*************************
+	 * SET UP MULTICAST SEND *
+	 *************************/
+	dbg_printf(4, "Setting up ipv4 multicast send (%s:%d)\n", addr, port);
+	sock = socket(PF_INET, SOCK_DGRAM, 0);
+	if (sock < 0) {
+		perror("socket");
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group.
+	 */
+	dbg_printf(4, "Joining IP Multicast group (pass 1)\n");
+	if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq,
+		       sizeof(mreq)) == -1) {
+		printf("Failed to add multicast membership to transmit "
+		       "socket %s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group.
+	 */
+	dbg_printf(4, "Joining IP Multicast group (pass 2)\n");
+	if (setsockopt(sock, IPPROTO_IP, IP_MULTICAST_IF, &src.sin_addr,
+		       sizeof(src.sin_addr)) == -1) {
+		printf("Failed to bind multicast transmit socket to "
+		       "%s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * set time to live to 2 hops.
+	 */
+	dbg_printf(4, "Setting TTL to 2 for fd%d\n", sock);
+	val = 2;
+	if (setsockopt(sock, SOL_IP, IP_MULTICAST_TTL, &val,
+		       sizeof(val)))
+		printf("warning: setting TTL failed %s\n", strerror(errno));
+
+	memcpy((struct sockaddr_in *)tgt, &mcast, sizeof(struct sockaddr_in));
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}
+
+
+
+/** 
+  Sets up a multicast receive (ipv6) socket
+ */
+int
+ipv6_recv_sk(char *addr, int port, unsigned int ifindex)
+{
+	int sock, val;
+	struct ipv6_mreq mreq;
+	struct sockaddr_in6 sin;
+
+	memset(&mreq, 0, sizeof(mreq));
+	memset(&sin, 0, sizeof(sin));
+	sin.sin6_family = PF_INET6;
+	sin.sin6_port = htons(port);
+	if (inet_pton(PF_INET6, addr,
+		      (void *)&sin.sin6_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+
+	memcpy(&mreq.ipv6mr_multiaddr, &sin.sin6_addr,
+	       sizeof(struct in6_addr));
+
+	mreq.ipv6mr_interface = ifindex;
+
+	/********************************
+	 * SET UP MULTICAST RECV SOCKET *
+	 ********************************/
+	dbg_printf(4, "Setting up ipv6 multicast receive (%s:%d)\n", addr, port);
+	sock = socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP);
+	if (sock < 0) {
+		printf("socket: %s\n", strerror(errno));
+		close(sock);
+		sock = -1;
+		return 1;
+	}
+
+	/*
+	 * When using Multicast, bind to the LOCAL address, not the MULTICAST
+	 * address.
+	 */
+	memset(&sin, 0, sizeof(sin));
+	sin.sin6_family = PF_INET6;
+	sin.sin6_port = htons(port);
+	sin.sin6_addr = in6addr_any;
+	if (bind(sock, (struct sockaddr *) &sin,
+		 sizeof(struct sockaddr_in6)) < 0) {
+		printf("bind failed: %s\n", strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	dbg_printf(4, "Disabling IP Multicast loopback\n");
+	val = 1;
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_LOOP, &val,
+		       sizeof(val)) != 0) {
+		printf("Failed to disable multicast loopback\n");
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join multicast group
+	 */
+	dbg_printf(4, "Joining IP Multicast group\n");
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_ADD_MEMBERSHIP, &mreq,
+		       sizeof(mreq)) == -1) {
+		printf("Failed to add multicast to socket %s: %s\n",
+		       addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}
+
+
+/**
+  Set up ipv6 multicast send socket
+ */
+int
+ipv6_send_sk(char *send_addr, char *addr, int port, struct sockaddr *tgt,
+	     socklen_t tgt_len)
+{
+	int val;
+	struct ipv6_mreq mreq;
+	struct sockaddr_in6 mcast;
+	struct sockaddr_in6 src;
+	int sock;
+
+	if (tgt_len < sizeof(struct sockaddr_in6)) {
+		errno = EINVAL;
+		return -1;
+	}
+
+	memset(&mreq, 0, sizeof(mreq));
+
+	/* Store multicast address */
+	mcast.sin6_family = PF_INET6;
+	mcast.sin6_port = htons(port);
+	if (inet_pton(PF_INET6, addr,
+		      (void *)&mcast.sin6_addr) < 0) {
+		printf("Invalid multicast address: %s\n", addr);
+		return -1;
+	}
+
+	memcpy(&mreq.ipv6mr_multiaddr, &mcast.sin6_addr,
+	       sizeof(struct in6_addr));
+
+	/* Store sending address */
+	src.sin6_family = PF_INET6;
+	src.sin6_port = htons(port);
+	if (inet_pton(PF_INET6, send_addr,
+		      (void *)&src.sin6_addr) < 0) {
+		printf("Invalid source address: %s\n", send_addr);
+		return -1;
+	}
+
+	/*************************
+	 * SET UP MULTICAST SEND *
+	 *************************/
+	dbg_printf(4, "Setting up ipv6 multicast send (%s:%d)\n", addr, port);
+	sock = socket(PF_INET6, SOCK_DGRAM, 0);
+	if (sock < 0) {
+		perror("socket");
+		return -1;
+	}
+
+	dbg_printf(4, "Disabling IP Multicast loopback\n");
+	val = 1;
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_LOOP, &val,
+		       sizeof(val)) != 0) {
+		printf("Failed to disable multicast loopback\n");
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group.
+	 */
+	dbg_printf(4, "Joining IP Multicast group\n");
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_ADD_MEMBERSHIP, &mreq,
+		       sizeof(mreq)) == -1) {
+		printf("Failed to add multicast membership to transmit "
+		       "socket %s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	/*
+	 * Join Multicast group (part 2)
+	 */
+	/*
+	if (setsockopt(sock, IPPROTO_IPV6, IP_MULTICAST_IF, &src.sin6_addr,
+		       sizeof(src.sin6_addr)) == -1) {
+		printf("Failed to bind multicast transmit socket to "
+		       "%s: %s\n", addr, strerror(errno));
+		close(sock);
+		return -1;
+	}
+	*/
+
+	/*
+	 * set time to live to 2 hops.
+	 */
+	val = 2;
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_MULTICAST_HOPS, &val,
+		       sizeof(val)))
+		printf("warning: setting TTL failed %s\n", strerror(errno));
+
+	memcpy((struct sockaddr_in *)tgt, &mcast, sizeof(struct sockaddr_in6));
+
+	dbg_printf(4, "%s: success, fd = %d\n", __FUNCTION__, sock);
+	return sock;
+}

common/options.c

@@ -0,0 +1,707 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "simple_auth.h"
+#include "mcast.h"
+#include "options.h"
+
+
+
+/* Assignment functions */
+
+static inline void
+assign_debug(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (!value) {
+		/* GNU getopt sets optarg to NULL for options w/o a param
+		   We rely on this here... */
+		args->debug++;
+		return;
+	}
+
+	args->debug = atoi(value);
+	if (args->debug < 0) {
+		args->debug = 1;
+	}
+}
+
+
+static inline void
+assign_foreground(fence_virt_args_t *args, struct arg_info *arg,
+		  char *value)
+{
+	args->flags |= F_FOREGROUND;
+}
+
+
+static inline void
+assign_family(fence_virt_args_t *args, struct arg_info *arg,
+	      char *value)
+{
+	if (!strcasecmp(value, "ipv4")) {
+		args->net.family = PF_INET;
+	} else if (!strcasecmp(value, "ipv6")) {
+		args->net.family = PF_INET6;
+	} else if (!strcasecmp(value, "auto")) {
+		args->net.family = 0;
+	} else {
+		printf("Unsupported family: '%s'\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_address(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (args->net.addr)
+		free(args->net.addr);
+	args->net.addr = strdup(value);
+}
+
+
+static inline void
+assign_port(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->net.port = atoi(value);
+	if (args->net.port <= 0 || args->net.port >= 65500) {
+		printf("Invalid port: '%s'\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_interface(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->net.ifindex = if_nametoindex(value);
+}
+
+
+static inline void
+assign_retrans(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->retr_time = atoi(value);
+	if (args->retr_time <= 0) {
+		printf("Invalid retransmit time: '%s'\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+static inline void
+assign_hash(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (!strcasecmp(value, "none")) {
+		args->net.hash = HASH_NONE;
+	} else if (!strcasecmp(value, "sha1")) {
+		args->net.hash = HASH_SHA1;
+	} else if (!strcasecmp(value, "sha256")) {
+		args->net.hash = HASH_SHA256;
+	} else if (!strcasecmp(value, "sha512")) {
+		args->net.hash = HASH_SHA512;
+	} else {
+		printf("Unsupported hash: %s\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_auth(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (!strcasecmp(value, "none")) {
+		args->net.auth = AUTH_NONE;
+	} else if (!strcasecmp(value, "sha1")) {
+		args->net.auth = AUTH_SHA1;
+	} else if (!strcasecmp(value, "sha256")) {
+		args->net.auth = AUTH_SHA256;
+	} else if (!strcasecmp(value, "sha512")) {
+		args->net.auth = AUTH_SHA512;
+	} else {
+		printf("Unsupported auth type: %s\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+static inline void
+assign_key(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	struct stat st;
+
+	if (args->net.key_file)
+		free(args->net.key_file);
+	args->net.key_file = strdup(value);
+
+	if (stat(value, &st) == -1) {
+		printf("Invalid key file: '%s' (%s)\n", value,
+		       strerror(errno));
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_op(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (!strcasecmp(value, "null")) {
+		args->op = FENCE_NULL;
+	} else if (!strcasecmp(value, "off")) {
+		args->op = FENCE_OFF;
+	} else if (!strcasecmp(value, "reboot")) {
+		args->op = FENCE_REBOOT;
+	} else if (!strcasecmp(value, "status")) {
+		args->op = FENCE_STATUS;
+	} else if (!strcasecmp(value, "devstatus")) {
+		args->op = FENCE_DEVSTATUS;
+	} else {
+		printf("Unsupported operation: %s\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_device(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->serial.device = strdup(value);
+}
+
+
+static inline void
+assign_params(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->serial.speed = strdup(value);
+}
+
+
+static inline void
+assign_domain(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (args->domain) {
+		printf("Domain/UUID may not be specified more than once\n");
+		args->flags |= F_ERR;
+		return;
+	}
+
+	args->domain = strdup(value);
+
+	if (strlen(value) <= 0) {
+		printf("Invalid domain name\n");
+		args->flags |= F_ERR;
+	}
+
+	if (strlen(value) >= MAX_DOMAINNAME_LENGTH) {
+		errno = ENAMETOOLONG;
+		printf("Invalid domain name: '%s' (%s)\n",
+		       value, strerror(errno));
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_uuid_lookup(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	if (!value) {
+		/* GNU getopt sets optarg to NULL for options w/o a param
+		   We rely on this here... */
+		args->flags |= F_USE_UUID;
+		return;
+	}
+
+	args->flags |= ( !!atoi(value) ? F_USE_UUID : 0);
+}
+
+
+static inline void
+assign_timeout(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->timeout = atoi(value);
+	if (args->timeout <= 0) {
+		printf("Invalid timeout: '%s'\n", value);
+		args->flags |= F_ERR;
+	}
+}
+
+
+static inline void
+assign_help(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->flags |= F_HELP;
+}
+
+
+static inline void
+assign_version(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->flags |= F_VERSION;
+}
+
+
+static inline void
+assign_noccs(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->flags |= F_NOCCS;
+}
+
+
+static inline void
+assign_nocluster(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+	args->flags |= F_NOCCS;
+	args->flags |= F_NOCLUSTER;
+}
+
+
+static inline void
+assign_uri(fence_virt_args_t *args, struct arg_info *arg, char *value)
+{
+#if 0
+	if (args->uri)
+		free(args->uri);
+
+	/* XXX NO validation yet */
+	args->uri = strdup(value);
+#endif
+}
+
+
+
+/** ALL valid command line and stdin arguments for this fencing agent */
+static struct arg_info _arg_info[] = {
+	{ '\xff', NULL, "agent",
+	  "Not user serviceable",
+	  NULL },
+
+	{ '\xff', NULL, "self",
+	  "Not user serviceable", 
+	  NULL },
+
+	{ 'd', "-d", "debug",
+	  "Specify (CCS) / increment (command line) debug level",
+	  assign_debug },
+
+	{ 'f', "-f", NULL,
+	  "Foreground mode (do not fork)",
+	  assign_foreground },
+
+	{ 'i', "-i <family>", "ip_family",
+	  "IP Family ([auto], ipv4, ipv6)",
+	  assign_family },
+
+	{ 'a', "-a <address>", "multicast_address",
+	  "Multicast address (default=225.0.0.12 / ff02::3:1)",
+	  assign_address },
+
+	{ 'p', "-p <port>", "port",
+	  "IP port (default=1229)",
+	  assign_port },
+
+	{ 'I', "-I <interface>", "multicast_address",
+	  "Network interface name to listen on",
+	  assign_interface },
+
+	{ 'r', "-r <retrans>", "retrans", 
+	  "Multicast retransmit time (in 1/10sec; default=20)",
+	  assign_retrans },
+
+	{ 'c', "-c <hash>", "hash",
+	  "Packet hash strength (none, sha1, [sha256], sha512)",
+	  assign_hash },
+
+	{ 'C', "-C <auth>", "auth",
+	  "Authentication (none, sha1, [sha256], sha512)",
+	  assign_auth },
+
+	{ 'k', "-k <file>", "key_file",
+	  "Shared key file (default=/etc/cluster/fence_virt.key)",
+	  assign_key },
+
+	{ 'D', "-D <device>", "serial_device",
+	  "Shared key file (default=" DEFAULT_SERIAL_DEVICE  ")",
+	  assign_device },
+
+	{ 'P', "-P <param>", "serial_params",
+	  "Serial Parameters (default=" DEFAULT_SERIAL_SPEED ")",
+	  assign_params },
+
+	{ 'o', "-o <operation>", "option",
+	  "Fencing operation (null, off, [reboot], status, devstatus)",
+	  assign_op },
+
+	{ 'H', "-H <domain>", "domain",
+	  "Xen host (domain name) to fence",
+	  assign_domain },
+
+	{ 'u', "-u", "use_uuid",
+	  "Treat <domain> as UUID instead of domain name",
+	  assign_uuid_lookup },
+
+	{ 't', "-t <timeout>", "timeout",
+	  "Fencing timeout (in seconds; default=30)",
+	  assign_timeout },
+
+	{ 'h', "-h", NULL,
+ 	  "Help",
+	  assign_help },
+
+	{ '?', "-?", NULL,
+ 	  "Help (alternate)", 
+	  assign_help },
+
+	{ 'X', "-X", NULL,
+ 	  "Do not connect to CCS for configuration", 
+	  assign_noccs }, 
+
+	{ 'L', "-L", NULL,
+ 	  "Local mode only (no cluster; implies -X)",
+	  assign_nocluster }, 
+
+	{ 'U', "-U", "uri",
+	  "URI for Hypervisor (default: auto detect)",
+	  assign_uri },
+	  
+	{ 'V', "-V", NULL,
+ 	  "Display version and exit", 
+	  assign_version },
+
+	/* Terminator */
+	{ 0, NULL, NULL, NULL, NULL }
+};
+
+
+struct arg_info *
+find_arg_by_char(char arg)
+{
+	int x = 0;
+
+	for (x = 0; _arg_info[x].opt != 0; x++) {
+		if (_arg_info[x].opt == arg)
+			return &_arg_info[x];
+	}
+
+	return NULL;
+}
+
+
+struct arg_info *
+find_arg_by_string(char *arg)
+{
+	int x = 0;
+
+	for (x = 0; _arg_info[x].opt != 0; x++) {
+		if (!_arg_info[x].stdin_opt)
+			continue;
+		if (!strcasecmp(_arg_info[x].stdin_opt, arg))
+			return &_arg_info[x];
+	}
+
+	return NULL;
+}
+
+
+/* ============================================================= */
+
+/**
+  Initialize an args structure.
+
+  @param args		Pointer to args structure to initialize.
+ */
+void
+args_init(fence_virt_args_t *args)
+{
+	args->domain = NULL;
+	//args->uri = NULL;
+	args->op = FENCE_REBOOT;
+	args->net.key_file = strdup(DEFAULT_KEY_FILE);
+	args->net.hash = DEFAULT_HASH;
+	args->net.auth = DEFAULT_AUTH;
+	args->net.addr = NULL;
+	args->net.port = 1229;
+	args->net.ifindex = 0;
+	args->net.family = PF_INET;
+	args->serial.device = strdup(DEFAULT_SERIAL_DEVICE);
+	args->serial.speed = strdup(DEFAULT_SERIAL_SPEED);
+	args->timeout = 30;
+	args->retr_time = 20;
+	args->flags = 0;
+	args->debug = 0;
+}
+
+
+#define _pr_int(piece) printf("  %s = %d\n", #piece, piece)
+#define _pr_str(piece) printf("  %s = %s\n", #piece, piece)
+
+
+/**
+  Prints out the contents of an args structure for debugging.
+
+  @param args		Pointer to args structure to print out.
+ */
+void
+args_print(fence_virt_args_t *args)
+{
+	printf("-- args @ %p --\n", args);
+	_pr_str(args->domain);
+	_pr_int(args->op);
+
+	_pr_str(args->net.key_file);
+	_pr_int(args->net.hash);
+	_pr_str(args->net.addr);
+	_pr_int(args->net.auth);
+	_pr_int(args->net.port);
+	_pr_int(args->net.ifindex);
+	_pr_int(args->net.family);
+	_pr_int(args->timeout);
+	_pr_int(args->retr_time);
+	_pr_int(args->flags);
+	_pr_int(args->debug);
+	printf("-- end args --\n");
+}
+
+
+/**
+  Print out arguments and help information based on what is allowed in
+  the getopt string optstr.
+
+  @param progname	Program name.
+  @param optstr		Getopt(3) style options string
+  @param print_stdin	0 = print command line options + description,
+			1 = print fence-style stdin args + description
+ */
+void
+args_usage(char *progname, char *optstr, int print_stdin)
+{
+	int x;
+	struct arg_info *arg;
+
+	if (!print_stdin) {
+		if (progname) {
+			printf("usage: %s [args]\n", progname);
+		} else {
+			printf("usage: fence_virt [args]\n");
+		}
+	}
+
+	for (x = 0; x < strlen(optstr); x++) {
+		arg = find_arg_by_char(optstr[x]);
+		if (!arg)
+			continue;
+
+		if (print_stdin) {
+			if (arg && arg->stdin_opt)
+				printf("  %-20.20s %-55.55s\n",
+				       arg->stdin_opt, arg->desc);
+		} else {
+			printf("  %-20.20s %-55.55s\n", arg->opt_desc,
+			       arg->desc);
+		}
+	}
+
+	printf("\n");
+}
+
+
+/**
+  Remove leading and trailing whitespace from a line of text.
+
+  @param line		Line to clean up
+  @param linelen	Max size of line
+  @return		0 on success, -1 on failure
+ */
+int
+cleanup(char *line, size_t linelen)
+{
+	char *p;
+	int x;
+	
+	/* Remove leading whitespace. */
+	p = line;
+	for (x = 0; x <= linelen; x++) {
+		switch (line[x]) {
+		case '\t':
+		case ' ':
+			break;
+		case '\n':
+		case '\r':
+			return -1;
+		default:
+			goto eol;
+		}
+	}
+eol:
+	/* Move the remainder down by as many whitespace chars as we
+	   chewed up */
+	if (x)
+		memmove(p, &line[x], linelen-x);
+
+	/* Remove trailing whitespace. */
+	for (x=0; x <= linelen; x++) {
+		switch(line[x]) {
+		case '\t':
+		case ' ':
+		case '\r':
+		case '\n':
+			line[x] = 0;
+		case 0:
+		/* End of line */
+			return 0;
+		}
+	}
+
+	return -1;
+}
+
+
+/**
+  Parse args from stdin and assign to the specified args structure.
+  
+  @param optstr		Command line option string in getopt(3) format
+  @param args		Args structure to fill in.
+ */
+void
+args_get_stdin(char *optstr, fence_virt_args_t *args)
+{
+	char in[256];
+	int line = 0;
+	char *name, *val;
+	struct arg_info *arg;
+
+	while (fgets(in, sizeof(in), stdin)) {
+		++line;
+
+		if (in[0] == '#')
+			continue;
+
+		if (cleanup(in, sizeof(in)) == -1)
+			continue;
+
+		name = in;
+		if ((val = strchr(in, '='))) {
+			*val = 0;
+			++val;
+		}
+
+		arg = find_arg_by_string(name);
+		if (!arg || (arg->opt != '\xff' && 
+			     !strchr(optstr, arg->opt))) {
+			fprintf(stderr,
+				"parse warning: "
+				"illegal variable '%s' on line %d\n", name,
+				line);
+			continue;
+		}
+
+		if (arg->assign)
+			arg->assign(args, arg, val);
+	}
+}
+
+
+/**
+  Parse args from stdin and assign to the specified args structure.
+  
+  @param optstr		Command line option string in getopt(3) format
+  @param args		Args structure to fill in.
+ */
+void
+args_get_getopt(int argc, char **argv, char *optstr, fence_virt_args_t *args)
+{
+	int opt;
+	struct arg_info *arg;
+
+	while ((opt = getopt(argc, argv, optstr)) != EOF) {
+
+		arg = find_arg_by_char(opt);
+
+		if (!arg) {
+			args->flags |= F_ERR;
+			continue;
+		}
+
+		if (arg->assign)
+			arg->assign(args, arg, optarg);
+	}
+}
+
+
+void
+args_finalize(fence_virt_args_t *args)
+{
+	char *addr = NULL;
+
+	if (!args->net.addr) {
+		switch(args->net.family) {
+		case 0:
+		case PF_INET:
+			addr = IPV4_MCAST_DEFAULT;
+			break;
+		case PF_INET6:
+			addr = IPV6_MCAST_DEFAULT;
+			break;
+		default:
+			args->flags |= F_ERR;
+		break;
+		}
+	}
+
+	if (!args->net.addr)
+		args->net.addr = addr;
+
+	if (!args->net.addr) {
+		printf("No multicast address available\n");
+		args->flags |= F_ERR;
+	}
+
+	if (!args->net.addr)
+		return;
+	if (args->net.family)
+		return;
+
+	/* Set family */
+	if (strchr(args->net.addr, ':'))
+		args->net.family = PF_INET6;
+	if (strchr(args->net.addr, '.'))
+		args->net.family = PF_INET;
+	if (!args->net.family) {
+		printf("Could not determine address family\n");
+		args->flags |= F_ERR;
+	}
+}

common/simple_auth.c

@@ -0,0 +1,414 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#include <sys/types.h>
+#include <string.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sechash.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <errno.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "simple_auth.h"
+#include "debug.h"
+
+
+void
+print_hash(unsigned char *hash, size_t hashlen)
+{
+	int x; 
+
+	for (x = 0; x < hashlen; x++)
+		printf("%02x", (hash[x]&0xff));
+}
+
+
+static void
+sha_sign(fence_req_t *req, void *key, size_t key_len)
+{
+	unsigned char hash[SHA512_LENGTH];
+	HASHContext *h;
+	HASH_HashType ht;
+	unsigned int rlen;
+	int devrand;
+
+	switch(req->hashtype) {
+		case HASH_SHA1:
+			ht = HASH_AlgSHA1;
+			break;
+		case HASH_SHA256:
+			ht = HASH_AlgSHA256;
+			break;
+		case HASH_SHA512:
+			ht = HASH_AlgSHA512;
+			break;
+		default:
+			return;
+	}
+
+	dbg_printf(4, "Opening /dev/urandom\n");
+	devrand = open("/dev/urandom", O_RDONLY);
+	if (devrand >= 0) {
+		if (read(devrand, req->random, sizeof(req->random)) < 0) {
+			perror("read /dev/urandom");
+		}
+		close(devrand);
+	}
+
+	memset(hash, 0, sizeof(hash));
+	h = HASH_Create(ht);
+	if (!h)
+		return;
+
+	HASH_Begin(h);
+	HASH_Update(h, key, key_len);
+	HASH_Update(h, (void *)req, sizeof(*req));
+	HASH_End(h, hash, &rlen, sizeof(hash));
+	HASH_Destroy(h);
+
+	memcpy(req->hash, hash, sizeof(req->hash));
+}
+
+
+static int
+sha_verify(fence_req_t *req, void *key, size_t key_len)
+{
+	unsigned char hash[SHA512_LENGTH];
+	unsigned char pkt_hash[SHA512_LENGTH];
+	HASHContext *h = NULL;
+	HASH_HashType ht;
+	unsigned int rlen;
+	int ret;
+
+	switch(req->hashtype) {
+		case HASH_SHA1:
+			ht = HASH_AlgSHA1;
+			break;
+		case HASH_SHA256:
+			ht = HASH_AlgSHA256;
+			break;
+		case HASH_SHA512:
+			ht = HASH_AlgSHA512;
+			break;
+		default:
+			dbg_printf(3, "%s: no-op (HASH_NONE)\n", __FUNCTION__);
+			return 0;
+	}
+
+	memset(hash, 0, sizeof(hash));
+	h = HASH_Create(ht);
+	if (!h)
+		return 0;
+
+	memcpy(pkt_hash, req->hash, sizeof(pkt_hash));
+	memset(req->hash, 0, sizeof(req->hash));
+
+	HASH_Begin(h);
+	HASH_Update(h, key, key_len);
+	HASH_Update(h, (void *)req, sizeof(*req));
+	HASH_End(h, hash, &rlen, sizeof(hash));
+	HASH_Destroy(h);
+
+	memcpy(req->hash, pkt_hash, sizeof(req->hash));
+
+	ret = !memcmp(hash, pkt_hash, sizeof(hash));
+	if (!ret) {
+		printf("Hash mismatch:\nPKT = ");
+		print_hash(pkt_hash, sizeof(pkt_hash));
+		printf("\nEXP = ");
+		print_hash(hash, sizeof(hash));
+		printf("\n");
+	}
+
+	return ret;
+}
+
+
+int
+sign_request(fence_req_t *req, void *key, size_t key_len)
+{
+	memset(req->hash, 0, sizeof(req->hash));
+	switch(req->hashtype) {
+	case HASH_NONE:
+		dbg_printf(3, "%s: no-op (HASH_NONE)\n", __FUNCTION__);
+		return 0;
+	case HASH_SHA1:
+	case HASH_SHA256:
+	case HASH_SHA512:
+		sha_sign(req, key, key_len);
+		return 0;
+	default:
+		break;
+	}
+	return -1;
+}
+
+
+int
+verify_request(fence_req_t *req, fence_hash_t min,
+	       void *key, size_t key_len)
+{
+	if (req->hashtype < min) {
+		printf("Hash type not strong enough (%d < %d)\n",
+		       req->hashtype, min);
+		return 0;
+	}
+	switch(req->hashtype) {
+	case HASH_NONE:
+		return 1;
+	case HASH_SHA1:
+	case HASH_SHA256:
+	case HASH_SHA512:
+		return sha_verify(req, key, key_len);
+	default:
+		break;
+	}
+	return 0;
+}
+
+
+int
+sha_challenge(int fd, fence_auth_type_t auth, void *key,
+	      size_t key_len, int timeout)
+{
+	fd_set rfds;
+	struct timeval tv;
+	unsigned char hash[MAX_HASH_LENGTH];
+	unsigned char challenge[MAX_HASH_LENGTH];
+	unsigned char response[MAX_HASH_LENGTH];
+	int devrand;
+	int ret;
+	HASHContext *h;
+	HASH_HashType ht;
+	unsigned int rlen;
+
+	devrand = open("/dev/urandom", O_RDONLY);
+	if (read(devrand, challenge, sizeof(challenge)) < 0) {
+		perror("read /dev/urandom");
+		return 0;
+	}
+	close(devrand);
+
+	if (write(fd, challenge, sizeof(challenge)) < 0) {
+		perror("write");
+		return 0;
+	}
+
+	switch(auth) {
+		case HASH_SHA1:
+			ht = HASH_AlgSHA1;
+			break;
+		case HASH_SHA256:
+			ht = HASH_AlgSHA256;
+			break;
+		case HASH_SHA512:
+			ht = HASH_AlgSHA512;
+			break;
+		default:
+			return 0;
+	}
+
+	memset(hash, 0, sizeof(hash));
+	h = HASH_Create(ht);
+	if (!h)
+		return 0;
+
+	HASH_Begin(h);
+	HASH_Update(h, key, key_len);
+	HASH_Update(h, challenge, sizeof(challenge));
+	HASH_End(h, hash, &rlen, sizeof(hash));
+	HASH_Destroy(h);
+
+	memset(response, 0, sizeof(response));
+
+	FD_ZERO(&rfds);
+	FD_SET(fd, &rfds);
+	tv.tv_sec = timeout;
+	tv.tv_usec = 0;
+	if (select(fd + 1, &rfds, NULL, NULL, &tv) <= 0) {
+		perror("select");
+		return 0;
+	}
+
+	if (read(fd, response, sizeof(response)) < sizeof(response)) {
+		perror("read");
+		return 0;
+	}
+
+	ret = !memcmp(response, hash, sizeof(response));
+	if (!ret) {
+		printf("Hash mismatch:\nC = ");
+		print_hash(challenge, sizeof(challenge));
+		printf("\nH = ");
+		print_hash(hash, sizeof(hash));
+		printf("\nR = ");
+		print_hash(response, sizeof(response));
+		printf("\n");
+	}
+
+	return ret;
+}
+
+
+int
+sha_response(int fd, fence_auth_type_t auth, void *key,
+	     size_t key_len, int timeout)
+{
+	fd_set rfds;
+	struct timeval tv;
+	unsigned char challenge[MAX_HASH_LENGTH];
+	unsigned char hash[MAX_HASH_LENGTH];
+	HASHContext *h;
+	HASH_HashType ht;
+	unsigned int rlen;
+
+	FD_ZERO(&rfds);
+	FD_SET(fd, &rfds);
+	tv.tv_sec = timeout;
+	tv.tv_usec = 0;
+	if (select(fd + 1, &rfds, NULL, NULL, &tv) <= 0) {
+		perror("select");
+		return 0;
+	}
+
+	if (read(fd, challenge, sizeof(challenge)) < 0) {
+		perror("read");
+		return 0;
+	}
+
+	switch(auth) {
+		case AUTH_SHA1:
+			ht = HASH_AlgSHA1;
+			break;
+		case AUTH_SHA256:
+			ht = HASH_AlgSHA256;
+			break;
+		case AUTH_SHA512:
+			ht = HASH_AlgSHA512;
+			break;
+		default:
+			dbg_printf(3, "%s: no-op (AUTH_NONE)\n", __FUNCTION__);
+			return 0;
+	}
+
+	memset(hash, 0, sizeof(hash));
+	h = HASH_Create(ht); /* */
+	if (!h)
+		return 0;
+
+	HASH_Begin(h);
+	HASH_Update(h, key, key_len);
+	HASH_Update(h, challenge, sizeof(challenge));
+	HASH_End(h, hash, &rlen, sizeof(hash));
+	HASH_Destroy(h);
+
+	if (write(fd, hash, sizeof(hash)) < sizeof(hash)) {
+		perror("read");
+		return 0;
+	}
+
+	return 1;
+}
+
+
+int
+tcp_challenge(int fd, fence_auth_type_t auth, void *key, size_t key_len,
+	      int timeout)
+{
+	switch(auth) {
+	case AUTH_NONE:
+		dbg_printf(3, "%s: no-op (AUTH_NONE)\n", __FUNCTION__);
+		return 1;
+	case AUTH_SHA1:
+	case AUTH_SHA256:
+	case AUTH_SHA512:
+		return sha_challenge(fd, auth, key, key_len, timeout);
+	default:
+		break;
+	}
+	return -1;
+}
+
+
+int
+tcp_response(int fd, fence_auth_type_t auth, void *key, size_t key_len,
+	     int timeout)
+{
+	switch(auth) {
+	case AUTH_NONE:
+		dbg_printf(3, "%s: no-op (AUTH_NONE)\n", __FUNCTION__);
+		return 1;
+	case AUTH_SHA1:
+	case AUTH_SHA256:
+	case AUTH_SHA512:
+		return sha_response(fd, auth, key, key_len, timeout);
+	default:
+		break;
+	}
+	return -1;
+}
+
+
+int
+read_key_file(char *file, char *key, size_t max_len)
+{
+	int fd;
+	int nread, remain = max_len;
+	char *p;
+
+	dbg_printf(3, "Reading in key file %s into %p (%d max size)\n",
+		file, key, (int)max_len);
+	fd = open(file, O_RDONLY);
+	if (fd < 0) {
+		dbg_printf(2, "Error opening key file: %s\n", strerror(errno));
+		return -1;
+	}
+
+	memset(key, 0, max_len);
+	p = key;
+	remain = max_len;
+
+	while (remain) {
+		nread = read(fd, p, remain);
+		if (nread < 0) {
+			if (errno == EINTR)
+				continue;
+			dbg_printf(2, "Error from read: %s\n", strerror(errno));
+			close(fd);
+			return -1;
+		}
+
+		if (nread == 0) {
+			dbg_printf(3, "Stopped reading @ %d bytes",
+				(int)max_len-remain);
+			break;
+		}
+		
+		p += nread;
+		remain -= nread;
+	}
+
+	close(fd);	
+	dbg_printf(3, "Actual key length = %d bytes", (int)max_len-remain);
+	
+	return (int)(max_len - remain);
+}

common/tcp.c

@@ -0,0 +1,300 @@
+/*
+  Copyright Red Hat, Inc. 2002-2004, 2006
+  Copyright Mission Critical Linux, 2000
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+                                                                                
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+                                                                                
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge,
+  MA 02139, USA.
+*/
+/** @file
+ *
+ * @author Lon H. Hohberger <lhh at redhat.com>
+ * @author Jeff Moyer <jmoyer at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <sys/types.h>
+#include <arpa/inet.h>
+
+#include "debug.h"
+
+static int connect_nb(int fd, struct sockaddr *dest, socklen_t len, int timeout);
+
+/**
+  Set close-on-exec bit option for a socket.
+
+   @param fd		Socket to set CLOEXEC flag
+   @return		0 on success, -1 on failure
+   @see			fcntl
+ */
+static int 
+set_cloexec(int fd)
+{
+	int flags = fcntl(fd, F_GETFD, 0);
+	flags |= FD_CLOEXEC;
+	return fcntl(fd, F_SETFD, flags);
+}
+
+
+/**
+  Bind to a port on the local IPv6 stack
+
+  @param port		Port to bind to
+  @param backlog	same as backlog for listen(2)
+  @return		0 on success, -1 on failure
+  @see			ipv4_bind
+ */
+int
+ipv6_listen(uint16_t port, int backlog)
+{
+	struct sockaddr_in6 _sin6;
+	int fd, ret;
+
+	dbg_printf(4, "%s: Setting up ipv6 listen socket\n", __FUNCTION__);
+	fd = socket(PF_INET6, SOCK_STREAM, 0);
+	if (fd < 0)
+		return -1;
+
+	memset(&_sin6, 0, sizeof(_sin6));
+	_sin6.sin6_family = PF_INET6;
+	_sin6.sin6_port = htons(port);
+	_sin6.sin6_flowinfo = 0;
+	_sin6.sin6_addr = in6addr_any;
+
+	ret = 1;
+	setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, (void *)&ret, sizeof (ret));
+
+	ret = set_cloexec(fd);
+	if (ret < 0) {
+		close(fd);
+		return -1;
+	}
+
+	ret = bind(fd, (struct sockaddr *)&_sin6, sizeof(_sin6));
+	if (ret < 0) {
+		close(fd);
+		return -1;
+	}
+
+	if (listen(fd, backlog) < 0){
+		close(fd);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: Success; fd = %d\n", __FUNCTION__, fd);
+	return fd;
+}
+
+
+/**
+  Bind to a port on the local IPv4 stack
+
+  @param port		Port to bind to
+  @param backlog	same as backlog for listen(2)
+  @return		0 on success, -1 on failure
+  @see			ipv6_bind
+ */
+int
+ipv4_listen(uint16_t port, int backlog)
+{
+	struct sockaddr_in _sin;
+	int fd, ret;
+
+	dbg_printf(4, "%s: Setting up ipv4 listen socket\n", __FUNCTION__);
+	fd = socket(PF_INET, SOCK_STREAM, 0);
+	if (fd < 0)
+		return -1;
+
+	_sin.sin_family = PF_INET;
+	_sin.sin_port = htons(port);
+	_sin.sin_addr.s_addr = htonl(INADDR_ANY);
+
+	ret = 1;
+	setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, (void *)&ret, sizeof (ret));
+	
+	ret = set_cloexec(fd);
+	if (ret < 0) {
+		close(fd);
+		return -1;
+	}
+
+	ret = bind(fd, (struct sockaddr *)&_sin, sizeof(_sin));
+	if (ret < 0) {
+		close(fd);
+		return -1;
+	}
+
+	if (listen(fd, backlog) < 0){
+		close(fd);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: Success; fd = %d\n", __FUNCTION__, fd);
+	return fd;
+}
+
+
+
+/**
+  Connect via ipv6 socket to a given IP address and port.
+
+  @param in6_addr	IPv6 address to connect to
+  @param port		Port to connect to
+  @param timeout	Timeout, in seconds, to wait for a completed
+  			connection
+  @return 		0 on success, -1 on failure
+  @see			connect_nb, ipv4_connect
+ */
+int
+ipv6_connect(struct in6_addr *in6_addr, uint16_t port, int timeout)
+{
+	struct sockaddr_in6 _sin6;
+	int fd, ret;
+
+	dbg_printf(4, "%s: Connecting to client\n", __FUNCTION__);
+	fd = socket(PF_INET6, SOCK_STREAM, 0);
+	if (fd < 0)
+		return -1;
+
+	memset(&_sin6, 0, sizeof(_sin6));
+	_sin6.sin6_family = PF_INET6;
+	_sin6.sin6_port = htons(port);
+	_sin6.sin6_flowinfo = 0;
+	memcpy(&_sin6.sin6_addr, in6_addr, sizeof(_sin6.sin6_addr));
+
+	ret = connect_nb(fd, (struct sockaddr *)&_sin6, sizeof(_sin6), timeout);
+	if (ret < 0) {
+		close(fd);
+		return -1;
+	}
+	dbg_printf(4, "%s: Success; fd = %d\n", __FUNCTION__, fd);
+	return fd;
+}
+
+
+/**
+  Connect via ipv4 socket to a given IP address and port.
+
+  @param in_addr	IPv4 address to connect to
+  @param port		Port to connect to
+  @param timeout	Timeout, in seconds, to wait for a completed
+  			connection
+  @return 		0 on success, -1 on failure
+  @see			connect_nb, ipv6_connect
+ */
+int
+ipv4_connect(struct in_addr *in_addr, uint16_t port, int timeout)
+{
+	struct sockaddr_in _sin;
+	int fd, ret;
+
+	dbg_printf(4, "%s: Connecting to client\n", __FUNCTION__);
+	fd = socket(PF_INET, SOCK_STREAM, 0);
+	if (fd < 0)
+		return -1;
+
+	_sin.sin_family = PF_INET;
+	_sin.sin_port = htons(port);
+	memcpy(&_sin.sin_addr, in_addr, sizeof(_sin.sin_addr));
+
+	ret = connect_nb(fd, (struct sockaddr *)&_sin, sizeof(_sin), timeout);
+	if (ret < 0) {
+		close(fd);
+		return -1;
+	}
+
+	dbg_printf(4, "%s: Success; fd = %d\n", __FUNCTION__, fd);
+	return fd;
+}
+
+
+/**
+  Connect in a non-blocking fashion to the designated address.
+
+  @param fd		File descriptor to connect
+  @param dest		sockaddr (ipv4 or ipv6) to connect to.
+  @param len		Length of dest
+  @param timeout	Timeout, in seconds, to wait for a completed
+  			connection.
+  @return		0 on success, -1 on failure.
+ */
+static int
+connect_nb(int fd, struct sockaddr *dest, socklen_t len, int timeout)
+{
+	int ret, flags = 1, err;
+	unsigned l;
+	fd_set rfds, wfds;
+	struct timeval tv;
+
+	/*
+	 * Use TCP Keepalive
+	 */
+	if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, (void *)&flags,
+		       sizeof(flags))<0)
+		return -1;
+			
+	/*
+	   Set up non-blocking connect
+	 */
+	flags = fcntl(fd, F_GETFL, 0);
+	fcntl(fd, F_SETFL, flags | O_NONBLOCK);
+
+	ret = connect(fd, dest, len);
+
+	if ((ret < 0) && (errno != EINPROGRESS))
+		return -1;
+
+	if (ret != 0) {
+		FD_ZERO(&rfds);
+		FD_SET(fd, &rfds);
+		FD_ZERO(&wfds);
+		FD_SET(fd, &wfds);
+
+		tv.tv_sec = timeout;
+		tv.tv_usec = 0;
+		
+		if (select(fd + 1, &rfds, &wfds, NULL, &tv) == 0) {
+			errno = ETIMEDOUT;
+			return -1;
+		}
+		/* XXX check for -1 from select */
+
+		if (FD_ISSET(fd, &rfds) || FD_ISSET(fd, &wfds)) {
+			l = sizeof(err);
+			if (getsockopt(fd, SOL_SOCKET, SO_ERROR,
+				       (void *)&err, &l) < 0) {
+				close(fd);
+				return -1;
+			}
+
+			if (err != 0) {
+				close(fd);
+				errno = err;
+				return -1;
+			}
+
+			fcntl(fd, F_SETFL, flags);
+			return 0;
+		}
+	}
+
+	errno = EIO;
+	return -1;
+}

fence_virt.txt

@@ -0,0 +1,124 @@
+We need a fencing agent which can work in a variety of guest cluster
+configurations and host configurations.
+
+Requirements
+
+1. Nonrequirement of guest to host networking.  Virtual machines
+   may be configured to run using a nework unknown to the host
+   operating system.  Therefore, the ability to run without network
+   communication between the guest and the hsot is required.
+
+2. Ease of configuration.  The absolute minimum possible configuration
+   must be available.
+
+3. Nonrequirement of host clustering software.  Multiple layers of
+   configuration sucks.  While I fundamentally disagree with the general
+   idea that running CMAN on the host constitutes a "heavyweight
+   cluster", customer perception is important.
+
+4. Support for RHEV-M.  Be able to send fencing requests up to RHEV-M
+   for execution.  This is beneficial from a security standpoint.
+
+5. Upgrade compatibility with fence_xvm from a configuration standpoint.
+   This may be provided by a symlink over fence_xvm.  If this feature
+   can not be provided as a matter of design, a method to convert an
+   existing fence_xvm/fence_xvmd configuration to fence_virt must be
+   present.
+
+
+Guest to Host Interaction
+-------------------------
+
+The proposal is to use various communications media plugins in order
+to facilitate flexibility with respect to how virtual machine 
+environments are configured.
+
+There are at least 3 simple plugins for guest/client to host/server
+communications:
+
+ * Direct serial.  The guest sends fencing requests out via /dev/ttySX
+   in the guest.  The host is listening on a Unix domain socket[1],
+   and forwards fencing requests accordingly.
+
+   This satisifies most of the requirements, but adds a conundrum
+   when configuring guest clusters, as /dev/ttySX may be /dev/ttySY
+   on another guest.  So, either we must account for this per-guest
+   configuration discrepancy or we must make it an administrative
+   requirement to provide the same serial device on each host
+
+ * VM Channel over Serial.  This works like direct serial, but
+   instead of owning the whole device, the device may be shared between
+   multiple applications.  The server subscribes to a channel and
+   listens for fencing requests on the channel; the client in the
+   guest OS connects to the channel and issues fencing requests across
+   it.  One interesting thing is that it may be possible to provide
+   unprivileged users the ability to fence using this method (I
+   do not claim to know if this is useful or not).
+
+ * Multicast.  This violates requirement, however, provides for one
+   of the simpler configurations: all that is needed is the guest's
+   name or UUID.  The guest to host communications operates in the
+   same manner as fence_xvm/fence_xvmd, except that there is an
+   implied requirement on restricting the multicast packets accepted
+   to be from the local guests.
+
+Host to Hypervisor interaction
+------------------------------
+
+Similar to the way we have plugins for guest to host interaction, 
+we also have plugins which actually do the real work.  These plugins
+are responsible for all of the actual real work performed, including
+tracking VMs if required, forwarding requests to the appropriate hosts
+or management services, and handling the responses.
+
+We propose at 5 plugins in this case:
+
+ * Libvirt (local-only).  There is no intracommunication and no
+   migration support is provided 
+
+ * Cluster checkpoint (+ libvirt).  This the way fence_xvmd
+   operates today.  This setup has the most requirements on the
+   infrastructure, as it requires guest to host networking _and_
+   host-to-host clustering in order to keep track of virtual
+   machines.  The benefit is that it is self-contained and requires
+   no external management nodes.  VM states are stored in checkpoints
+   so that other hosts know the locations of other VMs and can make
+   some decisions about whether a VM is dead based on whether a host
+   is dead (i.e. if fencing is in use or can be performed on the
+   host).
+
+ * Libvirt-QMF ... ???  Subscription to the appropriate cluster
+   specific AMQP channel is required on the host side, but this
+   handles routing the message very easily.  The fencing request
+   is forwarded to the other listeners on the channel, the VM owner
+   takes the action requested and returns a value.  When new VMs
+   are created, the event is broadcast out via the AMQP channel so
+   other hosts know the locations of other VMs and can make some
+   decisions about whether a VM is dead based on whether a host
+   is dead (i.e. if fencing is in use or can be performed on the
+   host).
+
+ * oVirt Manager.  The request is forwarded to the oVirt Manager
+   and the oVirt manager is responsible for taking the appropriate
+   action and responding to the request.
+   
+ * RHEV-M.  The request is forwarded to the RHEV-M node, which is
+   responsible for taking the appropriate action and responding to
+   the request.
+
+These plugins have no requirements on which guest to host communication
+plugin is used (you could, if you wanted, use 'direct serial' with
+'cluster checkpoint', or 'multicast' with 'RHEV-H' for example).
+
+These plugins must also be able to discover where appropriate.  For
+example, the Checkpoint plugin can only be used if corosync/openais
+is running.  Likewise, the RHEV-M plugin may only be used in the
+cases where the host operating system is a RHEV-H.  A defined plugin
+preference order should be specified/documented so that the host
+daemon behaves in a predictable manner in absence of host-side
+configuration data (about which plugin to use).
+
+
+[1] TCP was also explored, however, the security is much better 
+    using a Unix domain socket, despite the additional complexity 
+    of listening for VM creation events.

include/bcast.h

@@ -0,0 +1,16 @@
+#ifndef _XVM_MCAST_H
+#define _XVM_MCAST_H
+
+#define IPV4_MCAST_DEFAULT "225.0.0.12"
+#define IPV6_MCAST_DEFAULT "ff05::3:1"
+
+int ipv4_recv_sk(char *addr, int port);
+int ipv4_send_sk(char *src_addr, char *addr, int port,
+		 struct sockaddr *src, socklen_t slen,
+		 int ttl);
+int ipv6_recv_sk(char *addr, int port);
+int ipv6_send_sk(char *src_addr, char *addr, int port,
+		 struct sockaddr *src, socklen_t slen,
+		 int ttl);
+
+#endif

include/client.h

@@ -0,0 +1,7 @@
+#ifndef _CLIENT_H
+#define _CLIENT_H
+
+int serial_fence_virt(fence_virt_args_t *args);
+int mcast_fence_virt(fence_virt_args_t *args);
+
+#endif

include/debug.h

@@ -0,0 +1,31 @@
+/*
+  Copyright Red Hat, Inc. 2007
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _DBG_H
+#define _DBG_H
+
+inline void dset(int);
+inline int dget(void);
+
+#define dbg_printf(level, fmt, args...) \
+do { \
+	if (dget()>=level) \
+		printf(fmt, ##args); \
+} while(0)
+
+#endif

include/fdops.h

@@ -0,0 +1,14 @@
+#ifndef _FDOPS_H
+#define _FDOPS_H
+#include <unistd.h>
+#include <sys/types.h>
+#include <fcntl.h>
+#include <errno.h>
+
+int _select_retry(int fdmax, fd_set * rfds, fd_set * wfds, fd_set * xfds,
+   		  struct timeval *timeout);
+ssize_t _write_retry(int fd, void *buf, int count, struct timeval * timeout);
+ssize_t _read_retry(int sockfd, void *buf, int count,
+		    struct timeval * timeout);
+
+#endif

include/ip_lookup.h

@@ -0,0 +1,40 @@
+/*
+  Copyright Red Hat, Inc. 2004,2006
+
+  The Magma Cluster API Library is free software; you can redistribute
+  it and/or modify it under the terms of the GNU Lesser General Public
+  License as published by the Free Software Foundation; either version
+  2.1 of the License, or (at your option) any later version.
+
+  The Magma Cluster API Library is distributed in the hope that it will
+  be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
+  of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  Lesser General Public License for more details.
+
+  You should have received a copy of the GNU Lesser General Public
+  License along with this library; if not, write to the Free Software
+  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+  USA.
+*/
+/** @file
+ * Header for ip_lookup.c
+ */
+#ifndef _IP_LOOKUP_H
+#define _IP_LOOKUP_H
+
+#include <sys/queue.h>
+
+typedef struct _ip_address {
+	TAILQ_ENTRY(_ip_address) ipa_entries;
+	char ipa_family;
+	char *ipa_address;
+} ip_addr_t;
+
+typedef TAILQ_HEAD(_ip_list, _ip_address) ip_list_t;
+
+int ip_search(ip_list_t *ipl, char *ip_name);
+int ip_free_list(ip_list_t *ipl);
+int ip_build_list(ip_list_t *ipl);
+int ip_lookup(char *, struct addrinfo **);
+
+#endif

include/mcast.h

@@ -0,0 +1,32 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _XVM_MCAST_H
+#define _XVM_MCAST_H
+
+#define IPV4_MCAST_DEFAULT "225.0.0.12"
+#define IPV6_MCAST_DEFAULT "ff05::3:1"
+
+int ipv4_recv_sk(char *addr, int port, unsigned int ifindex);
+int ipv4_send_sk(char *src_addr, char *addr, int port,
+		 struct sockaddr *src, socklen_t slen);
+int ipv6_recv_sk(char *addr, int port, unsigned int ifindex);
+int ipv6_send_sk(char *src_addr, char *addr, int port,
+		 struct sockaddr *src, socklen_t slen);
+
+#endif

include/options.h

@@ -0,0 +1,87 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _XVM_OPTIONS_H
+#define _XVM_OPTIONS_H
+
+typedef enum {
+	F_FOREGROUND	= 0x1,
+	F_NOCCS		= 0x2,
+	F_ERR		= 0x4,
+	F_HELP		= 0x8,
+	F_USE_UUID	= 0x10,
+	F_VERSION	= 0x20,
+	F_CCSERR	= 0x40,
+	F_CCSFAIL	= 0x80,
+	F_NOCLUSTER	= 0x100
+} arg_flags_t;
+
+typedef enum {
+	MODE_MULTICAST  = 0,
+	//MODE_BROADCAST  = 1,
+	MODE_SERIAL     = 2,
+	MODE_VMCHANNEL  = 3
+} client_mode_t;
+
+typedef struct {
+	char *domain;
+	fence_cmd_t op;
+	client_mode_t mode;
+	int debug;
+	int timeout;
+	int retr_time;
+	arg_flags_t flags;
+
+	struct network_args {
+		char *addr;
+		char *key_file;
+		int port;
+		fence_hash_t hash;
+		fence_auth_type_t auth;
+		int family;
+		unsigned int ifindex;
+	} net;
+
+	struct serial_args {
+		char *device;		/* Serial device */
+		char *speed;
+	} serial;
+} fence_virt_args_t;
+
+/* Private structure for commandline / stdin fencing args */
+struct arg_info {
+	char opt;
+	char *opt_desc;
+	char *stdin_opt;
+	char *desc;
+	void (*assign)(fence_virt_args_t *, struct arg_info *, char *);
+};
+
+
+/* Get options */
+void args_init(fence_virt_args_t *args);
+void args_finalize(fence_virt_args_t *args);
+
+void args_get_getopt(int argc, char **argv, char *optstr,
+		     fence_virt_args_t *args);
+void args_get_stdin(char *optstr, fence_virt_args_t *args);
+void args_get_ccs(char *optstr, fence_virt_args_t *args);
+void args_usage(char *progname, char *optstr, int print_stdin);
+void args_print(fence_virt_args_t *args);
+
+#endif

include/server_plugin.h

@@ -0,0 +1,80 @@
+
+/*  */
+
+#include <uuid/uuid.h>
+
+typedef void * srv_context_t;
+
+/* These callbacks hand requests off to the
+   appropriate backend. */
+   
+/* Do nothing.  Returns 1 (failure) to caller */
+typedef int (*fence_null_callback)(const char *vm_name,
+				   void *priv);
+
+/* Turn the VM 'off'.  Returns 0 to caller if successful or
+   nonzero if unsuccessful. */
+typedef int (*fence_off_callback)(const char *vm_name,
+				  void *priv);
+
+/* Turn the VM 'on'.  Returns 0 to caller if successful or
+   nonzero if unsuccessful. */
+typedef int (*fence_on_callback)(const char *vm_name,
+				 void *priv);
+
+/* Reboot a VM.  Returns 0 to caller if successful or
+   nonzero if unsuccessful. */
+typedef int (*fence_reboot_callback)(const char *vm_name,
+				     void *priv);
+
+/* Get status of a VM.  Returns 0 to caller if VM is alive or
+   nonzero if VM is not alive. */
+typedef int (*fence_status_callback)(const char *vm_name,
+				     void *priv);
+
+/* Get status of backend.  Returns 0 to caller if backend
+   is responding to requests. */
+typedef int (*fence_devstatus_callback)(void *priv);
+
+typedef struct _fence_callbacks {
+	fence_null_callback null;
+	fence_off_callback off;
+	fence_on_callback on;
+	fence_reboot_callback reboot;
+	fence_status_callback status;
+	fence_devstatus_callback devstatus;
+} fence_callbacks_t;
+
+
+extern fence_callbacks_t libvirt_callbacks;
+int libvirt_init(srv_context_t *c);
+int libvirt_shutdown(srv_context_t c);
+
+
+/* TODO: make these 'plugins' instead of static uses */
+typedef void serial_options;
+/* Directory path / hypervisor uri if using libvirt...
+   .. whatever you think you need...  */
+
+int serial_init(srv_context_t *, fence_callbacks_t *,
+		serial_options *, void *priv);
+
+/* NULL = no timeout; wait forever */
+int serial_dispatch(srv_context_t, struct timeval *timeout);
+int serial_shutdown(srv_context_t);
+
+typedef struct {
+	char *addr;
+	char *key_file;
+	int ifindex;
+	int family;
+	unsigned int port;
+	unsigned int hash;
+	unsigned int auth;
+} mcast_options;
+
+int mcast_init(srv_context_t *, fence_callbacks_t *,
+	       mcast_options *, void *priv);
+int mcast_dispatch(srv_context_t, struct timeval *timeout);
+int mcast_shutdown(srv_context_t);
+

include/simple_auth.h

@@ -0,0 +1,35 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _XVM_SIMPLE_AUTH_H
+#define _XVM_SIMPLE_AUTH_H
+
+#include <sys/types.h>
+
+/* 2-way challenge/response simple auth */
+#define DEFAULT_KEY_FILE "/etc/cluster/fence_xvm.key"
+
+int read_key_file(char *, char *, size_t);
+int tcp_challenge(int, fence_auth_type_t, void *, size_t, int);
+int tcp_response(int, fence_auth_type_t, void *, size_t, int);
+int sign_request(fence_req_t *, void *, size_t);
+int verify_request(fence_req_t *, fence_hash_t, void *, size_t);
+
+/* SSL certificate-based authentication TBD */
+
+#endif

include/tcp.h

@@ -0,0 +1,27 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _XVM_TCP_H
+#define _XVM_TCP_H
+
+int ipv4_connect(struct in_addr *in_addr, uint16_t port, int timeout);
+int ipv6_connect(struct in6_addr *in6_addr, uint16_t port, int timeout);
+int ipv4_listen(uint16_t port, int backlog);
+int ipv6_listen(uint16_t port, int backlog);
+
+#endif

include/xvm.h

@@ -0,0 +1,102 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _XVM_H
+#define _XVM_H
+
+#include <stdint.h>
+#include <sechash.h>
+#include <netinet/in.h>
+
+#define XVM_VERSION "1.9.0"
+
+#define MAX_DOMAINNAME_LENGTH 64 /* XXX MAXHOSTNAMELEN */
+#define MAX_ADDR_LEN		sizeof(struct sockaddr_in6)
+#define DOMAIN0NAME "Domain-0"
+#define DOMAIN0UUID "00000000-0000-0000-0000-000000000000"
+
+typedef enum {
+	HASH_NONE = 0x0,	/* No packet signing */
+	HASH_SHA1 = 0x1,	/* SHA1 signing */
+     	HASH_SHA256 = 0x2,      /* SHA256 signing */
+     	HASH_SHA512 = 0x3       /* SHA512 signing */
+} fence_hash_t;
+
+#define DEFAULT_HASH HASH_SHA256
+
+typedef enum {
+	AUTH_NONE = 0x0,	/* Plain TCP */
+	AUTH_SHA1 = 0x1,	/* Challenge-response (SHA1) */
+  	AUTH_SHA256 = 0x2,      /* Challenge-response (SHA256) */
+	AUTH_SHA512 = 0x3,      /* Challenge-response (SHA512) */
+     /* AUTH_SSL_X509 = 0x10        SSL X509 certificates */
+} fence_auth_type_t;
+
+#define DEFAULT_AUTH AUTH_SHA256
+
+typedef enum {
+	FENCE_NULL   = 0x0,	
+	FENCE_OFF    = 0x1,		/* Turn the VM off */
+	FENCE_REBOOT = 0x2,		/* Hit the reset button */
+	FENCE_ON     = 0x3,		/* Turn the VM on */
+	FENCE_STATUS = 0x4,		/* virtual machine status (off/on) */
+	FENCE_DEVSTATUS = 0x5	/* List virtual machines */
+} fence_cmd_t;
+
+#define DEFAULT_TTL 4
+
+#ifndef DEFAULT_HYPERVISOR_URI
+#define DEFAULT_HYPERVISOR_URI "qemu:///system"
+#endif
+
+#define MAX_HASH_LENGTH SHA512_LENGTH
+#define MAX_KEY_LEN 4096
+
+typedef struct __attribute__ ((packed)) _fence_req {
+	uint8_t  request;		/* Fence request */
+	uint8_t  hashtype;		/* Hash type used */
+	uint8_t  addrlen;		/* Length of address */
+	uint8_t  flags;			/* Special flags */
+#define RF_UUID 0x1			   /* Flag specifying UUID */
+	uint8_t  domain[MAX_DOMAINNAME_LENGTH]; /* Domain to fence*/
+	uint8_t  address[MAX_ADDR_LEN]; /* We're this IP */
+	uint16_t port;			/* Port we bound to */
+	uint8_t  random[10];		/* Random Data */
+	uint32_t family;		/* Address family */
+	uint8_t  hash[MAX_HASH_LENGTH];	/* Binary hash */
+} fence_req_t;
+
+
+#define DEFAULT_SERIAL_DEVICE "/dev/ttyS1"
+#define DEFAULT_SERIAL_SPEED "115200,8N1"
+#define SERIAL_MAGIC 0x61626261 /* endian doesn't matter */
+
+typedef struct __attribute__((packed)) _serial_fence_req {
+	uint32_t magic;
+	uint8_t request;
+	uint8_t flags;
+	uint8_t domain[MAX_DOMAINNAME_LENGTH];
+} serial_req_t;
+
+typedef struct __attribute__((packed)) _serial_fense_resp {
+	uint32_t magic;
+	uint8_t response;
+} serial_resp_t;
+
+
+#endif

server/Makefile

@@ -0,0 +1,45 @@
+###############################################################################
+###############################################################################
+##
+##  Copyright (C) 2006 Red Hat, Inc.
+##  
+##  This copyrighted material is made available to anyone wishing to use,
+##  modify, copy, or redistribute it subject to the terms and conditions
+##  of the GNU General Public License v.2.
+##
+###############################################################################
+###############################################################################
+
+
+TARGETS=fence_virtd
+
+fence_virtd_SOURCES = mcast.c libvirt.c main.c
+
+INCLUDES=-I../include\
+	-I/usr/include/openais -I/usr/include/libvirt \
+	-I/usr/include/nss3 -I/usr/include/nspr4 \
+	-I../../../cman/lib -I../../../ccs/lib -I/usr/include/libxml2 \
+	-I/usr/include/libvirt
+
+CFLAGS+=-DFENCE_RELEASE_NAME=\"devel\" \
+	-Wall -Werror -Wstrict-prototypes -Wshadow -ggdb -D_GNU_SOURCE
+
+LIBS+=-L/usr/lib64/openais -lnss3 -lxml2 -lSaCkpt -lccs -lvirt -lcman \
+      -L../common -lfence_virt
+
+all: ${TARGETS}
+
+fence_virtd: ${fence_virtd_SOURCES:.c=.o}
+	gcc -o $@ $^ $(LIBS) -L../common -lfence_virt
+
+%.o: %.c
+	gcc $(CFLAGS) -c -o $@ $^ $(INCLUDES)
+
+install: all
+	if [ ! -d ${sbindir} ]; then \
+		install -d ${sbindir}; \
+	fi
+	install -m755 ${TARGETS} ${sbindir}
+
+clean:
+	rm -f ${TARGETS} *.o *.d *~

server/libvirt.c

@@ -0,0 +1,303 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+/*
+ * Author: Lon Hohberger <lhh at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <pthread.h>
+#include <virterror.h>
+#include <nss.h>
+#include <libgen.h>
+//#include <uuid/uuid.h>
+#include <server_plugin.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "simple_auth.h"
+#include "options.h"
+#include "mcast.h"
+#include "tcp.h"
+#include "virt.h"
+#include "libcman.h"
+#include "debug.h"
+
+
+static inline int
+wait_domain(const char *vm_name, virConnectPtr vp, int timeout)
+{
+	int tries = 0;
+	int response = 1;
+	virDomainPtr vdp;
+	virDomainInfo vdi;
+
+	vdp = virDomainLookupByName(vp, vm_name);
+	if (!vdp)
+		return 0;
+
+	/* Check domain liveliness.  If the domain is still here,
+	   we return failure, and the client must then retry */
+	/* XXX On the xen 3.0.4 API, we will be able to guarantee
+	   synchronous virDomainDestroy, so this check will not
+	   be necessary */
+	do {
+		sleep(1);
+		vdp = virDomainLookupByName(vp, vm_name);
+		if (!vdp) {
+			dbg_printf(2, "Domain no longer exists\n");
+			response = 0;
+			break;
+		}
+
+		memset(&vdi, 0, sizeof(vdi));
+		virDomainGetInfo(vdp, &vdi);
+		virDomainFree(vdp);
+
+		if (vdi.state == VIR_DOMAIN_SHUTOFF) {
+			dbg_printf(2, "Domain has been shut off\n");
+			response = 0;
+			break;
+		}
+		
+		dbg_printf(4, "Domain still exists (state %d) after %d seconds\n",
+			vdi.state, tries);
+
+		if (++tries >= timeout)
+			break;
+	} while (1);
+
+	return response;
+}
+
+
+static int
+libvirt_null(const char *vm_name, void *priv)
+{
+	dbg_printf(5, "%s %s\n", __FUNCTION__, vm_name);
+	printf("NULL operation: returning failure\n");
+	return 1;
+}
+
+
+static int
+libvirt_off(const char *vm_name, void *priv)
+{
+	virConnectPtr vp = (virConnectPtr)priv;
+	virDomainPtr vdp;
+	virDomainInfo vdi;
+	int ret = -1;
+
+	dbg_printf(5, "%s %s\n", __FUNCTION__, vm_name);
+
+	vdp = virDomainLookupByName(vp, vm_name);
+
+	if (!vdp ||
+	    ((virDomainGetInfo(vdp, &vdi) == 0) &&
+	     (vdi.state == VIR_DOMAIN_SHUTOFF))) {
+		dbg_printf(2, "[NOCLUSTER] Nothing to "
+			   "do - domain does not exist\n");
+
+		if (vdp)
+			virDomainFree(vdp);
+		return 0;
+	}
+
+	dbg_printf(2, "[OFF] Calling virDomainDestroy\n");
+	ret = virDomainDestroy(vdp);
+	if (ret < 0) {
+		printf("virDomainDestroy() failed: %d\n", ret);
+		return 1;
+	}
+
+	if (ret) {
+		printf("Domain still exists; fencing failed\n");
+		return 1;
+	}
+
+	return 0;
+}
+
+
+static int
+libvirt_on(const char *vm_name, void *priv)
+{
+	dbg_printf(5, "%s %s\n", __FUNCTION__, vm_name);
+
+	return -ENOSYS;
+}
+
+
+static int
+libvirt_devstatus(void *priv)
+{
+	dbg_printf(5, "%s ---\n", __FUNCTION__);
+
+	if (priv)
+		return 0;
+	return 1;
+}
+
+
+static int
+libvirt_status(const char *vm_name, void *priv)
+{
+	virConnectPtr vp = (virConnectPtr)priv;
+	virDomainPtr vdp;
+	virDomainInfo vdi;
+	int ret = 0;
+
+	dbg_printf(5, "%s %s\n", __FUNCTION__, vm_name);
+
+	vdp = virDomainLookupByName(vp, vm_name);
+
+	if (!vdp || ((virDomainGetInfo(vdp, &vdi) == 0) &&
+	     (vdi.state == VIR_DOMAIN_SHUTOFF))) {
+		ret = 1;
+	}
+
+	if (vdp)
+		virDomainFree(vdp);
+	return ret;
+}
+
+
+static int
+libvirt_reboot(const char *vm_name, void *priv)
+{
+	virConnectPtr vp = (virConnectPtr)priv;
+	virDomainPtr vdp, nvdp;
+	virDomainInfo vdi;
+	char *domain_desc;
+	int ret;
+
+	//uuid_unparse(vm_uuid, uu_string);
+	dbg_printf(5, "%s %s\n", __FUNCTION__, vm_name);
+
+	vdp = virDomainLookupByName(vp, vm_name);
+
+	if (!vdp || ((virDomainGetInfo(vdp, &vdi) == 0) &&
+	     (vdi.state == VIR_DOMAIN_SHUTOFF))) {
+		dbg_printf(2, "[libvirt:REBOOT] Nothing to "
+			   "do - domain does not exist\n");
+		if (vdp)
+			virDomainFree(vdp);
+		return 0;
+	}
+
+	printf("Rebooting domain %s...\n", vm_name);
+	domain_desc = virDomainGetXMLDesc(vdp, 0);
+
+	if (!domain_desc) {
+		printf("Failed getting domain description from "
+		       "libvirt\n");
+	}
+
+	dbg_printf(2, "[REBOOT] Calling virDomainDestroy(%p)\n", vdp);
+	ret = virDomainDestroy(vdp);
+	if (ret < 0) {
+		printf("virDomainDestroy() failed: %d/%d\n", ret, errno);
+		free(domain_desc);
+		virDomainFree(vdp);
+		return 1;
+	}
+
+	ret = wait_domain(vm_name, vp, 15);
+
+	if (ret) {
+		printf("Domain still exists; fencing failed\n");
+		if (domain_desc)
+			free(domain_desc);
+		return 1;
+	}
+		
+	if (!domain_desc)
+		return 0;
+
+	/* 'on' is not a failure */
+	ret = 0;
+
+	dbg_printf(3, "[[ XML Domain Info ]]\n");
+	dbg_printf(3, "%s\n[[ XML END ]]\n", domain_desc);
+	dbg_printf(2, "Calling virDomainCreateLinux()...\n");
+
+	nvdp = virDomainCreateLinux(vp, domain_desc, 0);
+	if (nvdp == NULL) {
+		/* More recent versions of libvirt or perhaps the
+		 * KVM back-end do not let you create a domain from
+		 * XML if there is already a defined domain description
+		 * with the same name that it knows about.  You must
+		 * then call virDomainCreate() */
+		dbg_printf(2, "Failed; Trying virDomainCreate()...\n");
+		if (virDomainCreate(vdp) < 0) {
+			dbg_printf(1, "Failed to recreate guest"
+				   " %s!\n", vm_name);
+		}
+	}
+
+	free(domain_desc);
+
+	return ret;
+}
+
+int
+libvirt_init(srv_context_t *c)
+{
+	virConnectPtr vp;
+
+	vp = virConnectOpen(NULL);
+	if (!vp)
+		return -1;
+	*c = (void *)vp;
+	return 0;
+}
+
+
+int
+libvirt_shutdown(srv_context_t c)
+{
+	return virConnectClose((virConnectPtr)c);
+}
+
+
+fence_callbacks_t libvirt_callbacks = {
+	.null = libvirt_null,
+	.off = libvirt_off,
+	.on = libvirt_on,
+	.reboot = libvirt_reboot,
+	.status = libvirt_status,
+	.devstatus = libvirt_devstatus
+};

server/main.c

@@ -0,0 +1,43 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+
+/* Local includes */
+#include <server_plugin.h>
+#include <debug.h>
+
+extern fence_callbacks_t libvirt_callbacks; /* should be in a header */
+
+int
+main(int argc, char **argv)
+{
+	srv_context_t mcast_context;
+	srv_context_t libvirt_context; /*XXX these should be differently
+					 named context types */
+
+	dset(99);
+
+	/* Only backend we have right now is basic libvirt */
+
+	if (libvirt_init(&libvirt_context) < 0) {
+		printf("Libvirt failed to initialize\n");
+		return 1;
+	}
+
+	/* only client we have now is mcast (fence_xvm behavior) */
+	if (mcast_init(&mcast_context, &libvirt_callbacks, NULL,
+		       libvirt_context) < 0) {
+		printf("Failed initialization!\n");
+		return 1;
+	}
+
+	while (mcast_dispatch(mcast_context, NULL) >= 0);
+
+	mcast_shutdown(mcast_context);
+	libvirt_shutdown(libvirt_context);
+
+	return 0;
+}

server/mcast.c

@@ -0,0 +1,371 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+/*
+ * Author: Lon Hohberger <lhh at redhat.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <pthread.h>
+#include <nss.h>
+#include <libgen.h>
+#include <server_plugin.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "simple_auth.h"
+#include "options.h"
+#include "mcast.h"
+#include "tcp.h"
+#include "debug.h"
+
+#define MCAST_MAGIC 0xaabab1b34b911a
+
+#define VALIDATE(info) \
+do {\
+	if (!info || info->magic != MCAST_MAGIC)\
+		return -EINVAL;\
+} while(0)
+
+typedef struct _mcast_info {
+	uint64_t magic;
+	void *priv;
+	char key[MAX_KEY_LEN];
+	mcast_options *args;
+	fence_callbacks_t *cb;
+	ssize_t key_len;
+	int mc_sock;
+	int need_kill;
+} mcast_info;
+
+
+int cleanup_xml(char *xmldesc, char **ret, size_t *retsz);
+
+
+static int
+connect_tcp(fence_req_t *req, fence_auth_type_t auth,
+	    void *key, size_t key_len)
+{
+	int fd = -1;
+	struct sockaddr_in sin;
+	struct sockaddr_in6 sin6;
+	char buf[128];
+
+	switch(req->family) {
+	case PF_INET:
+		memset(&sin, 0, sizeof(sin));
+		memcpy(&sin.sin_addr, req->address,
+		       sizeof(sin.sin_addr));
+		sin.sin_family = PF_INET;
+		fd = ipv4_connect(&sin.sin_addr, req->port,
+				  5);
+		if (fd < 0) {
+			printf("Failed to call back\n");
+			return -1;
+		}
+		break;
+	case PF_INET6:
+		memset(&sin6, 0, sizeof(sin));
+		memcpy(&sin6.sin6_addr, req->address,
+		       sizeof(sin6.sin6_addr));
+		sin.sin_family = PF_INET6;
+		fd = ipv6_connect(&sin6.sin6_addr, req->port,
+				  5);
+
+		memset(buf,0,sizeof(buf));
+		inet_ntop(PF_INET6, &sin6.sin6_addr, buf, sizeof(buf));
+
+		if (fd < 0) {
+			printf("Failed to call back %s\n", buf);
+			return -1;
+		}
+		break;
+	default:
+		printf("Family = %d\n", req->family);
+		return -1;
+	}
+
+	/* Noops if auth == AUTH_NONE */
+	if (tcp_response(fd, auth, key, key_len, 10) <= 0) {
+		printf("Failed to respond to challenge\n");
+		close(fd);
+		return -1;
+	}
+
+	if (tcp_challenge(fd, auth, key, key_len, 10) <= 0) {
+		printf("Remote failed challenge\n");
+		close(fd);
+		return -1;
+	}
+	return fd;
+}
+
+
+static int
+do_fence_request_tcp(fence_req_t *req, mcast_info *info)
+{
+	int fd = -1;
+	char response = 1;
+
+	fd = connect_tcp(req, info->args->auth, info->key, info->key_len);
+	if (fd < 0) {
+		dbg_printf(2, "Could call back for fence request: %s\n", 
+			strerror(errno));
+		goto out;
+	}
+
+	switch(req->request) {
+	case FENCE_NULL:
+		response = info->cb->null((char *)req->domain, info->priv);
+		break;
+	case FENCE_ON:
+		response = info->cb->on((char *)req->domain, info->priv);
+		break;
+	case FENCE_OFF:
+		response = info->cb->off((char *)req->domain, info->priv);
+		break;
+	case FENCE_REBOOT:
+		response = info->cb->reboot((char *)req->domain, info->priv);
+		break;
+	case FENCE_STATUS:
+		response = info->cb->status((char *)req->domain, info->priv);
+		break;
+	case FENCE_DEVSTATUS:
+		response = info->cb->devstatus(info->priv);
+		break;
+	}
+
+	dbg_printf(3, "Sending response to caller...\n");
+	if (write(fd, &response, 1) < 0) {
+		perror("write");
+	}
+out:
+	if (fd != -1)
+		close(fd);
+
+	return 1;
+}
+
+
+int
+mcast_dispatch(srv_context_t c, struct timeval *timeout)
+{
+	mcast_info *info;
+	fence_req_t data;
+	fd_set rfds;
+	struct sockaddr_in sin;
+	int len;
+	int n;
+	socklen_t slen;
+
+	info = (mcast_info *)c;
+	VALIDATE(info);
+
+	FD_ZERO(&rfds);
+	FD_SET(info->mc_sock, &rfds);
+
+#if 0
+	if (reload_key) {
+		char temp_key[MAX_KEY_LEN];
+		int ret;
+
+		reload_key = 0;
+
+			ret = read_key_file(args->key_file, temp_key, sizeof(temp_key));
+			if (ret < 0) {
+				printf("Could not read %s; not updating key",
+					args->key_file);
+			} else {
+				memcpy(key, temp_key, MAX_KEY_LEN);
+				key_len = (size_t) ret;
+
+				if (args->auth == AUTH_NONE)
+					args->auth = AUTH_SHA256;
+				if (args->hash == HASH_NONE)
+					args->hash = HASH_SHA256;
+			}
+		}
+#endif
+		
+	n = select((info->mc_sock)+1, &rfds, NULL, NULL, timeout);
+	if (n < 0)
+		return n;
+	
+	/* 
+	 * If no requests, we're done 
+	 */
+	if (n == 0)
+		return 0;
+
+	slen = sizeof(sin);
+	len = recvfrom(info->mc_sock, &data, sizeof(data), 0,
+		       (struct sockaddr *)&sin, &slen);
+		
+	if (len <= 0) {
+		perror("recvfrom");
+		return len;
+	}
+
+	if (!verify_request(&data, info->args->hash, info->key,
+			    info->key_len)) {
+		printf("Key mismatch; dropping packet\n");
+		return 0;
+	}
+
+#if 0
+	if ((args->flags & F_USE_UUID) &&
+	    !(data.flags & RF_UUID)) {
+			printf("Dropping packet: Request to fence by "
+			       "name while using UUIDs\n");
+			continue;
+		}
+
+		if (!(args->flags & F_USE_UUID) &&
+		    (data.flags & RF_UUID)) {
+			printf("Dropping packet: Request to fence by "
+			       "UUID while using names\n");
+			continue;
+		}
+#endif
+
+	printf("Request %d domain %s\n", data.request, data.domain);
+		
+	switch(info->args->auth) {
+	case AUTH_NONE:
+	case AUTH_SHA1:
+	case AUTH_SHA256:
+	case AUTH_SHA512:
+		printf("Plain TCP request\n");
+		do_fence_request_tcp(&data, info);
+		break;
+	default:
+		printf("XXX Unhandled authentication\n");
+	}
+
+	return 0;
+}
+
+
+int
+mcast_init(srv_context_t *c, fence_callbacks_t *cb,
+	   mcast_options *args, void *priv)
+{
+	mcast_info *info;
+	int mc_sock;
+
+	/* Initialize NSS; required to do hashing, as silly as that
+	   sounds... */
+	if (NSS_NoDB_Init(NULL) != SECSuccess) {
+		printf("Could not initialize NSS\n");
+		return 1;
+	}
+
+	info = malloc(sizeof(*info));
+	if (!info)
+		return -1;
+	memset(info, 0, sizeof(*info));
+
+	info->args = args;
+	info->priv = priv;
+	info->cb = cb;
+
+	if (!info->args) {
+		info->args = malloc(sizeof(*info->args));
+		if (!info->args) {
+			free(info);
+			return -ENOMEM;
+		}
+		info->need_kill = 1;
+		info->args->key_file = strdup(DEFAULT_KEY_FILE);
+		info->args->hash = DEFAULT_HASH;
+		info->args->auth = DEFAULT_AUTH;
+		info->args->addr = strdup(IPV4_MCAST_DEFAULT);
+		info->args->port = 1229;
+		info->args->ifindex = if_nametoindex("eth0");
+		info->args->family = PF_INET;
+	}
+
+	dbg_printf(6, "info->args->ifindex = %d\n", info->args->ifindex);
+
+	if (info->args->auth != AUTH_NONE || info->args->hash != HASH_NONE) {
+		info->key_len = read_key_file(info->args->key_file,
+					info->key, sizeof(info->key));
+		if (info->key_len < 0) {
+			printf("Could not read %s; operating without "
+			       "authentication\n", info->args->key_file);
+			info->args->auth = AUTH_NONE;
+			info->args->hash = HASH_NONE;
+		}
+	}
+
+	if (info->args->family == PF_INET)
+		mc_sock = ipv4_recv_sk(info->args->addr,
+				       info->args->port,
+				       info->args->ifindex);
+	else
+		mc_sock = ipv6_recv_sk(info->args->addr,
+				       info->args->port,
+				       info->args->ifindex);
+	if (mc_sock < 0) {
+		printf("Could not set up multicast listen socket\n");
+		free(info);
+		return 1;
+	}
+
+	info->magic = MCAST_MAGIC;
+	info->mc_sock = mc_sock;
+	*c = (srv_context_t)info;
+	return 0;
+}
+
+
+int
+mcast_shutdown(srv_context_t c)
+{
+	mcast_info *info = (mcast_info *)c;
+
+	VALIDATE(info);
+	info->magic = 0;
+	if (info->need_kill) {
+		free(info->args->key_file);
+		free(info->args->addr);
+		free(info->args);
+	}
+	close(info->mc_sock);
+	free(info);
+
+	return 0;
+}

server/options-ccs.c

@@ -0,0 +1,118 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/un.h>
+#include <sys/socket.h>
+#include <sys/select.h>
+#include <sys/ioctl.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+
+/* Local includes */
+#include "xvm.h"
+#include "simple_auth.h"
+#include "mcast.h"
+#include "options.h"
+
+#include <ccs.h>
+
+struct arg_info *find_arg_by_char(char arg);
+struct arg_info *find_arg_by_string(char *arg);
+
+extern int _debug;
+
+/**
+  Parse args from ccs and assign to the specified args structure.
+  (This should only be called from fence_xvmd; not fence_xvm!!!)
+  
+  @param optstr		Command line option string in getopt(3) format
+  @param args		Args structure to fill in.
+ */
+void
+args_get_ccs(char *optstr, fence_xvm_args_t *args)
+{
+	char buf[256];
+	int ccsfd = -1, x, n;
+	char *val;
+	struct arg_info *arg;
+	
+	if (args->flags & (F_NOCCS | F_HELP | F_VERSION))
+		return;
+
+	ccsfd = ccs_connect();
+	if (ccsfd < 0) {
+		args->flags |= F_CCSFAIL;
+		return;
+	}
+
+	for (x = 0; x < strlen(optstr); x++) {
+		arg = find_arg_by_char(optstr[x]);
+		if (!arg)
+			continue;
+
+		if (!arg || (arg->opt != '\xff' && 
+			     !strchr(optstr, arg->opt))) {
+			continue;
+		}
+
+		if (!arg->stdin_opt)
+			continue;
+
+		n = snprintf(buf, sizeof(buf), "/cluster/fence_xvmd/@%s\n",
+			     arg->stdin_opt);
+		if (n == sizeof(buf)) {
+			args->flags |= F_CCSERR;
+			return;		
+		}
+
+		val = NULL;
+		if (ccs_get(ccsfd, buf, &val) != 0) {
+			if (val) {
+				free(val);
+				val = NULL;
+			}
+			continue;
+		}
+
+		if (!val)
+			continue;
+
+		if (arg->assign)
+			arg->assign(args, arg, val);
+
+		if (val) {
+			free(val);
+			val = NULL;
+		}
+	}
+	
+	ccs_disconnect(ccsfd);
+}

server/virt.c

@@ -0,0 +1,217 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#include <stdio.h>
+#include <sys/types.h>
+#include <stdlib.h>
+#include <libvirt/libvirt.h>
+#include <string.h>
+#include <malloc.h>
+#include <stdint.h>
+#include <errno.h>
+#include "virt.h"
+
+
+static int
+_compare_virt(const void *_left, const void *_right)
+{
+	virt_state_t *left = (virt_state_t *)_left,
+		     *right = (virt_state_t *)_right;
+
+	return strcasecmp(left->v_name, right->v_name);
+}
+
+
+virt_list_t *vl_get(virConnectPtr vp, int my_id)
+{
+	virt_list_t *vl = NULL;
+	int *d_ids = NULL;
+	int d_count, x;
+	char *d_name;
+	char d_uuid[MAX_DOMAINNAME_LENGTH];
+	virDomainPtr dom;
+	virDomainInfo d_info;
+
+	errno = EINVAL;
+	if (!vp)
+		return NULL;
+
+	d_count = virConnectNumOfDomains(vp);
+	if (d_count <= 0) {
+		if (d_count == 0) {
+			/* Successful, but no domains running */
+			errno = 0;
+			return NULL;
+		}
+		goto out_fail;
+	}
+
+	vl = malloc(sizeof(uint32_t) + sizeof(virt_state_t) * d_count );
+	if (!vl)
+		goto out_fail;
+
+	d_ids = malloc(sizeof(int) * d_count);
+	if (!d_ids)
+		goto out_fail;
+
+	if (virConnectListDomains(vp, d_ids, d_count) < 0)
+		goto out_fail;
+
+	vl->vm_count = d_count;
+
+	/* Ok, we have the domain IDs - let's get their names and states */
+	for (x = 0; x < d_count; x++) {
+		dom = virDomainLookupByID(vp, d_ids[x]);
+		if (!dom) {
+			/* XXX doom */
+			goto out_fail;
+		}
+
+		if (!(d_name = (char *)virDomainGetName(dom))) {
+			/* XXX no name for the domain?!! */
+			virDomainFree(dom);
+			goto out_fail;
+		}
+
+		if (virDomainGetUUIDString(dom, d_uuid) != 0) {
+			virDomainFree(dom);
+			goto out_fail;
+		}
+
+		if (virDomainGetInfo(dom, &d_info) < 0) {
+			/* XXX no info for the domain?!! */
+			virDomainFree(dom);
+			goto out_fail;
+		}
+
+		/* Store the name & state */
+		strncpy(vl->vm_states[x].v_name, d_name, MAX_DOMAINNAME_LENGTH);
+		strncpy(vl->vm_states[x].v_uuid, d_uuid, MAX_DOMAINNAME_LENGTH);
+		vl->vm_states[x].v_state.s_state = d_info.state;
+		vl->vm_states[x].v_state.s_owner = my_id;
+
+		virDomainFree(dom);
+	}
+
+	/* We have all the locally running domains & states now */
+	/* Sort */
+	free(d_ids);
+	qsort(&vl->vm_states[0], vl->vm_count, sizeof(vl->vm_states[0]),
+	      _compare_virt);
+	return vl;	
+
+out_fail:
+	x = errno;
+	if (d_ids)
+		free(d_ids);
+	if (vl)
+		free(vl);
+	errno = x;
+	return NULL;
+}
+
+
+/* Returns 0 if equal, nonzero if not */
+int
+vl_cmp(virt_list_t *left, virt_list_t *right)
+{
+	int x;
+
+	/* Quick checks */
+	if (!left->vm_count && !right->vm_count)
+		return 1;
+	if (left->vm_count != right->vm_count)
+		return 0;
+
+	for (x = 0; x < left->vm_count; x++) {
+		if (strcmp(left->vm_states[x].v_name,
+			   right->vm_states[x].v_name))
+			return 1;
+		/*
+		if (left->vm_states[x].v_state.s_state !=
+		    right->vm_states[x].v_state.s_state)
+			return 1;
+		 */
+	}
+
+	return 0;
+}
+
+
+void
+vl_print(virt_list_t *vl)
+{
+	int x;
+
+	printf("%-24.24s %-36.36s %-5.5s %-5.5s\n", "Domain", "UUID",
+	       "Owner", "State");
+	printf("%-24.24s %-36.36s %-5.5s %-5.5s\n", "------", "----",
+	       "-----", "-----");
+
+	if (!vl || !vl->vm_count)
+		return;
+
+	for (x = 0; x < vl->vm_count; x++) {
+		printf("%-24.24s %-36.36s %-5.5d %-5.5d\n",
+		       vl->vm_states[x].v_name,
+		       vl->vm_states[x].v_uuid,
+		       vl->vm_states[x].v_state.s_owner,
+		       vl->vm_states[x].v_state.s_state);
+	}
+}
+
+
+virt_state_t *
+vl_find_name(virt_list_t *vl, char *name)
+{
+	int x;
+
+	if (!vl || !name || !vl->vm_count)
+		return NULL;
+
+	for (x = 0; x < vl->vm_count; x++) {
+		if (!strcasecmp(vl->vm_states[x].v_name, name))
+			return &vl->vm_states[x];
+	}
+
+	return NULL;
+}
+
+
+virt_state_t *
+vl_find_uuid(virt_list_t *vl, char *uuid)
+{
+	int x;
+
+	if (!vl || !uuid || !vl->vm_count)
+		return NULL;
+
+	for (x = 0; x < vl->vm_count; x++) {
+		if (!strcasecmp(vl->vm_states[x].v_uuid, uuid))
+			return &vl->vm_states[x];
+	}
+
+	return NULL;
+}
+
+
+void
+vl_free(virt_list_t *old)
+{
+	free(old);
+}

server/virt.h

@@ -0,0 +1,80 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge, 
+  MA 02139, USA.
+*/
+#ifndef _VIRT_H
+#define _VIRT_H
+#include <libvirt/libvirt.h>
+#include <stdint.h>
+#include <netinet/in.h>
+
+#include "xvm.h"
+
+/*
+   Owner 0 = no owner.
+
+   checkpoint "xen-vm-states" {
+     section "vm-name0" {
+       owner_nodeid;
+       vm_state;
+     }
+     section "vm-name1" {
+       owner_nodeid;
+       vm_state;
+     }
+     ...
+   }
+ */
+
+typedef struct {
+	uint32_t s_owner;
+	int32_t s_state;
+} vm_state_t;
+
+typedef struct {
+	char v_name[MAX_DOMAINNAME_LENGTH];
+	char v_uuid[MAX_DOMAINNAME_LENGTH];
+	vm_state_t v_state;
+} virt_state_t;
+
+/**
+  This is stored in our private checkpoint section.
+ */
+typedef struct _virt_list {
+	uint32_t	vm_count;
+	virt_state_t	vm_states[0];
+} virt_list_t;
+
+virt_list_t *vl_get(virConnectPtr vp, int my_id);
+
+int vl_cmp(virt_list_t *left, virt_list_t *right);
+
+void vl_print(virt_list_t *vl);
+void vl_free(virt_list_t *old);
+virt_state_t * vl_find_uuid(virt_list_t *vl, char *name);
+virt_state_t * vl_find_name(virt_list_t *vl, char *name);
+
+
+typedef void ckpt_handle;
+int ckpt_read(void *hp, char *secid, void *buf, size_t maxlen);
+int ckpt_finish(void *hp);
+int ckpt_write(void *hp, char *secid, void *buf, size_t maxlen);
+void *ckpt_init(char *ckpt_name, int maxlen, int maxsec, int maxseclen,
+		int timeout);
+
+
+#endif

server/vm_states.c

@@ -0,0 +1,432 @@
+/*
+  Copyright Red Hat, Inc. 2006
+
+  This program is free software; you can redistribute it and/or modify it
+  under the terms of the GNU General Public License as published by the
+  Free Software Foundation; either version 2, or (at your option) any
+  later version.
+
+  This program is distributed in the hope that it will be useful, but
+  WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  General Public License for more details.
+
+  You should have received a copy of the GNU General Public License
+  along with this program; see the file COPYING.  If not, write to the
+  Free Software Foundation, Inc.,  675 Mass Ave, Cambridge,
+  MA 02139, USA.
+*/
+/** @file
+ * Distributed VM states using saCkpt interface
+ */
+#include <string.h>
+#include <errno.h>
+#include <sys/types.h>
+#include <sys/resource.h>
+#include <sys/wait.h>
+#include <stdlib.h>
+#include <sys/time.h>
+#include <pthread.h>
+#include <saAis.h>
+#include <saCkpt.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <assert.h>
+
+typedef struct {
+	uint32_t ck_ready;
+	int ck_timeout;
+	SaCkptCheckpointHandleT ck_checkpoint;
+	SaCkptHandleT ck_handle;
+	char *ck_name;
+} ckpt_handle;
+
+
+#define READY_MAGIC 0x13fd237c
+#define VALIDATE(h) \
+do { \
+	if (!h || h->ck_ready != READY_MAGIC) { \
+		errno = EINVAL; \
+		return -1; \
+	} \
+} while(0)
+
+int ais_to_posix(SaAisErrorT err);
+
+int
+ais_to_posix(SaAisErrorT err)
+{
+	switch (err) {
+	case SA_AIS_OK:
+		return 0;
+	case SA_AIS_ERR_LIBRARY:
+		return ELIBBAD;
+	case SA_AIS_ERR_VERSION:
+		return EPROTONOSUPPORT; //XXX
+	case SA_AIS_ERR_INIT:
+		return EFAULT; //XXX
+	case SA_AIS_ERR_TIMEOUT:
+		return ETIMEDOUT;
+	case SA_AIS_ERR_TRY_AGAIN:
+		return EAGAIN;
+	case SA_AIS_ERR_INVALID_PARAM:
+		return EINVAL;
+	case SA_AIS_ERR_NO_MEMORY:
+		return ENOMEM;
+	case SA_AIS_ERR_BAD_HANDLE:
+		return EBADF;
+	case SA_AIS_ERR_BUSY:
+		return EBUSY;
+	case SA_AIS_ERR_ACCESS:
+		return EACCES;
+	case SA_AIS_ERR_NOT_EXIST:
+		return ENOENT;
+	case SA_AIS_ERR_NAME_TOO_LONG:
+		return ENAMETOOLONG;
+	case SA_AIS_ERR_EXIST:
+		return EEXIST;
+	case SA_AIS_ERR_NO_SPACE:
+		return ENOSPC;
+	case SA_AIS_ERR_INTERRUPT:
+		return EINTR;
+	case SA_AIS_ERR_NAME_NOT_FOUND:
+		return ENOENT;
+	case SA_AIS_ERR_NO_RESOURCES:
+		return ENOMEM; //XXX
+	case SA_AIS_ERR_NOT_SUPPORTED:
+		return ENOSYS;
+	case SA_AIS_ERR_BAD_OPERATION:
+		return EINVAL; //XXX
+	case SA_AIS_ERR_FAILED_OPERATION:
+		return EIO; //XXX
+	case SA_AIS_ERR_MESSAGE_ERROR:
+		return EIO; // XXX
+	case SA_AIS_ERR_QUEUE_FULL:
+		return ENOBUFS;
+	case SA_AIS_ERR_QUEUE_NOT_AVAILABLE:
+		return ENOENT;
+	case SA_AIS_ERR_BAD_FLAGS:
+		return EINVAL;
+	case SA_AIS_ERR_TOO_BIG:
+		return E2BIG;
+	case SA_AIS_ERR_NO_SECTIONS:
+		return ENOENT; // XXX
+	/*case SA_AIS_ERR_SECURITY:
+		return EPERM;*/
+	default:
+		return EINVAL; /* XXX */
+	}
+
+	return -1;
+}
+
+
+static int 
+ckpt_open(ckpt_handle *h, const char *ckpt_name, int maxsize,
+	  int maxsec, int maxsecsize, int timeout)
+{
+	SaCkptCheckpointCreationAttributesT attrs;
+	SaCkptCheckpointOpenFlagsT flags;
+	SaNameT cpname;
+#if 0
+	SaCkptCheckpointDescriptorT status;
+#endif
+	SaAisErrorT err = SA_AIS_OK;
+
+	VALIDATE(h);
+
+	flags = SA_CKPT_CHECKPOINT_READ |
+		SA_CKPT_CHECKPOINT_WRITE;
+
+	snprintf((char *)cpname.value, SA_MAX_NAME_LENGTH-1,
+		 "%s", ckpt_name);
+	cpname.length = strlen(ckpt_name);
+
+	h->ck_timeout = timeout;
+
+	err = saCkptCheckpointOpen(h->ck_handle,
+				   &cpname,
+				   NULL,	
+				   flags,
+				   timeout,
+				   &h->ck_checkpoint);
+
+	if (err == SA_AIS_OK) {
+#if 0
+		saCkptCheckpointStatusGet(h->ck_handle,
+					  &status);
+
+		printf("Checkpoint Size = %d bytes\n", (int)
+			status.checkpointCreationAttributes.checkpointSize);
+		printf("Flags = ");
+		if (status.checkpointCreationAttributes.creationFlags &
+			SA_CKPT_WR_ALL_REPLICAS) {
+			printf("%s ", "SA_CKPT_WR_ALL_REPLICAS");
+		}
+		if (status.checkpointCreationAttributes.creationFlags &
+			SA_CKPT_WR_ACTIVE_REPLICA) {
+			printf("%s ", "SA_CKPT_WR_ACTIVE_REPLICA");
+		}
+		if (status.checkpointCreationAttributes.creationFlags &
+			SA_CKPT_WR_ACTIVE_REPLICA_WEAK) {
+			printf("%s ", "SA_CKPT_WR_ACTIVE_REPLICA_WEAK");
+		}
+		if (status.checkpointCreationAttributes.creationFlags &
+			SA_CKPT_CHECKPOINT_COLLOCATED) {
+			printf("%s ", "SA_CKPT_CHECKPOINT_COLLOCATED");
+		}
+		printf("\nMax sections = %d\n",
+			(int)status.checkpointCreationAttributes.maxSections);
+		printf("Max section size = %d\n",
+			(int)status.checkpointCreationAttributes.maxSectionSize);
+		printf("Max section ID size = %d\n",
+			(int)status.checkpointCreationAttributes.maxSectionIdSize);
+		printf("Section count = %d\n", status.numberOfSections);
+		printf("\n");
+#endif
+		goto good;
+	}
+
+	attrs.creationFlags = SA_CKPT_WR_ALL_REPLICAS;
+	attrs.checkpointSize = (SaSizeT)maxsize;
+	attrs.retentionDuration = SA_TIME_ONE_HOUR;
+	attrs.maxSections = maxsec;
+	attrs.maxSectionSize = (SaSizeT)maxsecsize;
+	attrs.maxSectionIdSize = (SaSizeT)32;
+
+	flags = SA_CKPT_CHECKPOINT_READ |
+		SA_CKPT_CHECKPOINT_WRITE |
+		SA_CKPT_CHECKPOINT_CREATE;
+
+	err = saCkptCheckpointOpen(h->ck_handle,
+				   &cpname,
+				   &attrs,
+				   flags,
+				   timeout,
+				   &h->ck_checkpoint);
+	if (err == SA_AIS_OK)
+		goto good;
+
+	/* No checkpoint */
+	errno = ais_to_posix(err);
+	return (errno == 0 ? 0 : -1);
+good:
+	printf("Opened ckpt %s\n", ckpt_name);
+	h->ck_name = strdup(ckpt_name);
+
+	errno = ais_to_posix(err);
+	return (errno == 0 ? 0 : -1);
+}
+
+
+void *
+ckpt_init(char *ckpt_name, int maxlen, int maxsec,
+	  int maxseclen, int timeout)
+{
+	ckpt_handle *h;
+	SaAisErrorT err;
+	SaVersionT ver;
+
+	if (!ckpt_name || !strlen(ckpt_name)) {
+		errno = EINVAL;
+		return NULL;
+	}
+	h = malloc(sizeof(*h));
+	if (!h)
+		return NULL;
+	memset(h, 0, sizeof(*h));
+
+	ver.releaseCode = 'B';
+	ver.majorVersion = 1;
+	ver.minorVersion = 1;
+
+	err = saCkptInitialize(&h->ck_handle, NULL, &ver);
+
+	if (err != SA_AIS_OK) {
+		free(h);
+		return NULL;
+	} else
+		h->ck_ready = READY_MAGIC;
+
+	if (ckpt_open(h, ckpt_name, maxlen, maxsec, maxseclen,
+		      timeout) < 0) {
+		saCkptCheckpointClose(h->ck_checkpoint);
+		if (h->ck_name) 
+			free(h->ck_name);
+		free(h);
+		return NULL;
+	}
+
+	return (void *)h;
+}
+
+
+int
+ckpt_write(void *hp, char *secid, void *buf, size_t maxlen)
+{
+	ckpt_handle *h = (ckpt_handle *)hp;
+	SaCkptIOVectorElementT iov = {SA_CKPT_DEFAULT_SECTION_ID,
+				      NULL, 0, 0, 0};
+	SaAisErrorT err;
+	SaCkptSectionCreationAttributesT attrs;
+
+	VALIDATE(h);
+
+	/* Set section ID here */
+	iov.sectionId.id = (uint8_t *)secid;
+	iov.sectionId.idLen = strlen(secid);
+	iov.dataBuffer = buf;
+	iov.dataSize = (SaSizeT)maxlen;
+	iov.dataOffset = 0;
+	iov.readSize = 0;
+
+	err = saCkptCheckpointWrite(h->ck_checkpoint, &iov, 1, NULL);
+
+	if (err == SA_AIS_ERR_NOT_EXIST) {
+		attrs.sectionId = &iov.sectionId;
+		attrs.expirationTime = SA_TIME_END;
+
+		err = saCkptSectionCreate(h->ck_checkpoint, &attrs, 
+					  buf, maxlen);
+	}
+
+	if (err == SA_AIS_OK)
+		saCkptCheckpointSynchronize(h->ck_checkpoint,
+					    h->ck_timeout);
+
+	errno = ais_to_posix(err);
+	if (errno)
+		return -1;
+	return maxlen; /* XXX */
+}
+
+
+int
+ckpt_read(void *hp, char *secid, void *buf, size_t maxlen)
+{
+	ckpt_handle *h = (ckpt_handle *)hp;
+	SaCkptIOVectorElementT iov = {SA_CKPT_DEFAULT_SECTION_ID,
+				      NULL, 0, 0, 0};
+	SaAisErrorT err;
+
+	VALIDATE(h);
+	//printf("reading ckpt %s\n", keyid);
+
+	iov.sectionId.id = (uint8_t *)secid;
+	iov.sectionId.idLen = strlen(secid);
+	iov.dataBuffer = buf;
+	iov.dataSize = (SaSizeT)maxlen;
+	iov.dataOffset = 0;
+	iov.readSize = 0;
+
+	err = saCkptCheckpointRead(h->ck_checkpoint, &iov, 1, NULL);
+
+	errno = ais_to_posix(err);
+	if (errno)
+		return -1;
+	return iov.readSize; /* XXX */
+}
+
+
+int
+ckpt_finish(void *hp)
+{
+	ckpt_handle *h = (ckpt_handle *)hp;
+	int ret = 0;
+	SaAisErrorT err;
+
+	saCkptCheckpointClose(h->ck_checkpoint);
+	err = saCkptFinalize(h->ck_handle);
+
+	if (err != SA_AIS_OK)
+		ret = -1;
+	else
+		h->ck_ready = 0;
+
+	if (h->ck_name)
+		free(h->ck_name);
+
+	if (ret != 0)
+		errno = ais_to_posix(err);
+	return ret;
+}
+
+
+#ifdef STANDALONE
+void
+usage(int ret)
+{
+	printf("usage: ckpt [-c ckpt_name] <-r key|-w key -d data>\n");
+	exit(ret);
+}
+
+int
+main(int argc, char **argv)
+{
+	char *ckptname = "ckpt_test";
+	char *sec = "default";
+	char *val;
+	void *h;
+	char buf[64];
+	int ret;
+	int op = 0;
+
+	while((ret = getopt(argc, argv, "c:w:r:d:j?")) != EOF) {
+		switch(ret) {
+		case 'c': 
+			ckptname = optarg;
+			break;
+		case 'w': 
+			op = 'w';
+			sec = optarg;
+			break;
+		case 'r':
+			op = 'r';
+			sec = optarg;
+			break;
+		case 'd':
+			val = optarg;
+			break;
+		case '?':
+		case 'h':
+			usage(0);
+		default:
+			usage(1);
+		}
+	}
+
+	if (!op) {
+		usage(1);
+	}
+
+	if (!sec) {
+		usage(1);
+	}
+
+	h = ckpt_init(ckptname, 262144, 4096, 64, 10);
+	if (!h) {
+		perror("ckpt_init");
+		return -1;
+	}
+
+	if (op == 'w') {
+		if (ckpt_write(h, sec, val, strlen(val)+1) < 0) {
+			perror("ckpt_write");
+			return 1;
+		}
+	} else if (op == 'r') {
+		ret = ckpt_read(h, sec, buf, sizeof(buf));
+		if (ret < 0) {
+			perror("ckpt_read");
+			return 1;
+		}
+
+		printf("%d bytes\nDATA for '%s':\n%s\n", ret, sec,
+		       buf);
+	}
+
+	ckpt_finish(h);
+
+	return 0;
+}
+#endif