ctdb-tools-ctdb: Drop disconnected nodes when filtering by capability

Commit ba69742ccd822562ca2135d2466e09bf1216644b missed the point of
filtering disconnected nodes while limiting the nodemap to those in
the NAT gateway group.  It was really to avoid trying to fetch
capabilities from disconnected nodes.  This should be explicitly done
in filter_nodemap_by_capabilities(), otherwise "ctdb natgwlist" simply
fails when there is a disconnected node.

Note that the alternate solution where filter_nodemap_by_flags() is
called before filter_nodemap_by_capabilities() would not be not
correct.  Filtering on flags first can produce a "healthier" set of
nodes where none of them have the NAT gateway capability.

Also extend stub for ctdb_ctrl_getcapabilities() to fail when trying
to get capabilities from a disconnected node and add a corresponding
test to confirm that "ctdb natgwlist" is no longer broken.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb/tests/src/ctdb_test_stubs.c

@@ -550,6 +550,15 @@ int ctdb_ctrl_getcapabilities_stub(struct ctdb_context *ctdb,
 				   struct timeval timeout, uint32_t destnode,
 				   uint32_t *capabilities)
 {
+	if (ctdb->nodes[destnode]->flags & NODE_FLAGS_DISCONNECTED) {
+		DEBUG(DEBUG_ERR,
+		      ("ctdb_control error: 'ctdb_control to disconnected node\n"));
+		/* Placeholder for line#, instead of __location__ */
+		DEBUG(DEBUG_ERR,
+		      ("__LOCATION__ ctdb_ctrl_getcapabilities_recv failed\n"));
+		return -1;
+	}
+
 	*capabilities = ctdb->nodes[destnode]->capabilities;
 	return 0;
 }

ctdb/tests/tool/stubby.natgwlist.008.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+
+. "${TEST_SCRIPTS_DIR}/unit.sh"
+
+define_test "3 nodes, all in natgw group, 1 disconnected"
+
+setup_natgw <<EOF
+192.168.20.41
+192.168.20.42
+192.168.20.43
+EOF
+
+required_result 0 <<EOF
+1 192.168.20.42
+Number of nodes:3
+pnn:0 192.168.20.41    DISCONNECTED|INACTIVE
+pnn:1 192.168.20.42    OK (THIS NODE)
+pnn:2 192.168.20.43    OK
+EOF
+
+simple_test <<EOF
+NODEMAP
+0       192.168.20.41   0x1
+1       192.168.20.42   0x0     CURRENT RECMASTER
+2       192.168.20.43   0x0
+
+VNNMAP
+654321
+0
+1
+2
+
+IFACES
+:Name:LinkStatus:References:
+:eth2:1:2:
+:eth1:1:4:
+EOF

ctdb/tools/ctdb.c

@@ -1191,9 +1191,16 @@ filter_nodemap_by_capabilities(struct ctdb_context *ctdb,
 	ret->num = 0;
 
 	for (i = 0; i < nodemap->num; i++) {
-		int res = ctdb_ctrl_getcapabilities(ctdb, TIMELIMIT(),
-						    nodemap->nodes[i].pnn,
-						    &capabilities);
+		int res;
+
+		/* Disconnected nodes have no capabilities! */
+		if (nodemap->nodes[i].flags & NODE_FLAGS_DISCONNECTED) {
+			continue;
+		}
+
+		res = ctdb_ctrl_getcapabilities(ctdb, TIMELIMIT(),
+						nodemap->nodes[i].pnn,
+						&capabilities);
 		if (res != 0) {
 			DEBUG(DEBUG_ERR, ("Unable to get capabilities from node %u\n",
 					  nodemap->nodes[i].pnn));