ctdb-scripts: Fix impending SM_NOTIFY versus record deletion race

SM_NOTIFYs are sent before client records are deleted.  Theoretically,
this means new records resulting from lock reclaim can be deleted.

This doesn't actually happen at the moment because any new "records"
resulting from lock reclaim are entered into the queue directory and
only dequeued to the database during a later monitor event.  Since a
monitor event can't collide with an ipreallocated event, no records
can be dequeeued into the database during the ipreallocated event, so
they can't be deleted by delete_records().

However, a subsequent commit will add direct writing of records into a
shared cluster filesystem directory.  This means that add-client
events will cause records to be added directly to that directory so,
without a fix, the race will be able to occur.

So, delete records before sending SM_NOTIFYs.  In theory, the script
could be killed before all SM_NOTIFYs are successfully sent, resulting
in loss of locks.  However, given the overall lack of error checking,
there are other, more likely problems.

Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb/tools/statd_callout_helper

@@ -321,8 +321,8 @@ notify)
 		exit 0
 	fi
 
-	send_notifies <"$statd_state"
 	delete_records <"$statd_state"
+	send_notifies <"$statd_state"
 
 	rm -f "$statd_state"