drs_util: Improve memory usage when joining large DB

drs_Replicate.replicate() could consume a large amount of memory when
replicating a large DB. This is not a leak - the memory gets freed when
the function returns (i.e. once the partition is fully replicated).
However, while the partition is in the process of being replicated, it
accumulates memory for each replication chunk it receives. This can have
considerable overhead with 1000s of objects/links in the partition.

This was exhausting memory when joining a VM with 1Gb RAM to a DC with
25K users (average ~15 group memberships per user).

It seems that by storing a reference to something that's on the ctr's
talloc tree, it doesn't free up the memory for each ctr message (until
the function actually returns and req is destroyed).

With 10K users (and average 15 group memberships per user), .replicate()
consumed 211Mb of memory, according to talloc.report_full(). With this
patch, it goes down to just the current ctr message (1-2Mb).

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Oct 17 08:56:42 CEST 2018 on sn-devel-144

python/samba/drs_utils.py

@@ -189,6 +189,16 @@ def drs_get_rodc_partial_attribute_set(samdb):
     return partial_attribute_set
 
 
+def drs_copy_highwater_mark(hwm, new_hwm):
+    """
+    Copies the highwater mark by value, rather than by object reference. (This
+    avoids lingering talloc references to old GetNCChanges reply messages).
+    """
+    hwm.tmp_highest_usn = new_hwm.tmp_highest_usn
+    hwm.reserved_usn = new_hwm.reserved_usn
+    hwm.highest_usn = new_hwm.highest_usn
+
+
 class drs_Replicate(object):
     '''DRS replication calls'''
 
@@ -353,7 +363,9 @@ class drs_Replicate(object):
 
             if ctr.more_data == 0:
                 break
-            req.highwatermark = ctr.new_highwatermark
+
+            # update the request's HWM so we get the next chunk
+            drs_copy_highwater_mark(req.highwatermark, ctr.new_highwatermark)
 
         return (num_objects, num_links)