samba-mirror/talloc_guide.txt

Using talloc in Samba4
----------------------

Andrew Tridgell
September 2004

The most current version of this document is available at
   http://samba.org/ftp/unpacked/samba4/talloc_guide.txt

If you are used to talloc from Samba3 then please read this carefully,
as talloc has changed a lot.

The new talloc is a hierarchical, reference counted memory pool system
with destructors. Quite a mounthful really, but not too bad once you
get used to it.

Perhaps the biggest change from Samba3 is that there is no distinction
between a "talloc context" and a "talloc pointer". Any pointer
returned from talloc() is itself a valid talloc context. This means
you can do this:

  struct foo *X = talloc_p(mem_ctx, struct foo);
  X->name = talloc_strdup(X, "foo");

and the pointer X->name would be a "child" of the talloc context "X"
which is itself a child of mem_ctx. So if you do talloc_free(mem_ctx)
then it is all destroyed, whereas if you do talloc_free(X) then just X
and X->name are destroyed, and if you do talloc_free(X->name) then
just the name element of X is destroyed.

If you think about this, then what this effectively gives you is an
n-ary tree, where you can free any part of the tree with
talloc_free().

If you find this confusing, then I suggest you run the LOCAL-TALLOC
smbtorture test to watch talloc in action. You may also like to add
your own tests to source/torture/local/talloc.c to clarify how some
particular situation is handled.


Performance
-----------

All the additional features of talloc() over malloc() do come at a
price. We have a simple performance test in Samba4 that measures
talloc() versus malloc() performance, and it seems that talloc() is
about 10% slower than malloc() on my x86 Debian Linux box. For Samba,
the great reduction in code complexity that we get by using talloc
makes this worthwhile, especially as the total overhead of
talloc/malloc in Samba is already quite small.


talloc API
----------

The following is a complete guide to the talloc API. Read it all at
least twice.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc(const void *context, size_t size);

The talloc() function is the core of the talloc library. It takes a
memory context, and returns a pointer to a new area of memory of the
given size.

The returned pointer is itself a talloc context, so you can use it as
the context argument to more calls to talloc if you wish.

The returned pointer is a "child" of the supplied context. This means
that if you talloc_free() the context then the new child disappears as
well. Alternatively you can free just the child.

The context argument to talloc() can be NULL, in which case a new top
level context is created. 


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_p(const void *context, type);

The talloc_p() macro is the equivalent of 

  (type *)talloc(ctx, sizeof(type))

You should use it in preference to talloc() whenever possible, as it
provides additional type safety. It also automatically calls the
talloc_set_name_const() function with the name being a string holding
the name of the type.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
int talloc_free(void *ptr);

The talloc_free() function frees a piece of talloc memory, and all its
children. You can call talloc_free() on any pointer returned by
talloc().

The return value of talloc_free() indicates success or failure, with 0
returned for success and -1 for failure. The only possible failure
condition is if the pointer had a destructor attached to it and the
destructor returned -1. See talloc_set_destructor() for details on
destructors.

If this pointer has an additional reference when talloc_free() is
called then the memory is not actually released, but instead the most
recently established reference is destroyed. See talloc_reference()
for details on establishing additional references.

For more control on which parent is removed, see talloc_unlink()

talloc_free() operates recursively on its children.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_reference(const void *context, const void *ptr);

The talloc_reference() function returns an additional reference to
"ptr", and makes this additional reference a child of "context".

The return value of talloc_reference() is always the original pointer
"ptr", unless talloc ran out of memory in creating the reference in
which case it will return NULL (each additional reference consumes
around 48 bytes of memory on intel x86 platforms).

After creating a reference you can free it in one of the following
ways:

  - you can talloc_free() a parent of the original pointer. That will
    destroy the reference and make the pointer a child of the
    "context" argument from the most recently called
    talloc_reference() on the pointer.

  - you can talloc_free() the pointer itself. That will destroy the
    most recently established reference to the pointer and leave the
    pointer as a child of its current parent.

  - you can talloc_free() the context where you placed the
    reference. That will destroy the reference, and leave the pointer
    where it is.

For more control on which parent to remove, see talloc_unlink()


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
int talloc_unlink(const void *context, const void *ptr);

The talloc_unlink() function removes a specific parent from ptr. The
context passed must either be a context used in talloc_reference()
with this pointer, or must be a direct parent of ptr. 

Note that if the parent has already been removed using talloc_free()
then this function will fail and will return -1.

Usually you can just use talloc_free() instead of talloc_unlink(), but
sometimes it is useful to have the additional control on which parent
is removed.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_set_destructor(const void *ptr, int (*destructor)(void *));

The function talloc_set_destructor() sets the "destructor" for the
pointer "ptr". A destructor is a function that is called when the
memory used by a pointer is about to be released. The destructor
receives the pointer as an argument, and should return 0 for success
and -1 for failure.

The destructor can do anything it wants to, including freeing other
pieces of memory. A common use for destructors is to clean up
operating system resources (such as open file descriptors) contained
in the structure the destructor is placed on.

You can only place one destructor on a pointer. If you need more than
one destructor then you can create a zero-length child of the pointer
and place an additional destructor on that.

To remove a destructor call talloc_set_destructor() with NULL for the
destructor.

If your destructor attempts to talloc_free() the pointer that it is
the destructor for then talloc_free() will return -1 and the free will
be ignored. This would be a pointless operation anyway, as the
destructor is only called when the memory is just about to go away.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_increase_ref_count(const void *ptr);

The talloc_increase_ref_count(ptr) function is exactly equivalent to:

  talloc_reference(NULL, ptr);

You can use either syntax, depending on which you think is clearer in
your code.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_set_name(const void *ptr, const char *fmt, ...);

Each talloc pointer has a "name". The name is used principally for
debugging purposes, although it is also possible to set and get the
name on a pointer in as a way of "marking" pointers in your code.

The main use for names on pointer is for "talloc reports". See
talloc_report() and talloc_report_full() for details. Also see
talloc_enable_leak_report() and talloc_enable_leak_report_full().


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_set_name_const(const void *ptr, const char *name);

The function talloc_set_name_const() is just like talloc_set_name(),
but it takes a string constant, and is much faster. It is extensively
used by the "auto naming" macros, such as talloc_p().


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_named(const void *context, size_t size, const char *fmt, ...);

The talloc_named() function creates a named talloc pointer. It is
equivalent to:

   ptr = talloc(context, size);
   talloc_set_name(ptr, fmt, ....);


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_named_const(const void *context, size_t size, const char *name);

This is equivalent to:

   ptr = talloc(context, size);
   talloc_set_name_const(ptr, name);


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
const char *talloc_get_name(const void *ptr);

This returns the current name for the given talloc pointer. See
talloc_set_name() for details.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_init(const char *fmt, ...);

This function creates a zero length named talloc context as a top
level context. It is equivalent to:

  talloc_named(NULL, 0, fmt, ...);


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_realloc(const void *context, void *ptr, size_t size);

The talloc_realloc() function changes the size of a talloc
pointer. It has the following equivalences:

  talloc_realloc(context, NULL, size) ==> talloc(context, size);
  talloc_realloc(context, ptr, 0)     ==> talloc_free(ptr);

The "context" argument is only used if "ptr" is not NULL, otherwise it
is ignored.

talloc_realloc() returns the new pointer, or NULL on failure. The call
will fail either due to a lack of memory, or because the pointer has
an reference (see talloc_reference()).


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_steal(const void *new_ctx, const void *ptr);

The talloc_steal() function changes the parent context of a talloc
pointer. It is typically used when the context that the pointer is
currently a child of is going to be freed and you wish to keep the
memory for a longer time. 

The talloc_steal() function returns the pointer that you pass it. It
does not have any failure modes.

NOTE: It is possible to produce loops in the parent/child relationship
if you are not careful with talloc_steal(). No guarantees are provided
as to your sanity or the safety of your data if you do this.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
off_t talloc_total_size(const void *ptr);

The talloc_total_size() function returns the total size in bytes used
by this pointer and all child pointers. Mostly useful for debugging.

Passing NULL is allowed, but it will only give a meaningful result if
talloc_enable_leak_report() or talloc_enable_leak_report_full() has
been called.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
off_t talloc_total_blocks(const void *ptr);

The talloc_total_blocks() function returns the total memory block
count used by this pointer and all child pointers. Mostly useful for
debugging.

Passing NULL is allowed, but it will only give a meaningful result if
talloc_enable_leak_report() or talloc_enable_leak_report_full() has
been called.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_report(const void *ptr, FILE *f);

The talloc_report() function prints a summary report of all memory
used by ptr. One line of report is printed for each immediate child of
ptr, showing the total memory and number of blocks used by that child.

You can pass NULL for the pointer, in which case a report is printed
for the top level memory context, but only if
talloc_enable_leak_report() or talloc_enable_leak_report_full() has
been called.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_report_full(const void *ptr, FILE *f);

This provides a more detailed report than talloc_report(). It will
recursively print the ensire tree of memory referenced by the
pointer. References in the tree are shown by giving the name of the
pointer that is referenced.

You can pass NULL for the pointer, in which case a report is printed
for the top level memory context, but only if
talloc_enable_leak_report() or talloc_enable_leak_report_full() has
been called.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_enable_leak_report(void);

This enables calling of talloc_report(NULL, stderr) when the program
exits. In Samba4 this is enabled by using the --leak-report command
line option.

For it to be useful, this function must be called before any other
talloc function as it establishes a "null context" that acts as the
top of the tree. If you don't call this function first then passing
NULL to talloc_report() or talloc_report_full() won't give you the
full tree printout.

Here is a typical talloc report:

talloc report on 'null_context' (total 267 bytes in 15 blocks)
        libcli/auth/spnego_parse.c:55  contains     31 bytes in   2 blocks
        libcli/auth/spnego_parse.c:55  contains     31 bytes in   2 blocks
        iconv(UTF8,CP850)              contains     42 bytes in   2 blocks
        libcli/auth/spnego_parse.c:55  contains     31 bytes in   2 blocks
        iconv(CP850,UTF8)              contains     42 bytes in   2 blocks
        iconv(UTF8,UTF-16LE)           contains     45 bytes in   2 blocks
        iconv(UTF-16LE,UTF8)           contains     45 bytes in   2 blocks


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void talloc_enable_leak_report_full(void);

This enables calling of talloc_report_full(NULL, stderr) when the
program exits. In Samba4 this is enabled by using the
--leak-report-full command line option.

For it to be useful, this function must be called before any other
talloc function as it establishes a "null context" that acts as the
top of the tree. If you don't call this function first then passing
NULL to talloc_report() or talloc_report_full() won't give you the
full tree printout.

Here is a typical full report:

full talloc report on 'root' (total 18 bytes in 8 blocks)
    p1                             contains     18 bytes in   7 blocks (ref 0)
        r1                             contains     13 bytes in   2 blocks (ref 0)
            reference to: p2
        p2                             contains      1 bytes in   1 blocks (ref 1)
        x3                             contains      1 bytes in   1 blocks (ref 0)
        x2                             contains      1 bytes in   1 blocks (ref 0)
        x1                             contains      1 bytes in   1 blocks (ref 0)


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_zero(const void *ctx, size_t size);

The talloc_zero() function is equivalent to:

  ptr = talloc(ctx, size);
  if (ptr) memset(ptr, 0, size);


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_memdup(const void *ctx, const void *p, size_t size);

The talloc_memdup() function is equivalent to:

  ptr = talloc(ctx, size);
  if (ptr) memcpy(ptr, p, size);


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
char *talloc_strdup(const void *ctx, const char *p);

The talloc_strdup() function is equivalent to:

  ptr = talloc(ctx, strlen(p)+1);
  if (ptr) memcpy(ptr, p, strlen(p)+1);


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
char *talloc_strndup(const void *t, const char *p, size_t n);

The talloc_strndup() function is the talloc equivalent of the C
library function strndup()


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
char *talloc_vasprintf(const void *t, const char *fmt, va_list ap);

The talloc_vasprintf() function is the talloc equivalent of the C
library function vasprintf()


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
char *talloc_asprintf(const void *t, const char *fmt, ...);

The talloc_asprintf() function is the talloc equivalent of the C
library function asprintf()


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
char *talloc_asprintf_append(char *s, const char *fmt, ...);

The talloc_asprintf_append() function appends the given formatted 
string to the given string. 


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_array_p(const void *ctx, type, uint_t count);

The talloc_array_p() macro is equivalent to:

  (type *)talloc(ctx, sizeof(type) * count);

except that it provides integer overflow protection for the multiply,
returning NULL if the multiply overflows.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_realloc_p(const void *ctx, void *ptr, type, uint_t count);

The talloc_realloc_p() macro is equivalent to:

  (type *)talloc_realloc(ctx, ptr, sizeof(type) * count);

except that it provides integer overflow protection for the multiply,
returning NULL if the multiply overflows.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
void *talloc_realloc_fn(const void *ctx, void *ptr, size_t size);

This is a non-macro version of talloc_realloc(), which is useful 
as libraries sometimes want a ralloc function pointer. A realloc()
implementation encapsulates the functionality of malloc(), free() and
realloc() in one call, which is why it is useful to be able to pass
around a single function pointer.