samba-mirror/selftest/ns/README

The scripts in this directory are experimental and are used to create testenvs
in separate linux namespaces. This avoids the need for socket-wrapper.

What are Namespaces
===================
Namespaces allow the kernel to segregate its system resources (files, CPU,
etc), so that different processes only see the set of resources they are
allowed to use. There are several different types of namespace: network,
user, process, file, IPC, and so on.

Key points to grasp are:
* Each type of namespace gets managed separately by the kernel, i.e. process
namespaces are managed separately to network namespaces, which are separate
to user namespaces. These scripts give each testenv its own network namespace,
but otherwise they all still share the same user/process/etc namespace.
(In future, we may want to give each testenv its own process and user
namespace, to better mimic a production DC).
* Namespaces are created using the 'unshare' utility. The new selftest
namespaces are anonymous/nameless, and so the different namespaces are
identified by the PID of the processes running within the namespace
(typically samba).
* Linux supports nesting namespaces within namespaces. In this case, each
testenv DC has its own network namespace, which is a child of the overarching
selftest namespace (which itself is a child of whatever namespace you run
'make test' from - usually this would be the root namespace).

How does it work?
=================
Normally when 'make test' is run, every testenv uses a 127.0.0.x IP address
and socket-wrapper passes the packets between them.

With namespaces, we can use real IP addresses and have the packets pass through
the kernel's IP stack normally, as it forwards them between namespaces.

We use veth interfaces for this. veth is a type of virtual interface supported
by the kernel. veth interfaces come in pairs, and act as a tunnel - any packets
sent on a veth interface simply end up as received packets on the pair veth
interface.

We create a new veth interface pair for each testenv, and use them to connect
up the namespaces. One end of the veth pair is added to the main selftest
namespace, and the other end is added to a new namespace that we'll run
samba in. E.g.

selftest.pl  veth21-br ------------------------ veth21 samba (ad_dc_ntvfs)
             10.0.0.11                          10.0.0.21
 Namespace 1                                       Namespace 2

However, we need to run multiple different testenvs and have them talk to
each other. So to do this, we need a bridge interface ('selftest0') to connect
up the namespaces, which essentially just acts as a hub. So connecting together
multiple testenvs looks more like this:

selftest.pl     +-- veth21-br ------------------------ veth21 samba (ad_dc_ntvfs)
                |                                      10.0.0.21
    selftest0 --+                                        Namespace 2
    10.0.0.11   |
                +-- veth22-br ------------------------ veth22 samba (vampire_dc)
                                                       10.0.0.22
 Namespace 1                                             Namespace 3      

The veth interfaces are named vethX and vethX-br, where X is the
SOCKET_WRAPPER_DEFAULT_IFACE for the testenv. The vethX-br interface is always
added to the selftest0 bridge interface. 

How do I use it?
================
To use namespaces instead of socket-wrapper, just add 'USE_NAMESPACES=1' to the
make command, e.g.

To run the 'quick' test cases using namespaces:
USE_NAMESPACES=1 make test TESTS=quick

To setup an ad_dc testenv using namespaces:
USE_NAMESPACES=1 SELFTEST_TESTENV=ad_dc make testenv

You can connect secondary shells to the namespace your testenv is running in.
The command to do this is a little complicated, so a helper 'nsenter.sh' script
gets autogenerated when the testenv is created. E.g. to connect to the testenv
that the ad_dc is running in, use:
./st/ad_dc/nsenter.sh

This script also sets up the shell with all the same $SERVER/$USERNAME/etc
variables that you normally get in xterm.

To run the ad-dc-backup autobuild job using namespaces:
USE_NAMESPACES=1 script/autobuild.py samba-ad-dc-backup --verbose --nocleanup \
 --keeplogs --tail --testbase /tmp/samba-testbase

Using the customdc testenv, you can basically now essentially your own
light-weight samba VM. E.g.
MY_BACKUP=/home/$USER/samba-backup-prod-domain.tar.bz2
USE_NAMESPACES=1 BACKUP_FILE=$MY_BACKUP SELFTEST_TESTENV=customdc make testenv

You can then talk to that DC in any other shell by using
./st/customdc/nsenter.sh which enters the DC's network namespace (with
all the $SERVER/etc env variables defined).

How to join VMs to the testenv
----------------------------------------
I haven't tried this (beyond basic IP connectivity), but using namespaces it
should now be possible to connect a Windows VM to a Samba testenv.

1. Work out the main selftest.pl namespace PID manually, e.g.
SELFTEST_PID= ps waux | grep selftest.pl

2. Create a new veth to bridge between the selftest namespace and your PC's
default namespace:
sudo ip link add dev testenv-veth0 type veth peer name testenv-veth1

3. Move one end of the veth tunnel into the selftest namespace:
sudo ip link set testenv-veth1 netns $SELFTEST_PID

4. Configure the veth end in the default namespace to be in the same subnet
as the selftest network:
sudo ip link set dev testenv-veth0 up
sudo ip addr add 10.0.0.63/24 dev testenv-veth0

5. Enter the selftest namespace, bring that end of the pipe up, and add it to
to the main selftest0 bridge (that connects all the DCs together). We also need
to add a default route from selftest back to your PC's default namespace.
nsenter -t $SELFTEST_PID --net --user --preserve-credentials
ip link set dev testenv-veth1 up
ip link set testenv-veth1 master selftest0
ip route add default via 10.0.0.63
logout

Your Windows VM and samba testenv should now be able to talk to each
other over IP!

6. The other step is to get DNS working. You probably need to add dns_hub
(10.0.0.64) as a nameserver (at least on your Windows VM).

This should work for using RSAT tools on samba, or joining Windows to Samba
(depending on the schema version). Joining samba to Windows is a bit more
tricky, as the namespaces are tied to the *running* samba process.

What you'd probably want to do is run the join command to the windows VM
outside of testenv, create an offline backup-file of the resulting DB, and
then plug that backup-file into the customdc testenv. (And then follow the
above veth/bridge steps to join samba to the VM).

Note that the namespace disappears once you stop the testenv, so you'd
need to do the above steps with creating the veth interface every time
you restarted the testenv.

Known limitations
=================
- When running a testenv, sometimes xterm can fail to startup, due to a
  permissions problem with /dev/pts. This seems to be a particular problem
  with the 'none' testenv.
  A short-term work-around is to use a terminal that doesn't try to access
  /dev/pts, e.g. just use bash as the terminal:
  TERMINAL=bash TERMINAL_ARGS='--norc' USE_NAMESPACES=1 \
    SELFTEST_TESTENV=none make testenv
- Some test cases rely on socket-wrapper, so will fail when run using
  namespaces.
- Currently USE_NAMESPACES maps you (i.e. $USER) to root in the new namespace.
  This means any test cases that rely on being a non-root user will fail (i.e.
  anything that fails under 'sudo make test' will also fail with namespaces).
- Namespaces should work within docker, but currently the 'unshare' system
  call is disallowed on the gitlab CI runners.