shaba/systemd
1
0
Fork 0
mirror of https://github.com/systemd/systemd.git synced 2025-03-28 02:50:16 +03:00
Code

docs: move container interface docs from wiki to markdown

Browse Source 
Let's move
https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
into our tree and convert it to Markdown.

This also incldues some minor fixes and other changes.
This commit is contained in:
Lennart Poettering 2020-01-06 18:12:19 +01:00
parent 14b6e6b6f3
commit 635dea2783
1 changed files with 294 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

294
docs/CONTAINER_INTERFACE.md Normal file
View File

@ -0,0 +1,294 @@
---
title: The Container Interface
category: Interfaces
layout: default
---
# The Container Interface
Also consult [Writing Virtual Machine or Container
Managers](http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers).
systemd has a number of interfaces for interacting with container managers,
when systemd is used inside of an OS container. If you work on a container
manager, please consider supporting the following interfaces.
## Execution Environment
1. If the container manager wants to control the hostname for a container
running systemd it should just set it before invoking systemd, and systemd
will leave it unmodified (that is unless there's an explicit hostname
configured in `/etc/hostname` which overrides whatever is pre-initialized by
the container manager).
2. Make sure to pre-mount `/proc/`, `/sys/` and `/sys/fs/selinux/` before
invoking systemd, and mount `/proc/sys/`, `/sys/` and `/sys/fs/selinux/`
read-only in order to avoid that the container can alter the host kernel's
configuration settings. (As special exception, if your container has network
namespaces enabled, feel free to make `/proc/sys/net/` writable). systemd
and various other subsystems (such as the SELinux userspace) have been
modified to detect whether these file systems are read-only, and will behave
accordingly. (It's OK to mount `/sys/` as `tmpfs` btw, and only mount a
subset of its sub-trees from the real `sysfs` to hide `/sys/firmware/`,
`/sys/kernel/` and so on. If you do that, still make sure to mark `/sys/`
read-only, as that condition is what systemd looks for, and is what is
considered to be the API in this context.)
3. Pre-mount `/dev/` as (container private) `tmpfs` for the container and bind
mount some suitable TTY to `/dev/console`. Also, make sure to create device
nodes for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`,
`/dev/urandom`, `/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to
create `/dev/fd` or `/dev/stdout`, as systemd will do that on its own. Make
sure to set up a `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 —
or the `devices` cgroup controller — on cgroupv1 — so that no other devices
but these may be created in the container. Note that many systemd services
these days use `PrivateDevices=`, which means that systemd will set up a
private `/dev/` for them for which it needs to be able to create these
device nodes. Dropping `CAP_MKNOD` for containers is hence generally not
advisable, but see below.
4. `systemd-udevd` is not available in containers (and refuses to start), and hence device
dependencies are unavailable. The `systemd-udevd` unit files will check for `/sys/`
being read-only, as an indication whether device management can work. Hence
make sure to mount `/sys/` read-only in the container (see above). Various
clients to `systemd-udevd` also check the read-only state of `/sys/` in some cases,
including PID 1 itself and `systemd-networkd`.
5. If systemd detects it is run in a container it will spawn a single shell on
`/dev/console`, and not care about VTs or multiple gettys on VTs. (But see
`$container_ttys` below.)
6. Either pre-mount all cgroup hierarchies in full into the container, or leave
that to systemd which will do so if they are missing. Note that it is
explicitly *not* OK to just mount a sub-hierarchy into the container as that
is incompatible with `/proc/$PID/cgroup` (which lists full paths). Also the
root-level cgroup directories tend to be quite different from inner
directories, and that distinction matters. It is OK however, to mount the
"upper" parts read-only of the hierarchies, and only allow write-access to
the cgroup sub-tree the container runs in. It's also a good idea to mount
all controller hierarchies with exception of `name=systemd` fully read-only
(this only applies to cgroupv1, of course), to protect the controllers from
alteration from inside the containers. Or to turn this around: only the
cgroup sub-tree of the container itself (on cgroupv2 in the unified
hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) must be writable
to the container.
7. Create the control group root of your container by either running your
container as a service (in case you have one container manager instance per
container instance) or creating one scope unit for each container instance
via systemd's transient unit API (in case you have one container manager
that manages all instances. Either way, make sure to set `Delegate=yes` in
it. This ensures that that the unit you created will be part of all cgroup
controllers (or at least the ones systemd understands). The latter may also
be done via `systemd-machined`'s `CreateMachine()` API. Make sure to use the
cgroup path systemd put your process in for all operations of the
container. Do not add new cgroup directories to the top of the tree. This
will not only confuse systemd and the admin, but also ensure your
implementation is not "stackable".
## Environment Variables
1. To allow systemd (and other code) to identify that it is executed within a
container, please set the `$container=` environment variable for PID 1 in
the container to a short lowercase string identifying your
implementation. With this in place the `ConditionVirtualization=` setting in
unit files will work properly. Example: `container=lxc-libvirt`
2. systemd has special support for allowing container managers to initialize
the UUID for `/etc/machine-id` to some manager supplied value. This is only
enabled if `/etc/machine-id` is empty (i.e. not yet set) at boot time of the
container. The container manager should set `$container_uuid=` as
environment variable for the container's PID 1 to the container UUID it
wants to set. (This is similar to the effect of `qemu`'s `-uuid`
switch). Note that you should pass only a UUID here that is actually unique
(i.e. only one running container should have a specific UUID), and gets
changed when a container gets duplicated. Also note that systemd will try to
persistently store the UUID in `/etc/machine-id` (if writable) when this
option is used, hence you should always pass the same UUID here. Keeping the
externally used UUID for a container and the internal one in sync is
hopefully useful to minimize surprise for the administrator.
3. systemd can automatically spawn login gettys on additional ptys. A container
manager can set the `$container_ttys=` environment variable for the
container's PID 1 to tell it on which ptys to spawn gettys. The variable
should take a space separated list of pty names, without the leading `/dev/`
prefix, but with the `pts/` prefix included. Note that despite the
variable's name you may only specify ptys, and not other types of ttys. Also
you need to specify the pty itself, a symlink will not suffice. This is
implemented in
[systemd-getty-generator(8)](https://www.freedesktop.org/software/systemd/man/systemd-getty-generator.html). Note
that this variable should not include the pty that `/dev/console` maps to if
it maps to one (see below). Example: if the container receives
`container_ttys=pts/7 pts/8 pts/14` it will spawn three additionally login
gettys on ptys 7, 8 and 14.
## Advanced Integration
1. Consider syncing `/etc/localtime` from the host file system into the
container. Make it a relative symlink to the containers's zoneinfo dir, as
usual. Tools rely on being able to determine the timezone setting from the
symlink value, and by making it relative it looks nice even if people list
the containers' `/etc/` from the host.
2. Make the container journal available in the host, by automatically
symlinking the container journal directory into the host journal
directory. More precisely, link `/var/log/journal/<container-machine-id>` of
the container into the same dir of the host. Administrators can then
automatically browse all container journals (correctly interleaved) by
issuing `journalctl -m`. The container machine ID you can determine from
`/etc/machine-id` in the container.
3. If the container manager wants to cleanly shutdown the container, it might
be a good idea to send `SIGRTMIN+3` to its init process. systemd will then do a
clean shutdown. Note however, that only systemd understands `SIGRTMIN+3` like
this, this might confuse other init systems.
4. To support [Socket Activated
Containers](http://0pointer.de/blog/projects/socket-activated-containers.html)
the container manager should be capable of being run as a systemd
service. It will then receive the sockets starting with FD 3, the number of
passed FDs in `$LISTEN_FDS` and its PID as `$LISTEN_PID`. It should take
these and pass them on to the container's init process, also setting
$LISTEN_FDS and `$LISTEN_PID` (basically, it can just leave the FDs and
`$LISTEN_FDS` untouched, but it needs to set `$LISTEN_PID` to for the
container init process). That's all that's necessary to make socket
activation work. The protocol to hand sockets from systemd to services is
hence the same as from a container manager to a container systemd. For
further details see the explanations of
[sd_listen_fds(1)](http://0pointer.de/public/systemd-man/sd_listen_fds.html)
and the [blog story for service
developers](http://0pointer.de/blog/projects/socket-activation.html).
5. Container managers should stay away from the `name=systemd` cgroup hierarchy
outside of the unit they created for their container. That's private
property of systemd, and no other code should interfere with it.
## Networking
1. Inside of a container, if a `veth` link is named `host0`, `systemd-networkd`
running inside of the container will by default do DHCPv4 client, DHCPv6
client and IPv4LL on it. It is thus recommended that container managers that
add a `veth` link to a container name it `host0`, to get automatically
configured networked, with no manual interference from outside.
2. Outside of a container, if a `veth` link is prefixed "ve-" will by default do
DHCPv4 server and DHCPv6 serer on it, as well as IPv4LL. It is thus recommended
that container managers that add a `veth` link to a container name the external
side `ve-` followed by the container name.
3. It is recommended to configure stable MAC addresses to container `veth`
devices, for example hashed out of the container names. That way it is more
likely that DHCP and IPv4LL will acquire stable addresses.
## What You Shouldn't Do
1. Do not drop `CAP_MKNOD` from the container. `PrivateDevices=` is a commonly
used service setting that provides a service with its own, private, minimal
version of `/dev/`. To set this up systemd in the container needs this
capability. If you take away the capability than all services that set this
flag will cease to work, and this are increasingly many, as we encourage
people to make use of this functionality. Use `BPF_PROG_TYPE_CGROUP_DEVICE`
BPF programs — on cgroupv2 — or the `devices` controller — on cgroupv1 — to
restrict what device nodes the container can create instead of taking away
the capability wholesale. (Also see section about fully unprivileged
containers below.)
2. Do not drop `CAP_SYS_ADMIN` from the container. A number of file system
namespacing related settings, such as `PrivateDevices=`, `ProtectHome=`,
`ProtectSystem=`, `MountFlags=`, `PrivateTmp=`, `ReadWriteDirectories=`,
`ReadOnlyDirectories=`, `InaccessibleDirectories=`, `MountFlags=` need to be
able to open new mount namespaces and the mount certain file system into
it. You break all services that make use of these flags if you drop the
flag. Note that already quite a number of services make use of this as we
actively encourage users to make use of this security functionality. Also
note that logind mounts `XDG_RUNTIME_DIR` as `tmpfs` for all logged in users
and won't work either if you take away the capability. (Also see section
about fully unprivileged containers below.)
3. Do not cross-link `/dev/kmsg` with `/dev/console`. They are different things,
you cannot link them to each other.
4. Do not pretend that the real VTs would be available in the containers. The
VT subsystem consists of all devices `/dev/tty*`, `/dev/vcs*`, `/dev/vcsa*`
plus their `sysfs` counterparts. They speak specific `ioctl()`s and
understand specific escape sequences, that other ptys don't
understand. Hence, it is explicitly not OK to mount a pty to `/dev/tty1`,
`/dev/tty2`, `/dev/tty3`. This is explicitly not supported.
5. Don't pretend that passing arbitrary devices to containers could really work
well. For example, do not pass device nodes for block devices, … to the
container. Device access (with the exception of network devices) is not
virtualized on Linux. Enumeration and probing of meta information from
`/sys/` and elsewhere is not possible to do correctly in a container. Simply
adding a specific device node to a container's `/dev/` is *not* *enough* to
do the job, as `systemd-udevd` and suchlike are not available at all, and no
devices will appear available or enumerable, inside the container.
6. Don't mount only a sub-tree of the `cgroupfs` into the container. This will not
work as `/proc/$PID/cgroup` lists full paths and cannot be matched up with
the actual `cgroupfs` tree visible, then. (You may "prune" some branches
though, see above.)
7. Do not make `/sys/` writable in the container. If you do,
`systemd-udevd.service` is started to manage your devices — inside the
container —, but that will cause conflicts and errors given that the Linux
device model is not virtualized for containers on Linux and thus the
containers and the host would try to manage the same devices, fighting for
ownership. Multiple other subsystems of systemd similarly test for `/sys/`
being writable to decide whether to use `systemd-udevd` or assume that
device management is properly available on the instance. Among them
`systemd-networkd` and `systemd-logind`. The conditionalization on the
read-only state of `/sys/` enables a nice automatism: as soon as `/sys/` and
the Linux device model are changed to be virtualized properly the container
payload can make use of that, simply by marking `/sys/` writable. (Note that
as special exception, the devices in `/sys/class/net/` are virtualized
already, if network namespacing is used. Thus it is OK to mount the relevant
sub-directories of `/sys/` writable, but make sure to leave the root of
`/sys/` read-only.)
## Fully Unprivileged Container Payload
First things first, to make this clear: Linux containers are not a security
technology right now. There are more holes in the model than in a swiss cheese.
For example: If you do not use user namespacing, and share root and other users
between container and host, the `struct user` structures will be shared between
host and container, and hence `RLIMIT_NPROC` and so of the container users affect
the host and other containers, and vice versa. This is a major security hole,
and actually is a real-life problem: since Avahi sets `RLIMIT_NPROC` of its user
to 2 (to effectively disallow `fork()`ing) you cannot run more than one Avahi
instance on the entire system...
People have been asking to be able to run systemd without `CAP_SYS_ADMIN` and
`CAP_SYS_MKNOD` in the container. This is now supported to some level in
systemd, but we recommend against it (see above). If `CAP_SYS_ADMIN` and
`CAP_SYS_MKNOD` are missing from the container systemd will now gracefully turn
off `PrivateTmp=`, `PrivateNetwork=`, `ProtectHome=`, `ProtectSystem=` and
others, because those capabilities are required to implement these options. The
services using these settings (which include many of systemd's own) will hence
run in a different, less secure environment when the capabilities are missing
than with them around.
With user namespacing in place things get much better. With user namespaces the
`struct user` issue described above goes away, and containers can keep
`CAP_SYS_ADMIN` safely for the user namespace, as capabilities are virtualized
and having capabilities inside a container doesn't mean one also has them
outside.
## Final Words
If you write software that wants to detect whether it is run in a container,
please check `/proc/1/environ` and look for the `container=` environment
variable. Do not assume the environment variable is inherited down the process
tree. It generally is not. Hence check the environment block of PID 1, not your
own. Note though that that file is only accessible to root. systemd hence early
on also copies the value into `/run/systemd/container`, which is readable for
everybody. However, that's a systemd-specific interface and other init systems
are unlikely to do the same.
Note that it is our intention to make systemd systems work flawlessly and
out-of-the-box in containers. In fact we are interested to ensure that the same
OS image can be booted on a bare system, in a VM and in a container, and behave
correctly each time. If you notice that some component in systemd does not work
in a container as it should, even though the container manager implements
everything documented above, please contact us.