lvm2/doc/lvmetad_design.txt

The design of LVMetaD
=====================

Invocation and setup
--------------------

The daemon should be started automatically by the first LVM command issued on
the system, when needed. The usage of the daemon should be configurable in
lvm.conf, probably with its own section. Say

    lvmetad {
        enabled = 1 # default
        autostart = 1 # default
        socket = "/path/to/socket" # defaults to /var/run/lvmetad or such
    }

Library integration
-------------------

When a command needs to access metadata, it currently needs to perform a scan
of the physical devices available in the system. This is a possibly quite
expensive operation, especially if many devices are attached to the system. In
most cases, LVM needs a complete image of the system's PVs to operate
correctly, so all devices need to be read, to at least determine presence (and
content) of a PV label. Additional IO is done to obtain or write metadata
areas, but this is only marginally related and addressed by Dave's
metadata-balancing work.

In the existing scanning code, a cache layer exists, under
lib/cache/lvmcache.[hc]. This layer is keeping a textual copy of the metadata
for a given volume group, in a format_text form, as a character string. We can
plug the lvmetad interface at this level: in lvmcache_get_vg, which is
responsible for looking up metadata in a local cache, we can, if the metadata
is not available in the local cache, query lvmetad. Under normal circumstances,
when a VG is not cached yet, this operation fails and prompts the caller to
perform a scan. Under the lvmetad enabled scenario, this would never happen and
the fall-through would only be activated when lvmetad is disabled, which would
lead to local cache being populated as usual through a locally executed scan.

Therefore, existing stand-alone (i.e. no lvmetad) functionality of the tools
would be not compromised by adding lvmetad. With lvmetad enabled, however,
significant portions of the code would be short-circuited.

Scanning
--------

Initially (at least), the lvmetad will be not allowed to read disks: it will
rely on an external program to provide the metadata. In the ideal case, this
will be triggered by udev. The role of lvmetad is then to collect and maintain
an accurate (up to the data it has received) image of the VGs available in the
system. I imagine we could extend the pvscan command (or add a new one, say
lvmetad_client, if pvscan is found to be inappropriate):

    $ pvscan --cache /dev/foo
    $ pvscan --cache --remove /dev/foo

These commands would simply read the label and the MDA (if applicable) from the
given PV and feed that data to the running lvmetad, using
lvmetad_{add,remove}_pv (see lvmetad_client.h).

We however need to ensure a couple of things here:

1) only LVM commands ever touch PV labels and VG metadata
2) when a device is added or removed, udev fires a rule to notify lvmetad

While the latter is straightforward, there are issues with the first. We
*might* want to invoke the dreaded "watch" udev rule in this case, however it
ends up being implemented. Of course, we can also rely on the sysadmin to be
reasonable and not write over existing LVM metadata without first telling LVM
to let go of the respective device(s).

Even if we simply ignore the problem, metadata write should fail in these
cases, so the admin should be unable to do substantial damage to the system. If
there were active LVs on top of the vanished PV, they are in trouble no matter
what happens there.

Incremental scan
----------------

There are some new issues arising with the "udev" scan mode. Namely, the
devices of a volume group will be appearing one by one. The behaviour in this
case will be very similar to the current behaviour when devices are missing:
the volume group, until *all* its physical volumes have been discovered and
announced by udev, will be in a state with some of its devices flagged as
MISSING_PV. This means that the volume group will be, for most purposes,
read-only until it is complete and LVs residing on yet-unknown PVs won't
activate without --partial. Under usual circumstances, this is not a problem
and the current code for dealing with MISSING_PVs should be adequate.

However, the code for reading volume groups from disks will need to be adapted,
since it currently does not work incrementally. Such support will need to track
metadata-less PVs that have been encountered so far and to provide a way to
update an existing volume group. When the first PV with metadata of a given VG
is encountered, the VG is created in lvmetad (probably in the form of "struct
volume_group") and it is assigned any previously cached metadata-less PVs it is
referencing. Any PVs that were not yet encountered will be marked as MISSING_PV
in the "struct volume_group". Upon scanning a new PV, if it belongs to any
already-known volume group, this PV is checked for consistency with the already
cached metadata (in a case of mismatch, the VG needs to be recovered or
declared conflicted), and is subsequently unmarked MISSING_PV. Care need be
taken not to unmark MISSING_PV on PVs that have this flag in their persistent
metadata, though.

The most problematic aspect of the whole design may be orphan PVs. At any given
point, a metadata-less PV may appear orphaned, if a PV of its VG with metadata
has not been scanned yet. Eventually, we will have to decide that this PV is
really an orphan and enable its usage for creating or extending VGs. In
practice, the decision might be governed by a timeout or assumed immediately --
the former case is a little safer, the latter is probably more transparent. I
am not very keen on using timeouts and we can probably assume that the admin
won't blindly try to re-use devices in a way that would trip up LVM in this
respect. I would be in favour of just assuming that metadata-less VGs with no
known referencing VGs are orphans -- after all, this is the same approach as we
use today. The metadata balancing support may stress this a bit more than the
usual contemporary setups do, though.

Automatic activation
--------------------

It may also be prudent to provide a command that will block until a volume
group is complete, so that scripts can reliably activate/mount LVs and such. Of
course, some PVs may never appear, so a timeout is necessary. Again, this is
something not handled by current tools, but may become more important in
future. It probably does not need to be implemented right away though.

The other aspect of the progressive VG assembly is automatic activation. The
currently only problem with that is that we would like to avoid having
activation code in lvmetad, so we would prefer to fire up an event of some sort
and let someone else handle the activation and whatnot.

Cluster support
---------------

When working in a cluster, clvmd integration will be necessary: clvmd will need
to instruct lvmetad to re-read metadata as appropriate due to writes on remote
hosts. Overall, this is not hard, but the devil is in the details. I would
possibly disable lvmetad for clustered volume groups in the first phase and
only proceed when the local mode is robust and well tested.

With lvmlockd, lvmetad state is kept up to date by flagging either an
individual VG as "invalid", or the global state as "invalid".  When either
the VG or the global state are read, this invalid flag is returned along
with the data.  The client command can check for this invalid state and
decide to read the information from disk rather than use the stale cached
data.  After the latest data is read from disk, the command may choose to
send it to lvmetad to update the cache.  lvmlockd uses version numbers
embedded in its VG and global locks to detect when cached data becomes
invalid, and it then tells lvmetad to set the related invalid flag.
dct, 2015-06-23

Protocol & co.
--------------

I expect a simple text-based protocol executed on top of an Unix Domain Socket
to be the communication interface for lvmetad. Ideally, the requests and
replies will be well-formed "config file" style strings, so we can re-use
existing parsing infrastructure.

Since we already have two daemons, I would probably look into factoring some
common code for daemon-y things, like sockets, communication (including thread
management) and maybe logging and re-using it in all the daemons (clvmd,
dmeventd and lvmetad). This shared infrastructure should live under
daemons/common, and the existing daemons shall be gradually migrated to the
shared code.

Future extensions
-----------------

The above should basically cover the use of lvmetad as a cache-only
daemon. Writes could still be executed locally, and the new metadata version
can be provided to lvmetad through the socket the usual way. This is fairly
natural and in my opinion reasonable. The lvmetad acts like a cache that will
hold metadata, no more no less.

Above this, there is a couple of things that could be worked on later, when the
above basic design is finished and implemented.

_Metadata writing_: We may want to support writing new metadata through
lvmetad. This may or may not be a better design, but the write itself should be
more or less orthogonal to the rest of the story outlined above.

_Locking_: Other than directing metadata writes through lvmetad, one could
conceivably also track VG/LV locking through the same.

_Clustering_: A deeper integration of lvmetad with clvmd might be possible and
maybe desirable. Since clvmd communicates over the network with other clvmd
instances, this could be extended to metadata exchange between lvmetad's,
further cutting down scanning costs. This would combine well with the
write-through-lvmetad approach.

Testing
-------

Since (at least bare-bones) lvmetad has no disk interaction and is fed metadata
externally, it should be very amenable to automated testing. We need to provide
a client that can feed arbitrary, synthetic metadata to the daemon and request
the data back, providing reasonable (nearly unit-level) testing infrastructure.

Battle plan & code layout
=========================

- config_tree from lib/config needs to move to libdm/
- daemon/common *client* code can go to libdm/ as well (say
  libdm/libdm-daemon.{h,c} or such)
- daemon/common *server* code stays, is built in daemon/ toplevel as a static
  library, say libdaemon-common.a
- daemon/lvmetad *client* code goes to lib/lvmetad
- daemon/lvmetad *server* code stays (links in daemon/libdaemon_common.a)