1
0
mirror of git://sourceware.org/git/lvm2.git synced 2024-12-21 13:34:40 +03:00
Commit Graph

559 Commits

Author SHA1 Message Date
David Teigland
fa9eb76a5d improve info about vgck updatemetadata
Add man page info about this option, and add log messages
pointing to this option.
2020-06-03 12:38:27 -05:00
David Teigland
2a304d7a75 lvmcache: free vginfo lock_type 2020-05-14 10:20:08 -05:00
David Teigland
4047a32128 use refresh_filters only where needed
Filters are changed and need refresh in only one
place (vgimportclone), so avoid doing the refresh
for every other command that doesn't need it.
2020-04-22 14:08:54 -05:00
David Teigland
d79afd4084 lvmcache: rework handling of VGs with duplicate vgnames
The previous method of managing duplicate vgnames prevented
vgreduce from working if a foreign vg with the same name
existed.
2020-04-21 14:40:34 -05:00
David Teigland
cc4051eec0 pass cmd struct through more functions
no functional change
2020-04-21 10:58:05 -05:00
David Teigland
3854931aea lvmcache_get_mda: remove unused function 2020-04-21 10:58:05 -05:00
David Teigland
0c1316cda8 scanning: optimize by checking text offset and checksum
After the VG lock is taken for vg_read, reread the mda_header
and compare the metadata text offset and checksum to what was
seen during label scan.  If it is unchanged, then the metadata
has not changed since the label scan, and the metadata does not
need to be reread under the lock for command processing.

For commands that do not make changes (e.g. reporting), the
mda_header is reread and checked on one mda to decide if the
full metadata rereading can be skipped.  For other commands
(e.g. modifying the vg) the mda_header is reread and checked
from all PVs.  (These could probably just check one mda also.)
2019-11-26 16:52:28 -06:00
Zdenek Kabelac
ad0343d8cb cov: remove unused headers 2019-11-14 18:06:42 +01:00
Zdenek Kabelac
44bf9c9a6a cov: fix memleak for duplicate device
For  dev_in_device_list() != 0 allocated  'devl' was
actually leaking - so instead allocate 'devl' only
when !dev_in_device_list() and indent code around.
2019-11-14 18:06:42 +01:00
Zdenek Kabelac
0c01a4c2a6 gcc: avoid warning: declaration of xxx shadows a global declaration
Fix some gcc complaints again shadowing global declarations
2019-10-21 15:32:35 +02:00
David Teigland
bd21736e8b vgck: let updatemetadata repair mismatched metadata
Let vgck --updatemetadata repair cases where different mdas
hold indepedently valid but unmatching copies of the metadata,
i.e. different text metadata checksums or text metadata sizes.
2019-10-11 12:57:39 -05:00
David Teigland
f836fe3836 scan: use PV device name hint for choosing duplicate PV
Prefer a device if its name matches the PV device name hint.
2019-09-30 11:38:10 -05:00
David Teigland
4910a31f6d scan: use PV size for choosing duplicate PV
Prefer a device if it matches the size of the PV.
2019-09-30 11:38:10 -05:00
David Teigland
f3084ee2e5 scan: add PV summary info to lvmcache
Expand the lvmcache info that is saved by the scan to
include PV info from the metadata.
2019-09-30 11:38:10 -05:00
David Teigland
dcbed38b33 fix duplicate pv size check
Fixes a segfault in the recent commit e01fddc57:
"improve duplicate pv handling for md components"

While choosing between duplicates, the info struct is
not always valid; it may have been dropped already.

Remove the code that was still using the info struct for
size comparisons.  The size comparisons were a bogus check
anyway because it was just preferring the dev that had
already been chosen, it wasn't actually comparing the
dev size to the PV size.  It would be good to use a
dev/PV size comparison in the duplicate handling code, but
the PV size is not available until after vg_read, not
from the scan.
2019-08-27 15:40:24 -05:00
David Teigland
e01fddc578 improve duplicate pv handling for md components
Eliminate md components at the start so they don't
interfere with actual duplicates, and don't need
to be removed later.  This also allows for choosing
no copy of a PVID if they all happen to be md
components.
2019-08-16 13:26:12 -05:00
David Teigland
ee4a32e992 lvmcache: use devl list helper 2019-08-16 13:26:12 -05:00
David Teigland
96dfad5022 lvmcache: replace found_duplicates variable
With just checking if the duplicates lists are empty.
2019-08-16 13:26:11 -05:00
David Teigland
677833ce6f lvmcache: renaming functions and variables
related to duplicates, no functional changes.
2019-08-16 13:26:11 -05:00
David Teigland
d16142f90f scanning: open devs rw when rescanning for write
When vg_read rescans devices with the intention of
writing the VG, the label rescan can open the devs
RW so they do not need to be closed and reopened
RW in dev_write_bytes.
2019-06-21 10:57:49 -05:00
David Teigland
4bb7d3da0e lvmcache: remove wrapper around lvmcache_get_vgnameids
This was left over from when there was an lvmetad
version of the function.
2019-06-11 14:10:14 -05:00
David Teigland
49b8846567 lvmcache: remove unused function
Drop lvmcache_fmt_from_vgname(), the way it was called made
it identical to the existing lvmcache_vginfo_from_vgname().
2019-06-10 10:38:32 -05:00
David Teigland
5036244ce8 lvmcache: remove unused code 2019-06-10 10:38:32 -05:00
David Teigland
a07cc8dbef reset cmd wipe_outdated_pvs
at the start of a command, which is needed in case the cmd
struct is reused.
2019-06-10 10:34:58 -05:00
David Teigland
ba7ff96faf improve reading and repairing vg metadata
The fact that vg repair is implemented as a part of vg read
has led to a messy and complicated implementation of vg_read,
and limited and uncontrolled repair capability.  This splits
read and repair apart.

Summary
-------

- take all kinds of various repairs out of vg_read
- vg_read no longer writes anything
- vg_read now simply reads and returns vg metadata
- vg_read ignores bad or old copies of metadata
- vg_read proceeds with a single good copy of metadata
- improve error checks and handling when reading
- keep track of bad (corrupt) copies of metadata in lvmcache
- keep track of old (seqno) copies of metadata in lvmcache
- keep track of outdated PVs in lvmcache
- vg_write will do basic repairs
- new command vgck --updatemetdata will do all repairs

Details
-------

- In scan, do not delete dev from lvmcache if reading/processing fails;
  the dev is still present, and removing it makes it look like the dev
  is not there.  Records are now kept about the problems with each PV
  so they be fixed/repaired in the appropriate places.

- In scan, record a bad mda on failure, and delete the mda from
  mda in use list so it will not be used by vg_read or vg_write,
  only by repair.

- In scan, succeed if any good mda on a device is found, instead of
  failing if any is bad.  The bad/old copies of metadata should not
  interfere with normal usage while good copies can be used.

- In scan, add a record of old mdas in lvmcache for later, do not repair
  them while reading, and do not let them prevent us from finding and
  using a good copy of metadata from elsewhere.  One result is that
  "inconsistent metadata" is no longer a read error, but instead a
  record in lvmcache that can be addressed separate from the read.

- Treat a dev with no good mdas like a dev with no mdas, which is an
  existing case we already handle.

- Don't use a fake vg "handle" for returning an error from vg_read,
  or the vg_read_error function for getting that error number;
  just return null if the vg cannot be read or used, and an error_flags
  arg with flags set for the specific kind of error (which can be used
  later for determining the kind of repair.)

- Saving an original copy of the vg metadata, for purposes of reverting
  a write, is now done explicitly in vg_read instead of being hidden in
  the vg_make_handle function.

- When a vg is not accessible due to "access restrictions" but is
  otherwise fine, return the vg through the new error_vg arg so that
  process_each_pv can skip the PVs in the VG while processing.
  (This is a temporary accomodation for the way process_each_pv
  tracks which devs have been looked at, and can be dropped later
  when process_each_pv implementation dev tracking is changed.)

- vg_read does not try to fix or recover a vg, but now just reads the
  metadata, checks access restrictions and returns it.
  (Checking access restrictions might be better done outside of vg_read,
   but this is a later improvement.)

- _vg_read now simply makes one attempt to read metadata from
  each mda, and uses the most recent copy to return to the caller
  in the form of a 'vg' struct.
  (bad mdas were excluded during the scan and are not retried)
  (old mdas were not excluded during scan and are retried here)

- vg_read uses _vg_read to get the latest copy of metadata from mdas,
  and then makes various checks against it to produce warnings,
  and to check if VG access is allowed (access restrictions include:
  writable, foreign, shared, clustered, missing pvs).

- Things that were previously silently/automatically written by vg_read
  that are now done by vg_write, based on the records made in lvmcache
  during the scan and read:
  . clearing the missing flag
  . updating old copies of metadata
  . clearing outdated pvs
  . updating pv header flags

- Bad/corrupt metadata are now repaired; they were not before.

Test changes
------------

- A read command no longer writes the VG to repair it, so add a write
  command to do a repair.
  (inconsistent-metadata, unlost-pv)

- When a missing PV is removed from a VG, and then the device is
  enabled again, vgck --updatemetadata is needed to clear the
  outdated PV before it can be used again, where it wasn't before.
  (lvconvert-repair-policy, lvconvert-repair-raid, lvconvert-repair,
   mirror-vgreduce-removemissing, pv-ext-flags, unlost-pv)

Reading bad/old metadata
------------------------

- "bad metadata": the mda_header or metadata text has invalid fields
  or can't be parsed by lvm.  This is a form of corruption that would
  not be caused by known failure scenarios.  A checksum error is
  typically included among the errors reported.

- "old metadata": a valid copy of the metadata that has a smaller seqno
  than other copies of the metadata.  This can happen if the device
  failed, or io failed, or lvm failed while commiting new metadata
  to all the metadata areas.  Old metadata on a PV that has been
  removed from the VG is the "outdated" case below.

When a VG has some PVs with bad/old metadata, lvm can simply ignore
the bad/old copies, and use a good copy.  This is why there are
multiple copies of the metadata -- so it's available even when some
of the copies cannot be used.  The bad/old copies do not have to be
repaired before the VG can be used (the repair can happen later.)

A PV with no good copies of the metadata simply falls back to being
treated like a PV with no mdas; a common and harmless configuration.

When bad/old metadata exists, lvm warns the user about it, and
suggests repairing it using a new metadata repair command.
Bad metadata in particular is something that users will want to
investigate and repair themselves, since it should not happen and
may indicate some other problem that needs to be fixed.

PVs with bad/old metadata are not the same as missing devices.
Missing devices will block various kinds of VG modification or
activation, but bad/old metadata will not.

Previously, lvm would attempt to repair bad/old metadata whenever
it was read.  This was unnecessary since lvm does not require every
copy of the metadata to be used.  It would also hide potential
problems that should be investigated by the user.  It was also
dangerous in cases where the VG was on shared storage.  The user
is now allowed to investigate potential problems and decide how
and when to repair them.

Repairing bad/old metadata
--------------------------

When label scan sees bad metadata in an mda, that mda is removed
from the lvmcache info->mdas list.  This means that vg_read will
skip it, and not attempt to read/process it again.  If it was
the only in-use mda on a PV, that PV is treated like a PV with
no mdas.  It also means that vg_write will also skip the bad mda,
and not attempt to write new metadata to it.  The only way to
repair bad metadata is with the metadata repair command.

When label scan sees old metadata in an mda, that mda is kept
in the lvmcache info->mdas list.  This means that vg_read will
read/process it again, and likely see the same mismatch with
the other copies of the metadata.  Like the label_scan, the
vg_read will simply ignore the old copy of the metadata and
use the latest copy.  If the command is modifying the vg
(e.g. lvcreate), then vg_write, which writes new metadata to
every mda on info->mdas, will write the new metadata to the
mda that had the old version.  If successful, this will resolve
the old metadata problem (without needing to run a metadata
repair command.)

Outdated PVs
------------

An outdated PV is a PV that has an old copy of VG metadata
that shows it is a member of the VG, but the latest copy of
the VG metadata does not include this PV.  This happens if
the PV is disconnected, vgreduce --removemissing is run to
remove the PV from the VG, then the PV is reconnected.
In this case, the outdated PV needs have its outdated metadata
removed and the PV used flag needs to be cleared.  This repair
will be done by the subsequent repair command.  It is also done
if vgremove is run on the VG.

MISSING PVs
-----------

When a device is missing, most commands will refuse to modify
the VG.  This is the simple case.  More complicated is when
a command is allowed to modify the VG while it is missing a
device.

When a VG is written while a device is missing for one of it's PVs,
the VG metadata is written to disk with the MISSING flag on the PV
with the missing device.  When the VG is next used, it is treated
as if the PV with the MISSING flag still has a missing device, even
if that device has reappeared.

If all LVs that were using a PV with the MISSING flag are removed
or repaired so that the MISSING PV is no longer used, then the
next time the VG metadata is written, the MISSING flag will be
dropped.

Alternative methods of clearing the MISSING flag are:

vgreduce --removemissing will remove PVs with missing devices,
or PVs with the MISSING flag where the device has reappeared.

vgextend --restoremissing will clear the MISSING flag on PVs
where the device has reappeared, allowing the VG to be used
normally.  This must be done with caution since the reappeared
device may have old data that is inconsistent with data on other PVs.

Bad mda repair
--------------

The new command:
vgck --updatemetadata VG

first uses vg_write to repair old metadata, and other basic
issues mentioned above (old metadata, outdated PVs, pv_header
flags, MISSING_PV flags).  It will also go further and repair
bad metadata:

. text metadata that has a bad checksum
. text metadata that is not parsable
. corrupt mda_header checksum and version fields

(To keep a clean diff, #if 0 is added around functions that
are replaced by new code.  These commented functions are
removed by the following commit.)
2019-06-07 15:54:04 -05:00
David Teigland
45b164f62c create separate lvmcache update functions for read and write
The vg read and vg write cases need to update lvmcache
differently, so create separate functions for them.

The read case now handles checking for outdated mdas
and moves them aside into a new list to be repaired in
a subsequent commit.
2019-06-07 15:54:04 -05:00
David Teigland
86d831b916 change args for text label read function
Have the caller pass the label_sector to the read
function so the read function can set the sector
field in the label struct, instead of having the
read function return a pointer to the label for
the caller to set the sector field.

Also have the read function return a flag indicating
to the caller that the scanned device was identified
as a duplicate pv.
2019-06-07 15:54:04 -05:00
David Teigland
889b5d3183 add mda arg to add_mda
Allow the caller of lvmcache_add_mda() to have the
new mda returned.
2019-06-07 15:54:04 -05:00
David Teigland
b2447e3538 keep track of which mdas have old metadata in lvmcache
This will be used for more advanced repair in a
subsequent commit.
2019-06-07 15:54:04 -05:00
David Teigland
0b18c25d93 ability to keep track of outdated pvs in lvmcache
Outdated PVs hold metadata for VG from which they
have been removed.  Add the ability to keep track
of these in lvmcache.
This will be used for more advanced repair in a
subsequent commit.
2019-06-07 15:54:04 -05:00
David Teigland
650524b955 ability to keep track of bad mdas in lvmcache
mda's that cannot be processed by lvm because of
some corruption can be kept on a separate list.
These will be used for more advanced repair in a
subsequent commit.
2019-06-07 15:54:04 -05:00
David Teigland
db98a6e362 Additional MD component checking
If udev info is missing for a device, (which would indicate
if it's an MD component), then do an end-of-device read to
check if a PV is an MD component.  (This is skipped when
using hints since we already know devs in hints are good.)

A new config setting md_component_checks can be used to
disable the additional end-of-device MD checks, or to
always enable end-of-device MD checks.

When both hints and udev info are disabled/unavailable,
the end of PVs will now be scanned by default.  If md
devices with end-of-device superblocks are not being
used, the extra I/O overhead can be avoided by setting
md_component_checks="start".
2019-06-07 13:27:16 -05:00
David Teigland
2bcd43c683 lvmcache: remove unused_duplicate_devs list from cmd
Save the previous duplicate PVs in a global list instead
of a list on the cmd struct.  dmeventd reuses the cmd struct
for multiple commands, and the list entries between commands
were being freed (apparently), causing a segfault in dmeventd
when it tried to use items in cmd->unused_duplicate_devs
that had been saved there by the previous command.
2019-06-07 10:14:33 -05:00
David Teigland
52586b1039 pvck: new dump option to extract metadata
The new command 'pvck --dump metadata PV' will extract
the current version of VG metadata from a PV for testing
and debugging.  --dump metadata_area extracts the entire
text metadata area.
2019-05-23 11:49:06 -05:00
David Teigland
90b94ead12 lvmcache: remove unused flag
The new label scan design is never called recursively,
so we don't need a flag to check for that.
2019-04-30 14:59:27 -05:00
David Teigland
8c87dda195 locking: unify global lock for flock and lockd
There have been two file locks used to protect lvm
"global state": "ORPHANS" and "GLOBAL".

Commands that used the ORPHAN flock in exclusive mode:
  pvcreate, pvremove, vgcreate, vgextend, vgremove,
  vgcfgrestore

Commands that used the ORPHAN flock in shared mode:
  vgimportclone, pvs, pvscan, pvresize, pvmove,
  pvdisplay, pvchange, fullreport

Commands that used the GLOBAL flock in exclusive mode:
  pvchange, pvscan, vgimportclone, vgscan

Commands that used the GLOBAL flock in shared mode:
  pvscan --cache, pvs

The ORPHAN lock covers the important cases of serializing
the use of orphan PVs.  It also partially covers the
reporting of orphan PVs (although not correctly as
explained below.)

The GLOBAL lock doesn't seem to have a clear purpose
(it may have eroded over time.)

Neither lock correctly protects the VG namespace, or
orphan PV properties.

To simplify and correct these issues, the two separate
flocks are combined into the one GLOBAL flock, and this flock
is used from the locking sites that are in place for the
lvmlockd global lock.

The logic behind the lvmlockd (distributed) global lock is
that any command that changes "global state" needs to take
the global lock in ex mode.  Global state in lvm is: the list
of VG names, the set of orphan PVs, and any properties of
orphan PVs.  Reading this global state can use the global lock
in sh mode to ensure it doesn't change while being reported.

The locking of global state now looks like:

lockd_global()
  previously named lockd_gl(), acquires the distributed
  global lock through lvmlockd.  This is unchanged.
  It serializes distributed lvm commands that are changing
  global state.  This is a no-op when lvmlockd is not in use.

lockf_global()
  acquires an flock on a local file.  It serializes local lvm
  commands that are changing global state.

lock_global()
  first calls lockf_global() to acquire the local flock for
  global state, and if this succeeds, it calls lockd_global()
  to acquire the distributed lock for global state.

Replace instances of lockd_gl() with lock_global(), so that the
existing sites for lvmlockd global state locking are now also
used for local file locking of global state.  Remove the previous
file locking calls lock_vol(GLOBAL) and lock_vol(ORPHAN).

The following commands which change global state are now
serialized with the exclusive global flock:

pvchange (of orphan), pvresize (of orphan), pvcreate, pvremove,
vgcreate, vgextend, vgremove, vgreduce, vgrename,
vgcfgrestore, vgimportclone, vgmerge, vgsplit

Commands that use a shared flock to read global state (and will
be serialized against the prior list) are those that use
process_each functions that are based on processing a list of
all VG names, or all PVs.  The list of all VGs or all PVs is
global state and the shared lock prevents those lists from
changing while the command is processing them.

The ORPHAN lock previously attempted to produce an accurate
listing of orphan PVs, but it was only acquired at the end of
the command during the fake vg_read of the fake orphan vg.
This is not when orphan PVs were determined; they were
determined by elimination beforehand by processing all real
VGs, and subtracting the PVs in the real VGs from the list
of all PVs that had been identified during the initial scan.
This is fixed by holding the single global lock in shared mode
while processing all VGs to determine the list of orphan PVs.
2019-04-29 13:01:05 -05:00
David Teigland
3584e0c0d5 io: warn when metadata size approaches io memory size
When a single copy of metadata gets within 1MB of the
current io_memory_size value, begin printing a warning
that the io_memory_size should be increased.
2019-03-04 12:13:09 -06:00
David Teigland
6620dc9475 add device hints to reduce scanning
Save the list of PVs in /run/lvm/hints.  These hints
are used to reduce scanning in a number of commands
to only the PVs on the system, or only the PVs in a
requested VG (rather than all devices on the system.)
2019-01-15 10:23:47 -06:00
David Teigland
3d2fd95af7 remove unused full filter
it's the same as cmd->filter
2018-12-04 14:06:46 -06:00
Zdenek Kabelac
9a6f0e64f9 debug: missing backtrace 2018-11-05 17:25:11 +01:00
Zdenek Kabelac
70e3d0a613 cov: remove unused assigns 2018-11-05 17:25:11 +01:00
David Teigland
117160b27e Remove lvmetad
Native disk scanning is now both reduced and
async/parallel, which makes it comparable in
performance (and often faster) when compared
to lvm using lvmetad.

Autoactivation now uses local temp files to record
online PVs, and no longer requires lvmetad.

There should be no apparent command-level change
in behavior.
2018-07-11 11:26:42 -05:00
David Teigland
db741e75a2 pvscan: autoactivate without lvmetad
When lvmetad is not used, use temporary files to record
which PVs have appeared.  Use these temp files to determine
when a VG is complete, to trigger autoactivation.

This change allows us to remove lvmetad while keeping the
same autoactivation behavior that lvmetad provides.

The temp files are created in /run/lvm/pvs_online/ and are
named for the PVID of the PV.  The files contain the
major:minor of the device the PV was read from.

e.g. if VG foo has dev1 and dev2, then:

. pvscan --cache -aay dev1
  reads vg metadata from dev1
  creates /run/lvm/pvs_online/<pvid-of-dev1>
  checks if all vg->pvs are online: no

. pvscan --cache -aay dev2
  reads vg metadata from dev2
  creates /run/lvm/pvs_online/<pvid-of-dev2>
  checks if all vg->pvs are online: yes
  autoactivates vg

A 'pvscan --cache dev' (without -aay) still records that
dev is online.

A 'pvscan --cache --major X --minor Y' after a device is
gone will remove the temp file for it.

A 'pvscan --cache [-aay]' (no devs) resets the state of
temp files by removing them all, then scanning all devs
and creating temp files for PVs that are found.

If no online files exist, the first pvscan --cache scans
all devs and creates temp files for any PVs found.

The scope of the temp files is only pvscan, and they are only
used for pvscan-based autoactivation.  No other commands are
concerned with or aware of these temp files.  When lvm creates
or removes PVs, no attempt is made to update the temp files.
2018-07-09 16:11:24 -05:00
David Teigland
3fd75d1bcd scan: use full md filter when md 1.0 devices are present
The md filter can operate in two native modes:
- normal: reads only the start of each device
- full: reads both the start and end of each device

md 1.0 devices place the superblock at the end of the device,
so components of this version will only be identified and
excluded when lvm uses the full md filter.

Previously, the full md filter was only used in commands
that could write to the device.  Now, the full md filter
is also applied when there is an md 1.0 device present
on the system.  This means the 'pvs' command can avoid
displaying md 1.0 components (at the cost of doubling
the i/o to every device on the system.)

(The md filter can operate in a third mode, using udev,
but this is disabled by default because there have been
problems with reliability of the info returned from udev.)
2018-06-15 12:21:25 -05:00
David Teigland
8eab37593e Add cmd arg to more functions
so that it can be used in the filter code
2018-06-15 11:03:55 -05:00
David Teigland
17f5572bc9 Remove independent metadata areas
in which metadata is stored in files on the local fs
instead of on PVs.
2018-06-13 12:25:19 -05:00
David Teigland
be3af7f93e Remove the unused lock_hash in lvmcache
It kept track of which VGs were locked, but is
no longer used, so remove it.
2018-06-12 11:29:56 -05:00
David Teigland
c4153a8dfc Remove checking for locked VGs
A few places were calling a function to check if a
VG lock was held.  The only place it was actually
needed is for pvcreate which wants to do its own
locking (and scanning) around process_each_pv.

The locking/scanning exceptions for pvcreate in
process_each_pv/vg_read can be enabled by just passing
a couple of flags instead of checking if the VG is
already locked.  This also means that these special
cases won't be enabled unknowingly in other places
where they shouldn't be used.
2018-06-12 09:46:04 -05:00
David Teigland
b48e10d9e6 Remove lvmcache CACHE_LOCKED flag
and the functions that set it.  It's no longer used.
2018-06-08 15:11:47 -05:00
David Teigland
669b1295ae Remove header declarations for removed functions 2018-06-08 10:01:05 -05:00
David Teigland
73b7e6fde7 Remove more code that was only used by liblvm2app 2018-06-08 09:29:11 -05:00
Joe Thornber
d5da55ed85 device_mapper: remove dbg_malloc.
I wrote dbg_malloc before we had valgrind.  These days there's just
no need.
2018-06-08 13:40:53 +01:00
David Teigland
e7aa51c70f Remove VG lock ordering check
Four commands lock two VGs at a time:

- vgsplit and vgmerge already have their own logic to
  acquire the locks in the correct order.

- vgimportclone and vgrename disable this ordering check.
2018-06-07 16:17:04 +01:00
David Teigland
3e781ea446 Remove clvmd and associated code
More code reduction and simplification can follow.
2018-06-05 11:09:13 -05:00
David Teigland
0625c7f372 devs: clear coverity warning about null info
a theoretical possibility.
2018-06-01 13:15:22 -05:00
David Teigland
b6f0f20da2 lvmlockd: primarily use vg_is_shared
to check if a vg uses an lvmlockd lock_type,
instead of the equivalent but longer is_lockd_type.
2018-06-01 13:15:22 -05:00
Joe Thornber
dbba1e9b93 Merge branch 'master' into 2018-05-11-fork-libdm 2018-06-01 13:04:12 +01:00
David Teigland
3c9ed33f83 scan: move warnings about duplicate devices
We have been warning about duplicate devices (and disabling lvmetad)
immediately when the dup was detected (during label_scan).  Move the
warnings (and the disabling) to happen later, after label_scan is
finished.

This lets us avoid an unwanted warning message about duplicates
in the special case were md components are eliminated during the
duplicate device resolution.
2018-05-21 16:48:02 -05:00
Joe Thornber
89fdc0b588 Merge branch 'master' into 2018-05-11-fork-libdm 2018-05-16 13:43:02 +01:00
David Teigland
11ceb77867 lvmcache: fix loop freeing infos
valgrind was concerned about loop through vginfo->infos,
so grab info from dev.
2018-05-14 13:45:55 -05:00
David Teigland
517d6cc418 scan: add some missing frees
some objects had been moved out of mem pools.
2018-05-14 13:38:16 -05:00
Joe Thornber
7f97c7ea9a build: Don't generate symlinks in include/ dir
As we start refactoring the code to break dependencies (see doc/refactoring.txt),
I want us to use full paths in the includes (eg, #include "base/data-struct/list.h").
This makes it more obvious when we're breaking abstraction boundaries, eg, including a file in
metadata/ from base/
2018-05-14 10:30:20 +01:00
David Teigland
09fcc8eaa8 scan: ignore duplicates that are md component devs
md devices using an older superblock version have
superblocks at the end of the md device.  For commands
that skip reading the end of devices during filtering,
the md component devs will be scanned, and will appear
as duplicate PVs to the original md device.  Remove
these md components from the list of unused duplicate
devices, so they are treated as if they had been
ignored during filtering.  This avoids the restrictions
that are placed on using PVs with duplicates.
2018-05-11 15:52:22 -05:00
David Teigland
73578e36fa dev_cache: remove the lvmcache check when closing fd
This is no longer used since devices are not held
open in dev_cache.
2018-05-11 14:30:10 -05:00
David Teigland
5c9dcd99fd scan: remove unused args from label_read 2018-05-11 14:16:49 -05:00
David Teigland
bbb8040456 dev_cache: drop open_list
devices are now held open only in bcache,
so drop the dev_cache list of open devices
which is unused.
2018-05-11 12:47:56 -05:00
David Teigland
57bb46c5e7 filter: use bcache for filter reads
Filters are still applied before any device reading or
the label scan, but any filter checks that want to read
the device are skipped and the device is flagged.

After bcache is populated, but before lvm looks for
devices (i.e. before label scan), the filters are
reapplied to the devices that were flagged above.
The filters will then find the data they need in
bcache.
2018-05-10 16:03:19 -05:00
David Teigland
f4a60fe004 clvmd: saved_vg code and comment formatting 2018-05-03 14:54:48 -05:00
David Teigland
822a8b62be clvmd: don't save cft and buf for saved_vg 2018-05-03 14:54:48 -05:00
David Teigland
c016b573ee clvmd: separate saved_vg from vginfo
The clvmd saved_vg data is independent from the normal lvm
lvmcache vginfo data, so separate saved_vg from vginfo.
Normal lvm doesn't need to use save_vg at all, and in clvmd,
lvmcache changes on vginfo can be made without worrying
about unwanted effects on saved_vg.
2018-05-03 14:54:48 -05:00
David Teigland
a5e13f2eef clvmd: defer freeing saved vgs
To avoid the chance of freeing a saved vg while another
code path is using it, defer freeing saved vgs until
all the lvmcache content is dropped for the vg.
2018-05-03 14:54:48 -05:00
David Teigland
a418f88b76 lvmcache: fix typo in lvmcache_get_saved_vg 2018-05-01 09:06:57 -05:00
David Teigland
c1cd18f21e Remove lvm1 and pool disk formats
There are likely more bits of code that can be removed,
e.g. lvm1/pool-specific bits of code that were identified
using FMT flags.

The vgconvert command can likely be reduced further.

The lvm1-specific config settings should probably have
some other fields set for proper deprecation.
2018-04-30 16:55:02 -05:00
David Teigland
89935ace29 clvmd: keep old saved_vg if it matches new
There is no need to release the old saved_vg
if it matches the new version.
2018-04-30 13:03:15 -05:00
Joe Thornber
513e9e3264 [lvmetad.h] Use static inline functions to stub out functions.
The macros were causing warnings because the arguments were percieved as
unused.
2018-04-30 09:45:13 +01:00
David Teigland
5b6e62dc1f clvmd: drop old saved_vg when returning new saved_vg
In some pvmove tests, clvmd uses the new (precommitted)
saved_vg, but then requests the old saved_vg, and
expects that the new saved_vg be returned instead of
the old.  So, when returning the new saved_vg, forget
the old one so we don't return it again.
2018-04-26 14:57:45 -05:00
David Teigland
cdb8400de2 scan: refresh filters before scan
The filters save information about devices that should
be ignored, so if we need to repeat a scan  (unusual,
but happens in clvmd), we need to update the filters.
2018-04-26 14:48:13 -05:00
David Teigland
4670e9f698 skip some clvmd-specific code in common cases
This, or something like it, can probably be done
in many other places.
2018-04-25 16:40:08 -05:00
David Teigland
1fec86571f clvmd: reuse a vg struct for sequential LV operations
After reading a VG, stash it in lvmcache as "saved_vg".
Before reading the VG again, try to use the saved_vg.
The saved_vg is dropped on VG lock operations.
2018-04-25 16:39:43 -05:00
David Teigland
f8616ac2d8 lvmcache: rename suspended_vg to saved_vg
The copy of the VG which clvmd stashes in lvmcache should
not only be used between suspend and resume, but between
sequential LV operations in clvmd, so that clvmd does not
need to reread the VG for each one.  Prepare for that by
renaming the stashed VG as "saved_vg".
2018-04-25 16:39:43 -05:00
David Teigland
aee27dc7ba scan: skip device rescan in vg_read
For reporting commands (pvs,vgs,lvs,pvdisplay,vgdisplay,lvdisplay)
we do not need to repeat the label scan of devices in vg_read if
they all had matching metadata in the initial label scan.  The
data read by label scan can just be reused for the vg_read.
This cuts the amount of device i/o in half, from two reads of
each device to one.  We have to be careful to avoid repairing
the VG if we've skipped rescanning.  (The VG repair code is very
poor, and will be redone soon.)
2018-04-20 11:23:14 -05:00
David Teigland
9b6a62f944 lvmcache: simplify
Recent changes allow some major simplification of the way
lvmcache works and is used.  lvmcache_label_scan is now
called in a controlled fashion at the start of commands,
and not via various unpredictable side effects.  Remove
various calls to it from other places.  lvmcache_label_scan
should not be called from anywhere during a command, because
it produces an incorrect representation of PVs with no MDAs,
and misclassifies them as orphans.  This has been a long
standing problem.  The invalid flag and rescanning based on
that is no longer used and removed.  The 'force' variation is
no longer needed and removed.
2018-04-20 11:22:48 -05:00
David Teigland
c0973e70a5 dev_cache: clean up scan
Pull out all of the twisted logic and simply call dev_cache_scan
at the start of the command prior to label scan.
2018-04-20 11:22:48 -05:00
David Teigland
ae21305ee7 scan: drop bcache between lvm shell commands
A running lvm shell keeps all lvm devices open
unless the bcache is dropped.
2018-04-20 11:22:48 -05:00
David Teigland
1717d4cb17 lvmcache: add shorter way to delete dev info
Don't make the caller look up the info first.
2018-04-20 11:22:47 -05:00
David Teigland
28255e3eee scan: always setup bcache for commands using lvmetad
Do this at the start of the command so that it doesn't
need to be checked and set up in every function that
could need it.
2018-04-20 11:22:46 -05:00
David Teigland
6c67c7557c scan: use separate fd for bcache
Create a new dev->bcache_fd that the scanning code owns
and is in charge of opening/closing.  This prevents other
parts of lvm code (which do various open/close) from
interfering with the bcache fd.  A number of dev_open
and dev_close are removed from the reading path since
the read path now uses the bcache.

With that in place, open(O_EXCL) for pvcreate/pvremove
can then be fixed.  That wouldn't work previously because
of other open fds.
2018-04-20 11:22:46 -05:00
David Teigland
f17c2cf7c6 pvremove: device check doesn't require label_read
It just needs to check if the device was found during
the scan, which means checking if it exists in lvmcache.
2018-04-20 11:22:45 -05:00
David Teigland
d9a77e8bb4 lvmcache: simplify metadata cache
The copy of VG metadata stored in lvmcache was not being used
in general.  It pretended to be a generic VG metadata cache,
but was not being used except for clvmd activation.  There
it was used to avoid reading from disk while devices were
suspended, i.e. in resume.

This removes the code that attempted to make this look
like a generic metadata cache, and replaces with with
something narrowly targetted to what it's actually used for.

This is a way of passing the VG from suspend to resume in
clvmd.  Since in the case of clvmd one caller can't simply
pass the same VG to both suspend and resume, suspend needs
to stash the VG somewhere that resume can grab it from.
(resume doesn't want to read it from disk since devices
are suspended.)  The lvmcache vginfo struct is used as a
convenient place to stash the VG to pass it from suspend
to resume, even though it isn't related to the lvmcache
or vginfo.  These suspended_vg* vginfo fields should
not be used or touched anywhere else, they are only to
be used for passing the VG data from suspend to resume
in clvmd.  The VG data being passed between suspend and
resume is never modified, and will only exist in the
brief period between suspend and resume in clvmd.

suspend has both old (current) and new (precommitted)
copies of the VG metadata.  It stashes both of these in
the vginfo prior to suspending devices.  When vg_commit
is successful, it sets a flag in vginfo as before,
signaling the transition from old to new metadata.

resume grabs the VG stashed by suspend.  If the vg_commit
happened, it grabs the new VG, and if the vg_commit didn't
happen it grabs the old VG.  The VG is then used to resume
LVs.

This isolates clvmd-specific code and usage from the
normal lvm vg_read code, making the code simpler and
the behavior easier to verify.

Sequence of operations:

- lv_suspend() has both vg_old and vg_new
  and stashes a copy of each onto the vginfo:
  lvmcache_save_suspended_vg(vg_old);
  lvmcache_save_suspended_vg(vg_new);

- vg_commit() happens, which causes all clvmd
  instances to call lvmcache_commit_metadata(vg).
  A flag is set in the vginfo indicating the
  transition from the old to new VG:
  vginfo->suspended_vg_committed = 1;

- lv_resume() needs either vg_old or vg_new
  to use in resuming LVs.  It doesn't want to
  read the VG from disk since devices are
  suspended, so it gets the VG stashed by
  lv_suspend:
  vg = lvmcache_get_suspended_vg(vgid);

If the vg_commit did not happen, suspended_vg_committed
will not be set, and in this case, lvmcache_get_suspended_vg()
will return the old VG instead of the new VG, and it will
resume LVs based on the old metadata.
2018-04-20 11:22:45 -05:00
David Teigland
79c4971210 label_scan: remove extra label scan and read for orphan PVs
When process_each_pv() calls vg_read() on the orphan VG, the
internal implementation was doing an unnecessary
lvmcache_label_scan() and two unnecessary label_read() calls
on each orphan.  Some of those unnecessary label scans/reads
would sometimes be skipped due to caching, but the code was
always doing at least one unnecessary read on each orphan.

The common format_text case was also unecessarily calling into
the format-specific pv_read() function which actually did nothing.

By analyzing each case in which vg_read() was being called on
the orphan VG, we can say that all of the label scans/reads
in vg_read_orphans are unnecessary:

1. reporting commands: the information saved in lvmcache by
the original label scan can be reported.  There is no advantage
to repeating the label scan on the orphans a second time before
reporting it.

2. pvcreate/vgcreate/vgextend: these all share a common
implementation in pvcreate_each_device().  That function
already rescans labels after acquiring the orphan VG lock,
which ensures that the command is using valid lvmcache
information.
2018-04-20 11:22:45 -05:00
David Teigland
e3e5beec74 lvmetad: use new label_scan for update from pvscan
Take advantage of the common implementation with aio
and reduced disk reads.
2018-04-20 11:22:43 -05:00
David Teigland
9c71fa0214 lvmetad: use new label_scan for update from lvmlockd
When lvmlockd indicates that the lvmetad cache is out of
date because of changes by another node, lvmetad_pvscan_vg()
rescans the devices in the VG to update lvmetad.  Use the
new label_scan in this function to use the common code and
take advantage of the new aio and reduced reads.
2018-04-20 11:21:41 -05:00
David Teigland
748f29b42a scan: do scanning at the start of a command
Move the location of scans to make it clearer and avoid
unnecessary repeated scanning.  There should be one scan
at the start of a command which is then used through the
rest of command processing.

Previously, the initial label scan was called as a side effect
from various utility functions.  This would lead to it being called
unnecessarily.  It is an expensive operation, and should only be
called when necessary.  Also, this is a primary step in the
function of the command, and as such it should be called prominently
at the top level of command processing, not as a hidden side effect
of a utility function.  lvm knows exactly where and when the
label scan needs to be done.  Because of this, move the label scan
calls from the internal functions to the top level of processing.

Other specific instances of lvmcache_label_scan() are still called
unnecessarily or unclearly by specific commands that do not use
the common process_each functions.  These will be improved in
future commits.

During the processing phase, rescanning labels for devices in a VG
needs to be done after the VG lock is acquired in case things have
changed since the initial label scan.  This was being done by way
of rescanning devices that had the INVALID flag set in lvmcache.
This usually approximated the right set of devices, but it was not
exact, and obfuscated the real requirement.  Correct this by using
a new function that rescans the devices in the VG:
lvmcache_label_rescan_vg().

Apart from being inexact, the rescanning was extremely well hidden.
_vg_read() would call ->create_instance(), _text_create_text_instance(),
_create_vg_text_instance() which would call lvmcache_label_scan()
which would call _scan_invalid() which repeats the label scan on
devices flagged INVALID.  lvmcache_label_rescan_vg() is now called
prominently by _vg_read() directly.
2018-04-20 11:21:38 -05:00
David Teigland
4507ba3596 scan: use new label_scan for lvmcache_label_scan
To do label scanning, lvm code calls lvmcache_label_scan().
Change lvmcache_label_scan() to use the new label_scan()
based on bcache.

Also add lvmcache_label_rescan_vg() which calls the new
label_scan_devs() which does label scanning on only the
specified devices.  This is for a subsequent commit and
is not yet used.
2018-04-20 11:19:32 -05:00
Joe Thornber
00f1b208a1 [io paths] Unpick agk's aio stuff 2018-04-20 11:03:58 -05:00
Zdenek Kabelac
e9cadbe105 cleanup: matching signess 2018-03-13 12:58:57 +01:00
Zdenek Kabelac
b6e7a0b490 cleanup: more usage of dm_strncpy
Use existing wrapper function arournd  strncpy + buf[] = 0;
2018-03-06 15:40:34 +01:00
Alasdair G Kergon
d6cabbbc53 device: Fix basic async I/O error handling 2018-02-08 20:19:21 +00:00
Alasdair G Kergon
db41fe6c5d lvmcache: Use asynchronous I/O when scanning devices. 2018-02-08 20:15:29 +00:00
Alasdair G Kergon
7a9af3cd0e device: Add flag to indicate that a code path can support AIO
Until the whole source supports AIO, library code can check for
AIO_SUPPORTED_CODE_PATH to determine whether or not it is OK
to use AIO.
2018-02-06 01:11:00 +00:00