1
0
mirror of git://sourceware.org/git/lvm2.git synced 2024-12-22 17:35:59 +03:00
Commit Graph

2964 Commits

Author SHA1 Message Date
Zdenek Kabelac
6b168afcad tests: use parametrized function
Shorten and make the test easily readable by moving same code into
function and removed one duplicated test for 512,4096 combination.

Always use scsi_debug - since default ramdisk or loop device backend
is unpredictible.
2020-09-19 17:30:51 +02:00
Zdenek Kabelac
f63aac5309 tests: use zero backend
Since we are not reading read - just use zero device as backend for
test, so we do not eat real disk space.
2020-09-19 17:30:51 +02:00
Zdenek Kabelac
f7c58c636d tests: use faster awk generator
Shortens log length.
2020-09-19 17:30:51 +02:00
Zdenek Kabelac
8e3e2c74ed tests: ensure mnt is defined before trap install 2020-09-19 17:30:51 +02:00
Zdenek Kabelac
530fc17b38 tests: reduce disk usage 2020-09-19 17:30:51 +02:00
David Teigland
1570e76233 bcache: use indirection table for fd
Add a "device index" (di) for each device, and use this
in the bcache api to the rest of lvm.  This replaces the
file descriptor (fd) in the api.  The rest of lvm uses
new functions bcache_set_fd(), bcache_clear_fd(), and
bcache_change_fd() to control which fd bcache uses for
io to a particular device.

. lvm opens a dev and gets and fd.
  fd = open(dev);

. lvm passes fd to the bcache layer and gets a di
  to use in the bcache api for the dev.
  di = bcache_set_fd(fd);

. lvm uses bcache functions, passing di for the dev.
  bcache_write_bytes(di, ...), etc.

. bcache translates di to fd to do io.

. lvm closes the device and clears the di/fd bcache state.
  close(fd);
  bcache_clear_fd(di);

In the bcache layer, a di-to-fd translation table
(int *_fd_table) is added.  When bcache needs to
perform io on a di, it uses _fd_table[di].

In the following commit, lvm will make use of the new
bcache_change_fd() function to change the fd that
bcache uses for the dev, without dropping cached blocks.
2020-09-18 15:10:11 -05:00
Zdenek Kabelac
4b07ae55f1 tests: printf to awk
Shorten trace logs.
2020-09-18 17:30:45 +02:00
Zdenek Kabelac
9fbcba1c40 tests: update integrity-dmeventd
Use tee.
Switch to more simple generator with awk
(which doesn't produce long debug trace)
Sync before sleep to provoke raid action.
2020-09-18 17:30:45 +02:00
Zdenek Kabelac
9448476202 tests: enhance low-disk-space behavior
Use new SKIP_WITH_LOW_SPACE and set higher requirement for free space.

But still this test can't run on system's tmpfs directories -
as they typically provide less then 2G of space and when the test
runs there it also provisioning for all READ pages!)
BRD (ramdisk) device should work.

Extend a _wait_recalc() loop for slower hw.
When creating large raid which do not need to be fully synchronized use
them on delay devices - so even less data needs read/write.
Remove unneeded lvchange as lvcreate is already leaving LV inactive.
Replace printf with awk as generator.

mm
2020-09-18 17:30:45 +02:00
Zdenek Kabelac
206620018e tests: inittest supports SKIP_WITH_LOW_SPACE
Test can set individually a higher value for required free space on
storage.

Note: it is not fully reliable since when 'brd' (ramdisk) device is used
this free space value is rather meanigul, but it might help
in case where a real filesystem is doing back-end for test devices.
2020-09-18 17:30:31 +02:00
Zdenek Kabelac
048e04e417 tests: utils better handle ouf of disk space
When the test exhausts all the available free space on storage device,
then during the fail we cannot write anything as well - yet
the teardown needs to finish it's work - otherwise we leave
basicaly overfilled filesystem for all remaining tests.
2020-09-18 17:29:26 +02:00
Zdenek Kabelac
b77595ac8b tests: aux better handle invalid table
In cases where internal functions like zero_dev, delay_dev pass-in
invalid parameter so resulting table can't work, resume at least
previous table line before failing out - so the cleaning process
later on is not stuck waiting on a suspended device.
2020-09-18 14:23:20 +02:00
Zdenek Kabelac
a4137412bf tests: also use sed to shorten log output 2020-09-18 00:31:59 +02:00
Zdenek Kabelac
8d40859e29 tests: resolve missing removal of loopdevice on error path
In case of test failure, loop device leaked and occupied space forever.
2020-09-18 00:31:11 +02:00
Zdenek Kabelac
a5e867139d tests: bigger data still needed for 0.7.0 2020-09-17 23:27:52 +02:00
Zdenek Kabelac
7f019f2580 tests: lower memory usage
Reduce memory needed by test at one time.
2020-09-17 23:27:45 +02:00
Zdenek Kabelac
52d3c4de6e tests: smaller delay and lowered version
See if this will still work. Some boxes are delayed too much.
Also try to check for raid extend progress from version 1.13.
2020-09-16 14:08:05 +02:00
Zdenek Kabelac
49292bccc3 tests: fix bash regex syntax
Typo before last commit.
2020-09-16 14:08:05 +02:00
Zdenek Kabelac
2c6bd480b2 tests: switch for checking version of installed tools
It looks like older tools were compacting metadata more.
2020-09-15 23:07:06 +02:00
Zdenek Kabelac
cf4fed3761 tests: skip kernel for this test
Kills this kernel ATM
2020-09-15 23:07:06 +02:00
Zdenek Kabelac
18a60c6340 tests: protect this test for another kernel
Thisi 3.10.0-862 kernel also dies with this test.
2020-09-14 00:15:14 +02:00
Zdenek Kabelac
9c0d11ecc9 tests: tune usage of smaller metadata
While the previous commit c9b40083fc
decresed version to 1.19 for using bigger datasets,  it's not
been quite right - so from our bb machine it looks like
bigger metadata consumption started with 1.19 and kernel 4.18
(fc27)
2020-09-14 00:15:14 +02:00
Zdenek Kabelac
1005fd7b06 tests: raise needed target version
Require higher version to avoid early bugs.
2020-09-14 00:15:14 +02:00
Zdenek Kabelac
57e1e037b6 tests: improve cache abort test
Use bigger volume and slowdown writing to cache device.
This allows more simple to reach 'dirty' state.
Also document exactly 1 SIGINT has to fire aborting of flushing.
2020-09-14 00:15:14 +02:00
Zdenek Kabelac
64c8827cf3 tests: check in_sync prints also dm status
It's more useful to see how the progress of status checking is moving.
2020-09-14 00:15:14 +02:00
Zdenek Kabelac
fe77d1a710 tests: avoid using string
String 'TEST WARNING' may not be present in the test script itself.
Add '\ ' to avoid 'grep' matching test as the test with warning.
2020-09-12 13:24:03 +02:00
Zdenek Kabelac
3008e1be08 tests: support for 16T is needed
Likely 32bit machines can't pass here.
2020-09-12 13:24:03 +02:00
Zdenek Kabelac
17dbb24f7c tests: change skip to die for upstream crash
So the failing test is not lost from sight.
2020-09-12 13:24:03 +02:00
Zdenek Kabelac
7bd015861d tests: skip test on failing kernel 2020-09-12 13:24:03 +02:00
Zdenek Kabelac
93e252c4a3 tests: check for boundary allocation sizes 2020-09-11 21:52:55 +02:00
Zdenek Kabelac
f84a7266bc tests: reduce disk space usage by pvck-dump
Lower disk usage for 'dd'.
2020-09-11 21:52:55 +02:00
Zdenek Kabelac
c9b40083fc tests: lower at_least to version 1.19
With this version already can be seen different metadata usage on
kernel side, so lower the target version.
2020-09-11 21:52:55 +02:00
Zdenek Kabelac
f233d9a909 tests: have_cache function checks for cache-pool
Check for cache-pool segment as plain cache can match writecache.
2020-09-11 21:52:55 +02:00
Zdenek Kabelac
3f2e9e3546 tests: aux collects stack trace when stuck is assumed
Automatically collect traces in this case.
2020-09-10 23:55:03 +02:00
Zdenek Kabelac
49d2b27a68 tests: adding "" around DM_DEV_DIR 2020-09-10 23:55:03 +02:00
Zdenek Kabelac
3dcd61d3d7 tests: disable dbustest with valgrind testing 2020-09-10 23:55:03 +02:00
Zdenek Kabelac
bb62af5b3d tests: tune extend test
For proper checking of extension progress require version 1.15

It looks with older versoin extension happens during very slow
resume within lvm command - although speed is still somewhat slow
with latest version.
2020-09-08 21:23:03 +02:00
Zdenek Kabelac
676ce47754 tests: check for writecache being compiled in 2020-09-08 21:23:03 +02:00
Zdenek Kabelac
8dea63d30f tests: check for cache_version that supports v2 2020-09-08 21:23:03 +02:00
David Teigland
dddf63ebc3 tests: fix pvck repair in hints.sh 2020-09-04 11:23:25 -05:00
Zdenek Kabelac
10fc3610c4 tests: use delayzero_dev
Speed-up a bit the first synchronization with just 50ms write delay,
but later set also delay on read to slowdown lvextend.

FIXME: there are still things to look at:

0 229376 raid raid1 2 AA 229376/229376 idle 0 0
0 229376 raid raid1 2 AA 0/229376 frozen 0 0 -
0 262144 raid raid1 2 AA 229376/262144 repair 0 0 -
0 262144 raid raid1 2 AA 229376/262144 repair 0 0 -
0 262144 raid raid1 2 AA 245888/262144 repair 0 0 -
2020-09-04 18:11:42 +02:00
Zdenek Kabelac
76b1f43e81 tests: add aux delayzero_dev support
Just like we have 'writeerror_dev' supporting creation of device
which 'readable' segment and segments where write will fail we
have now support for delay zero mappings.

This is useful if we want to 'fake' large writing areas where we do
not really care about the actual 'disk' content - since we test
operation logic and it doesn't matter we read and write zeroes.
With combination with 'delay' target we can create specific mappings
and avoid using large memory areas of ramdisk.
2020-09-04 18:11:42 +02:00
David Teigland
d8bb85d963 writecache: allow pvmove on origin
The removed check didn't actually prevent pvmoving the origin,
which was possible by naming the wcorig lv, or naming no lv.
2020-09-02 14:45:52 -05:00
David Teigland
f5a669f314 pvck: repair should clear hints
repairing a pv can cause the hint file to become incorrect
2020-09-02 14:21:17 -05:00
David Teigland
739827ef1c tests: add new integrity reporting fields 2020-09-01 17:13:46 -05:00
Zdenek Kabelac
9a06700017 tests: skip this test for current 5.8 5.9 kernels
Kernel is hitting not yet fixed kernel bug.
Skip the test to avoid killing testing machine.
2020-09-01 23:40:24 +02:00
Zdenek Kabelac
fa1290f40e tests: slightly faster
Use lvm shell to agregrate lots of lvm commands
Reduce initial zeroing.
2020-09-01 23:40:24 +02:00
Zdenek Kabelac
38d460ed6e tests: set skipping autoactivation
On test system with 'default' filter  (aka accept all) test
after enabling device can suffer from automatic system
activation - so for created LVs setup skipping this automatic
activation. This should prevent getting LVs into table
with pvscan service.
2020-09-01 23:40:24 +02:00
Zdenek Kabelac
ada5728c72 tests: skip test when gcore cannot catch securetest 2020-09-01 23:40:24 +02:00
Zdenek Kabelac
2fbc578cfa tests: filefrag needs to support -e
Skip on systems with 'too old' filefrag without -e support
2020-09-01 17:57:48 +02:00
Zdenek Kabelac
ca54afd701 tests: check we detect lvm.conf read failure
No coredumps with unreadable lvm.conf.
2020-08-28 21:43:02 +02:00
Vojtech Trefny
8f1068c02d lvmdbusd: Add support for LVM writecache 2020-08-06 13:54:34 -05:00
Marian Csontos
e12bdd591a tests: Adapt RAID test to changes
Change 3c9177fdc0 causes a conversion of raid1 volume to a raid1 with
the same number of legs succeed with a warning.
2020-07-28 17:36:57 +02:00
David Teigland
085760992d cachevol: generate a unique name when creating
When a cachevol is automatically created, if the default name
conflicts with an existing name, generate a new unique name.
2020-07-23 13:18:22 -05:00
David Teigland
ad773511c5 integrity: add initial size to metadata size
The metadata device size needs to include space for
the dm-integrity "initial_sectors" which hold journals.
2020-06-30 16:43:05 -05:00
Zdenek Kabelac
3f32f9811e tests: check pool metadata are zeroed 2020-06-24 15:01:03 +02:00
Zdenek Kabelac
094d6f80dd tests: failure of zeroing fails command 2020-06-24 15:01:03 +02:00
David Teigland
ae5634a8be tests: cachevol-cachedevice 2020-06-22 11:23:58 -05:00
David Teigland
950d2d59c1 integrity: wait for raid sync to complete 2020-06-16 12:29:41 -05:00
David Teigland
48872b0369 integrity: avoid increasing logical block size of active LV
When adding integrity to an active LV, avoid choosing an
integrity block size that would result in increasing the
logical block size of the LV.
2020-06-16 12:27:22 -05:00
David Teigland
a014c4f341 tests: integrity and block size 2020-06-15 16:04:40 -05:00
David Teigland
9f38e95a2f tests: fix typo in writecache-blocksize 2020-06-11 13:15:38 -05:00
David Teigland
f32e85ae51 tests: expand integrity-blocksize 2020-06-11 12:46:47 -05:00
David Teigland
6ea3654868 tests: writecache tests
backport updates from later commits
2020-06-10 16:09:36 -05:00
David Teigland
48c1a295a2 tests: writecache-blocksize 2020-06-10 12:16:31 -05:00
David Teigland
240062a183 writecache: remove from an active lv 2020-06-10 12:13:31 -05:00
David Teigland
7b04ed07ba tests: integrity wait for sync
The test was using a raid+integrity LV without
first waiting for the integrity sync, which could
cause the test to fail (depending on init speed)
where it depends on integrity to work in uninitialized
areas.

Also use cmp instead of diff.
2020-05-29 10:57:56 -05:00
Marian Csontos
be61bd6ff5 test: Warn and exit on problematic integrity device behavior
The first leg of integrity enabled raid device sometimes does not get
recalculated.
2020-05-28 17:04:35 +02:00
David Teigland
74a211cfd3 lvconvert: error when using existing cachevol
Check if LV is an existing cachevol before attempting
to use it again as a cachevol or cachepool.
2020-05-22 14:12:34 -05:00
Zdenek Kabelac
bb41ca86fa tests: also udev wait on clean-up path 2020-05-21 16:03:41 +02:00
Marian Csontos
53803821de test: Use printf to generate data
...to avoid unnecessary dependency on python
2020-05-21 15:33:24 +02:00
Marian Csontos
b5811b7c9c tests: Use python single liner to generate data 2020-05-21 15:11:22 +02:00
Zdenek Kabelac
d3b515cea5 tests: add wait on udev processing
Trying to avoid collision with udev watch rule preventing to
succeed 'dmsetup remove' becuase it keeps device open.
2020-05-20 16:01:20 +02:00
David Teigland
d79afd4084 lvmcache: rework handling of VGs with duplicate vgnames
The previous method of managing duplicate vgnames prevented
vgreduce from working if a foreign vg with the same name
existed.
2020-04-21 14:40:34 -05:00
David Teigland
d9e8895a96 Allow dm-integrity to be used for raid images
dm-integrity stores checksums of the data written to an
LV, and returns an error if data read from the LV does
not match the previously saved checksum.  When used on
raid images, dm-raid will correct the error by reading
the block from another image, and the device user sees
no error.  The integrity metadata (checksums) are stored
on an internal LV allocated by lvm for each linear image.
The internal LV is allocated on the same PV as the image.

Create a raid LV with an integrity layer over each
raid image (for raid levels 1,4,5,6,10):

lvcreate --type raidN --raidintegrity y [options]

Add an integrity layer to images of an existing raid LV:

lvconvert --raidintegrity y LV

Remove the integrity layer from images of a raid LV:

lvconvert --raidintegrity n LV

Settings

Use --raidintegritymode journal|bitmap (journal is default)
to configure the method used by dm-integrity to ensure
crash consistency.

Initialization

When integrity is added to an LV, the kernel needs to
initialize the integrity metadata/checksums for all blocks
in the LV.  The data corruption checking performed by
dm-integrity will only operate on areas of the LV that
are already initialized.  The progress of integrity
initialization is reported by the "syncpercent" LV
reporting field (and under the Cpy%Sync lvs column.)

Example: create a raid1 LV with integrity:

$ lvcreate --type raid1 -m1 --raidintegrity y -n rr -L1G foo
  Creating integrity metadata LV rr_rimage_0_imeta with size 12.00 MiB.
  Logical volume "rr_rimage_0_imeta" created.
  Creating integrity metadata LV rr_rimage_1_imeta with size 12.00 MiB.
  Logical volume "rr_rimage_1_imeta" created.
  Logical volume "rr" created.
$ lvs -a foo
  LV                  VG  Attr       LSize  Origin              Cpy%Sync
  rr                  foo rwi-a-r---  1.00g                     4.93
  [rr_rimage_0]       foo gwi-aor---  1.00g [rr_rimage_0_iorig] 41.02
  [rr_rimage_0_imeta] foo ewi-ao---- 12.00m
  [rr_rimage_0_iorig] foo -wi-ao----  1.00g
  [rr_rimage_1]       foo gwi-aor---  1.00g [rr_rimage_1_iorig] 39.45
  [rr_rimage_1_imeta] foo ewi-ao---- 12.00m
  [rr_rimage_1_iorig] foo -wi-ao----  1.00g
  [rr_rmeta_0]        foo ewi-aor---  4.00m
  [rr_rmeta_1]        foo ewi-aor---  4.00m
2020-04-15 12:10:32 -05:00
Zdenek Kabelac
3dd11d9ea8 test: repair of thin-pool used by foreign apps 2020-04-08 15:37:24 +02:00
Marian Csontos
e6b93dc24e test: Fix previous commit 2020-03-18 18:03:12 +01:00
Marian Csontos
fc32787c1b test: Can not attach writecache to active volume 2020-03-18 14:35:58 +01:00
David Teigland
4b5bfa779a tests: reduce sizes in pvck-dump and improve checks
Smaller devs can be used so tests can be run on small vms.
Improve checks.
2020-03-04 11:30:50 -06:00
David Teigland
f6667f94cb tests: pvck dump from larger metadata areas 2020-03-03 13:47:07 -06:00
Zdenek Kabelac
9532bb577a tests: validate vdo slab_size
New vdoformat can print this size - so check we pass proper bit count
matching preset value.
2020-02-26 13:29:21 +01:00
David Teigland
4829f27b76 writecache: drop real dm suffix
fixes the problem of adding writecache to an active LV
2020-02-17 13:07:06 -06:00
David Teigland
2a6078f961 writecache: fix splitcache when origin is raid 2020-02-04 16:12:09 -06:00
Zdenek Kabelac
62ad12d0d0 tests: compatible with older kernels
Older kernels just show syscall stacktrace.
2020-02-04 17:22:06 +01:00
David Teigland
c0de37ab18 tests: move vgsplit writecache
to a different file
2020-02-04 10:20:19 -06:00
David Teigland
379a7e1288 vgsplit: handle cachevol
attached to a cache or writecache LV.
Ensure PVs in cachevol are moved with the main LV.
2020-02-03 15:33:58 -06:00
Zdenek Kabelac
bab3b70e3a tests: add corruption write on PV test
Test a case where PV is readable, but fails on write updating.
Check the failure is reported only for a single PV.
2020-01-23 10:32:15 +01:00
Zdenek Kabelac
ecb77e9db3 tests: writeerror_dev
Intruduce aux function for easy simulation of disk areas,
that are 'normally' readable, but will fail on write.
2020-01-23 10:32:15 +01:00
Heming Zhao
2f6d0a6408 fix corosync.conf: no interface error
systemctl status corosync (version: 2.4.5) report error:
  parse error in config: No interfaces defined

Signed-off-by: Zhao Heming <heming.zhao@suse.com>
2020-01-15 09:46:54 -06:00
Zdenek Kabelac
30f4d0fcd4 tests: update for readahead 2020-01-13 17:42:31 +01:00
Vojtech Trefny
c496ba6505 lvmdbusd: Add function to convert LV into a VDO pool 2020-01-09 13:07:55 -06:00
Vojtech Trefny
c3ef41f620 lvmdbusd: Add VDO enable/disable compress & dedup
Added methods to vdo pool interface to allow enabling and
disabling of VDO:
 * Compression
 * Deduplication
2020-01-09 13:07:47 -06:00
Marian Csontos
87e88078c9 tests: Some lvmdbus tests require larger PVs 2019-12-18 15:33:58 +01:00
Marian Csontos
a2a993d995 tests: VDO detection in dbus tests 2019-12-16 12:06:42 +01:00
Zdenek Kabelac
4a52855899 tests: improve secure test
Validate we capture core while original task sleeps.
2019-12-10 15:44:16 +01:00
Zdenek Kabelac
611d4107a4 test: fix missing waiting on udev
After device creation we need to wait for a cookie so it's not forgotten
in the system.
2019-12-10 15:44:16 +01:00
Zdenek Kabelac
3b6defcf1f test: fail on device create
Correct validation of prepared device and fail if the device can't
be created.
2019-12-10 15:44:16 +01:00
Zdenek Kabelac
8ab1d489f3 test: aux setup
Avoid endless loop if there was no 'remove' progress.
2019-12-10 15:44:16 +01:00
David Teigland
3145a85583 pvck: repair headers and metadata
To write a new/repaired pv_header and label_header:

  pvck --repairtype pv_header --file <file> <device>

This uses the metadata input file to find the PV UUID,
device size, and data offset.

To write new/repaired metadata text and mda_header:

  pvck --repairtype metadata --file <file> <device>

This requires a good pv_header which points to one or two
metadata areas.  Any metadata areas referenced by the
pv_header are updated with the specified metadata and
a new mda_header. "--settings mda_num=1|2" can be used
to select one mda to repair.

To combine all header and metadata repairs:

  pvck --repair --file <file> <device>

It's best to use a raw metadata file as input, that was
extracted from another PV in the same VG (or from another
metadata area on the same PV.)  pvck will also accept a
metadata backup file, but that will produce metadata that
is not identical to other metadata copies on other PVs
and other areas.  So, when using a backup file, consider
using it to update metadata on all PVs/areas.

To get a raw metadata file to use for the repair, see
pvck --dump metadata|metadata_search.

List all instances of metadata from the metadata area:
  pvck --dump metadata_search <device>

Save one instance of metadata at the given offset to
the specified file (this file can be used for repair):

  pvck --dump metadata_search --file <file>
    --settings "metadata_offset=<off>" <device>
2019-11-27 11:13:47 -06:00
David Teigland
b400353c71 tests hints: update check for io count
Running a reporting command on a VG now includes one
additional read to check the mda_header for any change
to the vg between scan and lock.
2019-11-26 16:52:28 -06:00
David Teigland
bbd8badaef tests: update to md dev name
Restore WAIT_MD_DEV in teardown.

NOTE: The name of MD device may have changed.

(cherry picked from commit c2ff8876f9)
2019-11-26 10:34:51 +01:00
Marian Csontos
b690258518 tests: Find md name using lsblk
After stopping MD device and rescanning the leg, it is created with
different name.
2019-11-26 09:13:17 +01:00
Marian Csontos
4757ce4c2a Partial revert "tests: update to md dev name"
This partially reverts commit c2ff8876f9.

Not all MD devices are stopped. Something is missing there...
2019-11-25 09:23:02 +01:00
Marian Csontos
1e669ab315 test: Fix handling leftovers from previous tests
teardown fails current PREFIX is prefix of previously failed test with
leftovers in dmtable.
2019-11-20 15:27:03 +01:00
Zdenek Kabelac
f88f7c0fdc tests: add more tracing info 2019-11-15 12:37:44 +01:00
Zdenek Kabelac
496c368528 tests: reduce amount of written date
Since we reduced created LV to 4M - dd also just 4M.
2019-11-15 12:37:44 +01:00
Zdenek Kabelac
91df257b53 tests: enusure lib is initilized 2019-11-14 18:06:42 +01:00
Zdenek Kabelac
38617213f0 tests: add test of resize of different segtypes 2019-11-11 22:44:25 +01:00
Zdenek Kabelac
3b05fd4d07 tests: add extra settle
To avoid removing, while 'add' might not have been processed yet.
(when emulating reboot in pvmove-restart)
2019-11-11 22:44:25 +01:00
Zdenek Kabelac
ba313ac84a tests: skip unneeded status check
If 'remove' was succesful - we can break loop immediatelly.
2019-11-11 22:44:25 +01:00
Marian Csontos
8263e62b00 test: Fix metadata-zero-space with long VG names 2019-11-06 16:20:27 +01:00
Zdenek Kabelac
c67a03727b tests: be happy with less then 90 percent
Thin metadata evolve between kernel version, so it's not always
precisely predictible its usage - so let's met test happy,
when it gets bellow 90%.
2019-11-01 16:31:12 +01:00
Zdenek Kabelac
dbaa4cdcdf tests: skip test if scsi_debug is not available 2019-11-01 16:31:12 +01:00
Zdenek Kabelac
5f064e2221 tests: slowdown delay of raid
Slowdown 'delay' more.
2019-11-01 16:31:12 +01:00
Zdenek Kabelac
c935f8a327 tests: conversion only of exclusive lv
We can 'cache' only exclusively active LV in cluster.
2019-11-01 16:27:21 +01:00
Zdenek Kabelac
d1c5b3ae74 tests: avoid checking command result in cluster
When running cluster test with clvmd, the actual 'monitoring'
happens in cluster - so the 'already monitored' message
is also logged within clvmd code and the command cannot
see such effect.

clvmd was incapable to report this information back to command
so it cannot be displayed this way.

Add 'lvs -o+seg_monitor' validation which also works in clustered mode.
2019-11-01 16:27:21 +01:00
Zdenek Kabelac
50b50039d9 tests: reduce space requirements
Test well runs on smaller test machines.
2019-10-31 15:31:30 +01:00
Tony Asleson
508d1808b0 lvmdbustest.py: Use local data instead of fetching
Avoid making more dbus calls to get information we already have.  This
also avoids us getting an error where a dbus object representation is
being deleted by another process while we are trying to gather information
about it across the wire.
2019-10-30 10:38:40 -05:00
Tony Asleson
f91df163e2 lvmdbustest.py: Improve concurrent test handling
Filter out LVs too, so that we can run more than 1 instance of the
unit test at the same time.
2019-10-30 10:38:40 -05:00
Tony Asleson
f961311436 lvmdbustest.py: Add tests for LV interface
Add tests for all the different LV types with the standard LV dbus
interface.  These tests shook out a couple of new bugs.
2019-10-30 10:38:40 -05:00
Tony Asleson
9e15c83673 testlib.py: Add interface instance vars. 2019-10-30 10:38:40 -05:00
Tony Asleson
b7aab9ba59 testlib.py: WS corrections 2019-10-30 10:38:40 -05:00
Tony Asleson
89373761c8 lvmdbustest.py: Add basic vdo test 2019-10-30 10:38:40 -05:00
Tony Asleson
ed7e365ae5 testlib.py: Correct dbus signature verification
This allows us to fully verify introspection data matches what we are
getting.
2019-10-30 10:38:40 -05:00
Tony Asleson
5971da2c72 lvmdbusd: VDO Pool LV representation
VDO pool LVs are represented by a new dbus interface VgVdo.  Currently
the interface only has additional VDO properties, but when the
ability to support additional LV creation is added we can add a method
to the interface.
2019-10-30 10:38:40 -05:00
Tony Asleson
455498f206 lvmdbustest.py: Create common func. _create_cache_lv 2019-10-30 10:38:40 -05:00
Tony Asleson
c786636afb lvmdbustest.py: Add nested helper function major_minor 2019-10-30 10:38:40 -05:00
Tony Asleson
e1d3a6c552 lvmdbustest.py: WS corrections 2019-10-30 10:38:40 -05:00
Tony Asleson
5b224d58f7 lvmdbustest.py: Add cache LV rename test 2019-10-30 10:38:40 -05:00
Tony Asleson
9d2ef05c5d lvmdbusd: Add cfg.vdo_support
Will be used to add vdo interfaces on demand.
2019-10-30 10:38:40 -05:00
Tony Asleson
ceb808d26f lvmdbustest.py: Remove 2 TODOs
This issue has been resolved, sizes > 2**32-1 not supported.
2019-10-30 10:38:40 -05:00
Tony Asleson
c5f4f2efb6 lvmdbustest.py: Add func. _pv_scan 2019-10-30 10:38:40 -05:00
Tony Asleson
69d4847975 lvmdbustest.py: Use existing _create_lv 2019-10-30 10:38:40 -05:00
Tony Asleson
293f6d2795 lvmdbustest.py: Add func. _create_thin_lv 2019-10-30 10:38:40 -05:00
Tony Asleson
a4666f63ad lvmdbustest.py: Add func. _all_pv_object_paths
This is needed in a number of places.
2019-10-30 10:38:40 -05:00
Tony Asleson
b8d4969117 lvmdbustest.py: Add function for lv path check 2019-10-30 10:38:40 -05:00
Tony Asleson
22a22a735f lvmdbusdtest.py: Use common function for tag add 2019-10-30 10:38:40 -05:00
Tony Asleson
62136c056a lvmdbustest.py: Remove duplicate setup code
Remove the same copy & pasted code which simply creates a VG to
use.
2019-10-30 10:38:40 -05:00
Tony Asleson
bafe5d15b1 lvmdbusd: Add check for reserved name '_vdata'
Added for vdo support.
2019-10-30 10:38:40 -05:00
Tony Asleson
b7c64fe8e2 lvmdbustest.py: Add blurb about scan_lvs = 1
When developing and testing on a local system, to get the unit
test to pass the test_nesting test, editing the test conf will achieve
the success too.
2019-10-30 10:38:40 -05:00
Joe Thornber
5fdebf9bbf [bcache] add unit test
abort-forces-read
2019-10-29 10:33:31 +00:00
Joe Thornber
2938b4dcca [bcache] add bcache_abort()
This gives us a way to cope with write failures.
2019-10-28 15:00:53 +00:00
Zdenek Kabelac
e88fd2edfd tests: explicit testing of thin snapshot
Check merging of old snapshot of thin LV.
2019-10-26 00:49:16 +02:00
David Teigland
221edf4030 tests: lvmlockd-lv-types handle new cpool renaming 2019-10-24 13:26:33 -05:00
Zdenek Kabelac
b4e87e638e tests: enable with 6.2.1
Package vdo-6.2.2.18  goes with  kvdo version 6.2.1
Hopefully will get fixed soon.
Meanwhile to get test coverage...
2019-10-22 16:06:19 +02:00
Zdenek Kabelac
65efacd0e0 tests: more suffixes 2019-10-22 13:39:39 +02:00
Zdenek Kabelac
a0e40bad1d tests: used cache_pool adds _cpool 2019-10-21 15:32:35 +02:00
David Teigland
5714c8c9cc pvck: dump metadata search
Improve the implementation of extracting all text metadata
copies from the metadata area.  Use this for the existing
metadata_all dump option.

Add a new metadata_search dump option which does not use
lvm headers to find metadata, but looks in standard
locations.  This is useful if headers are damaged and
can't be used to locate metadata.

Adding '-v' to metadata_all or metadata_search will add
the description and creation_time to the printed list of
metadata instances that are found.
2019-10-18 12:26:29 -05:00
Zdenek Kabelac
4f860f79ff tests: cachevol needs FORMAT2
FORMAT2 is mandatory with cachevols...
2019-10-17 13:03:50 +02:00
Zdenek Kabelac
f07e5c1fe4 tests: skip when raid is not present 2019-10-17 13:03:50 +02:00
David Teigland
55bf692bff tests: metadata-zero-space long vgnames
In the hex dump output, grep for the vgname
followed by one space.  This allows for test pids
with up to seven digits, which are used to contruct
the variable vgname used by the test.  Otherwise
the long vgname wraps to the next line and fails to
match in grep.
2019-10-16 09:46:48 -05:00
David Teigland
998e7b1075 writecache: add cvol suffix to attached cachevol
When an LV is used as a writecache cachevol, give
it the LV name a _cvol suffix.  Remove the suffix
when the cachevol is detached, restoring the
original LV name.
2019-10-15 16:03:34 -05:00
Zdenek Kabelac
6666c39346 tests: cvol 2019-10-14 15:20:25 +02:00
Zdenek Kabelac
c70bab7531 tests: drop manipulation with 'internal' LV
Cachevol LV is 'internal' LV and should not be 'maintained'  this way
while it's being used as cachevol.
2019-10-14 15:20:25 +02:00
Zdenek Kabelac
e6691aa04b tests: disable caching of VDO
Disable until vdo gets fixed, then require fixed version.
Otherwise test kills kernel
2019-10-14 15:13:47 +02:00
Marian Csontos
df26b73b8e test: Minimize data written by metadata-zero-space
The test is checking only 2 megabytes, but is rewriting much more. Bad
for limited ramdisk.
2019-10-09 15:11:03 +02:00
David Teigland
a22729a557 tests: improve lib wipefs_a
in wipefs fallback, use limited dd and mdadm --zero-superblock
to avoid writing to entire dev in case dev space is provisioned
on write.
2019-10-04 11:06:38 -05:00
David Teigland
d693041149 tests: add wipefs wrapper
which falls back to using dd on old systems without
the wipefs command
2019-10-02 14:11:08 -05:00
David Teigland
c2ff8876f9 tests: update to md dev name
Use /dev/md33 instead of /dev/md0 to reduce chances of
conflicting with an existing name.

Only call 'mdadm --stop /dev/md33' for cleanup and don't
use 'mdadm --stop --scan' to avoid stopping other md devs.
2019-10-02 14:09:51 -05:00
Heinz Mauelshagen
b138a87f43 test: increase size of raid10 LV allowing tests to succeed on fast storage
Also add health char check.
2019-10-02 15:25:43 +02:00
Heinz Mauelshagen
242e2dccc3 test: add RAID lvextend resynchronization test
Due to a dm-raid target flaw fixed in target version 1.15.0,
extents of raid sets don't get resynchronized when new MD bitmp
pages have to be allocated due to the extension.

Introduce lvextend-raid.sh to test this flaw.

Related: rhbz1671964
2019-10-02 15:25:41 +02:00
Heinz Mauelshagen
c4aba47dd0 test: add checks for not 100% sync ratio after initiation of check/repair
Related: rhbz1640630
2019-10-02 15:25:30 +02:00
David Teigland
7368cf8e7d pvck: handle PVs with zero metadata copies 2019-09-30 16:20:17 -05:00
David Teigland
1bfae5bf8e tests: update duplicate md tests
adjust to recent improvements in duplicate handling
2019-09-30 11:38:10 -05:00
David Teigland
fd5b8b72da tests: mdadm stop in test cleanup
try to clear any existing md devs remaining after
a test
2019-09-27 12:51:34 -05:00
David Teigland
9680fd6d78 tests: skip md tests if system not clean 2019-09-27 12:40:42 -05:00
David Teigland
0673d13518 tests: writecache-split check have_writecache 2019-09-26 10:45:53 -05:00
David Teigland
3b36de573e tests: update cache-single-split
add some uncache commands
2019-09-24 15:51:05 -05:00
David Teigland
e35cf0f623 tests: add writecache-split 2019-09-23 16:35:01 -05:00
David Teigland
a353bfdd9f tests: update cache-single-split
enable splitting while LV is active which works
again since the cvol suffix was dropped.
2019-09-23 14:21:09 -05:00
David Teigland
338a60bb41 tests: update cache-single-split
the new use of uuid suffixes on internal dm devs
has impacted splitting when an LV is active,
so temporarily disable that until fixed.
2019-09-20 16:36:14 -05:00
David Teigland
8d8b5e4f75 tests: add cache-single-split 2019-09-20 09:59:37 -05:00
Zdenek Kabelac
38a3da532f tests: check vdo 2019-09-17 13:17:21 +02:00
Zdenek Kabelac
ee6b9b78ec tests: pool conversion
Test conversion of cached LV.
2019-09-17 13:13:49 +02:00
David Teigland
13fb57bbb1 Revert "tests: drop unnecessary pid from LVMTEST prefix"
This reverts commit 51c105702f.

it seems the test system still needs this for some reason
2019-09-16 13:01:50 -05:00
David Teigland
51c105702f tests: drop unnecessary pid from LVMTEST prefix 2019-09-12 14:52:13 -05:00
David Teigland
df36a8eadf tests: don't change real test output 2019-09-12 14:45:13 -05:00
David Teigland
196a9ed59f tests: metadata-zero-space more debugging 2019-09-12 14:38:20 -05:00
David Teigland
39a0de68bd tests: unnecessary braces causes test failures 2019-09-12 12:55:00 -05:00
David Teigland
585db87274 tests: skip some without wipefs 2019-09-12 11:23:30 -05:00
David Teigland
82b887a4cf tests: more debug in metadata-zero-space 2019-09-12 11:19:44 -05:00
David Teigland
1fe04f89ef tests: use standard md devices 2019-09-12 10:11:27 -05:00
David Teigland
772dbb1e39 tests: metadata-zero-space add info for debugging 2019-09-11 14:20:59 -05:00
David Teigland
447ba5f1c2 tests: skip without xxd 2019-09-11 14:01:55 -05:00
David Teigland
31b6a04e4c tests: open-file-limit skip without prlimit 2019-09-11 14:01:55 -05:00
David Teigland
69b7c00a77 tests: allow-mixed-block-sizes skip with older losetup 2019-09-11 14:01:55 -05:00
David Teigland
715d2c778f tests: cache-single tests require version 1 10 2019-09-11 14:01:55 -05:00
David Teigland
acb8050a30 tests: metadata-bad-text.sh with lvmlockd
skip part of test that doesn't apply
2019-09-05 15:27:52 -05:00
David Teigland
25b58310e3 pvscan: avoid full scan for activation
When an online PV completed a VG, the standard
activation functions were used to activate the VG.
These functions use a full scan of all devs.
When many pvscans are run during startup and need
to activate many VGs, scanning all devs from all
the pvscans can take a long time.

Optimize VG activation in pvscan to scan only the
devs in the VG being activated.  This makes use of
the online file info that was used to determine
the VG was complete.

The downside of this approach is that pvscan activation
will not detect duplicate PVs and block activation,
where a normal activation command (which scans all
devices) would.
2019-09-03 10:11:16 -05:00
David Teigland
987f2733c7 tests: duplicate-pvs-md
Testing duplicate PV handling for various md device setups/states.
2019-08-16 13:26:12 -05:00
Zdenek Kabelac
e653f43732 tests: simplify some var settings
scan_lvs now automatically comes with extend_filter_LVMTEST
2019-08-09 12:57:21 +02:00
Zdenek Kabelac
05c7e09e71 tests: for cluster testing we always need exclusive mirrors
Cluster test would fail without exlusively activated mirror.
FIXME: test is still broken, since LV2 is not active after split.
2019-08-09 12:57:21 +02:00
Zdenek Kabelac
505c1d56e8 tests: accept also value 512
Older kernels provide fixed value 512 instead of 0.
2019-08-09 12:57:21 +02:00
Zdenek Kabelac
1e553159d5 tests: add settle wait before issue remove
Some older BB with older cryptsetup tool do not 'retry' on remove
and when  remove is issued right after 'fsck'  - it might be
rejected with:

Device @PREFIX@-tcrypt2 is busy.

Try to use udevadm settle.
2019-08-09 12:57:21 +02:00
Zdenek Kabelac
d3903d94e9 tests: fix ra checking
Since with some installed package like 'tuned' the value of 'RA' on
PV origin device can be different, adapting tests to count with this.
2019-08-09 12:57:21 +02:00
Zdenek Kabelac
3596210e02 tests: replaces grep -q usage
Since we use 'set -euE -o pipefail' for shell execution,
any failure of any command in the 'piped' shell can result
in failure of whole executed chain - resulting in typically
unsually test skip, that was left unnoticed.

Since checked command have usually short output, the simplest
fix seems to be to let grep parse whole output instead
of quiting after first match.
2019-08-09 12:57:21 +02:00
David Teigland
3fcb4697f6 tests: open-file-limit 2019-08-08 15:45:03 -05:00
David Teigland
cd8a0133fa tests: allow-mixed-block-sizes 2019-08-01 10:13:41 -05:00
David Teigland
e3ba8561fa tests: large-physical-sector-size 2019-07-30 15:59:11 -05:00
Marian Csontos
dd19fa9ff3 tests: Fix unbound variable
Test `aux kernel_at_least 5 1` fails even for newer kernel
with `$3: unbound variable` when using `set -u`.
2019-07-24 16:30:15 +02:00
David Teigland
aa58f9bd9b tests: lvm-on-md use variable run dir
for hints file
2019-07-12 16:51:49 -05:00
David Teigland
bbca70a0b7 tests: metadata-zero-space
Test zero padding between copies of metadata.
2019-07-12 15:03:47 -05:00
David Teigland
7230aa891c tests: pvscan-autoactivate test unmatching dev and PV size 2019-07-11 11:38:12 -05:00
David Teigland
4eb0e65693 tests: extend lvm-on-md 2019-07-11 11:20:06 -05:00
David Teigland
f938545687 cache: warn and prompt for writeback with cachevol
The cache repair utility does not yet work with a cachevol
(where metadata and data exist on the same LV.)  So, warn
and prompt if writeback is specified with a cachevol.
2019-07-02 11:03:03 -05:00
Marian Csontos
ba9d152aa5 test: Remove now useless clvmd test 2019-06-27 11:14:00 +02:00
Marian Csontos
09f29570f2 test: Fix unbound variable
Test `aux kernel_at_least 5 1` fails even for newer kernel
with `$3: unbound variable` when using `set -u`.
2019-06-27 10:41:21 +02:00
David Teigland
9ba45d824a tests: add exported.sh
to test how commands work with exported VGs/PVs.
2019-06-25 15:45:47 -05:00
David Teigland
8fecd9c14e metadata: include description with command in metadata areas
Previously the VG metadata description field (which contains
the command line) was only included in backup/archive copies
of the metadata.  Now also include it in the metadata written
to the metadata areas.
2019-06-20 16:09:05 -05:00
David Teigland
208a09745d tests: aux have_writecache
function was never defined, causing writecache.sh to be skipped
2019-06-13 11:36:18 -05:00
Zdenek Kabelac
c9203a6106 tests: correct checked target name
So when the target name happened to be a suffix of another one,
the grep was filtering incorrect line
(i.e. dm-cache && dm-writecache) - so do a line head matching.
2019-06-11 16:43:14 +02:00
David Teigland
d7c1168c6a tests: add metadata-bad-mdaheader.sh
needs xxd command
2019-06-07 15:54:04 -05:00
David Teigland
878741502a tests: add metadata-bad-text.sh 2019-06-07 15:54:04 -05:00
David Teigland
4fa1638301 tests: add outdated-pv.sh 2019-06-07 15:54:04 -05:00
David Teigland
9156640b60 tests: add metadata-old.sh 2019-06-07 15:54:04 -05:00
David Teigland
d3636ff832 tests: add missing-pv missing-pv-unused 2019-06-07 15:54:04 -05:00
David Teigland
ba7ff96faf improve reading and repairing vg metadata
The fact that vg repair is implemented as a part of vg read
has led to a messy and complicated implementation of vg_read,
and limited and uncontrolled repair capability.  This splits
read and repair apart.

Summary
-------

- take all kinds of various repairs out of vg_read
- vg_read no longer writes anything
- vg_read now simply reads and returns vg metadata
- vg_read ignores bad or old copies of metadata
- vg_read proceeds with a single good copy of metadata
- improve error checks and handling when reading
- keep track of bad (corrupt) copies of metadata in lvmcache
- keep track of old (seqno) copies of metadata in lvmcache
- keep track of outdated PVs in lvmcache
- vg_write will do basic repairs
- new command vgck --updatemetdata will do all repairs

Details
-------

- In scan, do not delete dev from lvmcache if reading/processing fails;
  the dev is still present, and removing it makes it look like the dev
  is not there.  Records are now kept about the problems with each PV
  so they be fixed/repaired in the appropriate places.

- In scan, record a bad mda on failure, and delete the mda from
  mda in use list so it will not be used by vg_read or vg_write,
  only by repair.

- In scan, succeed if any good mda on a device is found, instead of
  failing if any is bad.  The bad/old copies of metadata should not
  interfere with normal usage while good copies can be used.

- In scan, add a record of old mdas in lvmcache for later, do not repair
  them while reading, and do not let them prevent us from finding and
  using a good copy of metadata from elsewhere.  One result is that
  "inconsistent metadata" is no longer a read error, but instead a
  record in lvmcache that can be addressed separate from the read.

- Treat a dev with no good mdas like a dev with no mdas, which is an
  existing case we already handle.

- Don't use a fake vg "handle" for returning an error from vg_read,
  or the vg_read_error function for getting that error number;
  just return null if the vg cannot be read or used, and an error_flags
  arg with flags set for the specific kind of error (which can be used
  later for determining the kind of repair.)

- Saving an original copy of the vg metadata, for purposes of reverting
  a write, is now done explicitly in vg_read instead of being hidden in
  the vg_make_handle function.

- When a vg is not accessible due to "access restrictions" but is
  otherwise fine, return the vg through the new error_vg arg so that
  process_each_pv can skip the PVs in the VG while processing.
  (This is a temporary accomodation for the way process_each_pv
  tracks which devs have been looked at, and can be dropped later
  when process_each_pv implementation dev tracking is changed.)

- vg_read does not try to fix or recover a vg, but now just reads the
  metadata, checks access restrictions and returns it.
  (Checking access restrictions might be better done outside of vg_read,
   but this is a later improvement.)

- _vg_read now simply makes one attempt to read metadata from
  each mda, and uses the most recent copy to return to the caller
  in the form of a 'vg' struct.
  (bad mdas were excluded during the scan and are not retried)
  (old mdas were not excluded during scan and are retried here)

- vg_read uses _vg_read to get the latest copy of metadata from mdas,
  and then makes various checks against it to produce warnings,
  and to check if VG access is allowed (access restrictions include:
  writable, foreign, shared, clustered, missing pvs).

- Things that were previously silently/automatically written by vg_read
  that are now done by vg_write, based on the records made in lvmcache
  during the scan and read:
  . clearing the missing flag
  . updating old copies of metadata
  . clearing outdated pvs
  . updating pv header flags

- Bad/corrupt metadata are now repaired; they were not before.

Test changes
------------

- A read command no longer writes the VG to repair it, so add a write
  command to do a repair.
  (inconsistent-metadata, unlost-pv)

- When a missing PV is removed from a VG, and then the device is
  enabled again, vgck --updatemetadata is needed to clear the
  outdated PV before it can be used again, where it wasn't before.
  (lvconvert-repair-policy, lvconvert-repair-raid, lvconvert-repair,
   mirror-vgreduce-removemissing, pv-ext-flags, unlost-pv)

Reading bad/old metadata
------------------------

- "bad metadata": the mda_header or metadata text has invalid fields
  or can't be parsed by lvm.  This is a form of corruption that would
  not be caused by known failure scenarios.  A checksum error is
  typically included among the errors reported.

- "old metadata": a valid copy of the metadata that has a smaller seqno
  than other copies of the metadata.  This can happen if the device
  failed, or io failed, or lvm failed while commiting new metadata
  to all the metadata areas.  Old metadata on a PV that has been
  removed from the VG is the "outdated" case below.

When a VG has some PVs with bad/old metadata, lvm can simply ignore
the bad/old copies, and use a good copy.  This is why there are
multiple copies of the metadata -- so it's available even when some
of the copies cannot be used.  The bad/old copies do not have to be
repaired before the VG can be used (the repair can happen later.)

A PV with no good copies of the metadata simply falls back to being
treated like a PV with no mdas; a common and harmless configuration.

When bad/old metadata exists, lvm warns the user about it, and
suggests repairing it using a new metadata repair command.
Bad metadata in particular is something that users will want to
investigate and repair themselves, since it should not happen and
may indicate some other problem that needs to be fixed.

PVs with bad/old metadata are not the same as missing devices.
Missing devices will block various kinds of VG modification or
activation, but bad/old metadata will not.

Previously, lvm would attempt to repair bad/old metadata whenever
it was read.  This was unnecessary since lvm does not require every
copy of the metadata to be used.  It would also hide potential
problems that should be investigated by the user.  It was also
dangerous in cases where the VG was on shared storage.  The user
is now allowed to investigate potential problems and decide how
and when to repair them.

Repairing bad/old metadata
--------------------------

When label scan sees bad metadata in an mda, that mda is removed
from the lvmcache info->mdas list.  This means that vg_read will
skip it, and not attempt to read/process it again.  If it was
the only in-use mda on a PV, that PV is treated like a PV with
no mdas.  It also means that vg_write will also skip the bad mda,
and not attempt to write new metadata to it.  The only way to
repair bad metadata is with the metadata repair command.

When label scan sees old metadata in an mda, that mda is kept
in the lvmcache info->mdas list.  This means that vg_read will
read/process it again, and likely see the same mismatch with
the other copies of the metadata.  Like the label_scan, the
vg_read will simply ignore the old copy of the metadata and
use the latest copy.  If the command is modifying the vg
(e.g. lvcreate), then vg_write, which writes new metadata to
every mda on info->mdas, will write the new metadata to the
mda that had the old version.  If successful, this will resolve
the old metadata problem (without needing to run a metadata
repair command.)

Outdated PVs
------------

An outdated PV is a PV that has an old copy of VG metadata
that shows it is a member of the VG, but the latest copy of
the VG metadata does not include this PV.  This happens if
the PV is disconnected, vgreduce --removemissing is run to
remove the PV from the VG, then the PV is reconnected.
In this case, the outdated PV needs have its outdated metadata
removed and the PV used flag needs to be cleared.  This repair
will be done by the subsequent repair command.  It is also done
if vgremove is run on the VG.

MISSING PVs
-----------

When a device is missing, most commands will refuse to modify
the VG.  This is the simple case.  More complicated is when
a command is allowed to modify the VG while it is missing a
device.

When a VG is written while a device is missing for one of it's PVs,
the VG metadata is written to disk with the MISSING flag on the PV
with the missing device.  When the VG is next used, it is treated
as if the PV with the MISSING flag still has a missing device, even
if that device has reappeared.

If all LVs that were using a PV with the MISSING flag are removed
or repaired so that the MISSING PV is no longer used, then the
next time the VG metadata is written, the MISSING flag will be
dropped.

Alternative methods of clearing the MISSING flag are:

vgreduce --removemissing will remove PVs with missing devices,
or PVs with the MISSING flag where the device has reappeared.

vgextend --restoremissing will clear the MISSING flag on PVs
where the device has reappeared, allowing the VG to be used
normally.  This must be done with caution since the reappeared
device may have old data that is inconsistent with data on other PVs.

Bad mda repair
--------------

The new command:
vgck --updatemetadata VG

first uses vg_write to repair old metadata, and other basic
issues mentioned above (old metadata, outdated PVs, pv_header
flags, MISSING_PV flags).  It will also go further and repair
bad metadata:

. text metadata that has a bad checksum
. text metadata that is not parsable
. corrupt mda_header checksum and version fields

(To keep a clean diff, #if 0 is added around functions that
are replaced by new code.  These commented functions are
removed by the following commit.)
2019-06-07 15:54:04 -05:00
David Teigland
db98a6e362 Additional MD component checking
If udev info is missing for a device, (which would indicate
if it's an MD component), then do an end-of-device read to
check if a PV is an MD component.  (This is skipped when
using hints since we already know devs in hints are good.)

A new config setting md_component_checks can be used to
disable the additional end-of-device MD checks, or to
always enable end-of-device MD checks.

When both hints and udev info are disabled/unavailable,
the end of PVs will now be scanned by default.  If md
devices with end-of-device superblocks are not being
used, the extra I/O overhead can be avoided by setting
md_component_checks="start".
2019-06-07 13:27:16 -05:00
David Teigland
c315112a3b tests: pvscan-autoactivate check for machine-id 2019-06-06 15:32:42 -05:00
David Teigland
356ea897cc tests: pvck-dump 2019-06-05 13:58:26 -05:00
Zdenek Kabelac
4d9f41b119 tests: check no_discard_passdown
Check reporting works
2019-06-05 15:48:44 +02:00
Zdenek Kabelac
ddd68fbead tests: automatically set scan_lvs when using extend_filter
When using 'aux extend_filter' we always want to use LV as PV.
2019-06-05 15:48:44 +02:00
Marian Csontos
669a834981 test: Increase latency in pvmove-resume-multiseg
The test was failing consistently on some VMs (F25), and inconsistently
on Rawhide.

With increased latency these failures are no longer reproducible.

Reproducer:

    make check_lvmpolld T=pvmove-resume-multiseg.sh
2019-06-03 16:57:49 +02:00
Marian Csontos
a9907bef99 test: Restore testing of D-Bus API 2019-05-31 08:58:30 +02:00
David Teigland
eebb5e9fff tests: add debug to pvscan-cache deactivation 2019-05-23 15:32:46 -05:00
David Teigland
e055b89d28 tests: pvscan-cache more attempts to fix 2019-05-23 14:55:57 -05:00
David Teigland
1022b88a66 tests: change mkfs usage in lvconvert raid tests
The "echo y | mkfs" was failing at times from echo y.
Remove echo y and replace with wipefs -a prior to mkfs.
2019-05-23 11:45:26 -05:00
David Teigland
6169c0a51b tests: fix error detection in lvconvert-raid-takeover.sh 2019-05-23 10:29:52 -05:00
David Teigland
2036608423 tests: pvscan-cache try to fix teardown problems
teardown after the test was failing, probably because
of uncoordinated udev actions running on the test
system.  Try to avoid this by doing some work before
teardown.
2019-05-22 11:55:48 -05:00
David Teigland
78afe75b08 tests: fsadm-crypt.sh update mkfs parameter
mkfs.xfs was rejecting previously working value
2019-05-21 14:46:01 -05:00
David Teigland
cf3f463929 tests: pvscan-autoactivate.sh switch system_id_source
to machineid instead of uname which would break if
the test system had no proper uname set.
2019-05-21 14:37:55 -05:00
David Teigland
99ca06ca46 tests: hints check if strace exists
avoid test failure if test system does not
have strace
2019-05-21 14:24:57 -05:00
Zdenek Kabelac
0c26aa13ca tests: check accepting out-of-range creation_time 2019-05-10 15:00:21 +02:00
Zdenek Kabelac
1f7c9da554 tests: split args
Here we want args to be splited into individual strings.
2019-05-06 13:02:45 +02:00
Zdenek Kabelac
4ff472b907 tests: drop call of wipefs
wipefs might not be present on test system.
Devices are also already zeroed by cleanup_md_dev
(which 'fakes' missing wipefs eventually)
2019-05-04 19:11:00 +02:00
David Teigland
6ff1583c1b tests: expand lvm-on-md
test both md raid0 and raid1
2019-05-03 14:39:42 -05:00
Zdenek Kabelac
ac627fd1ce tests: use luks1 for test
Since we do not need anywhere luks2 - pick older format
which does not require password for resize to keep
the rest of test unmodified.
2019-05-03 13:17:22 +02:00
Zdenek Kabelac
8c56e31134 tests: update resize value
Since we now properly extend also _pmspare - there was not enough free
space to add 8extents to both volumes.
2019-05-03 13:17:22 +02:00
David Teigland
8c87dda195 locking: unify global lock for flock and lockd
There have been two file locks used to protect lvm
"global state": "ORPHANS" and "GLOBAL".

Commands that used the ORPHAN flock in exclusive mode:
  pvcreate, pvremove, vgcreate, vgextend, vgremove,
  vgcfgrestore

Commands that used the ORPHAN flock in shared mode:
  vgimportclone, pvs, pvscan, pvresize, pvmove,
  pvdisplay, pvchange, fullreport

Commands that used the GLOBAL flock in exclusive mode:
  pvchange, pvscan, vgimportclone, vgscan

Commands that used the GLOBAL flock in shared mode:
  pvscan --cache, pvs

The ORPHAN lock covers the important cases of serializing
the use of orphan PVs.  It also partially covers the
reporting of orphan PVs (although not correctly as
explained below.)

The GLOBAL lock doesn't seem to have a clear purpose
(it may have eroded over time.)

Neither lock correctly protects the VG namespace, or
orphan PV properties.

To simplify and correct these issues, the two separate
flocks are combined into the one GLOBAL flock, and this flock
is used from the locking sites that are in place for the
lvmlockd global lock.

The logic behind the lvmlockd (distributed) global lock is
that any command that changes "global state" needs to take
the global lock in ex mode.  Global state in lvm is: the list
of VG names, the set of orphan PVs, and any properties of
orphan PVs.  Reading this global state can use the global lock
in sh mode to ensure it doesn't change while being reported.

The locking of global state now looks like:

lockd_global()
  previously named lockd_gl(), acquires the distributed
  global lock through lvmlockd.  This is unchanged.
  It serializes distributed lvm commands that are changing
  global state.  This is a no-op when lvmlockd is not in use.

lockf_global()
  acquires an flock on a local file.  It serializes local lvm
  commands that are changing global state.

lock_global()
  first calls lockf_global() to acquire the local flock for
  global state, and if this succeeds, it calls lockd_global()
  to acquire the distributed lock for global state.

Replace instances of lockd_gl() with lock_global(), so that the
existing sites for lvmlockd global state locking are now also
used for local file locking of global state.  Remove the previous
file locking calls lock_vol(GLOBAL) and lock_vol(ORPHAN).

The following commands which change global state are now
serialized with the exclusive global flock:

pvchange (of orphan), pvresize (of orphan), pvcreate, pvremove,
vgcreate, vgextend, vgremove, vgreduce, vgrename,
vgcfgrestore, vgimportclone, vgmerge, vgsplit

Commands that use a shared flock to read global state (and will
be serialized against the prior list) are those that use
process_each functions that are based on processing a list of
all VG names, or all PVs.  The list of all VGs or all PVs is
global state and the shared lock prevents those lists from
changing while the command is processing them.

The ORPHAN lock previously attempted to produce an accurate
listing of orphan PVs, but it was only acquired at the end of
the command during the fake vg_read of the fake orphan vg.
This is not when orphan PVs were determined; they were
determined by elimination beforehand by processing all real
VGs, and subtracting the PVs in the real VGs from the list
of all PVs that had been identified during the initial scan.
This is fixed by holding the single global lock in shared mode
while processing all VGs to determine the list of orphan PVs.
2019-04-29 13:01:05 -05:00
David Teigland
aa75b31db5 pvscan: handle case of scanning PV without metadata last
Handle the case where pvscan --cache -aay (with no dev args)
gets to the final PV, completing the VG, but that final PV does not
have VG metadata.  In this case, we need to use VG metadata from a
previously scanned PV in the same VG, which we saved for this
possibility.  Using this saved metadata, we can find which VG
this PVID belongs to, and then check if that VG is now complete,
and if so add the VG name to the list of complete VGs to be
autoactivated.
2019-04-15 11:27:49 -05:00
David Teigland
41ba2b568b tests: disable unworking pvscan case
and add corresponding fixme in the code
2019-04-12 15:40:38 -05:00
David Teigland
48e9f116ae tests: update pvscan-autoactivate for init change 2019-04-05 14:04:42 -05:00
Zdenek Kabelac
5d6fe796bd tests: check auto-growth of thin-pool meta 2019-04-03 13:28:56 +02:00
Zdenek Kabelac
7f757ab616 tests: vdo caching tests 2019-03-20 14:39:11 +01:00
Zdenek Kabelac
5139e5f1b3 tests: vdo dmevent autoresize 2019-03-20 14:39:11 +01:00
Zdenek Kabelac
ac31bfd6fd tests: check vgsplit works with cache 2019-03-20 14:38:05 +01:00
David Teigland
d84134c75b pvscan: fix ignoring foreign PVs
Fix to previous commit
  "pvscan: ignore online for shared and foreign PVs"

which was incorrectly considering a PV foreign if its
VG had no system ID when the host did have a system ID.
2019-03-13 16:03:02 -05:00
David Teigland
98b7a3a42d tests: check that pvscan --cache ignores certain PVs 2019-03-06 12:17:47 -06:00
David Teigland
a9eaab6beb Use "cachevol" to refer to cache on a single LV
and "cachepool" to refer to a cache on a cache pool object.

The problem was that the --cachepool option was being used
to refer to both a cache pool object, and to a standard LV
used for caching.  This could be somewhat confusing, and it
made it less clear when each kind would be used.  By
separating them, it's clear when a cachepool or a cachevol
should be used.

Previously:

- lvm would use the cache pool approach when the user passed
  a cache-pool LV to the --cachepool option.

- lvm would use the cache vol approach when the user passed
  a standard LV in the --cachepool option.

Now:

- lvm will always use the cache pool approach when the user
  uses the --cachepool option.

- lvm will always use the cache vol approach when the user
  uses the --cachevol option.
2019-02-27 08:52:34 -06:00
David Teigland
74a388cca1 pvscan: autoactivate a VG once
When a VG has multiple PVs, and all those PVs come online
at the same time, concurrent pvscans for each PV will all
create the individual pvid files, and all will often see
the VG is now complete.  This causes each of the pvscan
commands to think it should activate the VG, so there
are multiple activations of the same VG.  The vg lock
serializes them, and only the first pvscan actually does
the activation, but there is still a lot of extra overhead
and time used by the other pvscans that attempt to
activate the already active VG.  This can lead to a backlog
of pvscans and timeouts.

To fix this, this adds a new /run/lvm/vgs_online/ dir that
works like the existing /run/lvm/pvs_online/ dir.  Each pvscan
that wants to activate a VG will first try to exlusively create
the file vgs_online/<vgname>.  Only the first pvscan will
succeed, and that one will do the VG activation. The other
pvscans will find the vgname file exists and will not do the
activation step.

When a PV goes offline, the vgs_online file for the corresponding
VG is removed.  This allows the VG to be autoactivated again
when the PV comes online again.  This requires that the vgname be
stored in the pvid files.
2019-02-21 15:17:41 -06:00