1
0
mirror of git://sourceware.org/git/lvm2.git synced 2024-12-22 17:35:59 +03:00
Commit Graph

1316 Commits

Author SHA1 Message Date
Zdenek Kabelac
0e42b31dc3 libdm: fix errpath for control node creation
When control node failure happens, umask was not restored.
Fix it by using common exit path.
2017-08-22 10:23:29 +02:00
Alasdair G Kergon
e6afe9e782 ioctl: Allow minor without major.
There's no need to insist on a major number being supplied when the
code's going to override it if it's wrong anyway.
2017-08-04 14:45:20 +01:00
Zdenek Kabelac
92b53a8077 configure: improve test for realtime clock
Check first if we need to even link -lrt  - since clock functions
are normally emebeded with recent  glibc (>=2.17)
Use standard  RT_LIBS name.
Avoid duplicate test for realtime clock with lvmlockd
Show better error message when realtime clock support is missing or
disabled.
Link  RT_LIBS explicitely with lvmlockd and lvmetad.
2017-08-01 14:03:54 +02:00
Zdenek Kabelac
2232e82d25 makefiles: fixing linking
Avoid adding -g more then once for debug builds.
Avoid enabling  DEBUG_MEM when we build multithreaded tools.
Link executables with -fPIE -pie and --export-dynamic LDFLAGS
Introduce PROGS_FLAGS to add option to pass flags for external libs.
Link  lvm2 internally library only when really used.
Link DAEMON_LIBS with daemons.
Pass VALGRIND_CFLAGS internally
Set shell failure mode on couple places.
2017-08-01 11:53:30 +02:00
Zdenek Kabelac
57727bb4eb makefiles: RT_LIB
Link -lrt directly only with  libdevmapper.
All other users get dependency transiently.
2017-08-01 11:53:13 +02:00
Zdenek Kabelac
2593777f65 libdm: log_warn
No real effect anyway - function returs '1' in all paths,
but just make sure we do not have 'log_error()' on non-error path.
2017-08-01 11:53:12 +02:00
Zdenek Kabelac
0bf836aa14 tidy: prefer not using else after return
clang-tidy: avoid using  'else' after return - give more readable code,
and also saves indention level.
2017-07-20 11:18:29 +02:00
Zdenek Kabelac
0d0a3397c2 cleanup: add braces in macro 2017-07-20 11:18:29 +02:00
Zdenek Kabelac
b37e4e3f90 cleanup: constify used parameter 2017-07-20 11:18:29 +02:00
Zdenek Kabelac
f7e62bc55c cleanup: drop extra compare
dm_free() already validates for NULL itself.
2017-07-17 12:32:18 +02:00
Zdenek Kabelac
ba9820b142 numbers: strtod or strtoul need reset of errno
API for strtod() or strtoul() needs reset of errno, before it's being
called. So add missing resets in missing places and some also some
errno validation for out-of-range numbers.
2017-07-17 12:32:18 +02:00
Eric Ren
7617e08c03 cleanup: fix some typos
form -> from
accomodate -> accommodate

Signed-off-by: Eric Ren <zren@suse.com>
2017-07-10 14:58:33 +02:00
Zdenek Kabelac
5ff6260071 coverity: move initilization of count variable
Make code more understandble for Coverity, so it sees connection
between 'extents' and 'count' and in fact code is more readable.
2017-06-27 12:16:33 +02:00
Zdenek Kabelac
4e4067dd94 libdm: fix initialization of head for reused structure
Dmeventd reuses  'dm_task' struct for some STATUS operation, but due to
missing reinitization of dm_task target list, it has caused misprocesing
of recieved events as the parsed target has been simply added to the
list of existing status and cause multiple actions being called for
single event.
2017-06-26 19:45:11 +02:00
Zdenek Kabelac
feed61f3fa libdm: use rounded float for percent print
Use new added  dm_percent_to_round_float to enhance print
of percentage values.
2017-06-24 17:44:42 +02:00
Zdenek Kabelac
2ef8da61eb libdm: implement dm_percent_to_round_float
Add function to adjust printing of percent values in better way.
Rounding here is going along following rules:

0% & 100% are always clearly reported with  .0 decimal points.

Values slightly above 0% we make sure a nearest bigger
non zero value with given precission is printed
(i.e. 0.01  for  %.2f  will be shown)

For values closely approaching 100% we again detect and adjust value
that is less then 100 when printed.
(i.e. 99.99  for %.2f will be shown).

For other values we are leaving them with standard rounding mechanism
since we care mainly about corner values 0 & 100 which need to be
printed precisely.
2017-06-24 17:44:40 +02:00
Zdenek Kabelac
529dcaf6a3 libdm: workarounds reported raid status info
Current existing kernels reports status sometimes in weird form.

Instead of showing what is the exact progress, we need to estimate
this in-sync state from several surrounding states.

Main reason here is to never report 100% sync state for a raid device
which will be undergoing i.e. recovery.
2017-06-16 17:04:00 +02:00
Bryn M. Reeves
744f2920db libdm: allow truncated files in dm_stats_update_regions_from_fd()
It's not an error to attempt to update regions from an fd that has
been truncated (or otherwise no longer has any allocated extents):
in this case, the call should remove all regions corresponding to
the group, and return an empty region table.
2017-06-13 19:45:50 +01:00
Zdenek Kabelac
635e7e0c92 libdm: set min,maj
Reset derefered pointers instead of pointers.
2017-04-12 21:33:47 +02:00
Heinz Mauelshagen
01b5820d03 lvconvert: add segment type raid10_near
Introducing this alias for "raid10", avoid allocating
reshape space when converting between them.

Resolves: rhbz1441347
2017-04-12 01:28:22 +02:00
Bryn M. Reeves
febda60f3d libdm: use correct FMTu32 format specifier for uint32_t 2017-03-30 16:17:19 +01:00
Bryn M. Reeves
e8d5e05ff5 libdm: fix incorrect format specifier in _stats_resize_group()
The bitmap size in dm_bitset_t[0] is a uint32_t, not uint64_t: use
%u instead of FMTu64.
2017-03-30 10:31:36 +01:00
Bryn M. Reeves
c7e4b97abc libdm: use uint64_t where possible in filemap routines
Use uint64_t where possible and explicit casts elsewhere to avoid
compiler warnings when converting between int, int64_t and
uint64_t.
2017-03-30 10:02:54 +01:00
Bryn M. Reeves
b823646ace libdm: use size_t for buflen in _stats_group_file_regions() 2017-03-30 10:02:45 +01:00
Bryn M. Reeves
833b02106d libdm: make _stats_resize_group() num_regions argument uint64_t 2017-03-30 10:02:39 +01:00
Bryn M. Reeves
722542fabb libdm: make _find_extent() nr_extents argument uint64_t 2017-03-30 10:02:35 +01:00
Bryn M. Reeves
ac6089ee8d libdm: use DM_STATS_GROUP_* constants in dm_stats_create_regions_from_fd 2017-03-30 09:57:58 +01:00
Bryn M. Reeves
4b3d71212f libdm: use log_sys_error if dmfilemapd exec fails
Use log_sys_error rather than log_error if execvp() fails:

  /mnt/redhat/xdoio.13752.XIORQ: Created new group with 2 region(s) as group ID 0.
  # execvp() failed.

vs:

  /var/lib/libvirt/images/rhel7-vm1.qcow2: Created new group with 884 region(s) as group ID 0.
  dmfilemapd: execvp failed: No such file or directory
2017-03-29 21:38:15 +01:00
Bryn M. Reeves
6740eb1c2b libdm: do not attempt filemap cleanup if dm_stats_list() fails (coverity) 2017-03-29 18:34:43 +01:00
Bryn M. Reeves
f86c1f5d0a libdm: abort filemap update if pool allocation fails (coverity) 2017-03-29 18:34:38 +01:00
Bryn M. Reeves
bc5d67884f libdm: do not leak FIEMAP extent buffer if pool allocation fails (coverity) 2017-03-29 18:34:30 +01:00
Alasdair G Kergon
396377bc03 pre-release
Removing some unused new lines and changing some incorrect "can't
release until this is fixed" comments.  Rename license.txt to make
it clear its merely an included file, not itself a licence.
2017-03-28 16:11:35 +01:00
Zdenek Kabelac
e3a3cf01eb cleanup: use more common FMTd64 type
We use 'd' for plain singed integers.
2017-03-27 20:50:19 +02:00
Heinz Mauelshagen
fe3b9bb7d4 libdm: typo 2017-03-24 00:12:41 +01:00
Heinz Mauelshagen
b84bf3e8cd raid: adjust to misordered raid table line output
This commit supersedes reverted 1e4462dbfb
to avoid changes to liblvm and the libdm API completely.

The libdevmapper interface compares existing table line retrieved from
the kernel to new table line created to decide if it can suppress a reload.
Any difference between input and output of the table line is taken to be a
change thus causing a table reload.

The dm-raid target started to misorder the raid parameters (e.g. 'raid10_copies')
starting with dm-raid target version 1.9.0 up to (excluding) 1.11.0.  This causes
runtime failures (limited to raid10 as of tests) and needs to be reversed to allow
e.g. old lvm2 uspace to run properly.

Check for the aforementioned version range in libdm and adjust creation of the table line
to the respective (mis)ordered sequence inside and correct order outside the range
(as described for the raid target in the kernels Documentation/device-mapper/dm-raid.txt).
2017-03-23 01:20:00 +01:00
Heinz Mauelshagen
1bf90dac77 Revert "raid: adjust to misordered raid table line output"
This reverts commit 1e4462dbfb
in favour of an enhanced solution avoiding changes in liblvm
completetly by checking the target versions in libdm and emitting
the respective parameter lines.
2017-03-23 01:19:41 +01:00
Heinz Mauelshagen
1e4462dbfb raid: adjust to misordered raid table line output
The libdevmapper interface compares existing table line retrieved from
the kernel to new table line created to decide if it can suppress a reload.
Any difference between input and output of the table line is taken to be a
change thus causing a table reload.

The dm-raid target started to misorder the raid parameters (e.g. 'raid10_copies')
starting with dm-raid target version 1.9.0 up to (excluding) 1.11.0.  This causes
runtime failures (limited to raid10 as of tests) and needs to be reversed to allow
e.g. old lvm2 uspace to run properly.

Check for the aforementioned version range and adjust creation of the table line
to the respective (mis)ordered sequence inside and correct order outside the range
(as described for the raid target in the kernels Documentation/device-mapper/dm-raid.txt).
2017-03-21 18:17:42 +01:00
Zdenek Kabelac
42b970d4f3 libdm: fix endless loop
Coverity noticed endless loop (i not being changed).

TODO: test coverage
2017-03-16 01:02:10 +01:00
Bryn M. Reeves
77a7ed065f dmfilemapd: ensure path argument is absolute
Require that the path argument to dmfilemapd be an absolute path
and document this in tool output, libdevmapper.h and dmfilemapd.8.

The check is also enforced by dm_stats_start_filemapd() to avoid
forking a new process with an invalid path argument.
2017-03-13 11:44:05 +00:00
Zdenek Kabelac
a11b875074 cleanup: use FMTu
Replace some PRIu with FMTu for formating strings.
2017-03-10 19:33:01 +01:00
Zdenek Kabelac
bb20fac4ab libdm: maintain binary interface for new FEATURE flag
Older library version was not detecting unknown 'feature' bits
and could let start target without needed option.

New versioned symbol now checks for supported feature bits.
_Base version keeps accepting only previously known features and
mask/ignores unknown bits.

NB: if the older binary passed in 'random' bits, it will not get
metadata2 by chance. New linked binary get new validation function.
Library user is required to not pass 'trash' for unsupported bits,
as such calls will be rejected.
2017-03-10 19:33:01 +01:00
Zdenek Kabelac
ddd5a76801 libdm: support cache metadata2 feature flag
Dm cache target version 1.10 introduces new cache metadata format
(upstream kernel >=4.11).

New format is enable by passing new target feature flag metadata2.
Interace side on libdm uses DM_CACHE_FEATURE_METADATA2.
This feature bit is now also recognized on status
and set in 'feature_flags' field of dm_status_cache structure.

Code also adds check for 'highest' supported feature flag bit.
So it rejects properly any 'unknown' feature bit set by application.
2017-03-10 19:33:01 +01:00
Zdenek Kabelac
bf79fb1a33 libdm: better code to enforce writethrough
Better code to enforce writethrough caching for cleaner policy.
Only check for cleaner when DM_CACHE_FEATURE_PASSTHROUGH or
DM_CACHE_FEATURE_WRITEBACK is set.
2017-03-10 19:33:01 +01:00
Zdenek Kabelac
edf5e43f9a libdm: passthrough checks traling space char
When checking for passthrough length, check also for extra ' ' char
just like with other feature strings.
2017-03-10 19:33:01 +01:00
Bryn M. Reeves
8c6b8e90e3 libdm: use correct value for NR_FILEMAPD_ARGS
The argument count for the file mapping daemon needs to account
for the program's name in argv[0] to avoid indexing off the end
of the argument array.
2017-03-10 16:52:27 +00:00
Bryn M. Reeves
d28a50ec76 libdm: make dm_stats_destroy(NULL) a no-op
Test for NULL in dm_stats_destroy() and return immediately if
the struct dm_stats pointer is NULL (similar to free(NULL)).

This simplifies cleanup code which otherwise needs to:

  out:
    if (dms)
        dm_stats_destroy(dms);
    return;
2017-03-10 16:41:28 +00:00
Bryn M. Reeves
93644be44b libdm: dm_stats_start_filemapd() mode can never be < 0 2017-03-10 14:42:23 +00:00
Bryn M. Reeves
146b2582b0 libdm: move initialisation of group_id in _aggregate_histogram()
Older compilers are not able to determine that although group_id
is only assigned in one branch of a conditional, it is never used
used when the other branch is taken:

  libdm-stats.c:3319: warning: "group_id" may be used uninitialized in this function

Avoid this by always initialising the variable when it is
declared.
2017-03-10 11:45:08 +00:00
Bryn M. Reeves
0104fd6c66 libdm: don't nest FIEMAP and DMFILEMAPD ifdefs 2017-03-09 21:20:27 +00:00
Bryn M. Reeves
8b78982297 daemons: add dmfilemapd
Add a daemon that can be launched to monitor a group of regions
corresponding to the extents of a file, and to update the regions as the
file's allocation changes.

The daemon is intended to be started from a library interface, but can
also be run from the command line:

  dmfilemapd <fd> <group_id> <path> <mode> [<foreground>[<log_level>]]

Where fd is a file descriptor open on the mapped file, group_id is the
group identifier of the mapped group and mode is either "inode" or
"path". E.g.:

  # dmfilemapd 3 0 vm.img inode 1 3 3<vm.img
  ...

If foreground is non-zero, the daemon will not fork to run in the
background. If verbose is non-zero, libdm and daemon log messages will
be printed.

It is possible for the group identifier to change when regions are
re-mapped: this occurs when the group leader is deleted (regroup=1 in
dm_stats_update_regions_from_fd()), and another region is created before
the daemon has a chance to recreate the leader region.

The operation is inherently racey since there is currently no way to
atomically move or resize a dm_stats region while retaining its
region_id.

Detect this condition and update the group_id value stored in the
filemap monitor.

A function is also provided in the the stats API to launch the filemap
monitoring daemon:

  int dm_stats_start_filemapd(int fd, uint64_t group_id, const char *path,
                              dm_filemapd_mode_t mode, unsigned foreground,
                              unsigned verbose);

This carries out the first fork and execs dmfilemapd with the arguments
specified.

A dm_filemapd_mode_t value is specified by the mode argument: either
DM_FILEMAPD_FOLLOW_INODE, or DM_FILEMAPD_FOLLOW_PATH. A helper function,
dm_filemapd_mode_from_string(), is provided to parse a string containing
a valid mode name into the appropriate dm_filemapd_mode_t value.
2017-03-09 18:50:15 +00:00
Bryn M. Reeves
c98868181f libdm: remove unnecessary backtrace in _stats_group_id_present()
It's not an error to call dm_stats_group_present() on a handle
that contains no regions.

This causes dmfilemap to log a false backtrace during shutdown
if all regions are removed from the corresponding device:

  exiting _filemap_monitor_get_events() with deleted=0, check=0
  waiting for FILEMAPD_WAIT
  dm message   (253:1) [ opencount flush ]  @stats_list dmstats [32768] (*1)
  <backtrace>
  Filemap group removed: exiting.

Change this to only emit a backtrace if the handle is NULL.
2017-03-09 17:53:15 +00:00
Heinz Mauelshagen
fb42874a4f lvconvert: libdm RAID API compatibility versioning; remove new function
Commit 80a6de616a versioned the dm_tree_node_add_raid_target_with_params()
and dm_tree_node_add_raid_target() APIs for compatibility reasons.

There's no user of the latter function, remove it.

Related: rhbz834579
Related: rhbz1191935
Related: rhbz1191978
2017-03-01 18:58:48 +01:00
Heinz Mauelshagen
80a6de616a lvconvert: libdm RAID API compatibility versioning
Commit 27384c52cf lowered the maximum number of devices
back to 64 for compatibility.

Because more members have been added to the API in
'struct dm_tree_node_raid_params *', we have to version
the public libdm RAID API to not break any existing users.

Changes:

- keep the previous 'struct dm_tree_node_raid_params' and
  dm_tree_node_add_raid_target_with_params()/dm_tree_node_add_raid_target()
  in order to expose the already released public RAID API

- introduce 'struct dm_tree_node_raid_params_v2' and additional functions
  dm_tree_node_add_raid_target_with_params_v2()/dm_tree_node_add_raid_target_v2()
  to be used by the new lvm2 lib reshape extentions

With this new API, the bitfields for rebuild/writemostly legs in
'struct dm_tree_node_raid_params_v2' can be raised to 256 bits
again (253 legs maximum supported in MD kernel).

Mind that we can limit the maximum usable number via the
DEFAULT_RAID{1}_MAX_IMAGES definition in defaults.h.

Related: rhbz834579
Related: rhbz1191935
Related: rhbz1191978
2017-02-28 22:34:00 +01:00
Heinz Mauelshagen
27384c52cf lvconvert: limit libdm to maximum of 64 RAID devices
Commit 64a2fad5d6 raised the maximum number of RAID devices to 64.

Commit e2354ea344 introduced RAID_BITMAP_SIZE as 4 to have
256 bits (4 * 64 bit array members), thus changing the libdm API
unnecessarilly for the time being.

To not change the API, reduce RAID_BITMAP_SIZE to 1.
Remove an unneeded definition of it from libdm-common.h.

If we ever decide to raise past 64, we'll version the API.

Related: rhbz834579
Related: rhbz1191935
Related: rhbz1191978
2017-02-27 21:42:15 +01:00
Heinz Mauelshagen
e2354ea344 lvconvert: add infrastructure for RaidLV reshaping support
In order to support striped raid5/6/10 LV reshaping (change
of LV type, stripesize or number of legs), this patch
introduces infrastructure prerequisites to be used
by raid_manip.c extensions in followup patches.

This base is needed for allocation of out-of-place
reshape space required by the MD raid personalities to
avoid writing over data in-place when reading off the
current RAID layout or number of legs and writing out
the new layout or to a different number of legs
(i.e. restripe)

Changes:
- add members reshape_len to 'struct lv_segment' to store
  out-of-place reshape length per component rimage
- add member data_copies to struct lv_segment
  to support more than 2 raid10 data copies
- make alloc_lv_segment() aware of both reshape_len and data_copies
- adjust all alloc_lv_segment() callers to the new API
- add functions to retrieve the current data offset (needed for
  out-of-place reshaping space allocation) and the devices count
  from the kernel
- make libdm deptree code aware of reshape_len
- add LV flags for disk add/remove reshaping
- support import/export of the new 'struct lv_segment' members
- enhance lv_extend/_lv_reduce to cope with reshape_len
- add seg_is_*/segtype_is_* macros related to reshaping
- add target version check for reshaping
- grow rebuilds/writemostly bitmaps to 246 bit to support kernel maximal
- enhance libdm deptree code to support data_offset (out-of-place reshaping)
  and delta_disk (legs add/remove reshaping) target arguments

Related: rhbz834579
Related: rhbz1191935
Related: rhbz1191978
2017-02-24 05:20:58 +01:00
Zdenek Kabelac
3350eb67cc libdm: move code for _stats_resize_group
Code is only needed when "fiemap.h" so keep it under single #ifdef.
2017-02-14 10:24:56 +01:00
Zdenek Kabelac
a3579aafc5 cleanup: use matching signed number comparation 2017-02-13 10:06:19 +01:00
Zdenek Kabelac
7cbee8f31a cleanup: use matching const type 2017-02-13 10:06:18 +01:00
Zdenek Kabelac
b6301aa977 cleanup: use fall through
gcc gets 'selective' on having commented fall through case.
2017-02-13 10:06:18 +01:00
Zdenek Kabelac
717d0c6b94 cleanup: use proper printf specifier 2017-02-13 10:06:18 +01:00
Zdenek Kabelac
416f951283 coverity: fix double free
Do not try to free hist_arg twice.
2017-02-12 17:28:44 +01:00
Zdenek Kabelac
a7d2ee4bc2 coverity: fix mem leak on error path in dm stats
Free allocated resouces on error path.
2017-02-12 17:28:13 +01:00
Zdenek Kabelac
c908a8b131 libdm: avoid resume if preloaded device is smaller
When we preload device with smaller size, we avoid its resume,
so later suspend/resume of full device tree my process all
existing in flight bios.

Also update comment and avoid using confusing opposite meaning.
2017-02-10 20:29:11 +01:00
Zdenek Kabelac
811d137d3f cleanup: hide gcc warning
Gcc is not clever enough to see these vars are actually initialize in
given code path so let's just make sure it has a value.
2017-02-06 11:43:07 +01:00
Zdenek Kabelac
9fe8c2da36 debug: add space before uuid
With commit 8853462528 we added
uuid right after device name. Add space between them.
(Also fix some indenting)
2017-02-05 17:55:37 +01:00
Heinz Mauelshagen
a4bbaa3b89 lvconvert: add segtypes raid6_{ls,rs,la,ra}_6 and conversions to/from it
Add:
- support for segment types raid6_{ls,rs,la,ra}_6
  (striped raid with dedicated last Q-Syndrome SubLVs)
- conversion support from raid5_{ls,rs,la,ra} to/from raid6_{ls,rs,la,ra}_6
- setting convenient segtypes on conversions from/to raid4/5/6
- related tests to lvconvert-raid-takeover.sh factoring
  out _lvcreate,_lvconvert funxtions

Related: rhbz1366296
2017-02-05 00:56:27 +01:00
Heinz Mauelshagen
3673ce48e0 lvconvert: add segtype raid6_n_6 and conversions to/from it
Add:
- support for segment type raid6_n_6 (striped raid with dedicated last parity/Q-Syndrome SubLVs)
- conversion support from striped/raid0/raid0_meta/raid4 to/from raid6_n_6
- related tests to lvconvert-raid-takeover.sh

Related: rhbz1366296
2017-02-04 01:42:21 +01:00
Heinz Mauelshagen
60ddd05f16 lvconvert: add segtype raid5_n and conversions to/from it
Add:
- support for segment type raid5_n (striped raid with dedicated last parity SubLVs)
- conversion support from striped/raid0/raid0_meta/raid4 to/from raid5_n
- related tests to lvconvert-raid-takeover.sh

Related: rhbz1366296
2017-02-03 20:40:26 +01:00
Bryn M. Reeves
e0d19feb85 libdm: add dm_stats_update_regions_from_fd()
Add a call to update the regions corresponding to a file mapped
group of regions. The regions to be updated must be grouped, to
allow us to correctly identify extents that have been deallocated
since the map was created.

Tables are built of the file extents, and the extents currently
mapped to dmstats regions: if a region no longer has a matching
file extent, it is deleted, and new regions are created for any
file extents without a matching region.

The FIEMAP call returns extents that are currently in-memory (or
journaled) and awaiting allocation in the file system. These have
the FIEMAP_EXTENT_UNKNOWN | FIEMAP_EXTENT_DELALLOC flag bits set
in the fe_flags field - these extents are skipped until they
have a known disk location.

Since it is possile for the 0th extent of the file to have been
deallocated this must also handle the possible deletion and
re-creation of the group leader: if no other region allocation
is taking place the group identifier will not change.
2017-01-25 16:15:21 +00:00
Bryn M. Reeves
1c00bb5da3 libdm: test for DM_STATS_GROUP_NOT_PRESENT in _stats_group_id_present
If the group_id passed to _stats_group_id_present is equal to the
special value DM_STATS_GROUP_NOT_PRESENT there is no need to perform
any further tests: return false immediately.
2017-01-25 15:29:35 +00:00
Bryn M. Reeves
ca427a711a libdm: fix stats comment formatting in libdevmapper.h 2017-01-24 09:29:31 +00:00
Zdenek Kabelac
f8234d6e5f libdm: add human R|readable units
When showing sizes with 'H|human' units we do use standard rounding.
This however is confusing users from time to time,
when the printed number uses some biger units i.e. GiB and there is just
tiny fraction of space missing.

So here is some real-life example with new 'r' unit.

$lvs

  LV    VG Attr       LSize  Pool Origin
  lvol0 vg -wi-a-----  1.99g
  lvol1 vg -wi-a----- <2.00g
  lvol2 vg -wi-a----- <2.01g

Meaning is - lvol1 has 'slightly' less then 2.00g - from sign '<' user
can be aware the LV doesn't have full 2.00GiB in size so he
will be less surpriced allocation of 2G volume will not succeed.

$ vgs
  VG #PV #LV #SN Attr   VSize  VFree
  vg   2   2   0 wz--n- <6,00g <2,01g

For uses needing  'old'  undecorated human unit simply will continue
to use 'H|h' units.

The new R|r  may further change when we would recongnize some
other way how to improve readability.
2017-01-20 23:52:17 +01:00
Bryn M. Reeves
c90e9392e4 libdm: add dm_stats_bind_from_fd()
dmsetup already has a version of this function, and dmfilemapd will
need it too: move it to libdevmapper to avoid copying it around.
2016-12-18 20:47:17 +00:00
Bryn M. Reeves
009b711834 libdm: clear region table in dm_stats_list()
Call _stats_regions_destroy() from dm_stats_list() if dms->regions
is non-NULL. This avoids leaking any pool allocations and ensures
the handle is in a known state: if an error occurs during the list,
dms->regions will be NULL and the handle will appear empty.
2016-12-18 20:44:31 +00:00
Zdenek Kabelac
954c59779d libdm: drop callback on revert path
The system is likely in some very inconsisten state.
Do not try to make it even more problematic with trying
to invoke tools like thin_check via callback.
2016-12-18 19:29:08 +01:00
Bryn M. Reeves
35791689ba libdm: use destination size as limit in dm_bit_copy()
The dm_bit_copy() macro uses the source (bs1) bitset size as the
limit for memcpy:

    memcpy((bs1) + 1, (bs2) + 1, ((*(bs1) / DM_BITS_PER_INT) + 1)..)

This is safe if the destination bitset is smaller than the source,
or if the two bitsets are of the same size.

With a destination that is larger (e.g. when resizing a bitmap to
add more capacity), the memcpy will overrun the source bitset and
set garbage bits in the destination.

There are nine uses of the macro currently (8 in libdm/regex, and
1 in daemons/cmirrord): in each case the two bitsets are always of
equal size so the behaviour is unchanged.

Fix the macro to use bs2's size to simplify resizing bitsets and
avoid the need for another copy macro.
2016-12-14 11:28:11 +00:00
Bryn M. Reeves
f4401fe351 libdm: ensure first extent is always counted
If FIEMAP returns a single extent after the first call, no extent
boundary is detected and the first extent is not counted by the
normal mechanism.

In this case, increment nr_extents at the same time the extent is
added to the region table, before returning.
2016-12-13 21:41:31 +00:00
Bryn M. Reeves
7dff632c11 libdm: add min_num_bits to dm_bitset_parse_list()
It's useful to be able to specify a minimum number of bits for a
new bitmap parsed from a list, for e.g. to allow for expansing a
group without needing to copy/reallocate the bitmap.

Add a backwards compatible symbol for programs linked against old
versions of the library.
2016-12-13 21:02:18 +00:00
Bryn M. Reeves
e8d966bc31 libdm: use dm_bit_get_last() in _stats_group_tag_fill()
Instead of iterating over all bits, use dm_bit_get_last() to find
the last set bit in the group bitmap.
2016-12-13 21:02:18 +00:00
Bryn M. Reeves
5d1d65e735 libdm: add dm_bit_get_last()/dm_bit_get_prev()
It is sometimes convenient to iterate over the set bits in a dm
bitset in reverse order (from the highest set bit toward zero), or
to quickly find the last set bit.

Add dm_bit_get_last() and dm_bit_get_prev(), mirroring the existing
dm_bit_get_first() and dm_bit_get_next().

dm_bit_get_prev() uses __builtin_clz when available to efficiently
test the bitset in reverse.
2016-12-13 21:01:58 +00:00
Bryn M. Reeves
930b0b4c9e libdm: fix start of file detection in _stats_map_extents() 2016-12-13 20:25:47 +00:00
Bryn M. Reeves
eb65572217 libdm: break up _stats_get_extents_for_file()
Split out the loop that iterates over each batch of FIEMAP
extent data from the function that sets up and calls the ioctl
to reduce nesting and simplify local variable use:

  _stats_get_extents_for_file()
  ->  _stats_map_extents()

The _stats_map_extents() function is responsible for detecting
eof and extent boundaries and adding whole, allocated extents
to the file extent table for region creation.
2016-12-13 20:25:45 +00:00
Bryn M. Reeves
ea9af3e290 libdm: fix dm_stats_foreach_group() macro 2016-12-13 20:01:00 +00:00
Bryn M. Reeves
8e33972828 libdm: check for non-existent region_id values in groups
Check that all region_id values specified in a group bitmap are
actually present: although this should not normally happen when
using the dmstats tool, it is possible as a result of manual
changes (or bugs) for a group descriptor to contain one or more
group_id values that do not exist.

Check for this situation when reading group descriptors, warn
the user the user, and clear these bits in the bitmap when
formatting it for output.
2016-12-13 15:37:48 +00:00
Bryn M. Reeves
99b6d82e2d libdm: fix segfault with invalid group descriptor
If a region has a a DMS_GROUP tag in aux_data where the first
region_id in the bitmap is not the same as the containing region,
dmstats will segfault:

  # '2' is never a valid group bitset list for region_id == 0
  # dmsetup message vg_hex/root 0 "@stats_set_aux 0 DMS_GROUP=img:2#"

  # dmsetup message vg_hex/root 0 "@stats_list"
  0: 45383680+16384 16384 dmstats DMS_GROUP=img:2#
  1: 46071808+32768 32768 dmstats -
  2: 47382528+16384 16384 dmstats -

  # dmstats list
  Segmentation fault (core dumped)

The crash will occur in some arbitrary dm_stats_get_* property
method - this happens while processing the 1st region_id in the
bitset, because the region is marked as grouped, but there is
no group bitmap present at dms->groups[2]->regions.

Fix this by detecting a mismatch between the expected region_id
and dm_bit_get_first() for the parsed bitset during
_parse_aux_data_group().
2016-12-13 14:37:41 +00:00
Bryn M. Reeves
138e4336fd libdm: fix region overlap tests 2016-12-13 09:09:29 +00:00
Bryn M. Reeves
93f420caf4 libdm: fix _stats_get_extents_for_file()
Handle files that contain multiple logical extents in a single
physical extent properly:

  - In FIEMAP terms a logical extent is a contiguous range of
    sectors in the file's address space.

  - One or more physically adjacent logical extents comprise a
    physical extent: these are the disk areas that will be mapped
    to regions.

  - An extent boundary occurs when the start sector of extent
    n+1 is not equal to (n.start + n.length).

This requires that we accumulate the length values of extents
returned by FIEMAP until a discontinuity is found (since each
struct fiemap_extent returned by FIEMAP only represents a single
logical extent, which may be contiguous with other logical
extents on-disk).

This avoids creating large numbers of regions for physically
adjacent (logical) extents and fixes the earlier behaviour which
would only map the first logical extent of the physical extent,
leaving gaps in the region table for these files.
2016-12-13 09:09:25 +00:00
Zdenek Kabelac
48d33e5fb6 cov: use unsigned for single bit values
Avoid using signed int.
2016-12-12 11:21:42 +01:00
Zdenek Kabelac
6f84d3c69c libdm: validate vsnprintf
Avoid using buffer when no output has been generated.
Missed in ee13f265f0.
2016-12-11 23:24:17 +01:00
Bryn M. Reeves
b311122a56 libdm: fix filemap cleanup loop condition 2016-12-10 13:31:12 +00:00
Bryn M. Reeves
d8ba8ee9ae libdm: use a private pool for filemap extent table
When mapping regions to a file descriptor, a temporary table of
extent descriptors is built using the dm_pool object building
interface.

Previously this use borrowed the dms->mem region and counter
table pool (since nothing can interleave with the allocation
while the caller is still in dm_stats_create_regions_from_fd()).

This turns out to be problematic for error recovery. When a
region creation operation fails partway through file mapping,
we need to roll back the set of already created regions and
this requires a listed handle: the dm_stats_list() will then
allocate from the same pool as the extents; we either have
to throw away valid list data, or leak the extent table, to
return the handle in a valid state.

Avoid this problem by creating a new, temporary mem pool in
_stats_create_file_regions() to hold the extent data, and
discarding it on exit from the function.
2016-12-10 13:31:12 +00:00
Bryn M. Reeves
2d1dbb9edd libdm: fix performance of failed filemap cleanup
While cleaning up the table of already created regions during a
failed dm_stats_create_regions_from_fd(), list the handle once,
and call _stats_delete_region() directly. This avoids sending a
@stats_list message for each region deleted, reducing runtime
from 6s to 0.7s when cleaning up ~250 out of ~10000 regions:

  # time dmstats create --filemap b.img
  device-mapper: message ioctl on (253:0) failed: Cannot allocate memory
  Failed to create region 246 of 309 at 9388032.
  Could not create regions from file /root/b.img
  << pauses here >>
  Command failed

  real	0m6.267s
  user	0m3.770s
  sys	0m2.487s

  # time dmstats create --filemap b.img
  device-mapper: message ioctl on (253:0) failed: Cannot allocate memory
  Failed to create region 246 of 309 at 9388032.
  Could not create regions from file /root/b.img
  Command failed

  real	0m0.716s
  user	0m0.034s
  sys	0m0.581s

Testing the error path requires region creation to start to
fail part way through the operation (in order to have regions
to clean up): the simplest way is to ensure the system is
close to the kernel limit of 1/4 RAM or 1/2 vmalloc space
consumed by dmstats data.
2016-12-10 11:59:16 +00:00
Bryn M. Reeves
97c4490cc5 libdm: split off internal _stats_delete_region()
Split dm_stats_delete_region() so that internal callers can manage
the handle state themselves.

dm_stats_delete_region() now just handles checking the state of the
handle, reporting validation errors, and calling dm_stats_list() if
necessary, before calling _stats_delete_region().

The new _stats_delete_region() function performs the actual group
member removal and region deletion, and requires a fully listed
handle to operate.

Callers that repeatedly delete regions can use a single listed
handle for many operations on the same device, avoiding one
message ioctl per region deleted: since @stats_list with many
regions is expensive, this yields large runtime improvements.
2016-12-10 11:57:14 +00:00
Bryn M. Reeves
30ad254d84 libdm: use correct region_id when cleaning up a failed filemap
If we fail to create a region during dm_stats_create_regions_from_fd(),
we must remove all regions that were created to do this to date. This
needs to loop over the table of region_id values that were populated
by _stats_create_file_regions() before the error.

The code for this failure case in the out_remove branch incorrectly
uses the table index as the region_id:

    for (--i; i != DM_STATS_REGION_NOT_PRESENT; i--) {
            if (!dm_stats_delete_region(dms, i))
                    log_error("Could not delete region " FMTu64 ".", i);
    }

This causes the cleanup code to delete a completely unrelated set
of regions (since the index here will always be nr_regions..0).

Fix it to pass the actual region_id stored in regions[i] instead.
2016-12-09 16:04:13 +00:00
Bryn M. Reeves
7fd2fa22dd libdm-stats: clear dms->groups in _stats_groups_destroy() 2016-12-09 16:04:13 +00:00
Bryn M. Reeves
cb8c04760f libdm-stats: clear dms->regions in _stats_regions_destroy() 2016-12-09 16:04:13 +00:00
Bryn M. Reeves
6dd0bd0255 libdm-stats: fix dm_stats_delete_region() performance
Fix a silly bug in dm_stats_delete_region() that hugely inflates
runtimes when deleting a large number of regions.

For ~50,000 regions this change reduces the runtime from 98s to
6s on my test systems (a ~93% reduction).

The bug exists because dm_stats_delete_region() applies a truth
test to the return value of dm_stats_get_nr_areas(); this is
never correct usage - it will walk the entire region table and
calculate area counts for each region (which is roughly O(n^2)
in the number of regions, as dm_stats_delete_region() is being
called inside a region walk).

Although the individual area calculation is not that costly,
uselessly running anything 2,500,000,000 times over gets a bit
slow.

A much cheaper test (which is always true if the areas check is
true) is to just test dm_stats_get_nr_regions() or dms->regions;
if either is true it implies at least one area exists.

Old:

 Performance counter stats for 'dmstats delete --allregions --alldevices':

      98117.791458      task-clock (msec)         #    1.000 CPUs utilized
               127      context-switches          #    0.001 K/sec
                 3      cpu-migrations            #    0.000 K/sec
             6,631      page-faults               #    0.068 K/sec
   307,711,724,562      cycles                    #    3.136 GHz
   544,762,959,577      instructions              #    1.77  insn per cycle
    84,287,824,115      branches                  #  859.047 M/sec
         2,538,875      branch-misses             #    0.00% of all branches

      98.119578733 seconds time elapsed

New:

 Performance counter stats for 'dmstats delete --allregions --alldevices':

       6427.251074      task-clock (msec)         #    1.000 CPUs utilized
                 6      context-switches          #    0.001 K/sec
                 0      cpu-migrations            #    0.000 K/sec
             6,634      page-faults               #    0.001 M/sec
    21,613,018,724      cycles                    #    3.363 GHz
     3,794,755,445      instructions              #    0.18  insn per cycle
       852,974,026      branches                  #  132.712 M/sec
           808,625      branch-misses             #    0.09% of all branches

       6.428953647 seconds time elapsed
2016-12-09 10:55:39 +00:00
Zdenek Kabelac
1d58074d9f debug: more stacktrace corrections
Continue previous patch dropping some unneeded stack traces
after printed log_error/warn messages.
2016-11-25 14:58:28 +01:00
Zdenek Kabelac
74923c213f cleanup: add doc for raid status states
Show possible values for raid fields user may get ATM.
2016-11-23 17:55:03 +01:00
Zdenek Kabelac
cc3895623c cleanup: hide gcc warning 2016-11-23 17:55:03 +01:00