1
0
mirror of git://sourceware.org/git/lvm2.git synced 2024-12-21 13:34:40 +03:00
Commit Graph

1248 Commits

Author SHA1 Message Date
Bryn M. Reeves
e0d19feb85 libdm: add dm_stats_update_regions_from_fd()
Add a call to update the regions corresponding to a file mapped
group of regions. The regions to be updated must be grouped, to
allow us to correctly identify extents that have been deallocated
since the map was created.

Tables are built of the file extents, and the extents currently
mapped to dmstats regions: if a region no longer has a matching
file extent, it is deleted, and new regions are created for any
file extents without a matching region.

The FIEMAP call returns extents that are currently in-memory (or
journaled) and awaiting allocation in the file system. These have
the FIEMAP_EXTENT_UNKNOWN | FIEMAP_EXTENT_DELALLOC flag bits set
in the fe_flags field - these extents are skipped until they
have a known disk location.

Since it is possile for the 0th extent of the file to have been
deallocated this must also handle the possible deletion and
re-creation of the group leader: if no other region allocation
is taking place the group identifier will not change.
2017-01-25 16:15:21 +00:00
Bryn M. Reeves
1c00bb5da3 libdm: test for DM_STATS_GROUP_NOT_PRESENT in _stats_group_id_present
If the group_id passed to _stats_group_id_present is equal to the
special value DM_STATS_GROUP_NOT_PRESENT there is no need to perform
any further tests: return false immediately.
2017-01-25 15:29:35 +00:00
Bryn M. Reeves
ca427a711a libdm: fix stats comment formatting in libdevmapper.h 2017-01-24 09:29:31 +00:00
Zdenek Kabelac
f8234d6e5f libdm: add human R|readable units
When showing sizes with 'H|human' units we do use standard rounding.
This however is confusing users from time to time,
when the printed number uses some biger units i.e. GiB and there is just
tiny fraction of space missing.

So here is some real-life example with new 'r' unit.

$lvs

  LV    VG Attr       LSize  Pool Origin
  lvol0 vg -wi-a-----  1.99g
  lvol1 vg -wi-a----- <2.00g
  lvol2 vg -wi-a----- <2.01g

Meaning is - lvol1 has 'slightly' less then 2.00g - from sign '<' user
can be aware the LV doesn't have full 2.00GiB in size so he
will be less surpriced allocation of 2G volume will not succeed.

$ vgs
  VG #PV #LV #SN Attr   VSize  VFree
  vg   2   2   0 wz--n- <6,00g <2,01g

For uses needing  'old'  undecorated human unit simply will continue
to use 'H|h' units.

The new R|r  may further change when we would recongnize some
other way how to improve readability.
2017-01-20 23:52:17 +01:00
Bryn M. Reeves
c90e9392e4 libdm: add dm_stats_bind_from_fd()
dmsetup already has a version of this function, and dmfilemapd will
need it too: move it to libdevmapper to avoid copying it around.
2016-12-18 20:47:17 +00:00
Bryn M. Reeves
009b711834 libdm: clear region table in dm_stats_list()
Call _stats_regions_destroy() from dm_stats_list() if dms->regions
is non-NULL. This avoids leaking any pool allocations and ensures
the handle is in a known state: if an error occurs during the list,
dms->regions will be NULL and the handle will appear empty.
2016-12-18 20:44:31 +00:00
Zdenek Kabelac
954c59779d libdm: drop callback on revert path
The system is likely in some very inconsisten state.
Do not try to make it even more problematic with trying
to invoke tools like thin_check via callback.
2016-12-18 19:29:08 +01:00
Bryn M. Reeves
35791689ba libdm: use destination size as limit in dm_bit_copy()
The dm_bit_copy() macro uses the source (bs1) bitset size as the
limit for memcpy:

    memcpy((bs1) + 1, (bs2) + 1, ((*(bs1) / DM_BITS_PER_INT) + 1)..)

This is safe if the destination bitset is smaller than the source,
or if the two bitsets are of the same size.

With a destination that is larger (e.g. when resizing a bitmap to
add more capacity), the memcpy will overrun the source bitset and
set garbage bits in the destination.

There are nine uses of the macro currently (8 in libdm/regex, and
1 in daemons/cmirrord): in each case the two bitsets are always of
equal size so the behaviour is unchanged.

Fix the macro to use bs2's size to simplify resizing bitsets and
avoid the need for another copy macro.
2016-12-14 11:28:11 +00:00
Bryn M. Reeves
f4401fe351 libdm: ensure first extent is always counted
If FIEMAP returns a single extent after the first call, no extent
boundary is detected and the first extent is not counted by the
normal mechanism.

In this case, increment nr_extents at the same time the extent is
added to the region table, before returning.
2016-12-13 21:41:31 +00:00
Bryn M. Reeves
7dff632c11 libdm: add min_num_bits to dm_bitset_parse_list()
It's useful to be able to specify a minimum number of bits for a
new bitmap parsed from a list, for e.g. to allow for expansing a
group without needing to copy/reallocate the bitmap.

Add a backwards compatible symbol for programs linked against old
versions of the library.
2016-12-13 21:02:18 +00:00
Bryn M. Reeves
e8d966bc31 libdm: use dm_bit_get_last() in _stats_group_tag_fill()
Instead of iterating over all bits, use dm_bit_get_last() to find
the last set bit in the group bitmap.
2016-12-13 21:02:18 +00:00
Bryn M. Reeves
5d1d65e735 libdm: add dm_bit_get_last()/dm_bit_get_prev()
It is sometimes convenient to iterate over the set bits in a dm
bitset in reverse order (from the highest set bit toward zero), or
to quickly find the last set bit.

Add dm_bit_get_last() and dm_bit_get_prev(), mirroring the existing
dm_bit_get_first() and dm_bit_get_next().

dm_bit_get_prev() uses __builtin_clz when available to efficiently
test the bitset in reverse.
2016-12-13 21:01:58 +00:00
Bryn M. Reeves
930b0b4c9e libdm: fix start of file detection in _stats_map_extents() 2016-12-13 20:25:47 +00:00
Bryn M. Reeves
eb65572217 libdm: break up _stats_get_extents_for_file()
Split out the loop that iterates over each batch of FIEMAP
extent data from the function that sets up and calls the ioctl
to reduce nesting and simplify local variable use:

  _stats_get_extents_for_file()
  ->  _stats_map_extents()

The _stats_map_extents() function is responsible for detecting
eof and extent boundaries and adding whole, allocated extents
to the file extent table for region creation.
2016-12-13 20:25:45 +00:00
Bryn M. Reeves
ea9af3e290 libdm: fix dm_stats_foreach_group() macro 2016-12-13 20:01:00 +00:00
Bryn M. Reeves
8e33972828 libdm: check for non-existent region_id values in groups
Check that all region_id values specified in a group bitmap are
actually present: although this should not normally happen when
using the dmstats tool, it is possible as a result of manual
changes (or bugs) for a group descriptor to contain one or more
group_id values that do not exist.

Check for this situation when reading group descriptors, warn
the user the user, and clear these bits in the bitmap when
formatting it for output.
2016-12-13 15:37:48 +00:00
Bryn M. Reeves
99b6d82e2d libdm: fix segfault with invalid group descriptor
If a region has a a DMS_GROUP tag in aux_data where the first
region_id in the bitmap is not the same as the containing region,
dmstats will segfault:

  # '2' is never a valid group bitset list for region_id == 0
  # dmsetup message vg_hex/root 0 "@stats_set_aux 0 DMS_GROUP=img:2#"

  # dmsetup message vg_hex/root 0 "@stats_list"
  0: 45383680+16384 16384 dmstats DMS_GROUP=img:2#
  1: 46071808+32768 32768 dmstats -
  2: 47382528+16384 16384 dmstats -

  # dmstats list
  Segmentation fault (core dumped)

The crash will occur in some arbitrary dm_stats_get_* property
method - this happens while processing the 1st region_id in the
bitset, because the region is marked as grouped, but there is
no group bitmap present at dms->groups[2]->regions.

Fix this by detecting a mismatch between the expected region_id
and dm_bit_get_first() for the parsed bitset during
_parse_aux_data_group().
2016-12-13 14:37:41 +00:00
Bryn M. Reeves
138e4336fd libdm: fix region overlap tests 2016-12-13 09:09:29 +00:00
Bryn M. Reeves
93f420caf4 libdm: fix _stats_get_extents_for_file()
Handle files that contain multiple logical extents in a single
physical extent properly:

  - In FIEMAP terms a logical extent is a contiguous range of
    sectors in the file's address space.

  - One or more physically adjacent logical extents comprise a
    physical extent: these are the disk areas that will be mapped
    to regions.

  - An extent boundary occurs when the start sector of extent
    n+1 is not equal to (n.start + n.length).

This requires that we accumulate the length values of extents
returned by FIEMAP until a discontinuity is found (since each
struct fiemap_extent returned by FIEMAP only represents a single
logical extent, which may be contiguous with other logical
extents on-disk).

This avoids creating large numbers of regions for physically
adjacent (logical) extents and fixes the earlier behaviour which
would only map the first logical extent of the physical extent,
leaving gaps in the region table for these files.
2016-12-13 09:09:25 +00:00
Zdenek Kabelac
48d33e5fb6 cov: use unsigned for single bit values
Avoid using signed int.
2016-12-12 11:21:42 +01:00
Zdenek Kabelac
6f84d3c69c libdm: validate vsnprintf
Avoid using buffer when no output has been generated.
Missed in ee13f265f0.
2016-12-11 23:24:17 +01:00
Bryn M. Reeves
b311122a56 libdm: fix filemap cleanup loop condition 2016-12-10 13:31:12 +00:00
Bryn M. Reeves
d8ba8ee9ae libdm: use a private pool for filemap extent table
When mapping regions to a file descriptor, a temporary table of
extent descriptors is built using the dm_pool object building
interface.

Previously this use borrowed the dms->mem region and counter
table pool (since nothing can interleave with the allocation
while the caller is still in dm_stats_create_regions_from_fd()).

This turns out to be problematic for error recovery. When a
region creation operation fails partway through file mapping,
we need to roll back the set of already created regions and
this requires a listed handle: the dm_stats_list() will then
allocate from the same pool as the extents; we either have
to throw away valid list data, or leak the extent table, to
return the handle in a valid state.

Avoid this problem by creating a new, temporary mem pool in
_stats_create_file_regions() to hold the extent data, and
discarding it on exit from the function.
2016-12-10 13:31:12 +00:00
Bryn M. Reeves
2d1dbb9edd libdm: fix performance of failed filemap cleanup
While cleaning up the table of already created regions during a
failed dm_stats_create_regions_from_fd(), list the handle once,
and call _stats_delete_region() directly. This avoids sending a
@stats_list message for each region deleted, reducing runtime
from 6s to 0.7s when cleaning up ~250 out of ~10000 regions:

  # time dmstats create --filemap b.img
  device-mapper: message ioctl on (253:0) failed: Cannot allocate memory
  Failed to create region 246 of 309 at 9388032.
  Could not create regions from file /root/b.img
  << pauses here >>
  Command failed

  real	0m6.267s
  user	0m3.770s
  sys	0m2.487s

  # time dmstats create --filemap b.img
  device-mapper: message ioctl on (253:0) failed: Cannot allocate memory
  Failed to create region 246 of 309 at 9388032.
  Could not create regions from file /root/b.img
  Command failed

  real	0m0.716s
  user	0m0.034s
  sys	0m0.581s

Testing the error path requires region creation to start to
fail part way through the operation (in order to have regions
to clean up): the simplest way is to ensure the system is
close to the kernel limit of 1/4 RAM or 1/2 vmalloc space
consumed by dmstats data.
2016-12-10 11:59:16 +00:00
Bryn M. Reeves
97c4490cc5 libdm: split off internal _stats_delete_region()
Split dm_stats_delete_region() so that internal callers can manage
the handle state themselves.

dm_stats_delete_region() now just handles checking the state of the
handle, reporting validation errors, and calling dm_stats_list() if
necessary, before calling _stats_delete_region().

The new _stats_delete_region() function performs the actual group
member removal and region deletion, and requires a fully listed
handle to operate.

Callers that repeatedly delete regions can use a single listed
handle for many operations on the same device, avoiding one
message ioctl per region deleted: since @stats_list with many
regions is expensive, this yields large runtime improvements.
2016-12-10 11:57:14 +00:00
Bryn M. Reeves
30ad254d84 libdm: use correct region_id when cleaning up a failed filemap
If we fail to create a region during dm_stats_create_regions_from_fd(),
we must remove all regions that were created to do this to date. This
needs to loop over the table of region_id values that were populated
by _stats_create_file_regions() before the error.

The code for this failure case in the out_remove branch incorrectly
uses the table index as the region_id:

    for (--i; i != DM_STATS_REGION_NOT_PRESENT; i--) {
            if (!dm_stats_delete_region(dms, i))
                    log_error("Could not delete region " FMTu64 ".", i);
    }

This causes the cleanup code to delete a completely unrelated set
of regions (since the index here will always be nr_regions..0).

Fix it to pass the actual region_id stored in regions[i] instead.
2016-12-09 16:04:13 +00:00
Bryn M. Reeves
7fd2fa22dd libdm-stats: clear dms->groups in _stats_groups_destroy() 2016-12-09 16:04:13 +00:00
Bryn M. Reeves
cb8c04760f libdm-stats: clear dms->regions in _stats_regions_destroy() 2016-12-09 16:04:13 +00:00
Bryn M. Reeves
6dd0bd0255 libdm-stats: fix dm_stats_delete_region() performance
Fix a silly bug in dm_stats_delete_region() that hugely inflates
runtimes when deleting a large number of regions.

For ~50,000 regions this change reduces the runtime from 98s to
6s on my test systems (a ~93% reduction).

The bug exists because dm_stats_delete_region() applies a truth
test to the return value of dm_stats_get_nr_areas(); this is
never correct usage - it will walk the entire region table and
calculate area counts for each region (which is roughly O(n^2)
in the number of regions, as dm_stats_delete_region() is being
called inside a region walk).

Although the individual area calculation is not that costly,
uselessly running anything 2,500,000,000 times over gets a bit
slow.

A much cheaper test (which is always true if the areas check is
true) is to just test dm_stats_get_nr_regions() or dms->regions;
if either is true it implies at least one area exists.

Old:

 Performance counter stats for 'dmstats delete --allregions --alldevices':

      98117.791458      task-clock (msec)         #    1.000 CPUs utilized
               127      context-switches          #    0.001 K/sec
                 3      cpu-migrations            #    0.000 K/sec
             6,631      page-faults               #    0.068 K/sec
   307,711,724,562      cycles                    #    3.136 GHz
   544,762,959,577      instructions              #    1.77  insn per cycle
    84,287,824,115      branches                  #  859.047 M/sec
         2,538,875      branch-misses             #    0.00% of all branches

      98.119578733 seconds time elapsed

New:

 Performance counter stats for 'dmstats delete --allregions --alldevices':

       6427.251074      task-clock (msec)         #    1.000 CPUs utilized
                 6      context-switches          #    0.001 K/sec
                 0      cpu-migrations            #    0.000 K/sec
             6,634      page-faults               #    0.001 M/sec
    21,613,018,724      cycles                    #    3.363 GHz
     3,794,755,445      instructions              #    0.18  insn per cycle
       852,974,026      branches                  #  132.712 M/sec
           808,625      branch-misses             #    0.09% of all branches

       6.428953647 seconds time elapsed
2016-12-09 10:55:39 +00:00
Zdenek Kabelac
1d58074d9f debug: more stacktrace corrections
Continue previous patch dropping some unneeded stack traces
after printed log_error/warn messages.
2016-11-25 14:58:28 +01:00
Zdenek Kabelac
74923c213f cleanup: add doc for raid status states
Show possible values for raid fields user may get ATM.
2016-11-23 17:55:03 +01:00
Zdenek Kabelac
cc3895623c cleanup: hide gcc warning 2016-11-23 17:55:03 +01:00
Bryn M. Reeves
2a86f54b09 libdm: separate dm_stats_populate() error cases
There are two possible errors in _dm_stats_populate_region():

  * No region struct in dms->regions[region_id]
  * Failure to parse data from @stats_print

These have very different causes: the first occurs where a client
program is populating one region at a time (region_id is a single
region identifier), and has not previously called dm_stats_list()
to dimension the region tables; this is an API usage error.

The second occurs when either we read unparseable data from the
kernel (kernel bug), or where various resource allocations fail.

Separate these two cases out and log separate messages for each
(allocation failures in the path already have their own distinct
message), since the "failed to parse.." message in the un-listed
handle case is confusing and misleading.
2016-11-17 11:39:43 +00:00
Zdenek Kabelac
4e26024add cleanup: use WARNING prefix for log_warn
Use capital WARNING prefix for log_warn() messages.
2016-11-03 17:49:07 +01:00
Zdenek Kabelac
9cbe4c1af9 log_info to log_very_verbose
Translate log_info() into log_very_verbose() which is macro
supposed to be used by our code.

log_info() is internal macro with eventually some 'symbolic' meaning
in syslogging daemons.
2016-11-03 17:49:07 +01:00
Zdenek Kabelac
ee13f265f0 libdm: use dm_log_with_errno always
Instead of compiling 2 log call for 2 different logging functions,
and runtime decide which version to use - use only 'newer' function
and when user sets his own OLD dm_log logging translate it runtime
for old arg list set.

The positive part is - we get shorter generated library,
on the negative part this translation means, we always have evaluate
all args and print the message into local on stack buffer, before
we can pass this buffer to the users' logging function with proper
expected parameters (and such function may later decide to discard
logging based on message level so whole printing was unnecessary).
2016-11-03 17:49:07 +01:00
Zdenek Kabelac
1db4b81d5a cleanup: drop unused attribute
In this function we pass args through so make the function
header look the same as with _default_log().
2016-11-03 12:43:09 +01:00
Zdenek Kabelac
e50d434a35 libdm: report logging with errno as changed default
When user changes logging with 'dm_log_with_errno_init()'
also report this as non default dm logging.
2016-11-03 12:43:09 +01:00
Zdenek Kabelac
6af26273cb logging: add more log macros
Introduce macros:

log_level(), log_stderr(), log_once(), log_bypass_report()

For easier and more consisten way how to 'decoder' bits
of info from passed 'level'.

This patch fixes potential problem when 'level' of message
might not have always masked right bits.
2016-11-03 12:43:09 +01:00
Zdenek Kabelac
d70f112762 libdm: check for mem when _canonicalize_field_ids
Add missing check for dm_pool_strdup() call (Coverity).
2016-10-03 17:46:26 +02:00
Bryn M. Reeves
e95a252974 libdm: convert FIEMAP buffer allocation from stack to dm_zalloc 2016-10-03 15:14:33 +01:00
Bryn M. Reeves
56c90ffa5e libdm: fix dm_stats_delete_region() backwards compat
The dm_stats_delete_region() call removes a region from the bound
device, and, if the region is grouped, from the group leader
group descriptor stored in aux_data.

To do this requires a listed handle: previous versions of the
library do not since no dependencies exist between regions without
grouping.

This leads to strange behaviour when a command built against an old
version of the library is used with one supporting groups. Deleting
a region with dmstats succeeds, but logs errors:

  # dmstats list
  Name             RgID RgSta RgSiz #Areas ArSize ProgID
  vg_hex-root         0     0 1.00g      1  1.00g dmstats
  vg_hex-root         1 1.00g 1.00g      1  1.00g dmstats
  vg_hex-root         2 2.00g 1.00g      1  1.00g dmstats
  # dmstats delete --regionid 2 vg_hex/root
  Region ID 2 does not exist
  Could not delete statistics region.
  Command failed
  # dmstats list
  Name             RgID RgSta RgSiz #Areas ArSize ProgID
  vg_hex-root         0     0 1.00g      1  1.00g dmstats
  vg_hex-root         1 1.00g 1.00g      1  1.00g dmstats

This happens because the call to dm_stats_delete_region() is inside
a dm_stats_walk_*() iterator: upon entry to the call, the iterator
is at its end conditions and about to terminate. Due to the call to
dm_stats_list() inside the function, it returns with an iterator at
the beginning of a walk and performs a further iteration before
exiting. This final loop makes a further attempt to delete the
(already deleted) region, leading to the confusing error messages.
2016-09-27 17:58:05 +01:00
Bryn M. Reeves
6ec8854fdb libdm: fix stats walk compatibility with older dmsetup
The current dmsetup.c handles DR_STATS and DR_STATS_META reports
separately in _display_info_cols(), meaning that the stats walk
functions are never called for these report types.

Versions before v2.02.159 have a loop using dm_stats_walk_do() and
dm_stats_walk_while(), that executes once for non-stats reports,
and once per region, or area, for DR_STATS/DR_STATS_META reports.

This older behaviour relies on the documented behaviour that the
walk functions will accept a NULL pointer as the struct dm_stats*
argument.

This was broken by commit f1f2df7b: the NULL test on dms and
dms->regions were incorrectly moved from the dm_stats_walk_end()
wrapper to the internal '_stats_walk_end()' helper.

Since the pointer is dereferenced in between these points, using
an older dmsetup with current libdm results in a segfault when
running a non-stats report:

  # dmsetup info -c vg00/lvol0
  Segmentation fault (core dumped)

Restore the NULL checks to the wrapper function as intended.
2016-09-27 14:46:00 +01:00
Peter Rajnoha
7563e69cf1 libdm: add dm_config_parse_without_dup_node_check
Introduce function for config parsing tree without checking
for duplicate nodes.
2016-09-21 18:15:18 +02:00
Bryn M. Reeves
c26cd48536 libdm: fix end-of-groups test in _stats_walk_end() 2016-09-16 13:09:22 +01:00
Zdenek Kabelac
a156fc9a54 libdm: cleaner debug message 2016-09-13 09:24:38 +02:00
Zdenek Kabelac
92d5a84410 cleanup: clean gcc6 minor/major types warnings
Put sys/sysmacros.h in front of sys/types.h header file as requested
by gcc6.
2016-08-29 20:51:16 +02:00
Peter Rajnoha
db0e34535c libdm: add some comments about DM_UDEV_DISABLE_LIBRARY_FALLBACK flag 2016-08-23 15:58:03 +02:00
Peter Rajnoha
7d1125e5b7 libdm: report: add dm_report_group_output_and_pop_all
The dm_report_group_output_and_pop_all calls dm_report_output and
dm_report_group_pop for all the items that are currently in report
group. This is just a shortcut that makes it easier to output and
pop group's content so the group handle can be reused again without
a need to initialize and configure it again.

The functionality of dm_report_group_output_and_pop_all is the
same as dm_report_destroy but without destroying the report group
handle.
2016-08-09 18:24:45 +02:00
Peter Rajnoha
d86caf952e libdm: report: postpone printing of JSON starting '{' character till it's needed
This patch moves printing of starting '{' character for JSON output up
untili it's known there's any further output following - either the
content or ending '}' character.

Also, remove unnecessary switch for different report group types and
calling individual functions to handle dm_report_group_create as that
code is shared for all existing types at the moment.
2016-08-09 18:24:45 +02:00
Peter Rajnoha
9c21139284 libdm: report: add dm_report_destroy_rows
Calling dm_report_destroy_rows makes it possible to destroy any report
content we have but at the same time it doesn't destroy the report
handle itself, thus it's possible to reuse that handle again for new
report content.

Functionally, this is the same as calling dm_report_output with the
report handle but omitting the output iself. This functionality may
be useful if we, for whatever reason, need to discard the report
content and start a fresh new one but with the same report configuration
and initialization and thus we can just reuse the existing handle.
2016-08-09 18:24:45 +02:00
Peter Rajnoha
f9697ea006 libdm: report: fix json reporting to escape '"' character that may appear in reported string 2016-07-26 12:27:41 +02:00
Bryn M. Reeves
252952ff33 libdm: document use of dm_free() with histogram bounds 2016-07-18 18:48:34 +01:00
Bryn M. Reeves
4ef1f34549 libdm: free hist_arg in _stats_create_file_regions() (Coverity) 2016-07-18 18:09:21 +01:00
Bryn M. Reeves
309bdfa224 libdm: add configure.in checks for fiemap.h and magic.h 2016-07-08 23:42:32 +01:00
Bryn M. Reeves
61cb58e549 libdm: use 'goto_bad' if extent pool allocation fails
Generate a backtrace if unable to extend the extent table.
2016-07-08 22:28:50 +01:00
Bryn M. Reeves
feb69966d4 libdm: use macro for boundary test in _stats_get_extents_for_file() 2016-07-08 22:21:14 +01:00
Bryn M. Reeves
2d1f03b616 libdm: use a constant for FIEMAP buffer size 2016-07-08 22:05:36 +01:00
Bryn M. Reeves
58bfea6a6e libdm: use SECTOR_SHIFT constant in _stats_add_extent() 2016-07-08 22:05:36 +01:00
Bryn M. Reeves
4bb57341bd libdm: enable creation of filemap regions with histograms 2016-07-08 17:27:52 +01:00
Bryn M. Reeves
db73d756e9 libdm: allow regions with histograms in dm_stats_create_group()
Allow regions with histograms to be grouped if all histograms have
the same number of bins and matching bounds.
2016-07-08 17:27:52 +01:00
Bryn M. Reeves
ae9cffba52 libdm: add aggregation support to dm_stats_get_histogram()
Support aggregate group and region histograms by allocating a new
histogram from the pool and populating it with a sum of the histogram
data for the areas contained in the region or group.

To avoid repeatedly summing the same histogram data, cache the pointer
in the group and regions structs for subsequent access. The aggregate
histograms are allocated from the same pool as the area histograms in
the corresponding handle and will be discarded at each list or populate
operation.
2016-07-08 17:27:52 +01:00
Bryn M. Reeves
e104825916 libdm: add dm_stats_create_regions_from_fd()
Add a call to create dmstats regions that correspond to the extents
present in a file descriptor open on a file in a local file system.
The file must reside on a file system type that correctly supports
physical extent location data in the FIEMAP ioctl.

Regions are optionally placed into a group with a user-defined alias.

File systems that do not support physical offsets in FIEMAP (btrfs
currently) are detected via fstatfs() - although attempting to map
a --filemap group on btrfs will fail anyway with the generic error
"Not on a device-mapper device" this is confusing; the file system
mount is on a device-mapper device, but btrfs' volume layer masks
this in the returned st_dev field since the returned logical file
extents may span multiple physical devices.
2016-07-08 14:34:41 +01:00
Bryn M. Reeves
ebc7fc67c8 libdm: fix group resource leak in dm_stats_delete_region()
The function _stats_remove_region_id_from_group() incorecctly set
the group_id to DM_STATS_GROUP_NOT_PRESENT _before_ the call to
_stats_group_destroy(). This will cause the destroy function to
return immediately without doing anything:

 339 static void _stats_group_destroy(struct dm_stats_group *group)
 340 {
 341         if (!_stats_group_present(group))
 342                 return;

Invalidating the ID in _stats_region_region_id_from_group() is
redundant anyway; it is rightly done as the last operation in
_stats_group_destroy (and it is not possible for anything to see
the old value between the two calls).

Remove the change to group_id to ensure that the alias and bitset
resources are correctly freed.
2016-07-08 12:30:09 +01:00
Bryn M. Reeves
cc4f036d36 libdm: improve comments in stats grouping functions
Add more detailed comments to dm_stats_create_group() and
_stats_group_check_overlap().
2016-07-08 11:16:12 +01:00
Bryn M. Reeves
059a383cf8 libdm: fix resource leak in dm_stats_set_alias()
When we fail to update aux_data the newly allocated group->alias must
be freed before reinstating old_alias.
2016-07-08 11:14:29 +01:00
Bryn M. Reeves
5e06b33c51 libdm: enclose dm_stats_walk_do/while() body in do..while
The call to dm_stats_walk_start() before the do statement makes
dm_stats_walk_do() behave inconsistently depending on context;
wrap them in an additional do { } while (0) so that the macro
always expands to a valid statement.
2016-07-08 11:14:22 +01:00
Bryn M. Reeves
bf1dfea393 libdm: check for empty aux_data in _parse_aux_data_group()
If after extracting stats arguments and group tags nothing remains
of aux_data but '-' set the region->aux_data field to the empty
string to match behaviour for non-grouped regions.
2016-07-06 16:31:30 +01:00
Bryn M. Reeves
a497b95db1 libdm: use log_err_once() for group histogram message 2016-07-06 11:16:12 +01:00
Bryn M. Reeves
28658541da libdm: do not permit grouping regions with histograms
Although not harmful do not allow a group containing regions with
histograms since it is not currently possible to present histogram
data aggregated for the group.
2016-07-06 11:10:23 +01:00
Bryn M. Reeves
95ef0cdb46 libdm: check non-zero io count in _average_{rd,wr}_wait_time (Coverity)
Although a non-zero value for the number of ticks spent doing IO
should imply a non-zero number of IOs in the interval test for
this explicitly to avoid a divide-by-zero in the event of bad
counter data.
2016-07-06 09:23:13 +01:00
Bryn M. Reeves
03e03e9c11 libdm: test for zero interval_ns in _utilization() (Coverity)
It's possible for interval_ns to be zero if the interval is not
set or the clock is misconfigured. Test for this before using the
value as the divisor in the utilisation calculation.
2016-07-06 09:14:43 +01:00
Bryn M. Reeves
5d3b136d38 libdm: restore missing braces in _stats_walk_end_areas
Jumping to the end of the region table must only happen if there
are no more present, non-skipped regions, and no group walk is
configured to begin.
2016-07-06 09:04:13 +01:00
Bryn M. Reeves
69c721dd68 libdm: fix mask leak in dm_bitset_parse_list
If an unexpected '-' is found jump to the error branch so that the
mask is properly freed before returning.
2016-07-06 08:59:09 +01:00
Bryn M. Reeves
4d4f48af9f libdm: cast walk flags to uint64_t when logging.
Walk flags are ULL constants; cast the result to a uint64_t before
logging with a FMTx64 format specifier to avoid a compiler warning:

  warning: format ‘%lx’ expects argument of type ‘long unsigned int’,
  but argument 5 has type ‘long long unsigned int’
2016-07-05 20:45:24 +01:00
Bryn M. Reeves
f1dd0258f1 libdm: ensure flags constants have ULL suffix
The walk flags used by libdm-stats use the upper portion of a 64b
value: use the ULL suffix to ensure the compiler knows the expected
size.
2016-07-05 20:21:49 +01:00
Bryn M. Reeves
0f64f2d5fc libdm: fix <backtrace> in dm_stats_populate 2016-07-05 19:53:17 +01:00
Bryn M. Reeves
53e92441a6 libdm: fix <backtrace> in dm_stats_get_nr_regions 2016-07-05 19:53:17 +01:00
Bryn M. Reeves
fef4832a85 libdm: clarify library's use of aux_data
Make it clear in libdevmapper.h, and in function argument names, that
libdm-stats uses the aux_data field internally and that any values set
for user_data are appended to the library values before being stored
with a region, and similarly, that internal data fields will be stripped
prior to returning any previously stored user_data.
2016-07-05 19:53:17 +01:00
Bryn M. Reeves
cda1622fef libdm: allow deleting regions with dm_stats_delete_group()
Add a flag to dm_stats_delete_group() to allow optional deletion
of all regions belonging to the group being removed.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
f1f2df7bc0 libdm: add stats group and region iterators and properties
Add support do dm_stats_walk*() to walk over the set of
available groups using the cursor embedded in the dm_stats
handle, and to obtain the type of the object at the current
stats cursor location. A set of flags is introduced to
control which objects are visited:

    DM_STATS_WALK_AREA
    DM_STATS_WALK_REGION
    DM_STATS_WALK_GROUP
    DM_STATS_WALK_ALL

A final flag suppresses visits to regions that contain only a
single area - since the aggregate of such a region is idential
to the area it contains this allows these duplicates to be
filtered out:

    DM_STATS_WALK_SKIP_SINGLE_AREA

If flags are not initialised before beginning a walk the default
set matches the behaviour of previous versions of the library.

Also accept group identifiers as immediate arguments to the
counter, metric, and property functions by adding control
flags to the region and area identifiers passed in.

Region and area properties are mapped to their equivalents for
the group (for example: group size is reported as the sum of
all regions contained in the group). Counter and metric values
are aggregated for the region or group.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
62050760aa libdm: use defined constants for buffer sizes
Introduce constants for the buffer sizes that libdm-stats uses:
one for messages sent to the kernel, one for rows of response data
returned, and a pair for the "start+len" range and histogram bounds
strings.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
2cb9794da2 libdm: add statistics groups
Add a grouping facility to the libdm-stats library that allows the
user to bind several regions together as a group. Groups may be
used to aggregate data from several regions for reporting, or to
select and sort among large sets of regions.

A textual descriptor ("group tag") is associated with each group
and is stored in the first group member's aux_data field. The
tag contains the group member list and an optional alias for the
group, allowing the user to assign meaningful names to groups of
regions.

These descriptors are parsed in @stats_list message responses and
populate the resulting region and area tables with the group
structure.

Groups with overlapping regions are permitted but since this will
result in some events being counted more than once a warning is
printed in this case.

Nested and overlapping groups are not currently supported and
attempting to create these configurations results in error.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
4cd3dcbbc2 libdm: rename 'region' to 'skip_region' in _stats_walk_next
In libdm-stats.c 'region' usually refers to a 'struct region*'.
Rename the argument to _stats_walk_start to avoid confusion.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
82e5766062 libdm: add enum based counter and metric calls
Add a new enum based interface for accessing counter and metric
values that uses a single function for each:

uint64_t dm_stats_get_counter(const struct dm_stats *dms,
                              dm_stats_counter_t counter
                              uint64_t region_id, uint64_t area_id);

int dm_stats_get_metric(const struct dm_stats *dms, int metric,
                        uint64_t region_id, uint64_t area_id,
                        double *value);

This simplifies the implementation of value aggregation for
groups of regions. The named function interface now calls the
enum interface internally so that all new functionality is
available regardless of the method used to retrieve values.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
69f808ac8d libdm: cache dm name in stats handle
Cache the device-mapper name of a bound device in the dm_stats
handle.

This will be used by stats groups to report a device name or
user defined alias for groups.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
4a66f02f88 libdm: rename dm_stats name, devno and uuid members
The device-mapper name, device numbers and uuid stored in the
dm_stats handle are used only to bind the handle to a specific
device in order to issue ioctls.

Rename them to "bind_*" to reflect this usage in preparation
for caching the device-mapper name of the bound device in the
dm_stats handle.

This will be used to allow optional aliases to be set for
dmstats groups.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
81fad9e853 libdm: add dm_bitset_parse_list()
Add a function to parse a list of integer values and ranges into
a dm_bitset representation. Individual values signify that that bit
is set in the resulting mask and ranges are given as a pair of
start and end values, M-N, such that M and N are the first and
last members of the range (inclusive).

The implementation is based on the kernel's __bitmap_parselist()
that is used for cpumasks and other set configuration passed in
string form from user space.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
d382e66035 libdm: fix histogram pool user-after-free (CWE-825) 2016-07-05 19:53:15 +01:00
Alasdair G Kergon
d8c2677ab9 raid0: Add raid0_meta segment type. 2016-07-01 22:20:54 +01:00
Peter Rajnoha
eac0706761 libdm: do not issue 'Failed to create directory' message for failure in dm_create_dir
There are detailed messages inside _create_dir_recursive that
dm_create_dir calls (except EROFS which where the message is not
generated, like anywhere else in the code).
2016-06-29 15:58:18 +02:00
Peter Rajnoha
3985b12a2d libdm: report: fix field width calculation when using dm_report_column_headings
This fixes commit 0ba5f4b8e9 which moved
field recalculation (field width and sort position) from
dm_report_object to dm_report_output but it didn't handle the case when
dm_report_column_headings was used separately to report headings (before
dm_report_outpout call) and hence we ended up with intial widths for
fields in the headings.

If we're using dm_report_column_headings, we need to recalculate
fields if we haven't done so yet, the same way as we do in
dm_report_output.
2016-06-28 02:28:40 +01:00
Peter Rajnoha
f0768f636e coverity: fix issues detected in recent code
Uninitialized variables  (UNINIT) /safe/guest2/covscan/LVM2.2.02.158/tools/toollib.c: 3520 in _process_pvs_in_vgs()
Uninitialized variables  (UNINIT) Using uninitialized value "do_report_ret_code".

Null pointer dereferences  (REVERSE_INULL) /safe/guest2/covscan/LVM2.2.02.158/libdm/libdm-report.c: 4745 in dm_report_output()
Null pointer dereferences  (REVERSE_INULL) Null-checking "rh" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.

Incorrect expression  (MISSING_COMMA) /safe/guest2/covscan/LVM2.2.02.158/lib/log/log.c: 280 in _get_log_level_name()
Incorrect expression  (MISSING_COMMA) In the initialization of "log_level_names", a suspicious concatenated string ""noticeinfo"" is produced.

Null pointer dereferences  (FORWARD_NULL) /safe/guest2/covscan/LVM2.2.02.158/tools/reporter.c: 816 in_get_report_options()
Null pointer dereferences  (FORWARD_NULL) Comparing "mem" to null implies that "mem" might be null.
2016-06-28 02:26:54 +01:00
Peter Rajnoha
751163a743 libdm: log: remove log_print_bypass_report calls and register new print_log_libdm for libdm during lvm initialization instead
This fixes commit f50d4011cd which
introduced a problem when using older lvm2 code with newer libdm.
In this case, the old LVM didn't recognize new _LOG_BYPASS_REPORT flag
that libdm-report code used. This ended up with no output at all
from libdm where log_print_bypass_report was called because the
_LOG_BYPASS_REPORT was not masked properly in lvm2's print_log fn
which was called as callback function for logging.

With this patch, the lvm2 registers separate print_log_libdm logging
function for libdm instead. The print_log_libdm is exactly the same
as print_log (used throughout lvm2 code) but it checks whether we're
printing common line on output where "common" means not going to stderr,
not a warning and not an error and if we are, it adds the
_LOG_BYPASS_REPORT flag so the log_print goes directly to output, not
to any log report.

So this achieves the same goal as in f50d4011cd,
just doing it in a way that newer libdm is still compatible with older
lvm2 code (libdm-report is the only code using log_print).

Looking at the opposite mixture - older libdm with newer lvm2 code,
that won't be compilable because the new log report functionality
that is in lvm2 also requires new dm_report_group_* libdm functions
so we don't need to care here.
2016-06-23 14:45:52 +02:00
Peter Rajnoha
5b93db6566 libdm: select: recognize special selection string 'all' as an alias for blank selection string 2016-06-20 11:33:43 +02:00
Peter Rajnoha
2078b842fb libdm: report: add dm_report_set_selection
Since we can do repeated dm_report_output calls now, we also like
to be able to set selection for each of these outputs.
2016-06-20 11:33:43 +02:00
Peter Rajnoha
f2facdc1d0 libdm: report: add DM_REPORT_OUTPUT_MULTIPLE_TIMES report flag to keep report data even after output is done
The DM_REPORT_OUTPUT_MULTIPLE_TIMES instructs reporting code to
keep rows even after dm_report_output call - the rows are not
destroyed in this case which makes it possible to call dm_report_output
multiple times.
2016-06-20 11:33:43 +02:00
Peter Rajnoha
0ba5f4b8e9 refactor: move field width calculation and sort preparation from _do_report_object to dm_report_output
This also prepares code for repeated dm_report_output calls.
2016-06-20 11:33:42 +02:00
Peter Rajnoha
102cc4c1e2 libdm: report: remember special field to display selection status in struct row's field_sel_status variable
This allows for moving parts of the code from dm_report_object to
dm_report_output which is important for subsequent patches that allow
for repeated dm_report_output, not destroying rows on each
dm_report_output call.
2016-06-20 11:33:42 +02:00
Peter Rajnoha
f50d4011cd log: also pass log_print through report and add log_print_bypass_report for use in libdm-report for direct print without report
log_print is used during cmd line processing to log the result of the
operation (e.g. "Volume group vg successfully changed" and similar).

We don't want output from log_print to be interleaved with current
reports from group where log is reported as well. Also, the information
printed by log_print belongs to the log report too, so it should be
rerouted to log report if it's set.

Since the code in libdm-report which is responsible for doing the report
output uses log_print too, we need to use a different kind of log_print
which bypasses any log report currently used for logging (...simply,
we can't call log_print to output the log report itself which in turn
would again reroute to report - the report would never get on output
this way).
2016-06-20 11:33:42 +02:00