1
0
mirror of git://sourceware.org/git/lvm2.git synced 2024-12-29 15:22:30 +03:00
Commit Graph

94 Commits

Author SHA1 Message Date
Bryn M. Reeves
e8d966bc31 libdm: use dm_bit_get_last() in _stats_group_tag_fill()
Instead of iterating over all bits, use dm_bit_get_last() to find
the last set bit in the group bitmap.
2016-12-13 21:02:18 +00:00
Bryn M. Reeves
930b0b4c9e libdm: fix start of file detection in _stats_map_extents() 2016-12-13 20:25:47 +00:00
Bryn M. Reeves
eb65572217 libdm: break up _stats_get_extents_for_file()
Split out the loop that iterates over each batch of FIEMAP
extent data from the function that sets up and calls the ioctl
to reduce nesting and simplify local variable use:

  _stats_get_extents_for_file()
  ->  _stats_map_extents()

The _stats_map_extents() function is responsible for detecting
eof and extent boundaries and adding whole, allocated extents
to the file extent table for region creation.
2016-12-13 20:25:45 +00:00
Bryn M. Reeves
8e33972828 libdm: check for non-existent region_id values in groups
Check that all region_id values specified in a group bitmap are
actually present: although this should not normally happen when
using the dmstats tool, it is possible as a result of manual
changes (or bugs) for a group descriptor to contain one or more
group_id values that do not exist.

Check for this situation when reading group descriptors, warn
the user the user, and clear these bits in the bitmap when
formatting it for output.
2016-12-13 15:37:48 +00:00
Bryn M. Reeves
99b6d82e2d libdm: fix segfault with invalid group descriptor
If a region has a a DMS_GROUP tag in aux_data where the first
region_id in the bitmap is not the same as the containing region,
dmstats will segfault:

  # '2' is never a valid group bitset list for region_id == 0
  # dmsetup message vg_hex/root 0 "@stats_set_aux 0 DMS_GROUP=img:2#"

  # dmsetup message vg_hex/root 0 "@stats_list"
  0: 45383680+16384 16384 dmstats DMS_GROUP=img:2#
  1: 46071808+32768 32768 dmstats -
  2: 47382528+16384 16384 dmstats -

  # dmstats list
  Segmentation fault (core dumped)

The crash will occur in some arbitrary dm_stats_get_* property
method - this happens while processing the 1st region_id in the
bitset, because the region is marked as grouped, but there is
no group bitmap present at dms->groups[2]->regions.

Fix this by detecting a mismatch between the expected region_id
and dm_bit_get_first() for the parsed bitset during
_parse_aux_data_group().
2016-12-13 14:37:41 +00:00
Bryn M. Reeves
138e4336fd libdm: fix region overlap tests 2016-12-13 09:09:29 +00:00
Bryn M. Reeves
93f420caf4 libdm: fix _stats_get_extents_for_file()
Handle files that contain multiple logical extents in a single
physical extent properly:

  - In FIEMAP terms a logical extent is a contiguous range of
    sectors in the file's address space.

  - One or more physically adjacent logical extents comprise a
    physical extent: these are the disk areas that will be mapped
    to regions.

  - An extent boundary occurs when the start sector of extent
    n+1 is not equal to (n.start + n.length).

This requires that we accumulate the length values of extents
returned by FIEMAP until a discontinuity is found (since each
struct fiemap_extent returned by FIEMAP only represents a single
logical extent, which may be contiguous with other logical
extents on-disk).

This avoids creating large numbers of regions for physically
adjacent (logical) extents and fixes the earlier behaviour which
would only map the first logical extent of the physical extent,
leaving gaps in the region table for these files.
2016-12-13 09:09:25 +00:00
Bryn M. Reeves
b311122a56 libdm: fix filemap cleanup loop condition 2016-12-10 13:31:12 +00:00
Bryn M. Reeves
d8ba8ee9ae libdm: use a private pool for filemap extent table
When mapping regions to a file descriptor, a temporary table of
extent descriptors is built using the dm_pool object building
interface.

Previously this use borrowed the dms->mem region and counter
table pool (since nothing can interleave with the allocation
while the caller is still in dm_stats_create_regions_from_fd()).

This turns out to be problematic for error recovery. When a
region creation operation fails partway through file mapping,
we need to roll back the set of already created regions and
this requires a listed handle: the dm_stats_list() will then
allocate from the same pool as the extents; we either have
to throw away valid list data, or leak the extent table, to
return the handle in a valid state.

Avoid this problem by creating a new, temporary mem pool in
_stats_create_file_regions() to hold the extent data, and
discarding it on exit from the function.
2016-12-10 13:31:12 +00:00
Bryn M. Reeves
2d1dbb9edd libdm: fix performance of failed filemap cleanup
While cleaning up the table of already created regions during a
failed dm_stats_create_regions_from_fd(), list the handle once,
and call _stats_delete_region() directly. This avoids sending a
@stats_list message for each region deleted, reducing runtime
from 6s to 0.7s when cleaning up ~250 out of ~10000 regions:

  # time dmstats create --filemap b.img
  device-mapper: message ioctl on (253:0) failed: Cannot allocate memory
  Failed to create region 246 of 309 at 9388032.
  Could not create regions from file /root/b.img
  << pauses here >>
  Command failed

  real	0m6.267s
  user	0m3.770s
  sys	0m2.487s

  # time dmstats create --filemap b.img
  device-mapper: message ioctl on (253:0) failed: Cannot allocate memory
  Failed to create region 246 of 309 at 9388032.
  Could not create regions from file /root/b.img
  Command failed

  real	0m0.716s
  user	0m0.034s
  sys	0m0.581s

Testing the error path requires region creation to start to
fail part way through the operation (in order to have regions
to clean up): the simplest way is to ensure the system is
close to the kernel limit of 1/4 RAM or 1/2 vmalloc space
consumed by dmstats data.
2016-12-10 11:59:16 +00:00
Bryn M. Reeves
97c4490cc5 libdm: split off internal _stats_delete_region()
Split dm_stats_delete_region() so that internal callers can manage
the handle state themselves.

dm_stats_delete_region() now just handles checking the state of the
handle, reporting validation errors, and calling dm_stats_list() if
necessary, before calling _stats_delete_region().

The new _stats_delete_region() function performs the actual group
member removal and region deletion, and requires a fully listed
handle to operate.

Callers that repeatedly delete regions can use a single listed
handle for many operations on the same device, avoiding one
message ioctl per region deleted: since @stats_list with many
regions is expensive, this yields large runtime improvements.
2016-12-10 11:57:14 +00:00
Bryn M. Reeves
30ad254d84 libdm: use correct region_id when cleaning up a failed filemap
If we fail to create a region during dm_stats_create_regions_from_fd(),
we must remove all regions that were created to do this to date. This
needs to loop over the table of region_id values that were populated
by _stats_create_file_regions() before the error.

The code for this failure case in the out_remove branch incorrectly
uses the table index as the region_id:

    for (--i; i != DM_STATS_REGION_NOT_PRESENT; i--) {
            if (!dm_stats_delete_region(dms, i))
                    log_error("Could not delete region " FMTu64 ".", i);
    }

This causes the cleanup code to delete a completely unrelated set
of regions (since the index here will always be nr_regions..0).

Fix it to pass the actual region_id stored in regions[i] instead.
2016-12-09 16:04:13 +00:00
Bryn M. Reeves
7fd2fa22dd libdm-stats: clear dms->groups in _stats_groups_destroy() 2016-12-09 16:04:13 +00:00
Bryn M. Reeves
cb8c04760f libdm-stats: clear dms->regions in _stats_regions_destroy() 2016-12-09 16:04:13 +00:00
Bryn M. Reeves
6dd0bd0255 libdm-stats: fix dm_stats_delete_region() performance
Fix a silly bug in dm_stats_delete_region() that hugely inflates
runtimes when deleting a large number of regions.

For ~50,000 regions this change reduces the runtime from 98s to
6s on my test systems (a ~93% reduction).

The bug exists because dm_stats_delete_region() applies a truth
test to the return value of dm_stats_get_nr_areas(); this is
never correct usage - it will walk the entire region table and
calculate area counts for each region (which is roughly O(n^2)
in the number of regions, as dm_stats_delete_region() is being
called inside a region walk).

Although the individual area calculation is not that costly,
uselessly running anything 2,500,000,000 times over gets a bit
slow.

A much cheaper test (which is always true if the areas check is
true) is to just test dm_stats_get_nr_regions() or dms->regions;
if either is true it implies at least one area exists.

Old:

 Performance counter stats for 'dmstats delete --allregions --alldevices':

      98117.791458      task-clock (msec)         #    1.000 CPUs utilized
               127      context-switches          #    0.001 K/sec
                 3      cpu-migrations            #    0.000 K/sec
             6,631      page-faults               #    0.068 K/sec
   307,711,724,562      cycles                    #    3.136 GHz
   544,762,959,577      instructions              #    1.77  insn per cycle
    84,287,824,115      branches                  #  859.047 M/sec
         2,538,875      branch-misses             #    0.00% of all branches

      98.119578733 seconds time elapsed

New:

 Performance counter stats for 'dmstats delete --allregions --alldevices':

       6427.251074      task-clock (msec)         #    1.000 CPUs utilized
                 6      context-switches          #    0.001 K/sec
                 0      cpu-migrations            #    0.000 K/sec
             6,634      page-faults               #    0.001 M/sec
    21,613,018,724      cycles                    #    3.363 GHz
     3,794,755,445      instructions              #    0.18  insn per cycle
       852,974,026      branches                  #  132.712 M/sec
           808,625      branch-misses             #    0.09% of all branches

       6.428953647 seconds time elapsed
2016-12-09 10:55:39 +00:00
Zdenek Kabelac
1d58074d9f debug: more stacktrace corrections
Continue previous patch dropping some unneeded stack traces
after printed log_error/warn messages.
2016-11-25 14:58:28 +01:00
Bryn M. Reeves
2a86f54b09 libdm: separate dm_stats_populate() error cases
There are two possible errors in _dm_stats_populate_region():

  * No region struct in dms->regions[region_id]
  * Failure to parse data from @stats_print

These have very different causes: the first occurs where a client
program is populating one region at a time (region_id is a single
region identifier), and has not previously called dm_stats_list()
to dimension the region tables; this is an API usage error.

The second occurs when either we read unparseable data from the
kernel (kernel bug), or where various resource allocations fail.

Separate these two cases out and log separate messages for each
(allocation failures in the path already have their own distinct
message), since the "failed to parse.." message in the un-listed
handle case is confusing and misleading.
2016-11-17 11:39:43 +00:00
Zdenek Kabelac
4e26024add cleanup: use WARNING prefix for log_warn
Use capital WARNING prefix for log_warn() messages.
2016-11-03 17:49:07 +01:00
Zdenek Kabelac
9cbe4c1af9 log_info to log_very_verbose
Translate log_info() into log_very_verbose() which is macro
supposed to be used by our code.

log_info() is internal macro with eventually some 'symbolic' meaning
in syslogging daemons.
2016-11-03 17:49:07 +01:00
Bryn M. Reeves
e95a252974 libdm: convert FIEMAP buffer allocation from stack to dm_zalloc 2016-10-03 15:14:33 +01:00
Bryn M. Reeves
56c90ffa5e libdm: fix dm_stats_delete_region() backwards compat
The dm_stats_delete_region() call removes a region from the bound
device, and, if the region is grouped, from the group leader
group descriptor stored in aux_data.

To do this requires a listed handle: previous versions of the
library do not since no dependencies exist between regions without
grouping.

This leads to strange behaviour when a command built against an old
version of the library is used with one supporting groups. Deleting
a region with dmstats succeeds, but logs errors:

  # dmstats list
  Name             RgID RgSta RgSiz #Areas ArSize ProgID
  vg_hex-root         0     0 1.00g      1  1.00g dmstats
  vg_hex-root         1 1.00g 1.00g      1  1.00g dmstats
  vg_hex-root         2 2.00g 1.00g      1  1.00g dmstats
  # dmstats delete --regionid 2 vg_hex/root
  Region ID 2 does not exist
  Could not delete statistics region.
  Command failed
  # dmstats list
  Name             RgID RgSta RgSiz #Areas ArSize ProgID
  vg_hex-root         0     0 1.00g      1  1.00g dmstats
  vg_hex-root         1 1.00g 1.00g      1  1.00g dmstats

This happens because the call to dm_stats_delete_region() is inside
a dm_stats_walk_*() iterator: upon entry to the call, the iterator
is at its end conditions and about to terminate. Due to the call to
dm_stats_list() inside the function, it returns with an iterator at
the beginning of a walk and performs a further iteration before
exiting. This final loop makes a further attempt to delete the
(already deleted) region, leading to the confusing error messages.
2016-09-27 17:58:05 +01:00
Bryn M. Reeves
6ec8854fdb libdm: fix stats walk compatibility with older dmsetup
The current dmsetup.c handles DR_STATS and DR_STATS_META reports
separately in _display_info_cols(), meaning that the stats walk
functions are never called for these report types.

Versions before v2.02.159 have a loop using dm_stats_walk_do() and
dm_stats_walk_while(), that executes once for non-stats reports,
and once per region, or area, for DR_STATS/DR_STATS_META reports.

This older behaviour relies on the documented behaviour that the
walk functions will accept a NULL pointer as the struct dm_stats*
argument.

This was broken by commit f1f2df7b: the NULL test on dms and
dms->regions were incorrectly moved from the dm_stats_walk_end()
wrapper to the internal '_stats_walk_end()' helper.

Since the pointer is dereferenced in between these points, using
an older dmsetup with current libdm results in a segfault when
running a non-stats report:

  # dmsetup info -c vg00/lvol0
  Segmentation fault (core dumped)

Restore the NULL checks to the wrapper function as intended.
2016-09-27 14:46:00 +01:00
Bryn M. Reeves
c26cd48536 libdm: fix end-of-groups test in _stats_walk_end() 2016-09-16 13:09:22 +01:00
Zdenek Kabelac
92d5a84410 cleanup: clean gcc6 minor/major types warnings
Put sys/sysmacros.h in front of sys/types.h header file as requested
by gcc6.
2016-08-29 20:51:16 +02:00
Bryn M. Reeves
4ef1f34549 libdm: free hist_arg in _stats_create_file_regions() (Coverity) 2016-07-18 18:09:21 +01:00
Bryn M. Reeves
309bdfa224 libdm: add configure.in checks for fiemap.h and magic.h 2016-07-08 23:42:32 +01:00
Bryn M. Reeves
61cb58e549 libdm: use 'goto_bad' if extent pool allocation fails
Generate a backtrace if unable to extend the extent table.
2016-07-08 22:28:50 +01:00
Bryn M. Reeves
feb69966d4 libdm: use macro for boundary test in _stats_get_extents_for_file() 2016-07-08 22:21:14 +01:00
Bryn M. Reeves
2d1f03b616 libdm: use a constant for FIEMAP buffer size 2016-07-08 22:05:36 +01:00
Bryn M. Reeves
58bfea6a6e libdm: use SECTOR_SHIFT constant in _stats_add_extent() 2016-07-08 22:05:36 +01:00
Bryn M. Reeves
4bb57341bd libdm: enable creation of filemap regions with histograms 2016-07-08 17:27:52 +01:00
Bryn M. Reeves
db73d756e9 libdm: allow regions with histograms in dm_stats_create_group()
Allow regions with histograms to be grouped if all histograms have
the same number of bins and matching bounds.
2016-07-08 17:27:52 +01:00
Bryn M. Reeves
ae9cffba52 libdm: add aggregation support to dm_stats_get_histogram()
Support aggregate group and region histograms by allocating a new
histogram from the pool and populating it with a sum of the histogram
data for the areas contained in the region or group.

To avoid repeatedly summing the same histogram data, cache the pointer
in the group and regions structs for subsequent access. The aggregate
histograms are allocated from the same pool as the area histograms in
the corresponding handle and will be discarded at each list or populate
operation.
2016-07-08 17:27:52 +01:00
Bryn M. Reeves
e104825916 libdm: add dm_stats_create_regions_from_fd()
Add a call to create dmstats regions that correspond to the extents
present in a file descriptor open on a file in a local file system.
The file must reside on a file system type that correctly supports
physical extent location data in the FIEMAP ioctl.

Regions are optionally placed into a group with a user-defined alias.

File systems that do not support physical offsets in FIEMAP (btrfs
currently) are detected via fstatfs() - although attempting to map
a --filemap group on btrfs will fail anyway with the generic error
"Not on a device-mapper device" this is confusing; the file system
mount is on a device-mapper device, but btrfs' volume layer masks
this in the returned st_dev field since the returned logical file
extents may span multiple physical devices.
2016-07-08 14:34:41 +01:00
Bryn M. Reeves
ebc7fc67c8 libdm: fix group resource leak in dm_stats_delete_region()
The function _stats_remove_region_id_from_group() incorecctly set
the group_id to DM_STATS_GROUP_NOT_PRESENT _before_ the call to
_stats_group_destroy(). This will cause the destroy function to
return immediately without doing anything:

 339 static void _stats_group_destroy(struct dm_stats_group *group)
 340 {
 341         if (!_stats_group_present(group))
 342                 return;

Invalidating the ID in _stats_region_region_id_from_group() is
redundant anyway; it is rightly done as the last operation in
_stats_group_destroy (and it is not possible for anything to see
the old value between the two calls).

Remove the change to group_id to ensure that the alias and bitset
resources are correctly freed.
2016-07-08 12:30:09 +01:00
Bryn M. Reeves
cc4f036d36 libdm: improve comments in stats grouping functions
Add more detailed comments to dm_stats_create_group() and
_stats_group_check_overlap().
2016-07-08 11:16:12 +01:00
Bryn M. Reeves
059a383cf8 libdm: fix resource leak in dm_stats_set_alias()
When we fail to update aux_data the newly allocated group->alias must
be freed before reinstating old_alias.
2016-07-08 11:14:29 +01:00
Bryn M. Reeves
bf1dfea393 libdm: check for empty aux_data in _parse_aux_data_group()
If after extracting stats arguments and group tags nothing remains
of aux_data but '-' set the region->aux_data field to the empty
string to match behaviour for non-grouped regions.
2016-07-06 16:31:30 +01:00
Bryn M. Reeves
a497b95db1 libdm: use log_err_once() for group histogram message 2016-07-06 11:16:12 +01:00
Bryn M. Reeves
28658541da libdm: do not permit grouping regions with histograms
Although not harmful do not allow a group containing regions with
histograms since it is not currently possible to present histogram
data aggregated for the group.
2016-07-06 11:10:23 +01:00
Bryn M. Reeves
95ef0cdb46 libdm: check non-zero io count in _average_{rd,wr}_wait_time (Coverity)
Although a non-zero value for the number of ticks spent doing IO
should imply a non-zero number of IOs in the interval test for
this explicitly to avoid a divide-by-zero in the event of bad
counter data.
2016-07-06 09:23:13 +01:00
Bryn M. Reeves
03e03e9c11 libdm: test for zero interval_ns in _utilization() (Coverity)
It's possible for interval_ns to be zero if the interval is not
set or the clock is misconfigured. Test for this before using the
value as the divisor in the utilisation calculation.
2016-07-06 09:14:43 +01:00
Bryn M. Reeves
5d3b136d38 libdm: restore missing braces in _stats_walk_end_areas
Jumping to the end of the region table must only happen if there
are no more present, non-skipped regions, and no group walk is
configured to begin.
2016-07-06 09:04:13 +01:00
Bryn M. Reeves
4d4f48af9f libdm: cast walk flags to uint64_t when logging.
Walk flags are ULL constants; cast the result to a uint64_t before
logging with a FMTx64 format specifier to avoid a compiler warning:

  warning: format ‘%lx’ expects argument of type ‘long unsigned int’,
  but argument 5 has type ‘long long unsigned int’
2016-07-05 20:45:24 +01:00
Bryn M. Reeves
0f64f2d5fc libdm: fix <backtrace> in dm_stats_populate 2016-07-05 19:53:17 +01:00
Bryn M. Reeves
53e92441a6 libdm: fix <backtrace> in dm_stats_get_nr_regions 2016-07-05 19:53:17 +01:00
Bryn M. Reeves
fef4832a85 libdm: clarify library's use of aux_data
Make it clear in libdevmapper.h, and in function argument names, that
libdm-stats uses the aux_data field internally and that any values set
for user_data are appended to the library values before being stored
with a region, and similarly, that internal data fields will be stripped
prior to returning any previously stored user_data.
2016-07-05 19:53:17 +01:00
Bryn M. Reeves
cda1622fef libdm: allow deleting regions with dm_stats_delete_group()
Add a flag to dm_stats_delete_group() to allow optional deletion
of all regions belonging to the group being removed.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
f1f2df7bc0 libdm: add stats group and region iterators and properties
Add support do dm_stats_walk*() to walk over the set of
available groups using the cursor embedded in the dm_stats
handle, and to obtain the type of the object at the current
stats cursor location. A set of flags is introduced to
control which objects are visited:

    DM_STATS_WALK_AREA
    DM_STATS_WALK_REGION
    DM_STATS_WALK_GROUP
    DM_STATS_WALK_ALL

A final flag suppresses visits to regions that contain only a
single area - since the aggregate of such a region is idential
to the area it contains this allows these duplicates to be
filtered out:

    DM_STATS_WALK_SKIP_SINGLE_AREA

If flags are not initialised before beginning a walk the default
set matches the behaviour of previous versions of the library.

Also accept group identifiers as immediate arguments to the
counter, metric, and property functions by adding control
flags to the region and area identifiers passed in.

Region and area properties are mapped to their equivalents for
the group (for example: group size is reported as the sum of
all regions contained in the group). Counter and metric values
are aggregated for the region or group.
2016-07-05 19:53:16 +01:00
Bryn M. Reeves
62050760aa libdm: use defined constants for buffer sizes
Introduce constants for the buffer sizes that libdm-stats uses:
one for messages sent to the kernel, one for rows of response data
returned, and a pair for the "start+len" range and histogram bounds
strings.
2016-07-05 19:53:16 +01:00