mirror of
git://sourceware.org/git/lvm2.git
synced 2024-12-21 13:34:40 +03:00
Initial import of document describing LVM's policies
surrounding device faults/failures.
This commit is contained in:
Jonathan Earl Brassow
parent
e78d473b43
commit
b5097c8462
221
doc/lvm_fault_handling.txt
Normal file
221
doc/lvm_fault_handling.txt
Normal file
@ -0,0 +1,221 @@
|
|||||||
|
LVM device fault handling
|
||||||
|
=========================
|
||||||
|
|
||||||
|
Introduction
|
||||||
|
------------
|
||||||
|
This document is to serve as the definitive source for information
|
||||||
|
regarding the policies and procedures surrounding device failures
|
||||||
|
in LVM. It codifies LVM's responses to device failures as well as
|
||||||
|
the responsibilities of administrators.
|
||||||
|
|
||||||
|
Device failures can be permanent or transient. A permanent failure
|
||||||
|
is one where a device becomes inaccessible and will never be
|
||||||
|
revived. A transient failure is a failure that can be recovered
|
||||||
|
from (e.g. a power failure, intermittent network outage, block
|
||||||
|
relocation, etc). The policies for handling both types of failures
|
||||||
|
is described herein.
|
||||||
|
|
||||||
|
Available Operations During a Device Failure
|
||||||
|
--------------------------------------------
|
||||||
|
When there is a device failure, LVM behaves somewhat differently because
|
||||||
|
only a subset of the available devices will be found for the particular
|
||||||
|
volume group. The number of operations available to the administrator
|
||||||
|
is diminished. It is not possible to create new logical volumes while
|
||||||
|
PVs cannot be accessed, for example. Operations that create, convert, or
|
||||||
|
resize logical volumes are disallowed, such as:
|
||||||
|
- lvcreate
|
||||||
|
- lvresize
|
||||||
|
- lvreduce
|
||||||
|
- lvextend
|
||||||
|
- lvconvert (unless '--repair' is used)
|
||||||
|
Operations that activate, deactivate, remove, report, or repair logical
|
||||||
|
volumes are allowed, such as:
|
||||||
|
- lvremove
|
||||||
|
- vgremove (will remove all LVs, but not the VG until consistent)
|
||||||
|
- pvs
|
||||||
|
- vgs
|
||||||
|
- lvs
|
||||||
|
- lvchange -a [yn]
|
||||||
|
- vgchange -a [yn]
|
||||||
|
Operations specific to the handling of failed devices are allowed and
|
||||||
|
are as follows:
|
||||||
|
|
||||||
|
- 'vgreduce --removemissing <VG>': This action is designed to remove
|
||||||
|
the reference of a failed device from the LVM metadata stored on the
|
||||||
|
remaining devices. If there are (portions of) logical volumes on the
|
||||||
|
failed devices, the ability of the operation to proceed will depend
|
||||||
|
on the type of logical volumes found. If an image (i.e leg or side)
|
||||||
|
of a mirror is located on the device, that image/leg of the mirror
|
||||||
|
is eliminated along with the failed device. The result of such a
|
||||||
|
mirror reduction could be a no-longer-redundant linear device. If
|
||||||
|
a linear, stripe, or snapshot device is located on the failed device
|
||||||
|
the command will not proceed without a '--force' option. The result
|
||||||
|
of using the '--force' option is the entire removal and complete
|
||||||
|
loss of the non-redundant logical volume. Once this operation is
|
||||||
|
complete, the volume group will again have a complete and consistent
|
||||||
|
view of the devices it contains. Thus, all operations will be
|
||||||
|
permitted - including creation, conversion, and resizing operations.
|
||||||
|
|
||||||
|
- 'lvconvert --repair <VG/LV>': This action is designed specifically
|
||||||
|
to operate on mirrored logical volumes. It is used on logical volumes
|
||||||
|
individually and does not remove the faulty device from the volume
|
||||||
|
group. If, for example, a failed device happened to contain the
|
||||||
|
images of four distinct mirrors, it would be necessary to run
|
||||||
|
'lvconvert --repair' on each of them. The ultimate result is to leave
|
||||||
|
the faulty device in the volume group, but have no logical volumes
|
||||||
|
referencing it. In addition to removing mirror images that reside
|
||||||
|
on failed devices, 'lvconvert --repair' can also replace the failed
|
||||||
|
device if there are spare devices available in the volume group. The
|
||||||
|
user is prompted whether to simply remove the failed portions of the
|
||||||
|
mirror or to also allocate a replacement, if run from the command-line.
|
||||||
|
Optionally, the '--use-policies' flag can be specified which will
|
||||||
|
cause the operation not to prompt the user, but instead respect
|
||||||
|
the policies outlined in the LVM configuration file - usually,
|
||||||
|
/etc/lvm/lvm.conf. Once this operation is complete, mirrored logical
|
||||||
|
volumes will be consistent and I/O will be allowed to continue.
|
||||||
|
However, the volume group will still be inconsistent - due to the
|
||||||
|
refernced-but-missing device/PV - and operations will still be
|
||||||
|
restricted to the aformentioned actions until either the device is
|
||||||
|
restored or 'vgreduce --removemissing' is run.
|
||||||
|
|
||||||
|
Device Revival (transient failures):
|
||||||
|
------------------------------------
|
||||||
|
During a device failure, the above section describes what limitations
|
||||||
|
a user can expect. However, if the device returns after a period of
|
||||||
|
time, what to expect will depend on what has happened during the time
|
||||||
|
period when the device was failed. If no automated actions (described
|
||||||
|
below) or user actions were necessary or performed, then no change in
|
||||||
|
operations or logical volume layout will occur. However, if an
|
||||||
|
automated action or one of the aforementioned repair commands was
|
||||||
|
manually run, the returning device will be perceived as having stale
|
||||||
|
LVM metadata. In this case, the user can expect to see a warning
|
||||||
|
concerning inconsistent metadata. The metadata on the returning
|
||||||
|
device will be automatically replaced with the latest copy of the
|
||||||
|
LVM metadata - restoring consistency. Note, while most LVM commands
|
||||||
|
will automatically update the metadata on a restored devices, the
|
||||||
|
following possible exceptions exist:
|
||||||
|
- pvs (when it does not read/update VG metadata)
|
||||||
|
|
||||||
|
Automated Target Response to Failures:
|
||||||
|
--------------------------------------
|
||||||
|
The only LVM target type (i.e. "personality") that has an automated
|
||||||
|
response to failures is a mirrored logical volume. The other target
|
||||||
|
types (linear, stripe, snapshot, etc) will simply propagate the failure.
|
||||||
|
[A snapshot becomes invalid if its underlying device fails, but the
|
||||||
|
origin will remain valid - presuming the origin device has not failed.]
|
||||||
|
There are three types of errors that a mirror can suffer - read, write,
|
||||||
|
and resynchronization errors. Each is described in depth below.
|
||||||
|
|
||||||
|
Mirror read failures:
|
||||||
|
If a mirror is 'in-sync' (i.e. all images have been initialized and
|
||||||
|
are identical), a read failure will only produce a warning. Data is
|
||||||
|
simply pulled from one of the other images and the fault is recorded.
|
||||||
|
Sometimes - like in the case of bad block relocation - read errors can
|
||||||
|
be recovered from by the storage hardware. Therefore, it is up to the
|
||||||
|
user to decide whether to reconfigure the mirror and remove the device
|
||||||
|
that caused the error. Managing the composition of a mirror is done with
|
||||||
|
'lvconvert' and removing a device from a volume group can be done with
|
||||||
|
'vgreduce'.
|
||||||
|
|
||||||
|
If a mirror is not 'in-sync', a read failure will produce an I/O error.
|
||||||
|
This error will propagate all the way up to the applications above the
|
||||||
|
logical volume (e.g. the file system). No automatic intervention will
|
||||||
|
take place in this case either. It is up to the user to decide what
|
||||||
|
can be done/salvaged in this senario. If the user is confident that the
|
||||||
|
images of the mirror are the same (or they are willing to simply attempt
|
||||||
|
to retreive whatever data they can), 'lvconvert' can be used to eliminate
|
||||||
|
the failed image and proceed.
|
||||||
|
|
||||||
|
Mirror resynchronization errors:
|
||||||
|
A resynchronization error is one that occurs when trying to initialize
|
||||||
|
all mirror images to be the same. It can happen due to a failure to
|
||||||
|
read the primary image (the image considered to have the 'good' data), or
|
||||||
|
due to a failure to write the secondary images. This type of failure
|
||||||
|
only produces a warning, and it is up to the user to take action in this
|
||||||
|
case. If the error is transient, the user can simply reactivate the
|
||||||
|
mirrored logical volume to make another attempt at resynchronization.
|
||||||
|
If attempts to finish resynchronization fail, 'lvconvert' can be used to
|
||||||
|
remove the faulty device from the mirror.
|
||||||
|
|
||||||
|
TODO...
|
||||||
|
Some sort of response to this type of error could be automated.
|
||||||
|
Since this document is the definitive source for how to handle device
|
||||||
|
failures, the process should be defined here. If the process is defined
|
||||||
|
but not implemented, it should be noted as such. One idea might be to
|
||||||
|
make a single attempt to suspend/resume the mirror in an attempt to
|
||||||
|
redo the sync operation that failed. On the other hand, if there is
|
||||||
|
a permanent failure, it may simply be best to wait for the user or the
|
||||||
|
automated response that is sure to follow from a write failure.
|
||||||
|
...TODO
|
||||||
|
|
||||||
|
Mirror write failures:
|
||||||
|
When a write error occurs on a mirror constituent device, an attempt
|
||||||
|
to handle the failure is automatically made. This is done by calling
|
||||||
|
'lvconvert --repair --use-policies'. The policies implied by this
|
||||||
|
command are set in the LVM configuration file. They are:
|
||||||
|
- mirror_log_fault_policy: This defines what action should be taken
|
||||||
|
if the device containing the log fails. The available options are
|
||||||
|
"remove" and "allocate". Either of these options will cause the
|
||||||
|
faulty log device to be removed from the mirror. The "allocate"
|
||||||
|
policy will attempt the further action of trying to replace the
|
||||||
|
failed disk log by using space that might be available in the
|
||||||
|
volume group. If the allocation fails (or the "remove" policy
|
||||||
|
is specified), the mirror log will be maintained in memory. Should
|
||||||
|
the machine be rebooted or the logical volume deactivated, a
|
||||||
|
complete resynchronization of the mirror will be necessary upon
|
||||||
|
the follow activation - such is the nature of a mirror with a 'core'
|
||||||
|
log. The default policy for handling log failures is "allocate".
|
||||||
|
The service disruption incurred by replacing the failed log is
|
||||||
|
negligible, while the benefits of having persistent log is
|
||||||
|
pronounced.
|
||||||
|
- mirror_image_fault_policy: This defines what action should be taken
|
||||||
|
if a device containing an image fails. Again, the available options
|
||||||
|
are "remove" and "allocate". Both of these options will cause the
|
||||||
|
faulty image device to be removed - adjusting the logical volume
|
||||||
|
accordingly. For example, if one image of a 2-way mirror fails, the
|
||||||
|
mirror will be converted to a linear device. If one image of a
|
||||||
|
3-way mirror fails, the mirror will be converted to a 2-way mirror.
|
||||||
|
The "allocate" policy takes the further action of trying to replace
|
||||||
|
the failed image using space that is available in the volume group.
|
||||||
|
Replacing a failed mirror image will incure the cost of
|
||||||
|
resynchronizing - degrading the performance of the mirror. The
|
||||||
|
default policy for handling an image failure is "remove". This
|
||||||
|
allows the mirror to still function, but gives the administrator the
|
||||||
|
choice of when to incure the extra performance costs of replacing
|
||||||
|
the failed image.
|
||||||
|
|
||||||
|
TODO...
|
||||||
|
The appropriate time to take permanent corrective action on a mirror
|
||||||
|
should be driven by policy. There should be a directive that takes
|
||||||
|
a time or percentage argument. Something like the following:
|
||||||
|
- mirror_fault_policy_WHEN = "10sec"/"10%"
|
||||||
|
A time value would signal the amount of time to wait for transient
|
||||||
|
failures to resolve themselves. The percentage value would signal the
|
||||||
|
amount a mirror could become out-of-sync before the faulty device is
|
||||||
|
removed.
|
||||||
|
|
||||||
|
A mirror cannot be used unless /some/ corrective action is taken,
|
||||||
|
however. One option is to replace the failed mirror image with an
|
||||||
|
error target, forgo the use of 'handle_errors', and simply let the
|
||||||
|
out-of-sync regions accumulate and be tracked by the log. Mirrors
|
||||||
|
that have more than 2 images would have to "stack" to perform the
|
||||||
|
tracking, as each failed image would have to be associated with a
|
||||||
|
log. If the failure is transient, the device would replace the
|
||||||
|
error target that was holding its spot and the log that was tracking
|
||||||
|
the deltas would be used to quickly restore the portions that changed.
|
||||||
|
|
||||||
|
One unresolved issue with the above scheme is how to know which
|
||||||
|
regions of the mirror are out-of-sync when a problem occurs. When
|
||||||
|
a write failure occurs in the kernel, the log will contain those
|
||||||
|
regions that are not in-sync. If the log is a disk log, that log
|
||||||
|
could continue to be used to track differences. However, if the
|
||||||
|
log was a core log - or if the log device failed at the same time
|
||||||
|
as an image device - there would be no way to determine which
|
||||||
|
regions are out-of-sync to begin with as we start to track the
|
||||||
|
deltas for the failed image. I don't have a solution for this
|
||||||
|
problem other than to only be able to handle errors in this way
|
||||||
|
if conditions are right. These issues will have to be ironed out
|
||||||
|
before proceeding. This could be another case, where it is better
|
||||||
|
to handle failures in the kernel by allowing the kernel to store
|
||||||
|
updates in various metadata areas.
|
||||||
|
...TODO
|
||||||
Loading…
Reference in New Issue
Block a user