5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2025-01-18 06:03:59 +03:00

1758 Commits

Author SHA1 Message Date
Fabian Ebner
904f06a82e pbs: update attribute: cleaner error message if not supported
Reported-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-12 16:53:27 +01:00
Fabian Grünbichler
c7aa1e28dd bump version to 7.0-15
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-10 14:25:26 +01:00
Fabian Ebner
6fbffac0a3 lvm thin: don't assume that a thin pool and its volumes are active
There are cases where autoactivation can fail, as reported in the
community forum [0]. And it could also be that a volume was
deactivated by something outside of our control.

It doesn't seem strictly necessary to activate the thin pool itself
(creating/removing/activating LVs within the pool still works if it's
not active), but it does not report usage information as long as
neither the pool nor any of its LVs are active. Activate the pool for
that, for being able to use the flag in status(), and it should also
serve as a good indicator that there's a problem with the pool if it
can't be activated.

Before activating, check the (cached) lv_state from lvm_list_volumes.
It's necessary to update the cache in activate_storage, because the
flag is re-used in status(). Also update it for other (de)activations
to be more future-proof.

[0]: https://forum.proxmox.com/threads/local-lvm-not-available-after-kernel-update-on-pve-7.97406

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 14:23:54 +01:00
Fabian Ebner
2668a86719 lvm thin: status: code cleanup
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 14:18:28 +01:00
Fabian Ebner
cde43c4880 api: disks: delete: add flag for cleaning up storage config
Update node restrictions to reflect that the storage is not available
anymore on the particular node. If the storage was only configured for
that node, remove it altogether.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>

slight style fixup

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
f81908eb58 api: disks: delete: add flag for wiping disks
For ZFS and directory storages, clean up the whole disk when the
layout is as usual to avoid left-overs.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
26082b7daf diskmanage: add helper for udev workaround
to avoid duplication. Current callers pass along at least one device,
but anticipate future callers that might call with the empty list. Do
nothing in that case, rather than triggering everything.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
a83d8eb178 api: disks: add DELETE endpoint for directory, lvm, lvmthin, zfs
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
a510449e2b api: list thin pools: add volume group to properties
So that DELETE can be called using only information from GET.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
b02db5fcea LVM: add lvm_destroy_volume_group
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:14:32 +01:00
Fabian Grünbichler
40a079bd87 bump version to 7.0-14
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 17:02:48 +01:00
Aaron Lauterer
95dfa44ca1 add disk rename feature
Functionality has been added for the following storage types:

* directory ones, based on the default implementation:
    * directory
    * NFS
    * CIFS
    * gluster
* ZFS
* (thin) LVM
* Ceph

A new feature `rename` has been introduced to mark which storage
plugin supports the feature.

Version API and AGE have been bumped.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>

the intention of this feature is to support the following use-cases:
- reassign a volume from one owning guest to another (which usually
  entails a rename, since the owning vmid is encoded in the volume name)
- rename a volume (e.g., to use a more meaningful name instead of the
  auto-assigned ...-disk-123)

only the former is implemented at the caller side in
qemu-server/pve-container for now, but since the lower-level feature is
basically the same for both, we can take advantage of the storage plugin
API bump now to get the building block for this future feature in place
already.

adapt ApiChangelog change to fix conflicts and added more detail above

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 17:02:29 +01:00
Fabian Grünbichler
93fbc01963 api changelog: add volume attributes change
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 16:54:33 +01:00
Fabian Ebner
bfba962647 pbs: integrate support for protected
free_image doesn't need to check for protection, because that will
happen on the server.

Getting/updating notes has also been refactored to re-use the code
for the PBS api calls.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>

add missing b-d and depend on libposix-strptime-perl

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 16:54:13 +01:00
Fabian Ebner
ecfe25058b prune: mark renamed and protected backups differently
While it makes no difference for pruning itself, protected backups are
additionally protected against removal. Avoid the potential to confuse
the two. Also update the description for the API return value and add
an enum constraint.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
56897a9203 fix #3307: make it possible to set protection for backups
A protected backup is not removed by free_image and ignored when
pruning.

The protection_file_path function is introduced in Storage.pm, so that
it can also be used by vzdump itself and in archive_remove.

For pruning, renamed backups already behaved similiar to how protected
backups will, but there are a few reasons to not just use that for
implementing the new feature:
1. It wouldn't protect against removal.
2. It would make it necessary to rename notes and log files too.
3. It wouldn't naturally extend to other volumes if that's needed.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
9a4c0e8471 prune mark: preserve additional information for the keep-all case
Currently, if an entry is already marked as 'protected'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
f1de828166 add generalized functions to manage volume attributes
replacing the ones for handling notes. To ensure backwards
compatibility with external plugins, all plugins that do not just call
another implementation need to call $class->{get, update}_volume_notes
when the attribute is 'notes' to catch any derived implementations.

This is mainly done to avoid the need to add new methods every time a
new attribute is added.

Not adding a timeout parameter like the notes functions have, because
it was not used and can still be added if it ever is needed in the
future.

For get_volume_attribute, undef will indicate that the attribute is
not supported. This makes it possible to distinguish "not supported"
from "error getting the attribute", which is useful when the attribute
is important for an operation. For example, free_image checking for
protection (introduced in a later patch) can abort if getting the
'protected' attribute fails.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
e0aa2070f6 dir plugin: get notes: return undef if notes are not supported
This avoids showing empty notes in the result of the content/{volid}
API call for volumes that do not even support notes. It's also in
preparation for the proposed get_volume_attribute generalization,
which expects undef to be returned when an attribute is not supported.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
ddb3263031 dir plugin: update notes: don't fail if file is already removed
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
a799f7529b bump APIVER and APIAGE
Added blockers parameter to volume_rollback_is_possible.
Replaced volume_snapshot_list with volume_snapshot_info.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:35:53 +01:00
Fabian Ebner
dc992e7b89 plugin: remove volume_snapshot_list
which was only used by replication, but now replication uses
volume_snapshot_info instead.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:35:53 +01:00
Fabian Ebner
8c20d8afa3 plugin: add volume_snapshot_info function
which allows for better choices of common replication snapshots.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:35:53 +01:00
Fabian Ebner
665f61a24b test: zfspool: extend some rollback is possible tests with new blockers parameter
and fix a few typos.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:34:00 +01:00
Fabian Ebner
9a5d50950c zfspool: add blockers parameter to volume_snapshot_is_possible
useful for rollback, so that only the required replication snapshots
can be removed, and it's possible to abort early without deleting any
replication snapshots if there are other non-replication snasphots
blocking rollback.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:34:00 +01:00
Fabian Ebner
ac5c1af57c zfspool: add zfs_get_sorted_snapshot_list helper
replacing the current zfs_get_latest_snapshot. For
volume_snapshot_list, ignore errors as before.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:34:00 +01:00
Dominik Csapak
bc7fecb082 cephfs: add support for multiple ceph filesystems
by optionally saving the name of the cephfs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-11-05 13:53:56 +01:00
Fabian Ebner
85043c0193 rbd plugin: free image: use actual command in error message
For linked clones, the base name was included, which is confusing.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-27 17:03:14 +02:00
Thomas Lamprecht
b206f7c471 bump version to 7.0-13
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-14 11:22:55 +02:00
Thomas Lamprecht
75b8555ac6 test: also pass format for backing base image
mirroring what commit 9177cc2eda87c9a8f85a5ba73fa5f8e45cdd44de did
for the plugin system already, with QEMU 6.1 this is now a hard
requirement

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-14 11:17:43 +02:00
Lorenz Stechauner
95ff5dbd64 fix #3580: plugins: make preallocation mode selectable for qcow2 and raw images
the plugins for file based storages
 * BTRFS
 * CIFS
 * Dir
 * Glusterfs
 * NFS
now allow the option 'preallocation'.

'preallocation' can have four values:
 * default
 * off
 * metadata
 * falloc
 * full
see man pages for `qemu-img` for what these mean exactly. [0]

the defualt value was chosen to be
 * qcow2: metadata (as previously)
 * raw: off

when using 'metadata' as preallocation mode, for raw images 'off'
is used.

[0] https://qemu.readthedocs.io/en/latest/system/images.html#disk-image-file-formats

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-14 11:00:23 +02:00
Thomas Lamprecht
2caa1194e9 ct templates: support zstd compressed archives
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 09:06:51 +02:00
Fabian Ebner
05d9171278 api: disks: create: set correct partition type
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
a2c34371e6 partially fix #2285: api: disks: allow partitions for creation paths
The calls for directory and ZFS need slight adaptations. Except for
those, the only thing that needs to be done is support partitions in
the disk_is_used helper.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
cc884f73d8 api: disks: initgpt: explicitly abort for partitions
In preparation to extend disk_is_used to support partitions. Without
this new check, initgpt would also allow partitions once disk_is_used
supports partitions, which is not desirable.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
ff91cfae35 diskmanage: don't set usage for unused partitions
The disk type is already 'partition' so there's no additional
information here. And it would need to serve as a code-word for
unused partitions. The cleaner approach is to not set the usage.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
bd46e59b5d diskmanage: wipe blockdev: also change partition type
when called with a partition. Since get_disks uses the partition type
(among other things) to detect LVM and ZFS volumes, such volumes would
still be seen as in-use after wiping. Thus, also change the partition
type and simply use 0x83 "Linux filesystem".

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
e8df8fb179 diskmanage: add change_parttype and is_partition helpers
For change_parttype, only GPT-partitioned disks are supported, as I
didn't see an option for sgdisk to make it also work with
MBR-partitioned disks. And while sfdisk could be used instead (or
additionally) it would be a new dependency, and AFAICS require some
conversion of partition type GUIDs to MBR types on our part.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
9524b31ee3 btrfs: free image: only remove snapshots for current subvol
instead of all in the same directory.

Reported in the community forum:
https://forum.proxmox.com/threads/error-could-not-statfs-no-such-file-or-directory.96057/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-06 13:55:14 +02:00
Thomas Lamprecht
4748fdf334 bump version to 7.0-12
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-05 06:25:08 +02:00
Stefan Reiter
b19ae5b47e import: don't check for 1K aligned size
TPM state disks on directory storages may have completely unaligned
sizes, this check doesn't make sense for them.

This appears to just be a (weak) safeguard and not serve an actual
functional purpose, so simply get rid of it to allow migration of TPM
state.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-10-05 06:19:39 +02:00
Thomas Lamprecht
838e0632f7 d/control: break libpve-http-server-perl (<< 4.0-3)
commit e4d56f096ed28761d6b9a9e348be0fc682928040 removes a `sleep 1`
hack for removal of a tempfile which earlier happened in the
pve-http-server but we do ourself now.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-04 10:23:38 +02:00
Lorenz Stechauner
855915b60d status: fix tmpfile cleanup
$tmpfilename already gets unlinked after executing the cmd.

furthermore, because this is a local file, it is wrong to delete
it via the ssh command on a remote node.

small change: added \n to the error message.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-10-04 10:22:06 +02:00
Lorenz Stechauner
0246225ca7 fix #3505: status: add checksum and algorithm to file upload
Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-10-04 10:22:06 +02:00
Lorenz Stechauner
e4d56f096e status: remove sleep(1) in file upload
this racey sleep(1) is only there for legacy reasons: because
we don't use apache anymore and only emulate its behabiour
regarding removing temp files, this is under our own control
now and so we can improve this whole situation.

this change requires a pve-http-server version, in which the
tmpfile gets not automatically removed anymore.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-10-04 10:22:06 +02:00
Fabian Ebner
a64aedd3eb diskmanage: allow passing partitions to get_disks
Requires that the $include_partitions parameter is set too, which:
1. Makes sense, because the partition won't be included in the result
   otherwise.
2. Ensures backwards compatibility for existing callers that don't
   use $include_partitions. No existing callers use both $disks and
   $include_partitions at the same time, so nothing learns to
   "support" partitions by accident.

Moving the strip_dev helper to the top, so it can be used everywhere.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:11:00 +02:00
Fabian Ebner
0f4533476d diskmanage: allow partitions for get_udev_info
both existing callers only call this with non-partitions currently, so
the change should be backwards compatible.

In preparation to enable ZFS creation on top of partitions (where the
udev info is used to get the stable by-id path of a device).

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:10:56 +02:00
Fabian Ebner
21a75847a8 api: disk: work around udev bug to ensure its database is updated
There is a udev bug [0] which can ultimately lead to the udev database
for certain devices not being actively updated. Determining whether a
disk is used or not in get_disks() (in part) relies upon lsblk, which
queries the udev database. Ensure the information is updated by
manually calling 'udevadm trigger' for the changed devices.

It's most important for the 'directory' API path, as mounting depends
on the '/dev/disk/by-uuid'-symlink to be generated.

[0]: https://github.com/systemd/systemd/issues/18525

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:04:25 +02:00
Fabian Ebner
e99bc248d4 api: disks: create: re-check disk after fork/lock
Because then it might not be unused anymore. If there really is a
race, this prevents e.g. sgdisk creating a partition on a device
already in use by LVM or LVM destroying a partitioned device.

For ZFS, also get the latest udev info once inside the worker.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:04:22 +02:00
Fabian Ebner
dcd8f3a3dd btrfs: call free_image correctly
Currently, 'PVE::Storage::DirPlugin' is implicitly passed along as
$class, which means that if the base class's free_image calls another
method (e.g.  filesystem_path) then the DirPlugin's method will be
used, rather than the one from BTRFSPlugin. Change it so that $class
itself is passed along.

See also commit 279d9de5108f6fc6f2d31f77f1b41d6dc7a67cb9 for context,
where the approach in this patch was suggested.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-22 08:52:53 +02:00