5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2024-12-22 13:34:16 +03:00
Commit Graph

1768 Commits

Author SHA1 Message Date
Fabian Grünbichler
7a158d0bc9 storage_migrate: pull out import/export_prepare
for re-use with remote migration, where import and export happen on
different clusters connected via a websocket instead of SSH tunnel.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-09 18:52:59 +01:00
Fabian Grünbichler
686b07376f storage_migrate_snapshot: skip for btrfs without snapshots
this allows migrating from btrfs to other raw+size accepting storages,
provided no snapshots exist.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-09 18:52:59 +01:00
Thomas Lamprecht
db3d1ef163 bump version to 7.1-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-04 18:08:09 +01:00
Thomas Lamprecht
c915afca7e rbd: followup code style cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-04 18:04:31 +01:00
Aaron Lauterer
ef2afce74a fix #1816: rbd: add support for erasure coded ec pools
The first step is to allocate rbd images correctly.

The metadata objects still need to be stored in a replicated pool, but
by providing the --data-pool parameter on image creation, we can place
the data objects on the erasure coded (EC) pool.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-02-04 17:51:48 +01:00
Fabian Grünbichler
05b07a67f5 storage_migrate: pull out snapshot decision
into new top-level helper for re-use with remote migration.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-04 17:30:42 +01:00
Fabian Grünbichler
f78f26ae60 volname_for_storage: parse volname before calling
to allow reusing this with remote migration, where parsing of the source
volid has to happen on the source node, but this call has to happen on
the target node.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-04 17:30:42 +01:00
Aaron Lauterer
e7bc1f03b7 CephConfig: ensure newline in $secret and $cephfs_secret parameter
Ensure that the user provided $secret ends in a newline. Otherwise we
will have Input/output errors from rados_connect.

For consistency and possible future proofing, also add a newline to
CephFS secrets.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
2022-01-25 10:57:58 +01:00
Fabian Ebner
73dfe360dd zfs: use -r parameter when listing snapshots
Some versions of ZFS do not automatically display the child snapshots
when '-t snapshot' is used, but require '-r' to be present
additionally[1]. And in general, it's cleaner to specify the flag
explicitly.

Because of that, commit ac5c1af led to a regression[0] in the context
of ZFS over iSCSI with zfs_get_sorted_snapshot_list. Fix it, by adding
a -r flag again.

The volume_snapshot_info function is currently only used in the
context of replication and that requires a local ZFS pool, but it
would be affected by the same issue if it is ever used in the context
of ZFS over iSCSI, so also add -r there.

[0]: https://forum.proxmox.com/threads/102683/
[1]: https://forum.proxmox.com/threads/102683/post-442577

Fixes: 8c20d8a ("plugin: add volume_snapshot_info function")
Fixes: ac5c1af ("zfspool: add zfs_get_sorted_snapshot_list helper")
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-01-10 13:33:31 +01:00
Fabian Ebner
b4616e5c4a lvm thin: add missing newline to error message
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-18 11:18:03 +01:00
Fabian Ebner
904f06a82e pbs: update attribute: cleaner error message if not supported
Reported-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-12 16:53:27 +01:00
Fabian Grünbichler
c7aa1e28dd bump version to 7.0-15
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-10 14:25:26 +01:00
Fabian Ebner
6fbffac0a3 lvm thin: don't assume that a thin pool and its volumes are active
There are cases where autoactivation can fail, as reported in the
community forum [0]. And it could also be that a volume was
deactivated by something outside of our control.

It doesn't seem strictly necessary to activate the thin pool itself
(creating/removing/activating LVs within the pool still works if it's
not active), but it does not report usage information as long as
neither the pool nor any of its LVs are active. Activate the pool for
that, for being able to use the flag in status(), and it should also
serve as a good indicator that there's a problem with the pool if it
can't be activated.

Before activating, check the (cached) lv_state from lvm_list_volumes.
It's necessary to update the cache in activate_storage, because the
flag is re-used in status(). Also update it for other (de)activations
to be more future-proof.

[0]: https://forum.proxmox.com/threads/local-lvm-not-available-after-kernel-update-on-pve-7.97406

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 14:23:54 +01:00
Fabian Ebner
2668a86719 lvm thin: status: code cleanup
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 14:18:28 +01:00
Fabian Ebner
cde43c4880 api: disks: delete: add flag for cleaning up storage config
Update node restrictions to reflect that the storage is not available
anymore on the particular node. If the storage was only configured for
that node, remove it altogether.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>

slight style fixup

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
f81908eb58 api: disks: delete: add flag for wiping disks
For ZFS and directory storages, clean up the whole disk when the
layout is as usual to avoid left-overs.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
26082b7daf diskmanage: add helper for udev workaround
to avoid duplication. Current callers pass along at least one device,
but anticipate future callers that might call with the empty list. Do
nothing in that case, rather than triggering everything.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
a83d8eb178 api: disks: add DELETE endpoint for directory, lvm, lvmthin, zfs
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
a510449e2b api: list thin pools: add volume group to properties
So that DELETE can be called using only information from GET.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:35:25 +01:00
Fabian Ebner
b02db5fcea LVM: add lvm_destroy_volume_group
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-10 12:14:32 +01:00
Fabian Grünbichler
40a079bd87 bump version to 7.0-14
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 17:02:48 +01:00
Aaron Lauterer
95dfa44ca1 add disk rename feature
Functionality has been added for the following storage types:

* directory ones, based on the default implementation:
    * directory
    * NFS
    * CIFS
    * gluster
* ZFS
* (thin) LVM
* Ceph

A new feature `rename` has been introduced to mark which storage
plugin supports the feature.

Version API and AGE have been bumped.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>

the intention of this feature is to support the following use-cases:
- reassign a volume from one owning guest to another (which usually
  entails a rename, since the owning vmid is encoded in the volume name)
- rename a volume (e.g., to use a more meaningful name instead of the
  auto-assigned ...-disk-123)

only the former is implemented at the caller side in
qemu-server/pve-container for now, but since the lower-level feature is
basically the same for both, we can take advantage of the storage plugin
API bump now to get the building block for this future feature in place
already.

adapt ApiChangelog change to fix conflicts and added more detail above

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 17:02:29 +01:00
Fabian Grünbichler
93fbc01963 api changelog: add volume attributes change
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 16:54:33 +01:00
Fabian Ebner
bfba962647 pbs: integrate support for protected
free_image doesn't need to check for protection, because that will
happen on the server.

Getting/updating notes has also been refactored to re-use the code
for the PBS api calls.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>

add missing b-d and depend on libposix-strptime-perl

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-11-09 16:54:13 +01:00
Fabian Ebner
ecfe25058b prune: mark renamed and protected backups differently
While it makes no difference for pruning itself, protected backups are
additionally protected against removal. Avoid the potential to confuse
the two. Also update the description for the API return value and add
an enum constraint.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
56897a9203 fix #3307: make it possible to set protection for backups
A protected backup is not removed by free_image and ignored when
pruning.

The protection_file_path function is introduced in Storage.pm, so that
it can also be used by vzdump itself and in archive_remove.

For pruning, renamed backups already behaved similiar to how protected
backups will, but there are a few reasons to not just use that for
implementing the new feature:
1. It wouldn't protect against removal.
2. It would make it necessary to rename notes and log files too.
3. It wouldn't naturally extend to other volumes if that's needed.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
9a4c0e8471 prune mark: preserve additional information for the keep-all case
Currently, if an entry is already marked as 'protected'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
f1de828166 add generalized functions to manage volume attributes
replacing the ones for handling notes. To ensure backwards
compatibility with external plugins, all plugins that do not just call
another implementation need to call $class->{get, update}_volume_notes
when the attribute is 'notes' to catch any derived implementations.

This is mainly done to avoid the need to add new methods every time a
new attribute is added.

Not adding a timeout parameter like the notes functions have, because
it was not used and can still be added if it ever is needed in the
future.

For get_volume_attribute, undef will indicate that the attribute is
not supported. This makes it possible to distinguish "not supported"
from "error getting the attribute", which is useful when the attribute
is important for an operation. For example, free_image checking for
protection (introduced in a later patch) can abort if getting the
'protected' attribute fails.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
e0aa2070f6 dir plugin: get notes: return undef if notes are not supported
This avoids showing empty notes in the result of the content/{volid}
API call for volumes that do not even support notes. It's also in
preparation for the proposed get_volume_attribute generalization,
which expects undef to be returned when an attribute is not supported.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
ddb3263031 dir plugin: update notes: don't fail if file is already removed
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 14:56:15 +01:00
Fabian Ebner
a799f7529b bump APIVER and APIAGE
Added blockers parameter to volume_rollback_is_possible.
Replaced volume_snapshot_list with volume_snapshot_info.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:35:53 +01:00
Fabian Ebner
dc992e7b89 plugin: remove volume_snapshot_list
which was only used by replication, but now replication uses
volume_snapshot_info instead.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:35:53 +01:00
Fabian Ebner
8c20d8afa3 plugin: add volume_snapshot_info function
which allows for better choices of common replication snapshots.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:35:53 +01:00
Fabian Ebner
665f61a24b test: zfspool: extend some rollback is possible tests with new blockers parameter
and fix a few typos.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:34:00 +01:00
Fabian Ebner
9a5d50950c zfspool: add blockers parameter to volume_snapshot_is_possible
useful for rollback, so that only the required replication snapshots
can be removed, and it's possible to abort early without deleting any
replication snapshots if there are other non-replication snasphots
blocking rollback.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:34:00 +01:00
Fabian Ebner
ac5c1af57c zfspool: add zfs_get_sorted_snapshot_list helper
replacing the current zfs_get_latest_snapshot. For
volume_snapshot_list, ignore errors as before.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-11-08 10:34:00 +01:00
Dominik Csapak
bc7fecb082 cephfs: add support for multiple ceph filesystems
by optionally saving the name of the cephfs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-11-05 13:53:56 +01:00
Fabian Ebner
85043c0193 rbd plugin: free image: use actual command in error message
For linked clones, the base name was included, which is confusing.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-27 17:03:14 +02:00
Thomas Lamprecht
b206f7c471 bump version to 7.0-13
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-14 11:22:55 +02:00
Thomas Lamprecht
75b8555ac6 test: also pass format for backing base image
mirroring what commit 9177cc2eda did
for the plugin system already, with QEMU 6.1 this is now a hard
requirement

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-14 11:17:43 +02:00
Lorenz Stechauner
95ff5dbd64 fix #3580: plugins: make preallocation mode selectable for qcow2 and raw images
the plugins for file based storages
 * BTRFS
 * CIFS
 * Dir
 * Glusterfs
 * NFS
now allow the option 'preallocation'.

'preallocation' can have four values:
 * default
 * off
 * metadata
 * falloc
 * full
see man pages for `qemu-img` for what these mean exactly. [0]

the defualt value was chosen to be
 * qcow2: metadata (as previously)
 * raw: off

when using 'metadata' as preallocation mode, for raw images 'off'
is used.

[0] https://qemu.readthedocs.io/en/latest/system/images.html#disk-image-file-formats

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-14 11:00:23 +02:00
Thomas Lamprecht
2caa1194e9 ct templates: support zstd compressed archives
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 09:06:51 +02:00
Fabian Ebner
05d9171278 api: disks: create: set correct partition type
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
a2c34371e6 partially fix #2285: api: disks: allow partitions for creation paths
The calls for directory and ZFS need slight adaptations. Except for
those, the only thing that needs to be done is support partitions in
the disk_is_used helper.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
cc884f73d8 api: disks: initgpt: explicitly abort for partitions
In preparation to extend disk_is_used to support partitions. Without
this new check, initgpt would also allow partitions once disk_is_used
supports partitions, which is not desirable.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
ff91cfae35 diskmanage: don't set usage for unused partitions
The disk type is already 'partition' so there's no additional
information here. And it would need to serve as a code-word for
unused partitions. The cleaner approach is to not set the usage.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
bd46e59b5d diskmanage: wipe blockdev: also change partition type
when called with a partition. Since get_disks uses the partition type
(among other things) to detect LVM and ZFS volumes, such volumes would
still be seen as in-use after wiping. Thus, also change the partition
type and simply use 0x83 "Linux filesystem".

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
e8df8fb179 diskmanage: add change_parttype and is_partition helpers
For change_parttype, only GPT-partitioned disks are supported, as I
didn't see an option for sgdisk to make it also work with
MBR-partitioned disks. And while sfdisk could be used instead (or
additionally) it would be a new dependency, and AFAICS require some
conversion of partition type GUIDs to MBR types on our part.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
9524b31ee3 btrfs: free image: only remove snapshots for current subvol
instead of all in the same directory.

Reported in the community forum:
https://forum.proxmox.com/threads/error-could-not-statfs-no-such-file-or-directory.96057/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-06 13:55:14 +02:00
Thomas Lamprecht
4748fdf334 bump version to 7.0-12
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-05 06:25:08 +02:00