5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2025-01-24 02:04:13 +03:00

1729 Commits

Author SHA1 Message Date
Thomas Lamprecht
75b8555ac6 test: also pass format for backing base image
mirroring what commit 9177cc2eda87c9a8f85a5ba73fa5f8e45cdd44de did
for the plugin system already, with QEMU 6.1 this is now a hard
requirement

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-14 11:17:43 +02:00
Lorenz Stechauner
95ff5dbd64 fix #3580: plugins: make preallocation mode selectable for qcow2 and raw images
the plugins for file based storages
 * BTRFS
 * CIFS
 * Dir
 * Glusterfs
 * NFS
now allow the option 'preallocation'.

'preallocation' can have four values:
 * default
 * off
 * metadata
 * falloc
 * full
see man pages for `qemu-img` for what these mean exactly. [0]

the defualt value was chosen to be
 * qcow2: metadata (as previously)
 * raw: off

when using 'metadata' as preallocation mode, for raw images 'off'
is used.

[0] https://qemu.readthedocs.io/en/latest/system/images.html#disk-image-file-formats

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-14 11:00:23 +02:00
Thomas Lamprecht
2caa1194e9 ct templates: support zstd compressed archives
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 09:06:51 +02:00
Fabian Ebner
05d9171278 api: disks: create: set correct partition type
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
a2c34371e6 partially fix #2285: api: disks: allow partitions for creation paths
The calls for directory and ZFS need slight adaptations. Except for
those, the only thing that needs to be done is support partitions in
the disk_is_used helper.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
cc884f73d8 api: disks: initgpt: explicitly abort for partitions
In preparation to extend disk_is_used to support partitions. Without
this new check, initgpt would also allow partitions once disk_is_used
supports partitions, which is not desirable.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
ff91cfae35 diskmanage: don't set usage for unused partitions
The disk type is already 'partition' so there's no additional
information here. And it would need to serve as a code-word for
unused partitions. The cleaner approach is to not set the usage.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
bd46e59b5d diskmanage: wipe blockdev: also change partition type
when called with a partition. Since get_disks uses the partition type
(among other things) to detect LVM and ZFS volumes, such volumes would
still be seen as in-use after wiping. Thus, also change the partition
type and simply use 0x83 "Linux filesystem".

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
e8df8fb179 diskmanage: add change_parttype and is_partition helpers
For change_parttype, only GPT-partitioned disks are supported, as I
didn't see an option for sgdisk to make it also work with
MBR-partitioned disks. And while sfdisk could be used instead (or
additionally) it would be a new dependency, and AFAICS require some
conversion of partition type GUIDs to MBR types on our part.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-07 14:39:26 +02:00
Fabian Ebner
9524b31ee3 btrfs: free image: only remove snapshots for current subvol
instead of all in the same directory.

Reported in the community forum:
https://forum.proxmox.com/threads/error-could-not-statfs-no-such-file-or-directory.96057/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-10-06 13:55:14 +02:00
Thomas Lamprecht
4748fdf334 bump version to 7.0-12
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-05 06:25:08 +02:00
Stefan Reiter
b19ae5b47e import: don't check for 1K aligned size
TPM state disks on directory storages may have completely unaligned
sizes, this check doesn't make sense for them.

This appears to just be a (weak) safeguard and not serve an actual
functional purpose, so simply get rid of it to allow migration of TPM
state.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-10-05 06:19:39 +02:00
Thomas Lamprecht
838e0632f7 d/control: break libpve-http-server-perl (<< 4.0-3)
commit e4d56f096ed28761d6b9a9e348be0fc682928040 removes a `sleep 1`
hack for removal of a tempfile which earlier happened in the
pve-http-server but we do ourself now.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-04 10:23:38 +02:00
Lorenz Stechauner
855915b60d status: fix tmpfile cleanup
$tmpfilename already gets unlinked after executing the cmd.

furthermore, because this is a local file, it is wrong to delete
it via the ssh command on a remote node.

small change: added \n to the error message.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-10-04 10:22:06 +02:00
Lorenz Stechauner
0246225ca7 fix #3505: status: add checksum and algorithm to file upload
Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-10-04 10:22:06 +02:00
Lorenz Stechauner
e4d56f096e status: remove sleep(1) in file upload
this racey sleep(1) is only there for legacy reasons: because
we don't use apache anymore and only emulate its behabiour
regarding removing temp files, this is under our own control
now and so we can improve this whole situation.

this change requires a pve-http-server version, in which the
tmpfile gets not automatically removed anymore.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-10-04 10:22:06 +02:00
Fabian Ebner
a64aedd3eb diskmanage: allow passing partitions to get_disks
Requires that the $include_partitions parameter is set too, which:
1. Makes sense, because the partition won't be included in the result
   otherwise.
2. Ensures backwards compatibility for existing callers that don't
   use $include_partitions. No existing callers use both $disks and
   $include_partitions at the same time, so nothing learns to
   "support" partitions by accident.

Moving the strip_dev helper to the top, so it can be used everywhere.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:11:00 +02:00
Fabian Ebner
0f4533476d diskmanage: allow partitions for get_udev_info
both existing callers only call this with non-partitions currently, so
the change should be backwards compatible.

In preparation to enable ZFS creation on top of partitions (where the
udev info is used to get the stable by-id path of a device).

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:10:56 +02:00
Fabian Ebner
21a75847a8 api: disk: work around udev bug to ensure its database is updated
There is a udev bug [0] which can ultimately lead to the udev database
for certain devices not being actively updated. Determining whether a
disk is used or not in get_disks() (in part) relies upon lsblk, which
queries the udev database. Ensure the information is updated by
manually calling 'udevadm trigger' for the changed devices.

It's most important for the 'directory' API path, as mounting depends
on the '/dev/disk/by-uuid'-symlink to be generated.

[0]: https://github.com/systemd/systemd/issues/18525

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:04:25 +02:00
Fabian Ebner
e99bc248d4 api: disks: create: re-check disk after fork/lock
Because then it might not be unused anymore. If there really is a
race, this prevents e.g. sgdisk creating a partition on a device
already in use by LVM or LVM destroying a partitioned device.

For ZFS, also get the latest udev info once inside the worker.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-30 18:04:22 +02:00
Fabian Ebner
dcd8f3a3dd btrfs: call free_image correctly
Currently, 'PVE::Storage::DirPlugin' is implicitly passed along as
$class, which means that if the base class's free_image calls another
method (e.g.  filesystem_path) then the DirPlugin's method will be
used, rather than the one from BTRFSPlugin. Change it so that $class
itself is passed along.

See also commit 279d9de5108f6fc6f2d31f77f1b41d6dc7a67cb9 for context,
where the approach in this patch was suggested.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-22 08:52:53 +02:00
Thomas Lamprecht
e1667a2253 cifs: negotiates the highest SMB2+ version supported by default
instead of hardcoding it to a potential outdated value.

For `smbclient` we only set max-protocol version and that could only
be smb2 or smb3 (no finer granularity) any how, so this was not
really correct.

Nowadays the kernel dropped SMB1 and tries to go for SMB2.1 or higher
by default, depending on what client and server supports. SMB2.1 is
Windows 7/2008R2 - both EOL since quite a bit, so ok as default lower
boundary.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 10:59:00 +02:00
Thomas Lamprecht
9fff8c7aca cifs: allow "3" and "default" for version
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-14 14:28:15 +02:00
Moayad Almalat
396ea58b65 fix #3609: cifs: add support to SMB 3.11
Added support for the SMB version SMB3_11 When the `min protocol =
SMB3_11` in the smb.conf, the CIFS mount will return with the
following error:
```
CIFS VFS: cifs_mount failed w/return code = -95
```
added an optional option to use the `vers=3.11`

Signed-off-by: Moayad Almalat <m.almalat@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
[ Thomas: move text from cover letter to commit message &
  add S-o-b ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-14 11:50:14 +02:00
Fabian Ebner
576e143ac1 fix #3610: properly build ZFS detail tree
Previously, top-level vdevs like log or special were wrongly added as
children of the previous outer vdev instead of the root.

Fix it by also showing the vdev with the same name as the pool and
start counting from level 1 (the pool itself serves as the root and
should be the only one with level 0). This results in the same kind
of structure as in PBS and (except for the root) zpool status itself.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-10 14:19:39 +02:00
Fabian Ebner
d5fc368503 fix prune-backups validation (again)
Commit a000e26ce7d30ba2b938402164a9a15e66dd123e caused a test failure
in pve-manager, because now 'keep-all=0' is not thrown out upon
validation anymore. Fix the issue the commit addressed differently,
by simply creating a copy of the (shallow) hash first, and using
the logic from before the commit.

Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-10 14:16:26 +02:00
Fabian Ebner
a000e26ce7 prune {validate, mark}: preserve input parameter
While the current way to detect settings like { 'keep-last' => 0 } is
concise, it's also wrong, because but the delete operation is visible
to the caller. This resulted in e.g.
    # $hash is { 'keep-all' => 1 }
    my $s = print_property_string($hash, 'prune-backups');
    # $hash is now {}, $s is 'keep-all=1'
because validation is called in print_property_string. The same issue
is present when calling prune_mark_backup_group.

Because validation complains when keep-all and something else is set,
this shouldn't have caused any real issues, besides vzdump with
keep-all=1 wrongly taking the removal path, but without any settings,
so not removing anything:
    INFO: prune older backups with retention:
    INFO: pruned 0 backup(s)

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-09 18:07:13 +02:00
Fabian Ebner
08ca395503 btrfs: style: add missing semicolon
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-08 14:26:12 +02:00
Fabian Ebner
6c315e4587 btrfs: avoid undef warnings with format
which is only set by parse_volname when the volume is a VM or
container image, but not for other content types.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-09-08 14:26:12 +02:00
Thomas Lamprecht
4aecade4dc bump version to 7.0-11
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-06 08:40:39 +02:00
Thomas Lamprecht
3de1c50a89 api: followup style/comment improvements
try to comment why not what, what is already described good enough by
the code here.

Also, we want to go up to 100cc text-width if it improves
readability, which for post-if's it most often does.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-06 08:32:20 +02:00
Lorenz Stechauner
f6aeefff54 status: move unlink from http-server to enpoint
this is the first step in which not the http server removes the
temporary file, but the worker itself.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-09-06 08:28:26 +02:00
Thomas Lamprecht
e2f8e86c83 btrfs: fix calling alloc_image from DirPlugin
similar to commit 279d9de5108f6fc6f2d31f77f1b41d6dc7a67cb9

This calling style is pretty dangerous in general for such plugin
systems...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-06 08:25:58 +02:00
Aaron Lauterer
22b68016f7 Ceph: add keyring parameter for external clusters
By adding the keyring for RBD storage or the secret for CephFS ones, it
is possible to add an external Ceph cluster with only one API call.

Previously the keyring / secret file needed to be placed in
/etc/pve/priv/ceph/$storeID.{keyring,secret} manually.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-08-26 18:15:30 +02:00
Aaron Lauterer
a4a1fe6419 CephConfig: add optional $secret parameter
This allows us to manually pass the used RBD keyring or CephFS secret.
Useful mostly when adding external Ceph clusters where we have no other
means to fetch them.

I renamed the previous $secret to $cephfs_secret to be able to use
$secret as parameter.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-08-26 18:15:30 +02:00
Fabian Ebner
ab3516a6d7 zfs: fix unmount request
by not dying when the dataset is already unmounted. Can be triggered
for a container by doing two rollbacks in a row.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-08-12 11:48:42 +02:00
Thomas Lamprecht
04bb277db4 buildsys: change upload dist to bullseye
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-30 15:37:13 +02:00
Thomas Lamprecht
c0a4b3d127 bump version to 7.0-10
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-30 15:23:19 +02:00
Dominik Csapak
ae098a191c api: disks: allow zstd compression for zfs pools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-30 15:21:01 +02:00
Hannes Laimer
279d9de510 fix #3555: BTRFS: call DirPlugin's free_image correctly
The method is only derived in the DirPlugin module from the base
Plugin, so we do not have it available there through a static module
method call using ::, but only when using a class dereference.

Other fix options would have been:

  PVE::Storage::Plugin::free_image(@_);

or:
  $class->SUPER::free_image($storeid, ...);

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
[ Thomas: add some background to the commit message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-30 14:52:51 +02:00
Thomas Lamprecht
b11d054be0 api: status: rework err cleanup
avoid open3 on local node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-29 17:12:40 +02:00
Thomas Lamprecht
a84804c432 api: status: import run_command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-29 17:12:40 +02:00
Lorenz Stechauner
31bd43648d api: status: fix unlink on file upload
after an error while copying the file to its destination the local
path of the destination was unlinked in every case, even when on the
destination was copied to via scp.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-07-29 17:12:40 +02:00
Thomas Lamprecht
0a65c237ea api: status: sort/cleanup module use-statements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-29 17:12:40 +02:00
Lorenz Stechauner
ca8c865871 status: add max length and disclaimer to file upload's filename
Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-07-21 16:47:01 +02:00
Lorenz Stechauner
1e96ffc954 status: add enum for file upload content type
the addition of this enum does not change API behaviour, because
it is checked for 'iso' or 'vztmpl' aftwerwards anyway.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-07-21 16:47:01 +02:00
Thomas Lamprecht
24adf51e7e bump version to 7.0-9
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-06 16:28:54 +02:00
Fabian Ebner
576e560d17 extract backup config: less precise matching for broken pipe detection
Extracting the config for zstd compressed vma files was broken:
    Failed to extract config from VMA archive: zstd: error 70 : Write
    error : cannot write decoded block : Broken pipe (500)
since the error message changed and wouldn't match anymore.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-06 16:24:22 +02:00
Thomas Lamprecht
49a2f518bc bump version to 7.0-8
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-06 12:41:42 +02:00
Stoiko Ivanov
5a16629577 lvm: wipe signatures on lvcreate
With PVE 7.0 we use upstream's lvm2 packages, which seem to detect
'more' signatures (and refuse creating lvs when they are present)

This prevents creating new disks on LVM (thick) storages as reported
on pve-user [0].

Adding -Wy to wipe signatures, and --yes (to actually wipe them
instead of prompting) fixes the aborted lvcreate.

Adding only to LVMPlugin and not to the lvcreate calls in
LvmThinPlugin, since I assume (and my quick tests confirm) that thin
pools are not affected by this issue..

Tested on a virtual test-setup with a LVM storage on a (virtual) iscsi
target and a local lvmthin storage.

[0] https://lists.proxmox.com/pipermail/pve-user/2021-July/172660.html

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2021-07-06 12:39:50 +02:00