5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2025-01-03 01:18:01 +03:00
Commit Graph

1749 Commits

Author SHA1 Message Date
Thomas Lamprecht
3de1c50a89 api: followup style/comment improvements
try to comment why not what, what is already described good enough by
the code here.

Also, we want to go up to 100cc text-width if it improves
readability, which for post-if's it most often does.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-06 08:32:20 +02:00
Lorenz Stechauner
f6aeefff54 status: move unlink from http-server to enpoint
this is the first step in which not the http server removes the
temporary file, but the worker itself.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-09-06 08:28:26 +02:00
Thomas Lamprecht
e2f8e86c83 btrfs: fix calling alloc_image from DirPlugin
similar to commit 279d9de510

This calling style is pretty dangerous in general for such plugin
systems...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-06 08:25:58 +02:00
Aaron Lauterer
22b68016f7 Ceph: add keyring parameter for external clusters
By adding the keyring for RBD storage or the secret for CephFS ones, it
is possible to add an external Ceph cluster with only one API call.

Previously the keyring / secret file needed to be placed in
/etc/pve/priv/ceph/$storeID.{keyring,secret} manually.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-08-26 18:15:30 +02:00
Aaron Lauterer
a4a1fe6419 CephConfig: add optional $secret parameter
This allows us to manually pass the used RBD keyring or CephFS secret.
Useful mostly when adding external Ceph clusters where we have no other
means to fetch them.

I renamed the previous $secret to $cephfs_secret to be able to use
$secret as parameter.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-08-26 18:15:30 +02:00
Fabian Ebner
ab3516a6d7 zfs: fix unmount request
by not dying when the dataset is already unmounted. Can be triggered
for a container by doing two rollbacks in a row.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-08-12 11:48:42 +02:00
Thomas Lamprecht
04bb277db4 buildsys: change upload dist to bullseye
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-30 15:37:13 +02:00
Thomas Lamprecht
c0a4b3d127 bump version to 7.0-10
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-30 15:23:19 +02:00
Dominik Csapak
ae098a191c api: disks: allow zstd compression for zfs pools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-30 15:21:01 +02:00
Hannes Laimer
279d9de510 fix #3555: BTRFS: call DirPlugin's free_image correctly
The method is only derived in the DirPlugin module from the base
Plugin, so we do not have it available there through a static module
method call using ::, but only when using a class dereference.

Other fix options would have been:

  PVE::Storage::Plugin::free_image(@_);

or:
  $class->SUPER::free_image($storeid, ...);

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
[ Thomas: add some background to the commit message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-30 14:52:51 +02:00
Thomas Lamprecht
b11d054be0 api: status: rework err cleanup
avoid open3 on local node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-29 17:12:40 +02:00
Thomas Lamprecht
a84804c432 api: status: import run_command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-29 17:12:40 +02:00
Lorenz Stechauner
31bd43648d api: status: fix unlink on file upload
after an error while copying the file to its destination the local
path of the destination was unlinked in every case, even when on the
destination was copied to via scp.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-07-29 17:12:40 +02:00
Thomas Lamprecht
0a65c237ea api: status: sort/cleanup module use-statements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-29 17:12:40 +02:00
Lorenz Stechauner
ca8c865871 status: add max length and disclaimer to file upload's filename
Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-07-21 16:47:01 +02:00
Lorenz Stechauner
1e96ffc954 status: add enum for file upload content type
the addition of this enum does not change API behaviour, because
it is checked for 'iso' or 'vztmpl' aftwerwards anyway.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-07-21 16:47:01 +02:00
Thomas Lamprecht
24adf51e7e bump version to 7.0-9
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-06 16:28:54 +02:00
Fabian Ebner
576e560d17 extract backup config: less precise matching for broken pipe detection
Extracting the config for zstd compressed vma files was broken:
    Failed to extract config from VMA archive: zstd: error 70 : Write
    error : cannot write decoded block : Broken pipe (500)
since the error message changed and wouldn't match anymore.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-06 16:24:22 +02:00
Thomas Lamprecht
49a2f518bc bump version to 7.0-8
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-06 12:41:42 +02:00
Stoiko Ivanov
5a16629577 lvm: wipe signatures on lvcreate
With PVE 7.0 we use upstream's lvm2 packages, which seem to detect
'more' signatures (and refuse creating lvs when they are present)

This prevents creating new disks on LVM (thick) storages as reported
on pve-user [0].

Adding -Wy to wipe signatures, and --yes (to actually wipe them
instead of prompting) fixes the aborted lvcreate.

Adding only to LVMPlugin and not to the lvcreate calls in
LvmThinPlugin, since I assume (and my quick tests confirm) that thin
pools are not affected by this issue..

Tested on a virtual test-setup with a LVM storage on a (virtual) iscsi
target and a local lvmthin storage.

[0] https://lists.proxmox.com/pipermail/pve-user/2021-July/172660.html

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2021-07-06 12:39:50 +02:00
Thomas Lamprecht
b4e88b7fd3 cifs: improve warning for password but no username set
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-06 07:50:29 +02:00
Thomas Lamprecht
02f43ab4a8 cifs: fix sensitive parameter name for on-update/add
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-06 07:50:06 +02:00
Thomas Lamprecht
54e315e91b bump version to 7.0-7
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-05 19:01:15 +02:00
Wolfgang Bumiller
38f0f4698e btrfs: fix path_is_mounted invocation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-28 08:40:03 +02:00
Thomas Lamprecht
3577266e39 bump version to 7.0-6
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-24 16:13:12 +02:00
Thomas Lamprecht
1c1589e60d btrfs: support newer prune-backups for backup retention
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-24 16:11:48 +02:00
Thomas Lamprecht
f31143701d bump version to 7.0-5
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-24 11:45:45 +02:00
Thomas Lamprecht
a1234a04df btrfs: add mkdir as option for now
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-24 11:45:45 +02:00
Thomas Lamprecht
f449cddc79 btrfs: do not reuse DirPlugins activate_storage directrly
as then the btrfs assertion would happen after we already created
subdirectories on some path, leaving those left-over..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-24 11:18:40 +02:00
Wolfgang Bumiller
f6abd82a6d btrfs: check for btrfs in on_add_hook and activate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-24 11:06:02 +02:00
Thomas Lamprecht
5d65bf5f12 bump version to 7.0-4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 22:51:02 +02:00
Thomas Lamprecht
fc015f3d9c api: file download: limit filename length and encode it for the worker id
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 22:40:40 +02:00
Thomas Lamprecht
d0a3db1bf5 api: file download: code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 22:40:05 +02:00
Lorenz Stechauner
837b194224 status: add download_url method
uses common function PVE::Tools::download_file_from_url to download
iso files.

Only users with permissions `Sys.Audit` and `Sys.Modify` on `/` are
permitted to perform this action. This restriction is due to the
fact, that the download function is able to download files from
internal networks (which are not visible/accessible from outside).
Users with these permissions anyway have the means to alter node
(network) config, so this does not create any further security risk.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-06-23 22:28:44 +02:00
Lorenz Stechauner
edda43ed4f status: factoring out normalize_content_filename
Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-06-23 22:28:44 +02:00
Fabian Ebner
0153334270 api: content: correctly handle warnings status for delayed task
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 22:21:55 +02:00
Thomas Lamprecht
347e677b78 btrfs: drop qcow2 and vmdk for now
the web-interface always prefers qcow2 once that is in the list,
itself a bug on it's own as the preferred one from the backend should
be preferred too, but still, vmdk support should not be extended we
can only cope with that in a limited way anyway, and both can always
get enabled later easily, if there's actual user-request for it.
Disabling is never that easy, at least if one cares about backward
compat.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 20:22:52 +02:00
Wolfgang Bumiller
d3c5cf2487 btrfs: make NOCOW optional
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-23 20:20:31 +02:00
Wolfgang Bumiller
a0e3e224ea btrfs: add 'btrfs' import/export format
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-23 20:20:31 +02:00
Wolfgang Bumiller
3cc29a0487 bump storage API: update import/export methods
Bumps APIVER to 9 and resets APIAGE to zero.

The import methods (volume_import, volume_import_formats):

These additionally get the '$snapshot' parameter which is
already present on the export side as an informational piece
to know which of the snapshots is the *current* one.
This parameter is inserted *in the middle* of the current
parameters, so the import & export format methods now have
the same signatures.
The current "disk" state will be set to this snapshot.
This, too, is required for our btrfs implementation.
  `volume_import_formats` can obviously not make much
*use* of this parameter, but it'll still be useful to know
that the information is actually available in the import
call, so its presence will be checked in the btrfs
implementation.

Currently this is intended to be used for btrfs send/recv
support, which in theory could also get additional metadata
similar to how we do the "tar+size" format, however, we
currently only really use this within this repository in
storage_migrate() which has this information readily
available anyway.

On the export side (volume_export, volume_export_formats):

The `$with_snapshots` option is now "defined" to be an
ordered array of snapshots to include, as a hint for
storages which need this. (As of the next commit this is
only btrfs, and only when also specifying a base snapshot,
which is a case we can currently not run into except on the
command line interface.)
  The current providers of the `with_snapshot` option will
still treat it as a boolean (since eg. for ZFS you cannot
really "skip" snapshots AFAIK).
  This is mainly intended for storages which do not have a
strong association between snapshots and the originals, or
an ordering (eg. btrfs and lvm-thin allow creating
arbitrary snapshot trees, and with btrfs you can even
create a "circular" connection between subvolumes, also we
could consider reflink based copies snapshots on xfs in
the future maybe?)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-23 20:20:31 +02:00
Wolfgang Bumiller
af50c2e671 add BTRFS storage plugin
This is mostly the same as a directory storage, with 2 major
differences:

* 'subvol' volumes are actual btrfs subvolumes and therefore
  allow snapshots
* 'raw' files are placed *into* a subvolume and therefore
  also allow snapshots, the raw file for volume
  `btrstore:100/vm-100-disk-1.raw` can be found under
  `$path/images/100/vm-100-disk-1/disk.raw`
* in both cases, snapshots add an '@name' suffix to the
  subvolume's directory name, so snapshot 'foo' of the above
  would be found under
  `$path/images/100/vm-100-disk-1@foo/disk.raw`
  or for format "subvol":
  `$path/images/100/subvol-100-disk-1.subvol@foo`

Note that qgroups aren't included in btrfs-send streams,
therefore for now we will only be using *unsized* subvolumes
for containers and place a regular raw+ext4 file for sized
containers.
We could extend the import/export stream format to include
the information at the front (similar to how we do the
"tar+size" format, but we need to include the size of all
the contained snapshots as well, since they can technically
change). (But before enabling quotas we should do some
performance testing on bigger file systems with multiple
snapshots as there are quite a few reports of the fs slowing
down considerably in such scenarios).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-23 20:20:31 +02:00
Lorenz Stechauner
bba10cf4af factoring out regex for vztmpl
stores the regex definition in PVE::Storage.

One test had to be adapted because it tested obsolete code. Namely:
it expects vztmpl to only end with .tar.gz, but the new regex also
includes .tar.xz, there is nothing against allowing .tar.xz files as
vztmpl files.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2021-06-23 20:19:09 +02:00
Thomas Lamprecht
339a4eb3c0 file size info: return early if we cannot parse json
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 08:28:48 +02:00
Thomas Lamprecht
d4e00f2bd5 file/volume size info: add actual errors to untaint messages
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 08:28:48 +02:00
Stoiko Ivanov
ac598d851e plugins: untaint volume_size_info retuns
the size returned by volume_size_info is used for creating the new
destination image in PVE::QemuServer::clone_disk (and probably
elsewhere). In certain cases the return values are tainted - they are
obtained by a run_command call and depending on the format and length
of the parsed output can still have their tainted attribute.

One example of a tainted return has been reported in our
community-forum:
https://forum.proxmox.com/threads/cannot-clone-vm-or-move-disk-with-more-than-13-snapshots.89628/

A qcow2 image with 13 snapshots generates a output > 4k in length from
`qemu-img info --output=json`, which in turn causes the output to be
considered tainted.

This patch untaints the returns where applicable. The other
storage-plugins are not affected:
* LVMPlugin returns a single number and a newline (thus gets untainted
  by run_command)
* RBDPlugin untaints the complete json before decoding
* ZFSPoolplugin and ISCSIDirectPlugin explicitly untaint their
  returns.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2021-06-23 08:28:48 +02:00
Thomas Lamprecht
ffc31266da tree-wide: fix typos with codespell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 08:28:48 +02:00
Fabian Grünbichler
5b955999b9 pbs: fix typo
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-06-22 13:44:06 +02:00
Thomas Lamprecht
046b64d40b bump version to 7.0-3
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-21 11:30:10 +02:00
Fabian Ebner
03c487e553 config: prevent empty content list when content type 'none' is not supported
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-06-21 11:21:45 +02:00
Fabian Ebner
d96b789aed vdisk_list: only scan storages with the correct content type(s)
The enabled check in the lower loop is now redundant and can be removed.

If storeid is provided, initialize the result hash accordingly, mainly for
backwards compatibility (needed by a caller in pve-manager's Ceph/Pools.pm and
the migration code in pve-container and qemu-server), but it also is less
surprising in general.

Remaining vdisk_list users that do not specify a content type are:
    1. pve-manager's Pool/Ceph.pm, but the content type for RBD can only be
       rootdir and images, so the storage is scanned (if enabled, same as
       before).
    2. pve-container migration
    3. qemu-server migration
For the latter two, it's planned to enforce content type, so the change is fine
too.

This also means that for iscsi(direct) plugins with content type 'none', i.e.
"use LUNs directly" does not return the list of images anymore, but that was
rather a bug anyways as they're not virtual disks then:
    0.0.0.scsi-36001405b8f2772e13a04b8e9390db13d
All of the remaining callers not using content types (see above) are fine with
that change too.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-06-21 11:21:45 +02:00