5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2024-12-22 13:34:16 +03:00
Commit Graph

1878 Commits

Author SHA1 Message Date
Fabian Grünbichler
9017e1abc7 download handling: adapt to method schema field rename
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 11:18:43 +02:00
Fabian Grünbichler
f9bdac0e8f api: annotate file-restore endpoint as download-enabled
to satisfy the new checks in the API handler that only allow downloading via
marked endpoints.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 11:08:25 +02:00
Thomas Lamprecht
f2b4f7f955 bump version to 8.2.4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-09-12 12:26:30 +02:00
Fiona Ebner
1fd1b53018 api: upload: correctly test for result of unlink
It's not enough to check whether $! is set. From "perldoc perlvar":

> Many system or library calls set "errno" if they fail, to
> indicate the cause of failure. They usually do not set "errno"
> to zero if they succeed and may set "errno" to a non-zero value
> on success. This means "errno", hence $!, is meaningful only
> *immediately* after a failure:

To protect against potential issues, check the return value of unlink
and only check $! if it failed.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-09-12 12:12:49 +02:00
Maximiliano Sandoval
851cc07a2a base plugin: do not decode the empty string
If the json was empty, for example if the qemu-img command times out, a
message

    warn "could not parse qemu-img info command output for '$filename' - $err\n";

would have been printed.

This message could lead one to think the issue lies in the contents of
the json, even if the previous warning said that there was a timeout.

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-09-06 19:06:27 +02:00
Maximiliano Sandoval
fe854c9658 fix #4272: btrfs: add rename feature
Adds the ability to change the owner of a guest image.

Btrfs does not need special commands to rename a subvolume and this can
be achieved the same as in Storage/plugin.pm's rename_volume taking
special care of how the directory structure used by Btrfs.

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Tested-By: Aaron Lauterer <a.lauterer@proxmox.com>
Reviewed-By: Aaron Lauterer <a.lauterer@proxmox.com>
2024-09-06 19:04:37 +02:00
Daniel Kral
ef9f955e7e esxi: fix #5587: add support for older version of vmx storage filepaths
Allow the ESXi storage disk entry property "fileName" to be flatcased
("filename") in addition to being camelcased ("fileName"). This adds
compatibility with older ESXi .vmx configuration files.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
2024-09-06 17:31:55 +02:00
Maximiliano Sandoval
70c480b9f1 btrfs: log executed command on failures
Having the complete command printed out makes debuging easier.

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-07-22 19:05:02 +02:00
Fiona Ebner
6cdcf65a2c volume import: assume target API version is at least 9
The storage API version has been bumped to at least 9 since
libpve-storage = 7.0-4. If the source node is on Proxmox VE 8, where
this change will come in, then the target node can be assumed to be
running either Proxmox VE 8 or, during upgrade, the latest version of
Proxmox VE 7.4, so it's safe to assume a storage API version of at
least 9 in all cases.

As reported by Maximiliano, the fact that the 'apiinfo' call was
guarded with a quiet eval could lead to strange errors for replication
on a customer system where an SSH connection could not always be
established, because the target's API version would fall back to 1.
Because of that, the '-base' argument would be missing for the import
call on the target which would in turn lead to an error about the
target ZFS volume already existing (rather than doing an incremental
sync).

Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
2024-07-03 14:23:29 +02:00
Fiona Ebner
4abfc054c9 plugin: move definition for 'port' option to base plugin
Commit 7020491 ("esxi: add 'port' config parameter") started using
the 'port' option in a second plugin, but the definition stayed in the
PBS plugin. Avoid the hidden dependency and move the definition to the
base plugin instead.

It is necessary to mark it as optional or it would be required always.

Clarify that the option is not used by NFS and CIFS.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-07-02 16:13:16 +02:00
Fabian Grünbichler
bc12c9c088 style: remove goto statements
these can just as well be `die` statements right there, there is no complicated
cleanup that would warrant a goto statement..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
2024-07-01 10:49:49 +02:00
Fiona Ebner
6b32cece09 fix reminder comment for template regex
The necessary change in pve-manager to remove this [0] only got
applied during PVE 8.2 [1], so removing this regex has to wait.

[0]: https://lists.proxmox.com/pipermail/pve-devel/2021-October/050514.html
[1]: https://git.proxmox.com/?p=pve-manager.git;a=commit;h=9c54c52d7996572956e65c4820459ff3a1cda778

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-06-27 15:11:08 +02:00
Wolfgang Bumiller
13a81873ba bump version to 8.2.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-06-21 13:56:32 +02:00
Wolfgang Bumiller
5f35d73d52 esxi: remove already-dealt-with FIXME comment
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-06-21 13:56:32 +02:00
Wolfgang Bumiller
9a594f594d Revert "esxi: improve error handling for fuse mount tool"
This reverts commit d80420bca8.

This diverts stderr of the fuse process to a pipe, which makes no
sense as it runs daemonized in a scope, also, the pipe fd was used as
a ready-signal, which now does not trigger anymore.
2024-06-21 13:46:36 +02:00
Wolfgang Bumiller
6d7dcb575b bump version to 8.2.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-06-07 13:23:35 +02:00
Dominik Csapak
d80420bca8 esxi: improve error handling for fuse mount tool
if the fuse tool encounters an error early, it prints it like:
   Error: some error message
on stderr.

Redirect STDERR of the child process (which mounts the ESXi instance) to
the pipe of the parent (API) process, so that it can pass a hopefully
more meaningful message to the user than just an erroneous return code.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-06-07 13:17:06 +02:00
Wolfgang Bumiller
98a5bc5c9b fixup error messages when getting file size info
The assignment happens before the 'die', so the error message would
always contain 'undef'.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-04-30 10:16:35 +02:00
Wolfgang Bumiller
22cb4b0b78 don't bail on whitespaces in backing devices
This prevents importing from vmdks with whitespaces in file names.
Further, some operations that include file sizes (like listing disks)
would potentially fail entirely if a custom disk with a badly name
backing device exists in a VM images directory since they don't expect
this. Specifically, since we don't necessarily know the actual naming
scheme of the current storage in the plain Plugin.pm version, we don't
check the full name anyway, so why bother with whitespaces...

See-also: https://forum.proxmox.com/threads/new-import-wizard-available-for-migrating-vmware-esxi-based-virtual-machines.144023/page-16#post-658697
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-04-30 10:16:23 +02:00
Thomas Lamprecht
bb72798064 bump version to 8.2.1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 17:20:47 +02:00
Thomas Lamprecht
b1d0effc99 esxi import: add "longhorn" to possible Windows Server 2008 OS types
While we detected "winLonghorn" already as w2k8, it seems that some
ESXi versions also use "longhorn" here.

Add that to our known Windows OS type mapping, so that it gets
correctly mapped to our w2k8 type, which affects the selected
hardware.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 10:50:07 +02:00
Thomas Lamprecht
a57207555b bump version to 8.2.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-19 09:07:15 +02:00
Thomas Lamprecht
0947165111 d/control: bump versioned dependency for libpve-cluster-perl
to have the new ssh related helpers available

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-19 08:41:24 +02:00
Fabian Grünbichler
0cb758c919 upload: use SSH helper to get ssh/scp options
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-04-19 08:39:59 +02:00
Fabian Grünbichler
a12671ae97 lvmthin: import: improve readability
the nested call combined with the long parameter list is not very nice to
format/read, let's split it..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-04-18 15:10:46 +02:00
Hannes Duerr
f8a368c63c fix #1611: implement import of base-images for LVM-thin Storage
for base images we call the volume_import of the parent plugin and pass
it as vm-image instead of base-image, then convert it back as base-image

Signed-off-by: Hannes Duerr <h.duerr@proxmox.com>
2024-04-18 15:05:59 +02:00
Thomas Lamprecht
3e5f91823a storage migrate: only output about cleaning-up snapshots if there are any
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-17 12:21:08 +02:00
Thomas Lamprecht
9e222078a9 storage migrate: tiny line reduction
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-17 12:20:36 +02:00
Mira Limbeck
b6fc9de14a fix insecure migration failing if waiting on lock
both STDOUT and STDERR are written into `$info` which is then parsed for
IP and port of the target socket listening.
when the ports file can't be locked immediately `trying to acquire
lock...` is printed on STDERR and in turn written into `$info`.
trying to parse the IP then fails, resulting in a migration or
replication failing.

the bare open3 call is replaced by the run_command wrapper from
pve-common to use a safe wrapper around open3 with the same
functionality.
STDERR is read separatey from STDOUT and the last line of STDERR is
kept in case of errors.

Fixes: 57acd6a ("fix #1452: also log stderr of remote command with
insecure storage migration")

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2024-04-17 12:10:05 +02:00
Fabian Grünbichler
d47460eec7 bump version to 8.1.5
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-04-11 14:33:56 +02:00
Max Carrara
f2c494b14a cephconfig: align our parser with Ceph's parser
This commit rewrites the entire parser for ceph.conf, aligning its
behaviour as closely as possible with Ceph's parser grammar [0].

The most notable improvements are as follows:

  1. The characters '#' and ';' now both mark comments, instead of
     just the '#' character.

  2. Any character, including comment literals ('#' and ';'), may now
     be escaped.

  3. Quoted values (single and double) are now supported.

  4. Line continuations are now supported (lines ending with '\').

  5. Repeated whitespace characters in keys are now treated as a
     single space character.

  6. Dashes '-' are not treated the same as spaces and underscores
     anymore, as Ceph's grammar doesn't treat them that way.
     * Paired with 5., this means that repeated whitespace is now
       equivalent to a single underscore.

  7. Escaped comment literals are now un-escaped.

  8. Although not too crucial, the parser now also supports empty
     sections and will just initialize them with an empty hash.

Furthermore, the original grammar's more quirky behaviours are also
respected where sanely possible.

[0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=ceph/src/common/ConfUtils.cc;h=2f78fd02bf9e27467275752e6f3bca0c5e3946ce;hb=e9fe820e7fffd1b7cde143a9f77653b73fcec748#l144

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
ad6bcc9e00 cephconfig: escape un-escaped comment literals on write
in order to prevent configuration errors or the configuration being
misinterpreted.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
d93f8aea9d cephconfig: align written key-value pairs by tab
instead of tab + space.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
f9d9bd3223 cephconfig: change order of written sections
in order to group related sections together.

Additionally, sections that are associated with the client, for
example '[client.foo]', are written directly after the '[client]'
section.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
8777f4a600 cephconfig: allow writing arbitrary sections
This adds support for writing arbitrary sections to 'ceph.conf' while
ensuring that already written sections are not duplicated.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
094f3f901f test: add tests for 'ceph.conf' parser and writer
These tests attempt to cover most of Ceph's config parser's grammar,
including all of its syntax quirks [0].

Each case is tested against two testing subroutines:

  1. The parser's output is compared with the expected output.

  2. The writer's output is parsed again ant then compared with the
     expected output.

[0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=ceph/src/common/ConfUtils.cc;h=2f78fd02bf9e27467275752e6f3bca0c5e3946ce;hb=e9fe820e7fffd1b7cde143a9f77653b73fcec748#l144

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
e8dbfc506c cephconfig: change code style inside config writer
This commit changes the code style of subroutine `write_ceph_config`
to match our style guide [0] more.

Furthermore, the repeated calls to the inner subroutine are replaced
with a loop, while the regular expressions used by the inner `sub` are
now quoted with `qr` to prevent any accidental mis-quotings in the
future.

[0]: https://pve.proxmox.com/wiki/Perl_Style_Guide

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Thomas Lamprecht
10facd3793 esxi: guest types: use quotes for all hash-map keys
A small coding style patch to make the list of guest-types mapping to
Windows easier to read.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-10 10:20:39 +02:00
Stefan Sterz
f72d3e12e1 esxi: add mapping for windows server 2016/2019
previously these were mapped to the linux 2.6 default

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2024-04-09 13:39:13 +02:00
Wolfgang Bumiller
27591290c7 use stderr from qemu-img invocation as error
Instead of just using it as a warning and then trying to parse an
empty string as json.

For example, trying to parse unsupported vmdks, previously we'd see
something like this:

  qemu-img: Could not open
  '/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
  Unsupported image type 'vsanSparse'
  could not parse qemu-img info command output for
  '/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk'
  - malformed JSON string, neither tag, array, object, number, string
  or atom, at character offset 0 (before "(end of string)") at
  src/PVE/Storage/Plugin.pm line 962, <DATA> line 960.

Now it simply shows:

  qemu-img: Could not open
  '/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
  Unsupported image type 'vsanSparse'

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-04-04 17:26:59 +02:00
Thomas Lamprecht
d09ed4bf76 esxi: status: mark as active if its mounted
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 19:12:06 +01:00
Thomas Lamprecht
73f4faefb3 esxi: drop unused variable and commented out code
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 19:11:33 +01:00
Thomas Lamprecht
0251b6d814 bump version to 8.1.4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 17:37:32 +01:00
Thomas Lamprecht
0d8a6f2901 d/control: bump versioned dependency for esxi-import-tool
to ensure new port-related CLI options are available

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 17:37:26 +01:00
Wolfgang Bumiller
702049157b esxi: add 'port' config parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-28 14:34:24 +01:00
Thomas Lamprecht
a71c404392 bump version to 8.1.3
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-27 13:13:57 +01:00
Thomas Lamprecht
897662bac5 esxi: reduce cache invalidation time to 30s
Reduce the time the cache stays valid from 60s to 30s, while this
could double the amount of requests in the worst case, it's still not
that frequent and also halves the maximal time a user has to wait to
see changes on the ESXi side to appear here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-27 13:13:14 +01:00
Aaron Lauterer
b29a8c3596 esxi: use mac address when static, generated and vpx
static -> defined manually
generated -> by ESXi
vpx -> generated by vCenter

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2024-03-27 10:52:07 +01:00
Thomas Lamprecht
291f8c7dbc bump version to 8.1.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-22 16:39:30 +01:00
Thomas Lamprecht
d0e43cfa4b d/control: bump versioned dependency for esxi-import-tool
to ensure new CLI options for dropping privileges are available

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-22 16:39:12 +01:00