5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2024-12-22 13:34:16 +03:00
Commit Graph

1905 Commits

Author SHA1 Message Date
Fabian Grünbichler
0cb758c919 upload: use SSH helper to get ssh/scp options
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-04-19 08:39:59 +02:00
Fabian Grünbichler
a12671ae97 lvmthin: import: improve readability
the nested call combined with the long parameter list is not very nice to
format/read, let's split it..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-04-18 15:10:46 +02:00
Hannes Duerr
f8a368c63c fix #1611: implement import of base-images for LVM-thin Storage
for base images we call the volume_import of the parent plugin and pass
it as vm-image instead of base-image, then convert it back as base-image

Signed-off-by: Hannes Duerr <h.duerr@proxmox.com>
2024-04-18 15:05:59 +02:00
Thomas Lamprecht
3e5f91823a storage migrate: only output about cleaning-up snapshots if there are any
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-17 12:21:08 +02:00
Thomas Lamprecht
9e222078a9 storage migrate: tiny line reduction
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-17 12:20:36 +02:00
Mira Limbeck
b6fc9de14a fix insecure migration failing if waiting on lock
both STDOUT and STDERR are written into `$info` which is then parsed for
IP and port of the target socket listening.
when the ports file can't be locked immediately `trying to acquire
lock...` is printed on STDERR and in turn written into `$info`.
trying to parse the IP then fails, resulting in a migration or
replication failing.

the bare open3 call is replaced by the run_command wrapper from
pve-common to use a safe wrapper around open3 with the same
functionality.
STDERR is read separatey from STDOUT and the last line of STDERR is
kept in case of errors.

Fixes: 57acd6a ("fix #1452: also log stderr of remote command with
insecure storage migration")

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2024-04-17 12:10:05 +02:00
Fabian Grünbichler
d47460eec7 bump version to 8.1.5
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-04-11 14:33:56 +02:00
Max Carrara
f2c494b14a cephconfig: align our parser with Ceph's parser
This commit rewrites the entire parser for ceph.conf, aligning its
behaviour as closely as possible with Ceph's parser grammar [0].

The most notable improvements are as follows:

  1. The characters '#' and ';' now both mark comments, instead of
     just the '#' character.

  2. Any character, including comment literals ('#' and ';'), may now
     be escaped.

  3. Quoted values (single and double) are now supported.

  4. Line continuations are now supported (lines ending with '\').

  5. Repeated whitespace characters in keys are now treated as a
     single space character.

  6. Dashes '-' are not treated the same as spaces and underscores
     anymore, as Ceph's grammar doesn't treat them that way.
     * Paired with 5., this means that repeated whitespace is now
       equivalent to a single underscore.

  7. Escaped comment literals are now un-escaped.

  8. Although not too crucial, the parser now also supports empty
     sections and will just initialize them with an empty hash.

Furthermore, the original grammar's more quirky behaviours are also
respected where sanely possible.

[0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=ceph/src/common/ConfUtils.cc;h=2f78fd02bf9e27467275752e6f3bca0c5e3946ce;hb=e9fe820e7fffd1b7cde143a9f77653b73fcec748#l144

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
ad6bcc9e00 cephconfig: escape un-escaped comment literals on write
in order to prevent configuration errors or the configuration being
misinterpreted.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
d93f8aea9d cephconfig: align written key-value pairs by tab
instead of tab + space.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
f9d9bd3223 cephconfig: change order of written sections
in order to group related sections together.

Additionally, sections that are associated with the client, for
example '[client.foo]', are written directly after the '[client]'
section.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
8777f4a600 cephconfig: allow writing arbitrary sections
This adds support for writing arbitrary sections to 'ceph.conf' while
ensuring that already written sections are not duplicated.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
094f3f901f test: add tests for 'ceph.conf' parser and writer
These tests attempt to cover most of Ceph's config parser's grammar,
including all of its syntax quirks [0].

Each case is tested against two testing subroutines:

  1. The parser's output is compared with the expected output.

  2. The writer's output is parsed again ant then compared with the
     expected output.

[0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=ceph/src/common/ConfUtils.cc;h=2f78fd02bf9e27467275752e6f3bca0c5e3946ce;hb=e9fe820e7fffd1b7cde143a9f77653b73fcec748#l144

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Max Carrara
e8dbfc506c cephconfig: change code style inside config writer
This commit changes the code style of subroutine `write_ceph_config`
to match our style guide [0] more.

Furthermore, the repeated calls to the inner subroutine are replaced
with a loop, while the regular expressions used by the inner `sub` are
now quoted with `qr` to prevent any accidental mis-quotings in the
future.

[0]: https://pve.proxmox.com/wiki/Perl_Style_Guide

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
2024-04-11 11:53:37 +02:00
Thomas Lamprecht
10facd3793 esxi: guest types: use quotes for all hash-map keys
A small coding style patch to make the list of guest-types mapping to
Windows easier to read.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-10 10:20:39 +02:00
Stefan Sterz
f72d3e12e1 esxi: add mapping for windows server 2016/2019
previously these were mapped to the linux 2.6 default

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2024-04-09 13:39:13 +02:00
Wolfgang Bumiller
27591290c7 use stderr from qemu-img invocation as error
Instead of just using it as a warning and then trying to parse an
empty string as json.

For example, trying to parse unsupported vmdks, previously we'd see
something like this:

  qemu-img: Could not open
  '/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
  Unsupported image type 'vsanSparse'
  could not parse qemu-img info command output for
  '/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk'
  - malformed JSON string, neither tag, array, object, number, string
  or atom, at character offset 0 (before "(end of string)") at
  src/PVE/Storage/Plugin.pm line 962, <DATA> line 960.

Now it simply shows:

  qemu-img: Could not open
  '/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
  Unsupported image type 'vsanSparse'

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-04-04 17:26:59 +02:00
Thomas Lamprecht
d09ed4bf76 esxi: status: mark as active if its mounted
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 19:12:06 +01:00
Thomas Lamprecht
73f4faefb3 esxi: drop unused variable and commented out code
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 19:11:33 +01:00
Thomas Lamprecht
0251b6d814 bump version to 8.1.4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 17:37:32 +01:00
Thomas Lamprecht
0d8a6f2901 d/control: bump versioned dependency for esxi-import-tool
to ensure new port-related CLI options are available

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-28 17:37:26 +01:00
Wolfgang Bumiller
702049157b esxi: add 'port' config parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-28 14:34:24 +01:00
Thomas Lamprecht
a71c404392 bump version to 8.1.3
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-27 13:13:57 +01:00
Thomas Lamprecht
897662bac5 esxi: reduce cache invalidation time to 30s
Reduce the time the cache stays valid from 60s to 30s, while this
could double the amount of requests in the worst case, it's still not
that frequent and also halves the maximal time a user has to wait to
see changes on the ESXi side to appear here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-27 13:13:14 +01:00
Aaron Lauterer
b29a8c3596 esxi: use mac address when static, generated and vpx
static -> defined manually
generated -> by ESXi
vpx -> generated by vCenter

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2024-03-27 10:52:07 +01:00
Thomas Lamprecht
291f8c7dbc bump version to 8.1.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-22 16:39:30 +01:00
Thomas Lamprecht
d0e43cfa4b d/control: bump versioned dependency for esxi-import-tool
to ensure new CLI options for dropping privileges are available

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-22 16:39:12 +01:00
Wolfgang Bumiller
9bb651effa esxi: have the fuse process drop its privileges
once it's mounted it only needs to make http requests and talk to the
open fuse fd

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-22 13:24:20 +01:00
Wolfgang Bumiller
06a28968d8 esxi: use listvms.py's stderr as error message
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-22 11:06:56 +01:00
Gabriel Goller
688cc11a29 esxi: detect correct os type in 'other' family
This patch introduces the conversion table for all possible OS Types
that are in the VMWare 'other' family and sets the pve counterpart.
Our default OS Type is 'linux', so including mappings to 'other' makes
sense.

Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2024-03-22 10:06:22 +01:00
Dominik Csapak
c9bece6986 esxi: only add scsihw if it's defined
otherwise we get `scsihw: null` from the api, which is not a valid
value, so just omit it.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-19 18:09:42 +01:00
Dominik Csapak
0f940f10fc esxi: add warning for losing efi state
we cannot import the state of the efivars (e.g. boot order)
so add a warning for that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ TL: add new warning to return schema ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-19 18:09:29 +01:00
Thomas Lamprecht
85fbbb2f96 bump version to 8.1.1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:52 +01:00
Thomas Lamprecht
0a2f75ffd6 esxi: deactivate storage if config gets updated
to ensure that the next access actually uses the updated config for
accessing the remote ESXi, for both manifest API and FUSE mount.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Thomas Lamprecht
68f3ec3f8b esxi: clean-up state-dir in deactivate storage
to ensure we always query a new one on quick add-delete-add cycles.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Thomas Lamprecht
72418e2f05 esxi: deactivate storage on deletion from config
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
2ff6f99c60 esxi: add 'skip-cert-verification' option
following pve-esxi-import-tools's commits:
3ee5c3b ("esxi-folder-fuse: add --insecure option")
c292c67 ("listvms.py: add --insecure parameter, verify cert by
          default")
34c87be ("rename --insecure option to --skip-cert-verification")

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 [ TL: rename 'insecure' to 'skip-cert-verification' to better convey
   what it means ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Dominik Csapak
4f50a578f4 import: rework plugin api to not need a blessed object
does not really make sense, and if the plugin wants that, it can still
be done, like we do here for the ESXiPlugin

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
c92d810053 import: drop target parameter
since disks are a separate hash now with no associated storage, we
don't need this anymore

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
10b5874078 esxi: add disk capacity to import metadata
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
1128100569 esxi: warn if the guest was running
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
aba709f247 esxi: add helper to get vm from manifest for vmx path
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
99ace9d447 esxi: let manifest file expire after 60s
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 [ TL: fix wrong comparison with >= and avoid undef warning if file
   does not yet exist at all ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
814bde1958 esxi: drop unused sub methods
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 [ TL: squash removal of both subs into one commit ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Thomas Lamprecht
1a94dceb6f api: import-metadata: make warnings structured & merge ignored-volumes
This allows the frontends to translate them and avoids somewhat
duplicated info by having some warnings explicitly (ignored-volumes)
while others are in the warnings array.

By passing along the key and the value the frontend can also show the
warnings in-line, e.g. by marking a disk-entry in a grid as having
potential problems.

Ideally we'd have a central list of known types used for the API
return schema enum and to check when calling the $warn closure, but as
we only got three warnings keep this as is and only add a comment.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Thomas Lamprecht
b628964289 api: import-metadata: complete return schema
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Thomas Lamprecht
da513e260c esxi plugin: do not support the images content-type
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Thomas Lamprecht
dde640eda7 esxi plugin: adapt to new libexec path for helpers
and rename the name of the constant for the FUSE tool, as otherwise it
sounds like this was the mountpoint to be used

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
5e934933ec esxi import: deal with serial devices
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-13 15:29:35 +01:00
Wolfgang Bumiller
fd16d98451 add source field to import-metadata
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-13 15:29:35 +01:00