5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2024-12-22 13:34:16 +03:00
Commit Graph

1948 Commits

Author SHA1 Message Date
Wolfgang Bumiller
bf453532b0 storage plugins: add 'import' content type
Add a new 'import' content type which will be the corner stone for a
better API and UI integrated way to import virtual guests into Proxmox
VE.

For starters this will be used to implement a ESXi adapter, so that
those VMs can get imported nicely.

Later we want to integrate the OVF/OVA import skeletons we got in
qemu-server to something more usable here.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 [ TL: add more commit message with some background ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-13 12:25:00 +01:00
Thomas Lamprecht
eaf6995afd bump version to 8.1.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-06 18:26:18 +01:00
Thomas Lamprecht
ee283b6d57 buildsys: cleanup all docs and shell comnpletion artefacts
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-05 11:26:30 +01:00
Thomas Lamprecht
8d060d6749 d/copyright: update years
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-01 10:13:24 +01:00
Thomas Lamprecht
71176c2b8c d/control: add dependency relation on libpve-access-control
We actually use it already (partially without perl use statement) and
the use of the new Sys.AccessNetwork privilege should be version
guarded too.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-02-28 15:51:25 +01:00
Thomas Lamprecht
17e0b01b7f fix #5254: api: allow usage of download-url with Sys.AccessNetwork
The download-url API endpoint has some implications that admins are
unaware of, namely that it basically allow to scan the whole network
via HTTP URLs, and potentially even download some image that the user
should not have access to and adding to a VM that the user controls.

That's why in addition to the Datastore.AllocateTemplate privilege on
the storage, the Sys.Modify on the whole Cluster was required to use
the API call. That design was chosen as we were not fully sure if a
separate privilege is warranted, but user feedback has shown that the
(not so big) cost of adding such a new privilege is justified.

Change the permission check to allow the combination of
Datastore.AllocateTemplate on the storage and either 'Sys.Modify' on
/, for backwards compatibility, or the newer 'Sys.AccessNetwork' on
the node that handles the download.
Using a node-specific ACL path allows admins to e.g. prepare one
specific node's firewall so that pveproxy can access only a safe set
of hosts via outgoing HTTP (not stemming from valid connection
tracking to the PVE API), and thus even further limit the privileges
of users or tools that are trusted to download images to a storage.

Buglink: https://bugzilla.proxmox.com/show_bug.cgi?id=5254
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Hannes Duerr <h.duerr@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-02-28 15:48:44 +01:00
Fiona Ebner
4abb327843 plugin schema: improve description of 'shared' property
It's not clear to users what this property does otherwise. Latest
report from the community forum:

https://forum.proxmox.com/threads/134393

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-01-30 12:42:36 +01:00
Fiona Ebner
0554c03683 plugin schema: improve description of 'nodes' property
The default description doesn't make it clear what the property is
used for in the context of storages.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-01-30 10:38:38 +01:00
Fiona Ebner
76aa80fc93 zfs: fix duplicate word typo in error message
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-01-30 10:25:41 +01:00
Thomas Lamprecht
a18a8b4b45 bump version to 8.0.5
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-21 21:20:57 +01:00
Thomas Lamprecht
5840bd18a8 btrfs: fix calling parent create_base method in fall-back
If we want to forward to the create_base of the directory plugin while
making that use our $class for the operations that call might do, we
cannot use the -> notation (which would resolve the next actual
implementation) but rather pass the class directly.
But, DirPlugin reuses the create_base method from the base Plugin
method, so we also need to call that, because on direct call notation
the inheritance fallback to super methods isn't available.

Reported in the forum:
https://forum.proxmox.com/threads/95684/post-606535

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-19 20:11:36 +01:00
Thomas Lamprecht
3cd71f42c0 bump version to 8.0.4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-13 18:12:48 +01:00
Aaron Lauterer
e1348a2792 disks: get: add osdid-list return parameter
It is possible to run multiple OSD daemons on one disk. The new
'osdid-list' parameter returns an array of all OSD IDs found on the
disk.

The old 'osdid' parameter is kept for compatibility. We might want to
deprecate / remove it in the future.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2023-11-13 16:48:08 +01:00
Thomas Lamprecht
29ca4abaee d/control: bump versioned dependency for libpve-common-perl to >= 8.0.10
to ensure the new tar parameter for the pbs-client is known

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-13 16:44:46 +01:00
Dominik Csapak
96c579a262 api/filerestore: add 'tar' parameter to 'download' api
to be able to download 'tar.zst' archives

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2023-11-13 16:44:07 +01:00
Thomas Lamprecht
c4f6c2ff7a iscsi: code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-27 13:44:05 +02:00
Thomas Lamprecht
6f072aaaa8 iscsi: always recheck if iscsiadm binary is available
this avoids the need for restarting all services that have pve-storage
modules loaded after the admin installed open-iscsi.

while at it make it a bit more clear that this might die by using
assert in the method name.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-27 13:43:18 +02:00
Yuri Konotopov via pve-devel
90c1b10cc5 fix #254: iscsi: add support for multipath targets
With this patch Proxmox now tries to login to all discovered portals
in case some of them are not logged yet.
In case of multipath configuration when initially configured portal is
missing for some reason Proxmox don't lose iSCSI storage now and can
successfully restore iSCSI connection between reboots.

Signed-off-by: Yuri Konotopov <ykonotopov@gnome.org>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
2023-10-27 13:18:52 +02:00
Thomas Lamprecht
9bd1c95a71 offline migration: use telling variable for capture group
With that the if/else branches are pretty clear, so drop the rather
noisy comments that describe what one can get from the code already
anyway.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-08-31 15:21:46 +02:00
Leo Nunner
aa82ad5c25 fix #3004: show progress of offline migration in task log
dd supports a 'status' flag, which enables it to show the copied bytes,
duration, and the transfer rate, which then get printed to stderr.

Signed-off-by: Leo Nunner <l.nunner@proxmox.com>
2023-08-31 15:21:11 +02:00
Fiona Ebner
122ccde412 cifs: bubble up NT_STATUS_INVALID_PARAMETER during connection check
instead of claiming that the storage is not online.

Would've made the issue fixed by b27da68 ("cifs: fix check connection
call") more obvious, because (the UI passes along an empty string for
domain if not set and) the smbclient call returns that status with
> -W ''
in Bookworm.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-08-30 11:27:45 +02:00
Wolfgang Bumiller
2a85ebe260 bump version to 8.0.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-08-18 10:40:16 +02:00
Wolfgang Bumiller
d99d45930a whitespace and formatting fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-08-18 10:12:57 +02:00
Wolfgang Bumiller
444d262fd4 remove unneeded comment
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-08-18 10:06:26 +02:00
Philipp Hufnagl
2197ff97dc fix #4849: download-url: allow download and decompression of compressed ISOs
adds information for how to decompress isos.

generates the compressor regex from a list of comression formats (to
avoid redundancy)
extends the download_url wtih the functionality to handley compression
for images

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
2023-08-18 09:58:06 +02:00
Philipp Hufnagl
57ec06621b fix whitespace
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>

FG: dropped non-indentation parts

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2023-08-04 13:49:03 +02:00
Philipp Hufnagl
618bdfd927 clean: fix whitspaces and minor code issues
removed Data::Dumper and a newline

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>

re-added the newline, but fixed up the indentation at that line..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2023-07-26 14:28:12 +02:00
Thomas Lamprecht
42c08f41cb bump version to 8.0.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-28 09:32:00 +02:00
Fiona Ebner
b27da68320 cifs: fix check connection call
by not having -W eat the next argument when $scfg->{domain} is defined
but empty.

As reported in the community forum [0], this could lead to issues
adding a storage in the UI.

[0]: https://forum.proxmox.com/threads/44983/post-566842

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-06-26 20:31:25 +02:00
Thomas Lamprecht
238d9b9aff bump version to 8.0.1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-17 15:34:04 +02:00
Thomas Lamprecht
8bd984a181 disk api: only ask for Datastore.Allocate if adding to storage config
The Proxmox VE storage systems doesn't cares at all if the
Datastore.Allocate privilege is present if no Proxmox VE storage will
be allocated.

Note, if we want to restrict this further as Sys.Modify on /, which
is already quite a powerful permission, we should probably add a new
one under the Sys. space, e.g., Sys.Disk.Use or the like.

This is a step in splitting the disk manage code out of the
pve-storage package, and maybe even repository

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-17 14:57:20 +02:00
Thomas Lamprecht
2deca27032 disk api: avoid using unrelated Datastore.Audit priv for disk management
Local disk and storage creation and listing is something rather
different than the Proxmox VE storage client ABI that provides an
abstract access to a variety of storage types, specifically targeted
to virtual guests images, templates and backups.

The Datastore.* privilege group is specifically made for auditing the
abstract configuration, here the name must be interpreted in context
and not just assumed that due to "datastore" sounding like it could
have to do something with disks or creation of local storage it just
must be a good fit.

Luckily, Sys.Audit was already used too, which is the correct one
here, this is for node specific (HW) details, not some config for
accessing datastore in a restricted way.

This is a step in splitting the disk manage code out of the
pve-storage package, and maybe even repository.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-17 14:57:20 +02:00
Thomas Lamprecht
b89854ad46 disk management api: comment how storages are added and cleanup style
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-17 14:57:20 +02:00
Thomas Lamprecht
c61e609eb6 fix #4785: avoid overly noisy mkdir deprecation warning for now
Move the warning over to creating and updating storage configs, which
is much less noisy as the constantly called activate storage (e.g.,
pvestatd).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-17 13:05:28 +02:00
Friedrich Weber
09f1f847ae content-dirs check: silently skip paths that cannot be resolved
Since commit 8e623a2930, the inequality
check for content-dirs prints a warning if a content directory path
could not be resolved, i.e., if `abs_path` returns undef. Among other
things, `abs_path` returns undef if the path has an inner (= any but
last) component that does not exist. This can happen for a storage
with content type `iso,vztmpl` and `create-subdirs` set to 0, in case
`template/` does not exist. In this case, the warnings printed by
pvestatd are quite noisy.

As missing content directories are not a problem per se, remove the
warning and just ignore the directory during the inequality check.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
2023-06-15 14:19:33 +02:00
Fiona Ebner
46b9c1fe31 api: config: add/update storage: check for type mismatch first
This avoids confusing errors about other properties when the storage
type doesn't match. By highlighting that the type doesn't match, users
should know right away what the issue is.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-06-15 14:19:33 +02:00
Thomas Lamprecht
da6be324d9 bump version to 8.0.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-09 14:01:31 +02:00
Thomas Lamprecht
030e5ad550 content dirs: skip creation if either mkdir or create-subdirs is false
This is slightly confusing due to both options, the legacy convoluted
one and the new targeted one, exist, but before the rework we skip if
either of those sub-expressions was true, so doing it needs both to
be true.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-09 13:57:17 +02:00
Thomas Lamprecht
8e623a2930 activate storage: ensure content directories are created before checking them
checking the content dirs for clashes via abs_path must be done after
the logic for creating them ran, as abs_path is working on actual
filesystem level, so it will return undf if the directory does not
exist, in which case we then set a hash entry for "undef", and the
next for loop round then resolved again to "undef", resulting in a
false-positive of the check.

Avoid the dangerous "return if" stanzas and reverse them to an actual
if block, which is much safer to adapt. Then move the check for
duplicate content-dir usage after that.

best viewed with white space change ignored: git show -w

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-09 13:32:11 +02:00
Thomas Lamprecht
dcc761c628 bump version to 8.0.0~2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-07 19:10:47 +02:00
Stefan Hrdlicka
13ee4fc859 fix #2920: cifs: add options parameter
This makes it possible to add all mount options offered by mount.cifs.
NFS & CIFS now share the options parameter since they use it for the
same purpose.

Signed-off-by: Stefan Hrdlicka <s.hrdlicka@proxmox.com>
 [FE: rebase + style fixes]
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
 [T: fix merge conflict ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-07 10:45:12 +02:00
Markus Frank
765e0da23f changed regex to allow @ in directory storage paths
@ is often used to signify btrfs subvolumes.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2023-06-07 08:58:04 +02:00
Thomas Lamprecht
6d9aac955d rbd: volume resize: wrap ceil in int, just to be sure
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-06 19:40:04 +02:00
Fiona Ebner
93e34f7259 rbd: volume resize: avoid passing floating point value to rbd
which causes an error "the argument for option '--size' is invalid".
Just round up to the nearest integer to have at least the requested
size. This is similar to what is done for ZFS with d3e3e5d ("When
resizing a ZFS volume, align size to 1M") and makes commands like 'qm
resize 102 scsi1 +0.01G' work.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-06-06 19:25:27 +02:00
Fiona Ebner
4ff0f451bd rbd: don't specify allow-shrink flag
It was introduced by commit 4b7dd9d ("allow --allow-shrink on RBD
resize"), but doesn't give a rationale. A mail gives more[0],
indicating that the user also uses the function to shrink images.
However, the volume_resize function is only reachable via the resize
API endpoints for VMs and containers, which have an explicit check to
disallow shrinkage. If somebody really wants to shrink the image, just
let them use the storage's tools directly. Calling into Proxmox VE's
perl functions directly is not supported.

[0]: https://lists.proxmox.com/pipermail/pve-devel/2016-November/024077.html

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-06-06 19:25:27 +02:00
Stefan Hanreich
0bbb358c91 volume export: explicitly state unit of bandwidth limit
Other places like replication explicitly state whether byte or bit
are used as a unit rather than only the abbreviation.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
2023-06-06 17:54:58 +02:00
Friedrich Weber
5f4b5bd11c content-dirs: check that all content dirs are pairwise inequal
This prevents strange interactions in case the same content directory
is used for multiple content types.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
2023-06-06 17:26:20 +02:00
Wolfgang Bumiller
49add58c4f bump version to 8.0.0~1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-06-05 14:46:22 +02:00
Wolfgang Bumiller
d81a9aea7b plugin: simplify and fix create-base-path vs mkdir logic
In the previous code, if `create-base-path` was explicitly
set to false, it would be treated the same as if it was
undef, falling through to whatever 'mkdir' was.

Instead, the new options should always be preferred, and the
logic can be simplified to a single line.

Here's the table showing the difference, 'u' being 'undef':

config: mkdir:  u 0 1 u 0 1 u 0 1
        create: u u u 0 0 0 1 1 1
        =========================
mkpath: old:    1 0 1 0 0 1 1 1 1
        new:    1 0 1 0 0 0 1 1 1

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-06-05 11:17:27 +02:00
Aaron Lauterer
7c242295c9 deprecate mkdir option for create-base-path and create-subdirs
The `mkdir` option has two meanings[0][1] which are split up in `create-path`
and `create-sub-dirs`.

The `create-base-path` option decides if the path to the storage is
automatically created or not.
The `create-subdirs` options decides if the default directory
structure (dump, images, ...) at the storage location is created.

The `mkdir` option is still working but will trigger a warning in the
logs.

As a side effect, this also fixes #3214 because the `create-base-path` option
is now run after the `is_mountpoint` check in the `activate_storage`
method in DirPlugin.pm.

The 'mkpath' command has been moved into a new helper function that
first determines if the conditions to create the path is true, called
'config_aware_base_mkdir'.

[0] https://lists.proxmox.com/pipermail/pve-devel/2020-December/046575.html
[1] https://lists.proxmox.com/pipermail/pve-devel/2020-December/046576.html

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2023-06-05 10:56:24 +02:00