5
0
mirror of git://git.proxmox.com/git/pve-storage.git synced 2025-01-22 22:03:59 +03:00

1504 Commits

Author SHA1 Message Date
Fabian Ebner
31ed94ccf8 Diskmanage: also set type for partitions
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-09 11:56:48 +01:00
Fabian Ebner
50853be2c5 remove lock from is_base_and_used check
and squash the __no_lock-variant into it.

This lock is not broad enough, because for a caller that plans to do or not do
some storage operation based on the result of the check, the following could
happen:
1. volume_is_base_and_used is called and the result is used to enter a branch
2. situation on the storage changes in the meantime
3. the branch chosen in 1. might not be the one that should be taken anymore

This means that callers are responsible for locking, and luckily the existing
callers do use their own locks already:
1. vdisk_free used the __no_lock-variant with a broader lock also covering
   the free operation.
2. vdisk_clone is not a caller, but is relevant and it does lock the storage
2. the calls during VM migration and VM destruction happen in the context of a
   locked VM config. Because the clone operation also locks the VM config, it
   cannot happen that a linked clone is created while the template VM is
   migrated away or destroyed or vice versa. And even if that were the case,
   the base disk would not be freed, because of what vdisk_free/vdisk_clone do.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 14:47:40 +01:00
Fabian Ebner
2949c537d9 Diskmanage: also include partitions with get_disks if flag is set
and have a parent key for partitions, to be able to see the associated disk in
the result without having to rely on naming heuristics (just adding a number at
the end doesn't work for NVMes).

The disk's usage will not be based on the partitions usage if the flag is set,
but will simply be 'partitions'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
6a1919b137 Diskmanage: save OSD information for individual partitions
in preparation to including partitions for get_disks()

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
41f93eceb2 Diskmanage: introduce ceph info helper
so it can be re-used for partitions.

Also changes the regular expression in get_ceph_volume_info to match the full
device/partition name the LV is on. Not only is this needed for partitions,
especially if there's multiple partitions with an OSD, but it also fixes
handling NVMe devices with an OSD as a side effect. Previuosly those were not
detected here, because of the digits in the name, e.g. /dev/nvme0n1

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
d3857eebb5 Diskmanage: also detect BIOS boot, EFI and ZFS reserved type partitions
as they are relevant to most PVE setups.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
01aa7d75fd Diskmanage: introduce usage helper
Note that this is a slight behavior change, because now the first
partition's usage which is not simply 'partition' will become the disk's
usage. Previously, if any partition was 'mounted', it would become the disk's
usage, then 'LVM', 'ZFS', etc.

A partitions usage defaults to 'partition' if nothing more specific can be
found, and is never treated as unused for now.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
89c27ea8ab Diskmanage: collect partitions in hash
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
40be5c5cab Diskmanage: introduce get_sysdir_size helper
to be used for partitions as well.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
59c03cd970 Diskmanage: also check for filesystem type when determining usage
Like this, a non-ZFS filesystem living on a whole disk will also be detected
when it is not mounted.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
b6bbc2ab28 Diskmanage: refactor and rename get_parttype_info
in preparation to also query the file system type from lsblk. Note that the
result now also includes devices without a parttype, so a definedness check in
get_devices_by_partuuid is needed. This will be useful when the whole device
contains a filesystem.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
0cca5356bb Diskmanage: replace closure with direct hash access
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Fabian Ebner
b2843044ba Disks: return correct journal disk candidates
Previously any GPT initialized disk without an osdid (i.e. equal to -1) would
be included in the list of journal disk candidates, for example a ZFS disk. But
the OSD creation API call will fail for those. To fix it, re-use the condition
from the corresponding check in that API call (in PVE/API2/Ceph/OSD.pm).
Now, included disks are unused disks, those with usage 'partitions' and GPT, and
those with usage 'LVM'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 13:52:20 +01:00
Dominik Csapak
acc9c3acbc api: storage/config: use extract_sensitive_params from tools
we have a more general version there

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 17:31:19 +01:00
Fabian Ebner
68f1fc2783 mark PBS storages as shared
Like this, the property will get added when parsing the storage configuration
and PBS storages will correctly show up as shared storages in API results.

AFAICT the only affected PBS operation is free_image via vdisk_free, which will
now be protected by a cluster-wide lock, and that shouldn't hurt.

Another issue this fixes, which is the reason this patch exists, was reported
in the forum[0]. The free space from PBS storages was counted once for each node
that had access to the storage.

[0]: https://forum.proxmox.com/threads/pve-6-3-the-storage-size-was-displayed-incorrectly.83136/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-01-28 16:43:20 +01:00
Dominic Jäger
d854a71803 lvm: Fix #3159: Show RAID LVs as storage content
LVM RAID logical volumes (including mirrors) can be valid disk images, so they
should show up in storage content listings (for example pvesm list).

Including LV types is safer than excluding, especially because of possible
additional types in the future.

Co-developed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
2021-01-26 19:19:02 +01:00
Alwin Antreich
89b9ac961a fix: check connection for nfs v4 only server
the check_connection is done by querying the exports of the nfs server
in question. With nfs v4 those exports aren't listed anymore since nfs
v4 employs a pseudo-filesystem starting from root (/).

rpcinfo allows to query the existence of an nfs v4 service.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2021-01-26 19:15:17 +01:00
Thomas Lamprecht
d7fedf266a bump version to 6.3-5
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-01-26 18:37:38 +01:00
Dominik Csapak
26a8f21ab2 add workaround for zfs rollback bug
as described in the zfs bug https://github.com/openzfs/zfs/issues/10931
the kernel keeps around cached data from mmaps after a rollback, thus
having invalid data in files that were allegedly rolled back

to workaround this (until a real fix comes along), we unmount the subvol,
invalidating the kernel cache anyway

the dataset gets mounted on the next 'activate_volume' again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 18:32:47 +01:00
Fabian Grünbichler
d3a5e30963 drop absolute udevadm path
the compat symlink from bin to sbin has been dropped with bullseye, and
we rely on PATH begin set properly in our daemons/CLI tools anyway..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 18:27:14 +01:00
Thomas Lamprecht
a7e0184190 debian/control: use https in homepage link
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-01-26 14:04:15 +01:00
Dominik Csapak
2c048efdaf Diskmanage: extend wearout detection for SAS disk
for some controllers/disks there the line is
Percentage used endurance indicator: x%

so extend the regex for that possibilty.
We even had a test-case for SAS but did not notice we could extract
that info from there...

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-15 14:42:16 +01:00
Dominik Csapak
a66db4ae55 tests: add ssd sas disk
copied from test 'sas' with rotational set to 0
this has then the type 'ssd', rpm: 0, and health: 'OK'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-15 14:42:16 +01:00
Fabian Ebner
189e67ffa6 fix #3199: by fixing usage of strftime
In a very early version I wanted to parse the date from the backup
name, and when switching to using the ctime and localtime() instead,
I forgot to update the usage of strftime.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-15 14:40:45 +01:00
Thomas Lamprecht
32dbc619a5 drbd: comment that the builtin plugin is depreacated
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-15 12:58:20 +01:00
Thomas Lamprecht
c6e632efb6 d/control: update
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-14 21:58:24 +01:00
Thomas Lamprecht
175d73fa75 d/compat: update to 10
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-14 21:53:13 +01:00
Thomas Lamprecht
b59f5e0eed bump version to 6.3-4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-14 16:15:07 +01:00
Fabian Ebner
10dfeb9ef0 prune mark: correctly keep track of already included backups
This needs to happen in a separate loop, because some time intervals are not
subsets of others, i.e. weeks and months. Previously, with a daily backup
schedule, having:
* a backup on Sun, 06 Dec 2020 kept by keep-daily
* a backup on Sun, 29 Nov 2020 kept by keep-weekly
would lead to the backup on Mon, 30 Nov 2020 to be selected for keep-monthly,
because the iteration did not yet reach the backup on Sun, 29 Nov 2020 that
would mark November as being covered.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-12-14 16:11:29 +01:00
Thomas Lamprecht
44fdfb2af6 nfs and cifs: implement backup notes helper
reuse the one from DirPlugin by directing the call to it, but with
the actual $class. This should stay stable, as we provide an ABI and
try to always use $class->helpers.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-07 16:13:09 +01:00
Thomas Lamprecht
f244e2aa7f api: content/backup: handle deletion of notes
Previous to this we did not called the plugins update_volume_notes at
all in the case where a user delted the textarea, which results to
passing a falsy value ('').

Also adapt the currently sole implementation to delete the notes field
in the undef or '' value case. This can be done safely, as we default
to returning an empty string if no notes file exists.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-07 16:10:09 +01:00
Thomas Lamprecht
20471dfd95 dir plugin: code cleanup
mostly re-ordering to improve statement grouping and avoiding the
need for an intermediate variable

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-07 16:09:26 +01:00
Thomas Lamprecht
a8e00d1a5d bump version to 6.3-3
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-03 17:24:59 +01:00
Thomas Lamprecht
b1f50fb5cc d/control: bump versioned dependency for libpve-common-perl
so the new get_repository helper for PBS is available

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-03 16:55:28 +01:00
Dominik Csapak
53003cb5ea PBSPlugin: use get_repository from PVE::PBSClient
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-03 16:53:53 +01:00
Wolfgang Bumiller
ab90c3b1f1 pbs: fix token auth with PVE::APIClient
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-12-03 16:53:43 +01:00
Thomas Lamprecht
405b247168 api: scan: note that USB is depreacated
It now got moved in /nodes/<node>/hardware/usb as envisioned[0], this
allows to sunset the usb scan API endpoint here and drop it with 7.0

[0]: https://lists.proxmox.com/pipermail/pve-devel/2018-November/034694.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-02 11:31:06 +01:00
Thomas Lamprecht
a57101434b bump version to 6.3-2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:27:54 +01:00
Thomas Lamprecht
39fc0c77e2 api/cli: add pbs scan endpoint and command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
11a942a18a pvesm: also map the password param for new style cifs scan command
not only for the old deprecated alias

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
b1ccf4f357 api: scan: move over index and usb scan from manager
Add the missing pieces allowing pve-manager to just point the
/nodes/<node>/scan api directory at this module, dropping it's
duplicated copy.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
e65abdb458 api: scan cifs: port over NT_STATUS filter from pve-manager
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
2892b6739d factor out scan CLI definition to real API module
we have a 1:1 copy of that code in pve-manager's PVE::API2::Scan,
which we can avoid by using a common module form pvesm CLI and the
API.

This is the first basic step of dropping the code duplication in
pve-manager.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
2cd10f58be pbs: activate storage: fully validate if storage config works
improves UX of on_update and on_add hooks *a lot*.

This is a bit more expensive than the TCP ping, or even just an
unauthenticated ping, but not as bad as a full datastore status - as
this only reads the datastore config file (which is normally in page
cache anyway).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
8b62ac6a0c pbs: add scan datastore helper
for use in both, the scan API and the on_add/on_update hooks

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Thomas Lamprecht
2f9eb6dc4c pbs: reuse pve apiclient for api connect helper
it is flexible enough to easily do so, and should do well until we
actually have cheap native bindings (e.g., through wolfgangs rust
permlod magic).

Make it a private helper, we do *not* want to expose it directly for
now.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-01 19:22:47 +01:00
Fabian Ebner
f3ccd0ef3d plugin: hooks: add explicit returns
to avoid returning something unexpected. Finish what
afeda182566292be15413d9b874720876eac14c9 already started for all the other
plugins. At least for ZFS's on_add_hook this is necessary (adding a ZFS storage
currently fails as reported here [0]), but it cannot hurt
in the other places either as the only hooks we expect to return something
currently are PBS's on_add_hook and on_update_hook.

[0]: https://forum.proxmox.com/threads/gui-add-zfs-storage-verification-failed-400-config-type-check-object-failed.79734/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-27 10:45:42 +01:00
Thomas Lamprecht
74dcca3a48 nfs: code cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-27 10:45:06 +01:00
Thomas Lamprecht
39591d92ec bump version to 6.3-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-24 23:20:58 +01:00
Thomas Lamprecht
878fe0177e api: content: pass encrypted status for PBS backups
Prefer the fingerprint, fallback to checking the files crypt-mode.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-24 23:18:05 +01:00