1874 Commits

Author SHA1 Message Date
Fabian Ebner
9c7674227f api: apt: switch to common Proxmox::RS::APT::Repositories package
which is shared between PVE and PMG.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-07-15 12:36:50 +02:00
Thomas Lamprecht
b23305d5ad pveceph install: clarify comment for conditional ceph-volume inclusion
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-07-05 08:38:51 +02:00
Stefan Sterz
3409fbeaff pveceph install: exclude 'ceph-volume' from older than quincy releases
when installing non-quincy versions, 'ceph-volume' is not contained in
the respective repositories and, thus, the install process would fail.

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
 [ T: reworded commit subject ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-07-05 08:19:28 +02:00
Thomas Lamprecht
2812907f23 pveceph: install split-out ceph-volume package
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-07-04 13:30:51 +02:00
Thomas Lamprecht
7271e6f67d pveceph: reuse supported ceph versions for non-default warning
albeit this is a bit odd safety "check" anyway

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-21 17:04:49 +02:00
Thomas Lamprecht
983921b9a1 pveceph: factor out supported/default ceph version to single source
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-21 17:04:49 +02:00
Thomas Lamprecht
ef25743f04 pveceph: warn on (soon) EOL octopus
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-21 16:52:54 +02:00
Thomas Lamprecht
9e81f36459 pveceph: add quincy installation support
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-21 16:52:25 +02:00
Dominik Csapak
0c8d746815 fix #4053: don't run vzdump jobs when they change from disabled->enabled
by updating the timestamp in the job state when enabled is changing
from 0 -> 1. We do it this way too in PBS for example, and is the more
sensible behaviour.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
2022-06-17 17:21:56 +02:00
Dominik Csapak
c61c192e17 fix #4026: add 'repeat-missed' option for jobs
like systemd-timers 'persistent'. so that the user can configure it to not be
run after powering up when it was previously missed

this reverses the default behaviour to not run missed jobs after pvescheduler
was started, since most of the time that's not the desired behaviour

since we don't use it for updated schedules anymore, rename
'updated_job_schedule' to 'update_last_runtime'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
2022-06-17 17:21:56 +02:00
Fabian Ebner
b70458b6dd fix #4111: replication: don't send mail when fail count is zero
which can happen when failing to obtain the guest's migration lock.
This led to a lot of mails being sent during migration (timeout for
obtaining lock is only 2 seconds and we run it in a loop).

One could argue that obtaining the lock should increase the fail
count, but without the lock, the job state should not be touched and
even the first three mails upon migration could be considered spam.

Fixes: fa4bb659 ("replication: sent always mail for first three tries and move helper")
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-06-17 17:20:06 +02:00
Oguz Bektas
61a687ac13 fix #3961: pveam: error on 'list' if storage doesn't exist
storage_check_enabled() already dies with an appropriate error message
so we don't have to handle it here

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2022-06-14 10:12:35 +02:00
Fabian Ebner
e6f55a13b0 api: ceph: mon: make checking for duplicate addresses more robust
Because $mon->{addr} might come with a port attached (affects monitors
created with PVE 5.4 as reported in the community forum [0]), or even
be a hostname (according to the code in Ceph/Services.pm). Although
the latter shouldn't happen for configurations created by PVE.

[0]: https://forum.proxmox.com/threads/105904/

Fixes: 9e989449 ("api: ceph: mon: fix handling of IPv6 addresses in assert_mon_prerequisites")
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-06-08 08:49:34 +02:00
Aaron Lauterer
ea4d55c99d report: add ceph device list
Can help a lot to identify which OSD is using which physical disk

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-06-02 09:09:58 +02:00
Thomas Lamprecht
7658f01078 minor code style clenaup and comment wording
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-28 09:12:21 +02:00
Fabian Ebner
3f488b618b vzdump: avoid 'requires' constraint when parsing defaults
to avoid warnings like
parse error in '/etc/vzdump.conf' - 'storage': missing property -
'notes-template' requires this property
when there is no default for the required property configured.

In new(), the defaults are mixed in with the regular CLI/API
parameters, so re-check if the required property is set. If it's not,
the defaults do not apply to the current run, and can be dropped.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-05-12 17:17:56 +02:00
Fabian Ebner
2078e35988 vzdump: generate notes: initialize potentially undef values
For VMs, $task->{hostname} might be undef and when running on a
stand-alone node, there is no cluster name.

Reported-by: Marco Gabriel <mgabriel@inett.de>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-05-12 17:17:56 +02:00
Fabian Ebner
facf65a6e0 vzdump: verify parameters: properly verify notes-template
instead of just checking for a newline, do a full check already.

Also do the check at the beginning of generate_notes() for consistency
and remove the check after expansion to avoid failing late for things
like '{{cl{{node}}er}}' (which can even expand to a valid variable
making the error even more confusing).

Co-developed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-05-12 17:17:48 +02:00
Thomas Lamprecht
d0dfd9b87f vzdump: fix unknown variable name error, add newline
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-04 08:19:35 +02:00
Fabian Ebner
1f581ff78e vzdump: notes-template: improve check for unknown variable
so that '{{foo}}{{bar}}' is not detected as being an unknown variable
named 'foo}}{{bar', but as 'foo' (and 'bar').

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-05-04 07:58:50 +02:00
Fabian Ebner
4ed6e1b182 vzdump: notes-template: avoid escaping meta-characters upon replace
which is caused by the quoting operators \Q...\E. The actual intention
was to avoid such surprises.

Fixes: e01438a7 ("partially close #438: vzdump: support setting notes-template")
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-05-04 07:57:45 +02:00
Thomas Lamprecht
ea9eea012a api: ceph pool: reword ec desc full textwidth and reword slightly
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-04 07:14:56 +02:00
Aaron Lauterer
1401cb4e43 ceph pools create: enhance erasure-code description
Mention which optional parameters will be used for the replicated
metadata pool but won't have an effect on the erasure coded data pool.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-05-04 07:12:00 +02:00
Aaron Lauterer
9569939a54 ceph pools create: remove crush_rule for ec pool data
The crush rule is an optional paramter which can be used for the
metadata pool, but the erasure coded data pool will always get its own
crush rule. Therefore this parameter can not be adapted.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-05-04 07:11:06 +02:00
Dominik Csapak
e5f94bc198 schedule-analyze: fix schedules with limited runs
When a schedule only has a limited amount of runs it can happen
(e.g. 2022-10-01 8:00/30), $next will be undef after the last run.
Exit early in that case.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-04 07:10:13 +02:00
Aaron Lauterer
fca1900c76 api: ceph pools: add type to returned properties
The osd dump already contains the pool type in numerical format.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-02 15:43:11 +02:00
Aaron Lauterer
5f4efb8817 ceph tools: set_pools: filter settings for erasure code pools
Erasure code pools cannot change certain settings after creation.
Trying to set them will cause errors on Cephs side.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-02 15:43:06 +02:00
Thomas Lamprecht
ec63b237de api: ceph: fix description indentation style
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-29 14:28:12 +02:00
Thomas Lamprecht
07316e6d04 api: followup: code locality
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-29 14:26:32 +02:00
Aaron Lauterer
4605d6fd22 api: ceph ec pools: make add_storages overridable default
The behavior of always adding the storage config was lost in commit
23c407e. But it is more sensible to make it a default that can be
changed if needed.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-04-29 14:24:04 +02:00
Aaron Lauterer
136f761ba7 api: ceph ec pools: schema fixes and enhancements
Ceph has a min value for 'k' of 2. Adding default and description where
missing.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-04-29 14:24:04 +02:00
Thomas Lamprecht
f35e7fcd8e api: ceph ec pools: move to format-str, create ec in worker, reuse $rados
moved to a format string 'erasurce-coded', that allows also to drop
most of the param existence checking as we can set the correct
optional'ness in there.  Also avoids bloating the API to much for
just this.

Reuse the $rados connection more often to avoid to much
overhead/lingering sockets (the rados connection stays around in the
background to allow efficient reuse)

really should be three separate commits, but too intertwined and too
late for me to care tbh.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-28 20:26:38 +02:00
Thomas Lamprecht
23c407e59b ceph tools: allow more rados connection reuse
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-28 20:26:38 +02:00
Aaron Lauterer
3bd128d7a0 ceph pools: allow to create erasure code pools
To use erasure coded (EC) pools for RBD storages, we need two pools. One
regular replicated pool that will hold the RBD omap and other metadata
and the EC pool which will hold the image data.

The coupling happens when an RBD image is created by adding the
--data-pool parameter. This is why we have the 'data-pool' parameter in
the storage configuration.

To follow already established semantics, we will create a 'X-metadata'
and 'X-data' pool. The storage configuration is always added as it is
the only thing that links the two together (besides naming schemes).

Different pg_num defaults are chosen for the replicated metadata pool as
it will not hold a lot of data.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-04-28 20:26:38 +02:00
Aaron Lauterer
34a2222dd8 ceph tools: add erasure code management functions
Functions to manage erasure code (EC) profiles:
* add
* remove
* check if exists
* get default prefixed name
* get pool properties
* destroy crush rule

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-04-28 20:26:38 +02:00
Aaron Lauterer
29fe1eea7a api: ceph: $get_storages check if data-pool too
When removing a pool, we check against any storage that might have that
pool configured.
We need to check if that pool is used as data-pool too.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-04-28 20:26:38 +02:00
Thomas Lamprecht
24f3f2bc29 ceph tools: make rados open if not reused shorter
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-28 20:26:38 +02:00
Thomas Lamprecht
0ac20586ac vzdump: transform hook script checks to post-if notation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-28 18:44:20 +02:00
Dylan Whyte
941ed98edd vzdump: add error if passed in hook script doesn't exist
Previously, if the '--script' argument was passed with a non-existent
file, it would state that a non-executable script was the reason for
failure. This adds a check to see if the hook script exists, in order
to provide a more accurate error message.

Also adds an 'Error:' prefix the 'script not executable' error.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2022-04-28 18:38:53 +02:00
Fabian Ebner
010ff16e9f vzdump: check that notes-template is a single line
While vzdump itself wouldn't mind about unescaped newlines, the
parameter isn't supposed to contain any, and when used as part of the
job config, it has to be a single line too, so make it consistent.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-28 14:16:33 +02:00
Fabian Ebner
31213d61d3 vzdump: generate notes: die upon unexpected escape character or variable
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-28 14:16:33 +02:00
Fabian Ebner
e01438a744 partially close #438: vzdump: support setting notes-template
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-28 13:45:53 +02:00
Fabian Grünbichler
c971330601 api2: network: anybridge: re-add regular bridges
commit 89d146f207225bb8ca2e01d7e79000bb37a227d1 introduced permission
checks here that caused all regular bridges to be removed from the
returned list as soon as the SDN package is installed, unless the user
is root@pam or there exists a VNET with the same ID.

this is arguably a breaking change, so limit the priv check to actually
defined VNETs for the time being, and add ALL regular bridges
uncondtionally like before.

get_local_vnets already filters by the same prvs, so we need to get the
full config to find out which IDs are VNETs and which are not.

once/iff we introduce ACL paths for *all* bridges in the future, we can
limit accordingly here.

CC: Alexandre Derumier <aderumier@odiso.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-04-27 13:55:57 +02:00
Thomas Lamprecht
ca65e09904 api: next-id: honor new datacenter.cfg upper/lower range option
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-27 13:52:01 +02:00
Fabian Ebner
bbd4cdd845 vzdump: support setting protected status
Check the number of protected backups early if the protected flag
is set.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-27 11:37:26 +02:00
Fabian Ebner
c527d28f0c fix #3955: vzdump: new: also send mail for error during job-init hook
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-27 10:36:19 +02:00
Fabian Ebner
3c5a7616cf vzdump: new: add add_error helper
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-27 10:36:19 +02:00
Thomas Lamprecht
5ac1eaa041 replication: split out error handling and include more info in mail
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-27 10:18:30 +02:00
Thomas Lamprecht
fa4bb659a3 replication: sent always mail for first three tries and move helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-27 09:48:28 +02:00
Fabian Ebner
237f00be03 api: replication: don't send mails about failed replication only once
but rather multiple times becoming exponentially less frequent.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-26 16:09:43 +02:00