7839 Commits

Author SHA1 Message Date
Christoph Heiss
fb378fe543 docs: installation: fix wrong product reference
This was probably copied verbatim from pve-docs and forgotten to be
appropriately changed.

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
2024-10-17 05:43:25 +02:00
Christian Ebner
4ee9264b00 docs: Fix typo for chunk directory naming and rewording
The chunks subdirectories are only using the chunk's 2 byte checksum
prefix given in hex notation.

Also, clarify that chunks are grouped into subdirectories.

Reported in the community forum:
https://forum.proxmox.com/threads/155751/

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-17 05:43:25 +02:00
Thomas Lamprecht
a61272c7ff debian: run wrap-and-sort -tkn to normalize control files
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-16 14:34:11 +02:00
Lukas Wagner
8038f96a53 api: metrics: check permissions before reading any data from the cache
Reading from the metric cache is somewhat expensive, so validate as many
of the required permissions as possible. For host metrics, we can
do the full check in advance. For datastores, we check if we have
audit permissions for *any* datastore. If we do not have privs for
either of those, we return early and avoid reading from the
cache altogether.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-10-15 16:03:37 +02:00
Wolfgang Bumiller
2feb4160f1 update d/control
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-10-15 15:32:58 +02:00
Wolfgang Bumiller
da409e0a62 api: optimize metrics permission checks a bit
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-10-15 14:53:15 +02:00
Lukas Wagner
c804763bdf api: add /status/metrics API
This one is modelled exactly as the one in PVE (there it
is available under /cluster/metrics/export).

The returned data format is quite simple, being an array of
metric records, including a value, a metric name, an id to identify
the object (e.g. datastore/foo, host), a timestamp and a type
('gauge', 'derive', ...). The latter property makes the format
self-describing and aids the metric collector in choosing a
representation for storing the metric data.

[
    ...
    {
	"metric": "cpu_avg1",
	"value": 0.12,
	"timestamp": 170053205,
	"id": "host",
	"type": "gauge"
    },
    ...
]

In terms of permissions, the new endpoint requires Sys.Audit
on /system/status for metrics of the 'host' object,
and Datastore.Audit on /datastore/{store} for 'datastore/{store}'
metric objects.

Via the 'history' and 'start-time' parameters one can query
the last 30mins of metric history. If these parameters
are not provided, only the most recent metric generation
is returned.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
da12adb1f9 metric collection: put metrics in a cache
Any pull-metric API endpoint can alter access the cache to
retrieve metric data for a limited time (30mins).

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
20753e1b53 metric collection: initialize metric cache on startup
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
8ecbb5f152 pbs-api-types: add types for the new metrics endpoint
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
3547cfb63e metric collection: move impl block for DiskStats to metric_server module
It is only needed there and could be considered an implementation detail
of how this module works.

No functional changes intended.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
98cb8ff86b metric collection: drop std::path prefix where not needed
No functional changes intended.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
da3612fade metric collection: rrd: remove rrd prefix from some function names
We have proper namespaces, so these are a bit redundant.

No functional changes intended.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
e8c70ec252 metric collection: rrd: restrict function visibility
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
b7ac1fc8aa metric collection: rrd: move rrd update function to rrd module
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
0a852e1927 metric_collection: split out push metric part
No functional changes intended.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
045fc7750c metric collection: move rrd_cache to new metric_collection module
No functional changes intended.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
b862c872e0 metric collection: add doc comments for public functions
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Lukas Wagner
72e1181830 proxy: server: move rrd stat/metric server to separate module
With the upcoming pull-metric system/metric caching, these
things should go into a sepearate module.

No functional changes intended.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2024-10-15 14:09:41 +02:00
Christian Ebner
d87c9771e4 server: sync: factor out namespace depth check into sync module
By moving and refactoring the check for a sync job exceeding the
global maximum namespace limit, the same function can be reused for
sync jobs in push direction.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-14 12:31:51 +02:00
Christian Ebner
a4f08cbbbb server: sync: make skip reason message more genenric
By specifying that the snapshot is being skipped because of the
condition met on the sync target instead of 'local', the same message
can be reused for the sync job in push direction without loosing
sense.
2024-10-14 12:31:51 +02:00
Christian Ebner
2f05d211c4 server: sync: move skip info/reason to common sync module
Make `SkipReason` and `SkipInfo` accessible for sync operations of
both direction variants, push and pull.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-14 12:31:51 +02:00
Christian Ebner
610dd9b8f3 server: sync: move source to common sync module
Rename the `PullSource` trait to `SyncSource` and move the trait and
types implementing it to the common sync module, making them
reusable for both sync directions, push and pull.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-10 11:11:25 +02:00
Christian Ebner
3f52a94624 server: sync: move reader trait to common sync module
Move the `PullReader` trait and the types implementing it to the
common sync module, so this can be reused for the push direction
variant for a sync job as well.

Adapt the naming to be more ambiguous by renaming `PullReader` trait to
`SyncSourceReader`, `LocalReader` to `LocalSourceReader` and
`RemoteReader` to `RemoteSourceReader`.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-10 11:11:25 +02:00
Christian Ebner
ffe1dd4369 server: sync: move sync related stats to common module
Move and rename the `PullStats` to `SyncStats` as well as moving the
`RemovedVanishedStats` to make them reusable for sync operations in
push direction as well as pull direction.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-10 11:11:25 +02:00
Christian Ebner
0a916665ae api: datastore: add missing whitespace in description
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-10-10 11:11:25 +02:00
Wolfgang Bumiller
6950c7e4ef update proxmox-acme to 0.5.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-10-03 09:54:33 +02:00
Wolfgang Bumiller
93e9e8b6ef update to rrd-api-types 1.0.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-09-19 15:21:45 +02:00
Wolfgang Bumiller
79ed296f2d update to proxmox-rrd 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-09-18 16:04:20 +02:00
Christian Ebner
0415304ca4 pxar: bin: rework and dynamically generate list test data
Commit f16c5de757 ("pxar: bin: test `pxar list` with payload-input")
introduced a regression test for listing of split pxar archives. This
test relies on a large pxar blob file, the large size (> 100M) being
overlooked when writing the test.

In order to not depend on this file any further in the future, drop
it and rewrite the test to dynamically generate the files, needed and
further extend the test thereby also cover the archive creation and
extraction for split pxar archives.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-09-11 16:04:46 +02:00
Wolfgang Bumiller
d1a5855e74 api: replace deprecated 'streaming' attribute with 'serializing'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-09-05 14:01:45 +02:00
Wolfgang Bumiller
bd8c677eab update to proxmox-router 3 and proxmox-rest-server 0.8
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-09-05 14:01:36 +02:00
Wolfgang Bumiller
c3b7862be5 pxar: list stuff to stdout instead of stderr
Our tooling really needs to stop doing outputs wrong...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-09-05 14:01:04 +02:00
Wolfgang Bumiller
ec1e78a4df bump proxmox-log dependency to 0.2.4 for stderr logging
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-09-05 14:00:59 +02:00
Gabriel Goller
1de4974eeb pxar-bin: remove log dependency, use tracing directly
When using the `log` to `tracing` translation layer, the messages get
padded with whitespaces. This bug will get fixed upstream [0], but in
the meantime we switch to the `tracing` macros.

[0]: https://github.com/tokio-rs/tracing/pull/3070

Tested-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2024-09-05 14:00:46 +02:00
Dominik Csapak
e97132bb64 tape: fix read element status for some changers
It seems some changers are setting the PVolTag/AVolTag flags in the
ELEMENT STATUS page response, but don't include the actual fields then.
To make it work with such changers, downgrade the errors to warnings, so
we can continue to decode the remaining data.

This is OK since one volume tag is optional and the other is skipped
anyway.

Reported in the forum:
https://forum.proxmox.com/threads/hpe-storeonce-vtl.152547/

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-09-03 09:22:08 +02:00
Gabriel Goller
dc0de0db10 move client binaries to tracing
Add tracing logger to all client binaries and remove env_logger.

The reason for this change is twofold: our migration to tracing, and the
behavior when the client calls an api handler directly. Currently the
proxmox-backup-manager calls the api handlers directly for some
commands. This results in no output (on console and task log), as no
tracing logger is instantiated.

Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2024-08-30 13:56:43 +02:00
Dominik Csapak
ff485aa320 fix #5622: backup client: properly handle rate/burst parameters
The rate and burst parameters are integers, so the mapping from value
with `.as_str()` will always return `None` effectively never
applying any rate limit at all.

Fix it by turning them into a HumanByte instead of an integer.

To not crowd the parameter section so much, create a
ClientRateLimitConfig struct that gets flattened into the parameter list
of the backup client.

To adapt the description of the parameters, add new schemas that copy
the `HumanByte` schema but change the description.

With this, the rate limit actually works, and there is no lower limit
any more.

The old TRAFFIC_CONTROL_RATE/BURST_SCHEMAs can be deleted since the
client was the only user of them.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-08-30 13:21:29 +02:00
Dominik Csapak
8f27262d42 data_blob: add TODO comment for zstd api
we currently use the behavior of zstd that is not part of the public
api, so this is at risk to be changed without notice.

There is a public api that we could use, but it's only available
with zstd_sys >= 2.0.9, which at this time, is not yet packaged for/by
us.

Add a comment that we can use the public api for this when the
new version of the crate gets available.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-08-30 09:57:32 +02:00
Gabriel Goller
2801fbf03c fix: proxmox-backup-manager network reload wait on worker
Make the `network reload` command in proxmox-backup-manager wait on the
api handler's workertask. Otherwise the task would be killed when the
client exits.

Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2024-08-29 14:13:10 +02:00
Christoph Heiss
4c0a8bc054 ui: user view: disable 'Unlock TFA' button by default
Without this, the button is enabled if no entry at all is selected (e.g.
when switching to the 'User Management' tab), with the button then
(obviously) being a noop.

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
2024-08-29 14:11:46 +02:00
Wolfgang Bumiller
e9152ee951 tests: replace static mut with a mutex
rustc warns about creating references to them (although it does allow
using `.as_ref()` on them for some reason), and this will become a
hard error with edition 2024.

Previously we could not use Mutex there as its ::new() was not a
`const fn` , but not we can, so let's drop the `mut`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-08-29 11:40:49 +02:00
Gabriel Goller
baacc3f2de log: retrieve ReaderEnvironment debug flag from tracing
Don't hardcode the debug flag but retrieve the currently enabled level
using tracing. This will change the default log-behavior and disable
some logs that have been printed previously. F.e.: the "protocol upgrade
done" message is not visible anymore per default because it is printed
with debug.

Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2024-08-29 11:23:44 +02:00
Wolfgang Bumiller
631b09b2eb bump d/control
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-08-14 12:15:40 +02:00
Maximiliano Sandoval
8f58b0bf60 cargo: remove unused dependencies
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 12:13:50 +02:00
Maximiliano Sandoval
bfca4f272d backup-client: remove unused dependencies
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 12:13:50 +02:00
Maximiliano Sandoval
59d9e62307 pxar-bin: remove unused dependencies
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 12:13:50 +02:00
Maximiliano Sandoval
ea3047b2c6 restore-daemon: remove unused dependencies
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 12:13:50 +02:00
Maximiliano Sandoval
0aab7980fc file-restore: remove unused deps
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 12:13:50 +02:00
Maximiliano Sandoval
2443b3f8d0 client: remove unused deps
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 12:13:50 +02:00