Commit Graph

6879 Commits

Author SHA1 Message Date
Wolfgang Bumiller
84283eabb0 rrd: add Entry::get() to access the data
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-24 13:53:49 +01:00
Daniel Tschlatscher
aaddf46a63 gui: change reporting of the estimated_time_full to "Full" if no space
is left in the datastore. Before, the GUI would report "Never" for the
estimated time full, because the value provided in the backend was in
the past. To get around this, the GUI now reports "Full" if the value
for available reaches 0.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
2022-11-24 13:26:29 +01:00
Daniel Tschlatscher
de923258d0 'available' field in rrd data in the API and change usage of 'total'
The API now exposes the field 'available' as well, with which the
unprivileged total is calculated in all corresponsing views in the
frontend.
The rrd charts now also display the total as the unprivileged total
if available, otherwise the absolute total is used.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
2022-11-24 13:26:27 +01:00
Daniel Tschlatscher
f362f8f075 fix #4077: Estimated Full metric on ext4 file systems
The rrd data now includes tracking the available field in disk usage.
The calculation for the estimated_time_full was adapted to use the
total for the unpriviliged user, which is the sum of used + available.

The total for unprivileged users is preferable, because datastores are
always written to by the backup user. Which means that any storage
space reserved for root is unusable for our purposes.

To avoid resetting the estimate when switching to this new version,
the backend will try to use the available value to calculate the
unprivileged total. When that is not an option, it will fall back to
using the absolute total.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
2022-11-24 13:26:24 +01:00
Wolfgang Bumiller
2943d00c84 cleanup for line limit & less indentation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-24 10:47:09 +01:00
Daniel Tschlatscher
67a5999aa6 make tasklog downloadable in the backup server backend
The read_tasklog API call now stream the whole log file if the query
parameter 'download' is set to true. If the limit parameter is set to
0, all lines in the tasklog will be returned in json format.

To make a file stream and a json response in the same API call work, I
had to use one of the lower level apimethod types from the
proxmox-router. Therefore, the routing declarations and parameter
schemas have been changed accordingly.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
2022-11-24 10:40:05 +01:00
Wolfgang Bumiller
9a087ce7a2 bump d/control
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-23 15:47:33 +01:00
Wolfgang Bumiller
593aa85a9e bump pxar dependency to 0.10.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-23 15:47:07 +01:00
Wolfgang Bumiller
45c7d758e5 distinguish between block/chardevs in diff output
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-23 10:07:23 +01:00
Lukas Wagner
52189f181f fix #3828: proxmox_backup_debug: Introduce diff archive subcommand.
This new subcommand compares a pxar archive in two different
snapshots and prints a list of added/modified/deleted file
entries.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2022-11-23 09:56:16 +01:00
Thomas Lamprecht
c93a8de89d ui: prune job edit: fix disabling jobs
The backend doesn't have an 'enable' option, but 'disable'. Convert
it to avoid a negative value that is checked "enabled".

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 15:26:22 +01:00
Wolfgang Bumiller
e5e252e990 api-types: derive Ord for BackupDir
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-15 10:16:54 +01:00
Thomas Lamprecht
7e13fe3b35 ui: update generated OnlineHelpInfo map
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:25:09 +01:00
Thomas Lamprecht
da6b16c3db ui: improve tape/drive icon positioning/size
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:24:40 +01:00
Thomas Lamprecht
2aaf3ef1e5 file-restore: move various formats to inline-string macros
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:08:09 +01:00
Thomas Lamprecht
7fa5be8b3f file-restore: split out internal QMP logic from unrelated fn
Capsule it in a small QMPSock struct impl, make the usage nicer as
the caller should not have to care & keep track of the initial socket
state+details.

A send_raw and send Value method should cover most needs.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:01:45 +01:00
Thomas Lamprecht
2f0f3e9979 file restore: allow to pass dimm size via env
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:01:45 +01:00
Thomas Lamprecht
0f2f523aa6 file restore: limit max dimm size to 512 MB
this is on top of the normal memory, and over 1.3 GB required is just
huge, sadly the commit adding this has zero details about what setups
fail and what work again with the change, so hard to tell, but any
setup that needs that much sounds like a bug in ZFS or remaining code
here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-14 15:59:50 +01:00
Thomas Lamprecht
69e3beb941 file restore: move allow-memory-hotplug param from CLI to environment
avoid the need to loop a parameter through a dozen function which all
don't care about it at all; iff this should be a global oncecell or
lock guarded param.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-14 15:49:49 +01:00
Thomas Lamprecht
fa1c3eaea1 file restore: rename dynamic-memory to auto-memory-hotplug
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Dominik Csapak
0ce86cb533 file-restore: make dynamic memory behaviour controllable
by adding 'dynamic-memory' parameter that controls if we automatically
increase the memory of the guest vm or not

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
a6c07eb9c2 limit ZFS Arc to more smalles possible value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
c190a32cb1 file restore: list api: fix description of "path"
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
5d0d3c61f9 file restore: switch some formats to inline template variables
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Fiona Ebner
a2e7d5eb6b fix #4287: d/control: recommend proxmox-mail-forward
which registers a binary in /root/.forward and handles mail forwarding
to the mail addresss configured for root@pam in PBS. Similar to how it
is done in PVE currently.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2022-11-10 11:48:58 +01:00
Dominik Csapak
72220d79d7 file-restore: dynamically increase memory of vm for zpools
when a backup contains a drive with zfs on it, the default memory
size (up to 384 MiB) is often not enough to hold the zfs metadata

to improve that situation, add memory dynamically (1GiB) when a path is
requested that is on zfs. Note that the image must be started with a
kernel capable of memory hotplug.

to achieve that, we also have to add a qmp socket to the vm, so that
we can later connect and add the memory backend and dimm

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-07 13:29:34 +01:00
Dominik Csapak
3e8ee886a5 file-restore: fix deprecated qemu parameters
server and nowait are deprecated, so we should use the longform:
server=on and wait=off

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-04 13:27:07 +01:00
Dominik Csapak
389f8c134d docs: add documentation about the 'sync-level' tuning
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-28 13:04:22 +02:00
Dominik Csapak
8d5039ed57 datastore: improve sync level code a bit
fixups for DatastoreFSyncLevel:
* use derive for Default
* add some more derives (Clone, Copy)

chunk store:
* drop to_owned for chunk_dir_path

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-28 13:04:22 +02:00
Fabian Grünbichler
4b3c010fd5 clippy fixes
the dropped .into() is guarded by the bumped build-dependency on
proxmox-sys 0.4.1, the missing Eq is a new clippy lint.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-10-24 12:10:19 +02:00
Aaron Lauterer
fcc8e35499 report: add arcstat
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-10-21 17:32:02 +02:00
Stefan Hanreich
3c78a1dec2 fix #4301: correctly pass rate limit parameters to API
With the old code the rate limit parameters got passed in their own
dictionary under the limit key, but the API expects the rate-limit
settings as top-level keys. This commit correctly sets the rate-limit
parameters so the API actually uses them.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
2022-10-20 16:06:09 +02:00
Dominik Csapak
647186ddf9 datastore: implement sync-level tuning for datastores
currently, we don't (f)sync on chunk insertion (or at any point after
that), which can lead to broken chunks in case of e.g. an unexpected
powerloss. To fix that, offer a tuning option for datastores that
controls the level of syncs it does:

* None (default): same as current state, no (f)syncs done at any point
* Filesystem: at the end of a backup, the datastore issues
  a syncfs(2) to the filesystem of the datastore
* File: issues an fsync on each chunk as they get inserted
  (using our 'replace_file' helper) and a fsync on the directory handle

a small benchmark showed the following (times in mm:ss):
setup: virtual pbs, 4 cores, 8GiB memory, ext4 on spinner

size                none    filesystem  file
2GiB (fits in ram)   00:13   0:41        01:00
33GiB                05:21   05:31       13:45

so if the backup fits in memory, there is a large difference between all
of the modes (expected), but as soon as it exceeds the memory size,
the difference between not syncing and syncing the fs at the end becomes
much smaller.

i also tested on an nvme, but there the syncs basically made no difference

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 14:59:15 +02:00
Dominik Csapak
37495b8eeb pbs-datastore: chunk_store: use replace_file in insert_chunk
it does the same as the current code

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 14:59:13 +02:00
Dominik Csapak
d2c79c54c2 docs: add information about chunk order option for datastores
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 14:59:11 +02:00
Dominik Csapak
f963bc0c81 proxmox-tape: inventory: add default to parameters
and convert the 'Option<bool>' to 'bool'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 13:22:47 +02:00
Dominik Csapak
57ee4e9ea4 ui: tape/ChangerStatus: adding parameter selection to inventory
namely 'catalog' and 'read-all-labels', by always opening a
window (with a drive now autoselected) and the two checkboxes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 13:22:46 +02:00
Dominik Csapak
4ebc97ad4b proxmox-tape: add 'catalog' option to 'proxmox-tape inventory'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 13:22:43 +02:00
Dominik Csapak
c658ea6185 api/tape/inventory: optionally try to restore catalogs
in a disaster recovery case, it is useful to not only re-invetorize
the labels + media-sets, but also to try to recover the catalogs
from the tape (to know whats on there). This adds an option to
the inventory api call that tries to do a fast catalog restore
from each tape to be inventorized.

also sets the correct default for 'read-all-labels' in the api and
converts to a bool

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 13:22:42 +02:00
Dominik Csapak
3921deb29a tape: replace '&Path' with 'AsRef<Path>' in function parameters
this way we can omit the pattern
```
let status_path = Path::new(TAPE_STATUS_DIR);
some_function(status_path);
```
and give the TAPE_STATUS_DIR directly. In some instances we now have to
give TAPE_STATUS_DIR more often, but most often we save a few
intermediary Paths.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-20 13:22:39 +02:00
Lukas Wagner
de7f180dab pbs-client: fix bash autocompletion for archive-files
Previously, autocompletion of archive names, for instance
in the case of

  $ proxmox-backup-client restore <snapshot> <TAB>

did not work if no namespace was provided via the --ns option.
The fix is to fall back to the root namespace if the option is
not provided by the user.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2022-10-20 13:20:02 +02:00
Wolfgang Bumiller
a616c3dfd2 pbs-client: use new non-Fd open from sys 0.4.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:28:10 +02:00
Wolfgang Bumiller
6cffe7229f bump d/control
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Wolfgang Bumiller
5c7ff2afd2 bump sys dep to 0.4.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Wolfgang Bumiller
a61d316c2d client: drop deprecated Fd usage
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Wolfgang Bumiller
76c2912e64 pbs-client: drop use of deprecated RawFdNum
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Wolfgang Bumiller
4cceaf3a2d pbs-client: drop deprecated Fd usage
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 13:51:13 +02:00
Wolfgang Bumiller
61dd5af5aa config: drop deprecated Fd usage
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 13:45:53 +02:00
Wolfgang Bumiller
a63a869e07 client: use BorrowedFd from std
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 11:00:36 +02:00
Wolfgang Bumiller
62c948a8d9 bump edition in rustfmt.toml
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-13 15:01:11 +02:00