5
0
mirror of git://git.proxmox.com/git/proxmox-backup.git synced 2025-01-06 13:18:00 +03:00
Commit Graph

88 Commits

Author SHA1 Message Date
Christian Ebner
082c801ebb file restore: show more error context when extraction fails
Otherwise the context swallows the actual, underlying error message.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
d4a22d05df file restore: cover split metadata and payload archives
Attach the payload data archive as input stream to the decoder
and accessor instances for split archives.
Allows to restore contents from split archives via the
`proxmox-file-restore extract` command, by passing the metadata
archive name.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
00b0fbc4b6 file restore: factor out getting pxar reader
Factor out the logic to get the pxar reader into a dedicated function
so it can be reused to get the payload data archive reader instance.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
8fb247b030 file restore: cover extension for split pxar archives
Cover the additional `.mpxar` for metadata archive and `.ppxar` for
the payload data for pxar archives written as split archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
08fe50520a client: mount: make split pxar archives mountable
Cover the cases where the pxar archive was uploaded as split payload
data and metadata streams. Instantiate the required reader and
decoder instances to access the metadata and payload data archives.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
e2784a594e client: pxar: switch to stack based encoder state
... and adapt to the new reader/writer variant for encoder or
decoder/accessor to attach a dedicated payload input/output for split
pxar archives.

In preparation for look-ahead caching, where a passing around of
per-directory level encoder instances with internal references is
not feasible.

Previously, for each directory level a new encoder instance has been
generated, restricting possible implementation errors. These encoder
instances have been internally linked by references to keep track of
the state changes in a parent child relationship.

This is however not feasible when the encoder has to be passed by
mutable reference, as required by the look-ahead cache
implementation. The encoder has therefore been adapted to use a
single instance implementation with an internal stack keeping track
of the state.

Depends on the bumped pxar library version, including the patches to
attach the corresponding variant for the pxar reader/writer
instantiation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Hannes Laimer
09683f1290 accept a ref to a HttpClient
... since the functions don't actually need to own the value.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Tested-by: Gabriel Goller <g.goller@proxmox.com>
2023-11-25 17:07:42 +01:00
Maximiliano Sandoval
b4b230fdff file restore: qemu helper: run rustfmt
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
 [T: fix commit subject ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-16 10:00:26 +02:00
Thomas Lamprecht
950fba5a47 file-restore: factor out detecting debug mode
to avoid that the detection logic gets out of sync

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-15 18:08:41 +02:00
Stoiko Ivanov
39531dfc80 file-restore: set loglevel considering PBS_QEMU_DEBUG
during some tests recently I wondered why a debug log-message was not
printed, despite running with PBS_QEMU_DEBUG.

This patch sets the loglevel for the cli logger to debug if the
variable is present and not-empty (see qemu_helper.rs for the other
usage).

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2023-06-15 13:03:36 +02:00
Stoiko Ivanov
ce3f15aeac file-restore: set zfs_arc_min to current minimum of 32M
zfs_arc_min was raised to 32M (for linux) in zfs-commit
121b3cae742a0670d902a51bc61d49dc4a3e4445

while the current logic would still set the min_size to 32M (it's
max(32M, allmem/32), which results to 32M for memory sizes up to
1024M), setting it explicitly to the minimum makes it clear, and will
still be kept should the restore vm have more than 1G of memory at
some point.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2023-06-15 13:03:36 +02:00
Stoiko Ivanov
18844d6e3c file-restore: add zfs. prefix to arc_min/max settings
Currently the values set for zfs_arc_min and zfs_arc_max are ignored
by the kernel:
```
Unknown kernel command line parameters... will be passed to user space
```
module parameters provided on the commandline usually need to be
prefixed with the modulename (e.g. zfs.zfs_arc_min, see [0] for a bit
on related information (the issue itself is not related)).

Paradoxically currently ZFS will print spurious warnings about
settings being ignored when they are actually set - see [1].

Booting the debug image and connecting the shell on the serial console
confirmed that the values did not seem to be set:
`grep '^c_' /proc/spl/kstat/zfs/arcstats` showed half of the memory
for c_max.

[0] https://github.com/openzfs/zfs/issues/698
[1] https://github.com/openzfs/zfs/issues/12504

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2023-06-15 13:03:36 +02:00
Thomas Lamprecht
3165853468 file restore: clippy and format argument fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-05-17 17:43:17 +02:00
Wolfgang Bumiller
1104d2a268 move pbs_config::key_config to pbs-key-config
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-12-12 14:19:52 +01:00
Fabian Grünbichler
21d4a68255 sort dependencies
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:08:56 +01:00
Fabian Grünbichler
781294e4b5 switch regular dependencies to workspace ones
where applicable.

notable changes:
- serde now uses 'derive' feature across the board
- serde removed from pbs-tools (not used)
- openssl bumped to 0.40 (and patched comment removed)
- removed invalid zstd comment

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:07:12 +01:00
Fabian Grünbichler
1fbdf8ef79 switch proxmox dependencies to workspace
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:05:27 +01:00
Fabian Grünbichler
cb33ed9ca5 workspace: inherit metadata
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:05:27 +01:00
Wolfgang Bumiller
d9f1ca9a46 tree-wide: bump edition to 2021
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-12-06 13:31:01 +01:00
Wolfgang Bumiller
593aa85a9e bump pxar dependency to 0.10.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-23 15:47:07 +01:00
Thomas Lamprecht
2aaf3ef1e5 file-restore: move various formats to inline-string macros
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:08:09 +01:00
Thomas Lamprecht
7fa5be8b3f file-restore: split out internal QMP logic from unrelated fn
Capsule it in a small QMPSock struct impl, make the usage nicer as
the caller should not have to care & keep track of the initial socket
state+details.

A send_raw and send Value method should cover most needs.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:01:45 +01:00
Thomas Lamprecht
2f0f3e9979 file restore: allow to pass dimm size via env
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:01:45 +01:00
Thomas Lamprecht
0f2f523aa6 file restore: limit max dimm size to 512 MB
this is on top of the normal memory, and over 1.3 GB required is just
huge, sadly the commit adding this has zero details about what setups
fail and what work again with the change, so hard to tell, but any
setup that needs that much sounds like a bug in ZFS or remaining code
here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-14 15:59:50 +01:00
Thomas Lamprecht
69e3beb941 file restore: move allow-memory-hotplug param from CLI to environment
avoid the need to loop a parameter through a dozen function which all
don't care about it at all; iff this should be a global oncecell or
lock guarded param.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-14 15:49:49 +01:00
Thomas Lamprecht
fa1c3eaea1 file restore: rename dynamic-memory to auto-memory-hotplug
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Dominik Csapak
0ce86cb533 file-restore: make dynamic memory behaviour controllable
by adding 'dynamic-memory' parameter that controls if we automatically
increase the memory of the guest vm or not

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
a6c07eb9c2 limit ZFS Arc to more smalles possible value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
c190a32cb1 file restore: list api: fix description of "path"
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
5d0d3c61f9 file restore: switch some formats to inline template variables
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Dominik Csapak
72220d79d7 file-restore: dynamically increase memory of vm for zpools
when a backup contains a drive with zfs on it, the default memory
size (up to 384 MiB) is often not enough to hold the zfs metadata

to improve that situation, add memory dynamically (1GiB) when a path is
requested that is on zfs. Note that the image must be started with a
kernel capable of memory hotplug.

to achieve that, we also have to add a qmp socket to the vm, so that
we can later connect and add the memory backend and dimm

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-07 13:29:34 +01:00
Dominik Csapak
3e8ee886a5 file-restore: fix deprecated qemu parameters
server and nowait are deprecated, so we should use the longform:
server=on and wait=off

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-04 13:27:07 +01:00
Wolfgang Bumiller
5c7ff2afd2 bump sys dep to 0.4.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Fabian Grünbichler
f097eaa80f clippy fixes
and one additional API fn "allow many parameters" addition.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-10-11 09:36:12 +02:00
Dominik Csapak
ba690c4023 file-restore: remove 'json-error' parameter from list_files
we can reuse the 'output_format' here
also remove the 'error: true' here. we can determine it was an error,
by checking if it's an object with a 'message' property

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-05 19:34:00 +02:00
Dominik Csapak
cc900ae2f7 file-restore: add 'format' and 'zstd' parameters to 'extract' command
if the target ist stdout, we can now specify the exact format by making use of
the new 'format' parameter of the restore daemons 'extract' api

note that extracting a pxar from a source pxar (container/host backups)
won't work currently since we would have to reencode as pxar first

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ T: fixed missing proxmox-compression dependency ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-10-05 18:40:51 +02:00
Fabian Grünbichler
f3cfeb20a1 update proxmox-router to 1.3.0
no real change for PBS usage - the ApiHandler enum is marked
non_exhaustive now because it has extra values if the new (enabled by
default) "server" feature is enabled.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-09-07 14:17:12 +02:00
Fabian Grünbichler
e762aea4e0 build: more missing features
these would cause failures when building the sub-crates directly from
their sub-directory.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-09-05 12:55:33 +02:00
Wolfgang Bumiller
135aaee2d9 bump proxmox-sys dep to 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-28 13:47:37 +02:00
Wolfgang Bumiller
3f7440916f file-restore: clippy fixes
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-28 09:55:43 +02:00
Wolfgang Bumiller
b58dd1d413 rustfmt
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-30 10:18:32 +02:00
Wolfgang Bumiller
40e47aa9cf file-restore: fix a hyperlink in docs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-30 10:16:03 +02:00
Wolfgang Bumiller
04dffc3ad7 bump proxmox-sys dep to 0.3.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-29 09:45:11 +02:00
Hannes Laimer
095b3c1c18 proxmox-file-restore: replace print with log macro
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Hannes Laimer
955aea8ae8 bins: init cli logger
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Wolfgang Bumiller
a3593c890e bump proxmox-router dep to 1.2.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Thomas Lamprecht
74cad4a8bd tree wide: typo fixes through codespell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-07 14:08:09 +02:00
Thomas Lamprecht
fbfb64a6b2 tree wide: clippy lint fixes
most (not all) where done automatically

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-02 15:59:55 +02:00
Fabian Grünbichler
11ca834317 update to nix 0.24 / rustyline 9 / proxmox-sys 0.3
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-06-02 14:33:33 +02:00
Wolfgang Bumiller
8ca7cccf5f file-restore: add namespace support to qemu part
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-12 13:35:34 +02:00