Commit Graph

82 Commits

Author SHA1 Message Date
Hannes Laimer
09683f1290 accept a ref to a HttpClient
... since the functions don't actually need to own the value.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Tested-by: Gabriel Goller <g.goller@proxmox.com>
2023-11-25 17:07:42 +01:00
Maximiliano Sandoval
b4b230fdff file restore: qemu helper: run rustfmt
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
 [T: fix commit subject ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-16 10:00:26 +02:00
Thomas Lamprecht
950fba5a47 file-restore: factor out detecting debug mode
to avoid that the detection logic gets out of sync

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-06-15 18:08:41 +02:00
Stoiko Ivanov
39531dfc80 file-restore: set loglevel considering PBS_QEMU_DEBUG
during some tests recently I wondered why a debug log-message was not
printed, despite running with PBS_QEMU_DEBUG.

This patch sets the loglevel for the cli logger to debug if the
variable is present and not-empty (see qemu_helper.rs for the other
usage).

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2023-06-15 13:03:36 +02:00
Stoiko Ivanov
ce3f15aeac file-restore: set zfs_arc_min to current minimum of 32M
zfs_arc_min was raised to 32M (for linux) in zfs-commit
121b3cae742a0670d902a51bc61d49dc4a3e4445

while the current logic would still set the min_size to 32M (it's
max(32M, allmem/32), which results to 32M for memory sizes up to
1024M), setting it explicitly to the minimum makes it clear, and will
still be kept should the restore vm have more than 1G of memory at
some point.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2023-06-15 13:03:36 +02:00
Stoiko Ivanov
18844d6e3c file-restore: add zfs. prefix to arc_min/max settings
Currently the values set for zfs_arc_min and zfs_arc_max are ignored
by the kernel:
```
Unknown kernel command line parameters... will be passed to user space
```
module parameters provided on the commandline usually need to be
prefixed with the modulename (e.g. zfs.zfs_arc_min, see [0] for a bit
on related information (the issue itself is not related)).

Paradoxically currently ZFS will print spurious warnings about
settings being ignored when they are actually set - see [1].

Booting the debug image and connecting the shell on the serial console
confirmed that the values did not seem to be set:
`grep '^c_' /proc/spl/kstat/zfs/arcstats` showed half of the memory
for c_max.

[0] https://github.com/openzfs/zfs/issues/698
[1] https://github.com/openzfs/zfs/issues/12504

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2023-06-15 13:03:36 +02:00
Thomas Lamprecht
3165853468 file restore: clippy and format argument fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-05-17 17:43:17 +02:00
Wolfgang Bumiller
1104d2a268 move pbs_config::key_config to pbs-key-config
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-12-12 14:19:52 +01:00
Fabian Grünbichler
21d4a68255 sort dependencies
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:08:56 +01:00
Fabian Grünbichler
781294e4b5 switch regular dependencies to workspace ones
where applicable.

notable changes:
- serde now uses 'derive' feature across the board
- serde removed from pbs-tools (not used)
- openssl bumped to 0.40 (and patched comment removed)
- removed invalid zstd comment

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:07:12 +01:00
Fabian Grünbichler
1fbdf8ef79 switch proxmox dependencies to workspace
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:05:27 +01:00
Fabian Grünbichler
cb33ed9ca5 workspace: inherit metadata
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:05:27 +01:00
Wolfgang Bumiller
d9f1ca9a46 tree-wide: bump edition to 2021
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-12-06 13:31:01 +01:00
Wolfgang Bumiller
593aa85a9e bump pxar dependency to 0.10.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-23 15:47:07 +01:00
Thomas Lamprecht
2aaf3ef1e5 file-restore: move various formats to inline-string macros
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:08:09 +01:00
Thomas Lamprecht
7fa5be8b3f file-restore: split out internal QMP logic from unrelated fn
Capsule it in a small QMPSock struct impl, make the usage nicer as
the caller should not have to care & keep track of the initial socket
state+details.

A send_raw and send Value method should cover most needs.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:01:45 +01:00
Thomas Lamprecht
2f0f3e9979 file restore: allow to pass dimm size via env
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-15 09:01:45 +01:00
Thomas Lamprecht
0f2f523aa6 file restore: limit max dimm size to 512 MB
this is on top of the normal memory, and over 1.3 GB required is just
huge, sadly the commit adding this has zero details about what setups
fail and what work again with the change, so hard to tell, but any
setup that needs that much sounds like a bug in ZFS or remaining code
here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-14 15:59:50 +01:00
Thomas Lamprecht
69e3beb941 file restore: move allow-memory-hotplug param from CLI to environment
avoid the need to loop a parameter through a dozen function which all
don't care about it at all; iff this should be a global oncecell or
lock guarded param.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-14 15:49:49 +01:00
Thomas Lamprecht
fa1c3eaea1 file restore: rename dynamic-memory to auto-memory-hotplug
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Dominik Csapak
0ce86cb533 file-restore: make dynamic memory behaviour controllable
by adding 'dynamic-memory' parameter that controls if we automatically
increase the memory of the guest vm or not

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
a6c07eb9c2 limit ZFS Arc to more smalles possible value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
c190a32cb1 file restore: list api: fix description of "path"
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Thomas Lamprecht
5d0d3c61f9 file restore: switch some formats to inline template variables
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-11-11 10:21:49 +01:00
Dominik Csapak
72220d79d7 file-restore: dynamically increase memory of vm for zpools
when a backup contains a drive with zfs on it, the default memory
size (up to 384 MiB) is often not enough to hold the zfs metadata

to improve that situation, add memory dynamically (1GiB) when a path is
requested that is on zfs. Note that the image must be started with a
kernel capable of memory hotplug.

to achieve that, we also have to add a qmp socket to the vm, so that
we can later connect and add the memory backend and dimm

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-07 13:29:34 +01:00
Dominik Csapak
3e8ee886a5 file-restore: fix deprecated qemu parameters
server and nowait are deprecated, so we should use the longform:
server=on and wait=off

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-11-04 13:27:07 +01:00
Wolfgang Bumiller
5c7ff2afd2 bump sys dep to 0.4.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Fabian Grünbichler
f097eaa80f clippy fixes
and one additional API fn "allow many parameters" addition.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-10-11 09:36:12 +02:00
Dominik Csapak
ba690c4023 file-restore: remove 'json-error' parameter from list_files
we can reuse the 'output_format' here
also remove the 'error: true' here. we can determine it was an error,
by checking if it's an object with a 'message' property

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-10-05 19:34:00 +02:00
Dominik Csapak
cc900ae2f7 file-restore: add 'format' and 'zstd' parameters to 'extract' command
if the target ist stdout, we can now specify the exact format by making use of
the new 'format' parameter of the restore daemons 'extract' api

note that extracting a pxar from a source pxar (container/host backups)
won't work currently since we would have to reencode as pxar first

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ T: fixed missing proxmox-compression dependency ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-10-05 18:40:51 +02:00
Fabian Grünbichler
f3cfeb20a1 update proxmox-router to 1.3.0
no real change for PBS usage - the ApiHandler enum is marked
non_exhaustive now because it has extra values if the new (enabled by
default) "server" feature is enabled.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-09-07 14:17:12 +02:00
Fabian Grünbichler
e762aea4e0 build: more missing features
these would cause failures when building the sub-crates directly from
their sub-directory.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-09-05 12:55:33 +02:00
Wolfgang Bumiller
135aaee2d9 bump proxmox-sys dep to 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-28 13:47:37 +02:00
Wolfgang Bumiller
3f7440916f file-restore: clippy fixes
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-28 09:55:43 +02:00
Wolfgang Bumiller
b58dd1d413 rustfmt
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-30 10:18:32 +02:00
Wolfgang Bumiller
40e47aa9cf file-restore: fix a hyperlink in docs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-30 10:16:03 +02:00
Wolfgang Bumiller
04dffc3ad7 bump proxmox-sys dep to 0.3.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-29 09:45:11 +02:00
Hannes Laimer
095b3c1c18 proxmox-file-restore: replace print with log macro
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Hannes Laimer
955aea8ae8 bins: init cli logger
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Wolfgang Bumiller
a3593c890e bump proxmox-router dep to 1.2.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Thomas Lamprecht
74cad4a8bd tree wide: typo fixes through codespell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-07 14:08:09 +02:00
Thomas Lamprecht
fbfb64a6b2 tree wide: clippy lint fixes
most (not all) where done automatically

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-02 15:59:55 +02:00
Fabian Grünbichler
11ca834317 update to nix 0.24 / rustyline 9 / proxmox-sys 0.3
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-06-02 14:33:33 +02:00
Wolfgang Bumiller
8ca7cccf5f file-restore: add namespace support to qemu part
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-12 13:35:34 +02:00
Wolfgang Bumiller
133d718fe4 split the namespace out of BackupGroup/Dir api types
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.

For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).

The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.

Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.

Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller
8c74349b08 api-types: add namespace to BackupGroup
Make it easier by adding an helper accepting either group or
directory

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Dominik Csapak
6ddd69c5ce file-restore: add 'timeout' and 'json-error' parameter
timeout limits the code with the given timeout in seconds, and
'json-error' return json to stdout when the call returns an error like
this:

{
    "msg": "error message",
    "error": true,
    "code": <HTTP_STATUS_CODE>, // if it was an http error
}

with both options set, a client can more easily determine if the call
ran into a timeout (since it will return a 503 error), and can poll
it again

both is done behind new parameters, so that we can stay backwards-compatible

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-27 19:19:57 +02:00
Dominik Csapak
25be1fa0d7 file-restore: factor out 'list_files'
we'll want to reuse that in a later patch

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-27 19:19:57 +02:00
Wolfgang Bumiller
db87d93efc make datastore BackupGroup/Dir ctors private
And use the api-types for their contents.

These are supposed to be instances for a datastore, the pure
specifications are the ones in pbs_api_types which should be
preferred in crates like clients which do not need to deal
with the datastore directly.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 11:56:23 +02:00
Wolfgang Bumiller
ebb85c1ca3 bump proxmox-schema dependency to 1.3.1 for streaming attribute
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:20:27 +02:00