Commit Graph

61 Commits

Author SHA1 Message Date
Fabian Grünbichler
8aa244641d trivial clippy fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-06-24 09:59:27 +02:00
Fabian Grünbichler
fd3f72820e build: use cargo wrapper when building package
else we don't pick up the options set by the wrapper, which include generation
of debug symbols. until rustc 1.77, this was not needed because compiled
binaries always included a non-stripped libstd. now, without this change, the
binaries built with `cargo build --release` have no debug symbols at all
trigger a warning. fix this and include debug symbols when building a package,
like was originally intended for release package builds.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-06-20 08:50:41 +02:00
Christian Ebner
cf75bc0db5 client: pxar: set cache limit based on nofile rlimit
The lookahead cache size requires the resource limit for open file
handles to be high in order to allow for efficient reuse of unchanged
file payloads.

Increase the nofile soft limit to the hard limit and dynamically adapt
the cache size to the new soft limit minus the half of the previous
soft limit.

The `PxarCreateOptions` and the `Archiver` are therefore extended by
an additional field to store the maximum cache size, with fallback to
a default size of 512 entries.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:42 +02:00
Christian Ebner
5a5d454083 client: chunk stream: switch payload stream chunker
Use the dedicated chunker with boundary suggestions for the payload
stream, by attaching the channel sender to the archiver and the
channel receiver to the payload stream chunker.

The archiver sends the file boundaries for the chunker to consume.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:42 +02:00
Christian Ebner
a43399da06 pxar: add optional payload input to mount archive
Allow to pass an optional input path to mount a split pxar archive
with dedicated payload data file.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
108764f95b pxar: bin: support creation of split pxar archives via cli
Add support to create split pxar archives by redirecting the payload
output to a dedicated file.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
f16c5de757 pxar: bin: test pxar list with payload-input
Add a unit test to check for correct listing of pxar archives with
split payload input.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
1bec755b50 pxar: bin: ignore version and prelude entries in listing
Do not list the pxar format version and the prelude entries in the
output of pxar list, these are not regular entries. Do include them
however when dumping with the debug environmet variable set.
Since the prelude is arbitrary in size, only show the content size.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
58bfb6bb49 pxar: bin: show padding in debug output on archive list
In addition to the entries, also show the padding encountered in-between
referenced payloads.

Example invocation: `PXAR_LOG=debug pxar list archive.mpxar`

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
ee478ef1dc client: pxar: allow to restore prelude to optional path
Pxar archives allow to store additional information in a prelude
entry since pxar format version 2.

Add an optional parameter to `pxar` and `proxmox-backup-client` to
specify the path to restore the prelude to and pass this to the
archive extraction by extending the `PxarExtractOptions` by a
corresponding field. If none is given, the prelude is simply skipped
during restore.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
fdea4e5327 client: implement prepare reference method
Implement a method that prepares the decoder instance to access a
previous snapshots metadata index and payload index in order to
pass it to the pxar archiver. The archiver than can utilize these
to compare the metadata for files to the previous state and gather
reusable chunks.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
7de35dc243 client: streams: add channels for dynamic entry injection
To reuse dynamic entries of a previous backup run and index them for
the new snapshot. Adds a non-blocking channel between the pxar
archiver and the chunk stream, as well as the chunk stream and the
backup writer.

The archiver sends forced boundary positions and the dynamic
entries to inject into the chunk stream following this boundary.

The chunk stream consumes this channel inputs as receiver whenever a
new chunk is requested by the upload stream, forcing a non-regular
chunk boundary in the pxar stream at the requested positions.

The dynamic entries to inject and the boundary are then send via the
second asynchronous channel to the backup writer's upload stream,
indexing them by inserting the dynamic entries as known chunks into
the upload stream.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
d83839ddf3 pxar: bin: add more context to extraction error
Show more of the extraction error context provided by the pxar decoder.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
cf5d30c53f pxar: bin: cover listing for split archives
Allows to list entries of split pxar archives. As the decoder skips
over the file payloads, the corresponding payload file has to be
provided. Otherwise the decoder would skip inside the metadata
archive, leading to incorrect decoding.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
0b789a96dd pxar: bin: add optional payload input for archive restore
Allows to pass the optional payload input to restore for cases where the
regular file payloads are stored in the split archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
0fd3bcebe7 client: pxar: combine writers into struct
Introduce a `PxarWriters` struct to bundle all writer instances
required for the pxar archive creation into a single object to limit
the number of function call parameters.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
e2784a594e client: pxar: switch to stack based encoder state
... and adapt to the new reader/writer variant for encoder or
decoder/accessor to attach a dedicated payload input/output for split
pxar archives.

In preparation for look-ahead caching, where a passing around of
per-directory level encoder instances with internal references is
not feasible.

Previously, for each directory level a new encoder instance has been
generated, restricting possible implementation errors. These encoder
instances have been internally linked by references to keep track of
the state changes in a parent child relationship.

This is however not feasible when the encoder has to be passed by
mutable reference, as required by the look-ahead cache
implementation. The encoder has therefore been adapted to use a
single instance implementation with an internal stack keeping track
of the state.

Depends on the bumped pxar library version, including the patches to
attach the corresponding variant for the pxar reader/writer
instantiation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Gabriel Goller
a602a885af fix #4975: client: ignore E2BIG error flag
Some filesystems (f.e. zfs) support xattrs bigger than 64kB, sadly we
can't get them because the kernel vfs limits us. The syscalls listxattr
and getxattr will return a E2BIG error in this case.
Added a flag --ignore-e2big-xattr to the client, this will ignore the
metadata (but still backup the file) if this error occurs.

Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2024-02-15 10:34:10 +01:00
Thomas Lamprecht
6685122c3b tree-wide: fix various typos
found with codespell

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-29 18:32:07 +01:00
Christian Ebner
70bca12324 fix: #4761: introduce overwrite bitflags for fine grained overwrites
Adds OverwriteFlags for granular control of which entry types should
overwrite entries present on the filesystem during a restore.

The original overwrite flag is refactored in order to cover all of the
other cases.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2023-08-17 14:00:44 +02:00
Fabian Grünbichler
21d4a68255 sort dependencies
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:08:56 +01:00
Fabian Grünbichler
781294e4b5 switch regular dependencies to workspace ones
where applicable.

notable changes:
- serde now uses 'derive' feature across the board
- serde removed from pbs-tools (not used)
- openssl bumped to 0.40 (and patched comment removed)
- removed invalid zstd comment

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:07:12 +01:00
Fabian Grünbichler
1fbdf8ef79 switch proxmox dependencies to workspace
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:05:27 +01:00
Fabian Grünbichler
cb33ed9ca5 workspace: inherit metadata
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 09:05:27 +01:00
Wolfgang Bumiller
d9f1ca9a46 tree-wide: bump edition to 2021
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-12-06 13:31:01 +01:00
Wolfgang Bumiller
d701808514 move fuse code from pbs-client to pbs-pxar-fuse
it's used by pxar-bin and proxmox-backup-client for
mounting, but pbs-client is used by more (eg. the
proxmox-backup-qemu library which really doesn't need to
pull in any fuse dependencies)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-12-06 09:17:04 +01:00
Wolfgang Bumiller
593aa85a9e bump pxar dependency to 0.10.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-11-23 15:47:07 +01:00
Wolfgang Bumiller
5c7ff2afd2 bump sys dep to 0.4.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Fabian Grünbichler
f3cfeb20a1 update proxmox-router to 1.3.0
no real change for PBS usage - the ApiHandler enum is marked
non_exhaustive now because it has extra values if the new (enabled by
default) "server" feature is enabled.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-09-07 14:17:12 +02:00
Markus Frank
95e910f153 pbs-client: added overwrite parameter to PxarExtractOptions.
If overwrite is true, O_TRUNC is set (to clean the leftovers)
instead of O_EXCL and therefore overwrites the files and
does not error out.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-08-19 13:36:37 +02:00
Wolfgang Bumiller
135aaee2d9 bump proxmox-sys dep to 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-28 13:47:37 +02:00
Wolfgang Bumiller
04dffc3ad7 bump proxmox-sys dep to 0.3.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-29 09:45:11 +02:00
Hannes Laimer
0608b36b30 replace print with log macro
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Hannes Laimer
955aea8ae8 bins: init cli logger
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Wolfgang Bumiller
a3593c890e bump proxmox-router dep to 1.2.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Thomas Lamprecht
fbfb64a6b2 tree wide: clippy lint fixes
most (not all) where done automatically

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-02 15:59:55 +02:00
Fabian Grünbichler
11ca834317 update to nix 0.24 / rustyline 9 / proxmox-sys 0.3
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-06-02 14:33:33 +02:00
Wolfgang Bumiller
ebb85c1ca3 bump proxmox-schema dependency to 1.3.1 for streaming attribute
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:20:27 +02:00
Wolfgang Bumiller
fb6e48f402 bump proxmox-router dependency to 1.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:17:08 +02:00
Thomas Lamprecht
48fcee6a50 pxar bin: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:58:04 +02:00
Wolfgang Bumiller
bd4562e4b1 bump proxmox-schema dep to 1.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-04 09:50:21 +01:00
Wolfgang Bumiller
9c890d72b9 bump proxmox-async dep to 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:25:37 +01:00
Fabian Grünbichler
9a37bd6c84 tree-wide: fix needless borrows
found and fixed via clippy

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 13:55:33 +01:00
Wolfgang Bumiller
821aa8eae6 bump proxmox-schema to 1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-12-16 11:25:02 +01:00
Dietmar Maurer
25877d05ac update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 10:32:27 +01:00
Dietmar Maurer
92a8f0bc82 depend on proxmox-async 0.2 2021-11-20 17:14:02 +01:00
Dietmar Maurer
9a1b24b6b1 use new proxmox-async crate
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-19 18:03:22 +01:00
Dietmar Maurer
860eaec58f use proxmox::tools::fd::fd_change_cloexec from proxmox 0.15.3
Depend on proxmox 0.15.3

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-18 13:43:41 +01:00
Dietmar Maurer
e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Dietmar Maurer
b3f279e2d9 use complete_file_name from proxmox-router 1.1 2021-10-13 14:10:02 +02:00