IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Payload entries are separated by headers of type PAYLOAD within the
payload stream of split pxar archives, used for consistency checks
when accessing the file contents via a reader instance.
Commit 5b8204d0 moved these consistency checks, so they only happen
when actually accessing the content, thereby drastically improving
performance when navigating contents via the metadata archive.
The commit however also incorrectly increased the `Decoder`s
`payload_consumed` field by the size of the header, in case the file
payload has not been accessed by the `content_reader`.
As this filed is used to account for consumed bytes while sequentially
reading to possibly skip over entries, this leads to incorrectly
skipping of bytes in the stream (less than required). The main
manifestation being that a pxar extract with provided match pattern
failed.
Therefore, drop the incorrect accounting of the payload header.
Fixes: 5b8204d0 ("decoder: move payload header check for split input")
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
the old location has been deprecated for a while, and rustc 1.78 will start to warn about it.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Add checks for split variant inputs when accessing the payload
contents via the accessor instance. Both cases, accessing via the
safe `contents` method and via the previousely unsafe
`open_contents_at_range` call are covered.
Reduce possible misuse by wrapping the current plain content range
into an opaque `ContentRange` type with an additional optional
payload reference field to check consistency between the payload
reference encoded in the metadata archive and the payload header'
found in the payload data archive.
Because of the additional type wrapping and the payload header check,
the `open_contents_at_range` is considered safe now, dropping the
previously unsafe implementation.
The corresponding interfaces have been adapted accordingly.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The `Decoder`s `contents` method call can fail because of an added
consistency check when using split variant inputs.
Therefore, the additional error has to be handled by the callers.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The payload entries in the payload output for split pxar archives are
separated by payload headers, which allow to perform consistency
checks for the payload references encoded in the metadata archive.
Currently, this consistency check is performed right after reading the
entry in the metadata archive, which however has the downside that the
payload has to be fetched and decoded just for this consistency check.
This greatly impacts performance when accessing a metadata archive
with attached payload input reader, e.g. in the fuse implementation to
mount pxar archives, being especially severe when accessed over the
network in combination with a remote chunk reader as the Proxmox
Backup Server does.
Therefore, move this check to the contents reader instantiation
instead and add an additional flag to the decoder's `InPayload` state.
Getting the decoder now needs to be async and the method must return
an error when the check fails.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The helper method will be used to check the payload header being
consistent with what was encoded as paylaod reference for split
pxar archives.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
this was leftover and not does not make sense with the current version of the
Prelude, which is an opaque blob of bytes.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Introduces a new pxar format entry type `Prelude` and the associated
encoder and decoder methods.
A prelude starts with header marker `PXAR_PRELUDE` followed by raw
byte content, used to store additional metadata associated with the
pxar archive, e.g. command line arguments passed on archive creation.
The prelude's content has no fixed encoding format but is stored as
an raw, arbitrary byte slice. A prelude entry is encoded right after
a pxar format version entry, both being encoded in the metadata
archive in case of an archive with dedicated payload output.
The prelude is not backwards compatible to pxar format version 1.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Introduces a new pxar format entry type `Version` and the associated
encoder and decoder methods. The format version entry is only allowed
once, as the first entry of the pxar archive, marked with a
`PXAR_FORMAT_VERSION` header followed by the encoded version number.
If not present, the default format version 1 is assumed as encoding
format for the archive.
The entry allows to early detect incompatibility with an encoded
archive and bail or switch mode based on the encountered version.
The format version entry is not backwards compatible to pxar format
version 1.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Mark the beginning of the payload stream with a magic number. Allows for
version and file type detection.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Mark the end of the optional payload stream, this makes sure that at
least some bytes are written to the stream (as empty archives are not
allowed by the proxmox backup server) and possible injected chunks
must be consumed.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Allows to advance the payload writer position by a given size.
This is used to update the encoders payload input position when
injecting reused chunks for files with unchanged metadata.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Allows to read the current payload offset from the dedicated payload
input stream. This is required to get the current offset for calculation
of forced boundaries in the proxmox-backup-client, when injecting reused
payload chunks into the payload stream.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Allows to encode regular files with a payload reference within a
separate payload archive rather than encoding the payload within the
regular archive.
Following the PXAR_PAYLOAD_REF marked header, the payload offset and
size are encoded.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
When accessing the file contents via the sequential file restore
the range of the payload contents cannot be inferred a-priori but need
to be calculated based on the payload references encountered during
decoding.
Extending the `SeqRead` trait by the method `update_range` allows to
set the range in the payload reader instance by implementing the
method for `SeqReadAtAdapter`, thereby setting the correct content
range to be accessed.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
When a pxar archive was encoded using the split stream output
variant, access to the payload of regular files has to be redirected
to the corresponding dedicated input.
Allow to pass the split input variant to the decoder and accessor
instances to handle the split streams accordingly and decode split
stream archives.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
During regular pxar archive encoding, the payload of regular files is
written as part of the archive.
This patch introduces functionality to instead attach a writer variant
with a split payload writer instance to redirect the payload to a
different output.
The separation of data and metadata streams allows for efficient
reuse of payload data by referencing the payload writer byte offset,
without having to reencode it.
Whenever the payload of regular files is redirected to a dedicated
output writer, encode a payload reference header followed by the
required data to locate the data, instead of adding the regular payload
header followed by the encoded payload to the archive.
This is in preparation for reusing payload chunks for unchanged files
of backups created via the proxmox-backup-client.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Introduces the header type `PXAR_PAYLOAD_REF` to mark regular file
entry payloads, not encoded within the regular pxar archive but
rather redirected to a dedicated payload output writer.
It therefore substitutes the `PXAR_PAYLOAD` header type for these
entries.
The header marks the start and size for a `PayloadRef` typed object
in the archive, storing the offset to the payload header offset in the
payload stream of the dedicated payload output as well as the payload
size.
The `PayloadRef` provides the means to store, serialize and
deserialize the entry.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In preparation for the proxmox-backup-client look-ahead caching,
where a passing around of different encoder instances with internal
references is not feasible.
Instead of creating a new encoder instance for each directory level
and keeping references to the parent state, use an internal stack.
Adds additional helper functions to solve borrow issues, when both
the state and writers have to be accessed by a mutable reference.
This is a breaking change in the pxar library API.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Introduce an enum which stores 2 different possible variants of
inputs or outputs to be passed to encoder and decoder/accessor
instances, depending whether to read/write a fully self contained
pxar archive or whether to split off the payload stream into a
separate input/output.
Co-authored-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Make the skip part reusable for a different input.
In preparation for skipping payload paddings in a separated input.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
This exposes `decoder::aio::TokioReader<T>` in a similar manner to
`decoder::sync::StandardReader<T>`, which is necessary if one wants
to remain generic over `T: tokio::io::AsyncRead`, e.g.:
> struct FooDecoder<T: tokio::io::AsyncRead> {
> decoder: aio::Decoder<aio::TokioReader<T>>,
> }
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
