Commit Graph

40 Commits

Author SHA1 Message Date
Maximiliano Sandoval
ba304a4f83 compression: make Deflate{De, En}coderBuilder public
These structs are returned by the public method
`Deflate{En,De}coder::builder`.

Reported-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-07-26 11:43:01 +02:00
Wolfgang Bumiller
2269153522 compression: bump version to 0.2.3-1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-10 12:15:54 +02:00
Maximiliano Sandoval
8b8957b5ba compression: deflate: add test module
We test the deflate encoder against the deflate decoder using (or not)
zlib and with different small buffer sizes. We also test compression and
decompression against the flate2 crate.

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
2024-07-10 11:37:36 +02:00
Maximiliano Sandoval
e6ca8d6049 compression: deflate: add a decoder
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
2024-07-10 11:37:36 +02:00
Maximiliano Sandoval
3f09610d86 compression: deflate: add builder pattern
This allows creating a encoder in a more general way and allows to
specify whether we want to set zlib headers. This is useful to compress
HTTP traffic, as per [1].

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding#directives

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
2024-07-10 11:37:36 +02:00
Maximiliano Sandoval
70a71e0c8e compression: deflate: move encoder into a mod
This allows to add a decompression mod inside the deflate mod. This does
not touch the public API.

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
2024-07-10 11:37:36 +02:00
Wolfgang Bumiller
0652d81977 tree-wide: enable doc_cfg and doc_auto_cfg for docs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-02 11:59:53 +02:00
Wolfgang Bumiller
eca6f31a22 compression: bump to 0.2.2-1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-06-20 13:59:46 +02:00
Wolfgang Bumiller
16d512b4d9 bump proxmox-compression to 0.2.1-1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-06-15 11:01:19 +02:00
Wolfgang Bumiller
5b55ea797a compression: match style fixup
if the match arms are this far away from the actual `match`
keyword, this needs to be split up...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-06-15 10:37:59 +02:00
Dominik Csapak
af46b655d4 compression: zip_directory: improve error handling
when zipping a directory, our intention was to skip over files that
cannot be zipped (e.g. the file can't be read/vanished/etc.), so we
ignored errors and simply logged it.

but when 'add_entry' fails, we will never actually restore, since every
error there is fatal to the point that the zip cannot be finished thats
because we take the 'target' sink out of self, and only insert it again
after all writes succeeded. so if an error occurs in between 'target' is
not put into self again (and never will be) and the zip cannot be
finished (even if we would catch all those intermediate errors and
restore 'target', we don't know in which state the output was, so we're
unable to finish a valid zip)

to fix that, split the actual 'add_entry' part there out of the async
move block and treat its errors always as fatal

without this, we generate heaps of log lines even after an error
occurred, and can never recover

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2023-06-15 10:28:50 +02:00
Wolfgang Bumiller
4f2a7d971b bump proxmox-compression to 0.2.0-1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-05-23 13:02:51 +02:00
Wolfgang Bumiller
818ac8e708 update zstd 0.6 -> 0.12 for bookworm
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-05-23 13:02:51 +02:00
Wolfgang Bumiller
8f8d52f148 update d/copyright files to debian copyright-format 1.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-05-23 13:02:39 +02:00
Fabian Grünbichler
ee8419cf2d workspace: switch remaining dependencies
while these are (currently) only used by a single member each, having *all*
dependency versions specified in the top level Cargo.toml only makes the whole
process of managing them less error-prone.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-12 11:05:30 +01:00
Fabian Grünbichler
46a675830d update d/control files
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-07 09:48:47 +01:00
Fabian Grünbichler
bdca6de588 update d/control files
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-07 09:48:47 +01:00
Fabian Grünbichler
e5abc0590e define workspace dependencies in workspace
so that we no longer have to (or forget to) bump the version in multiple places.

notable changes:
- outdated versions have been unified
- proxmox-metrics -> proxmox-async no longer uses explicit empty features
  (proxmox-async doesn't provide any anyway)
- proxmox-subscription -> proxmox-http no longer uses explicit default_features
  = false (proxmox-http has an empty default feature anyway)
- missing path dependencies added (mainly proxmox-rest-server)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-07 09:48:38 +01:00
Fabian Grünbichler
6c161bd5ab update d/control files
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-07 09:48:38 +01:00
Fabian Grünbichler
4189221470 inherit shared, external dependencies
noteworthy changes:
- proxmox-http had a default_features_false dep on hyper, which is dropped (the
  default feature is empty anyway)
- hyper, libc, nix, tokio and url versions are unified
- missing (cosmetic) bindgen feature on zstd enabled everywhere

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-07 09:48:38 +01:00
Fabian Grünbichler
64959d9ae0 move common metadata to workspace
and switch all crates to 2021 edition as well as a unified "authors" value.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-07 09:48:25 +01:00
Fabian Grünbichler
5ec765f842 update d/control files
debcargo 2.6 changed some minor details

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-06 11:21:43 +01:00
Fabian Grünbichler
538578c558 clippy 1.65 fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-05 11:17:37 +01:00
Wolfgang Bumiller
c7224c5f67 bump proxmox-compression to 0.1.2-1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-05 13:46:38 +02:00
Wolfgang Bumiller
ff86aa5f8a compression: more cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-05 13:23:14 +02:00
Wolfgang Bumiller
2d37cd92e0 compression: indentation cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-05 12:17:04 +02:00
Wolfgang Bumiller
cc6e5d7372 compression: minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-05 12:13:38 +02:00
Dominik Csapak
6e989a1c29 proxmox-compression: add 'tar_directory'
similar to 'zip_directory', this is intended to tar a local directory,
e.g. when we're in a restore vm.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-07-05 12:12:02 +02:00
Dominik Csapak
38db37dc5f proxmox-compression: make ZstdEncoder stream a bit more generic
by not requiring the 'anyhow::Error' type from the source, but only
that it implements 'Into<Error>'. This way, we can also accept a stream
that produces e.g. an io::Error

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-07-05 12:11:57 +02:00
Wolfgang Bumiller
30901c60f5 compression: clippy fixes
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-29 10:08:49 +02:00
Wolfgang Bumiller
c01b08fea9 bump proxmox-compression to 0.1.1-1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:35:31 +02:00
Wolfgang Bumiller
f8fe8f59a6 compression: limit ZstdEncoder constructors to usable ones
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:31:57 +02:00
Wolfgang Bumiller
79ac8d7344 compression: don't use tokio::main in doctest
because we have no rt feature enabled

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:31:26 +02:00
Wolfgang Bumiller
99add1733c compression: style changes
use where clauses where the parameter list is short enough
to become a single line

easier to read

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:31:10 +02:00
Wolfgang Bumiller
d4a09de520 compression: fmt
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:08:44 +02:00
Wolfgang Bumiller
fa5373c5c0 compression: clone_from_slice -> copy_from_slice
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:07:33 +02:00
Dominik Csapak
e7e4411f44 proxmox-compression: add streaming zstd encoder
similar to our DeflateEncoder, takes a Stream and implements it itself,
so that we can use it as an adapter for async api calls

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:02:54 +02:00
Dominik Csapak
d107d0b2eb proxmox-compression: add async tar builder
inspired by tar::Builder, but limited to the things we need and using
AsyncRead+AsyncWrite instead of the sync variants.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:02:51 +02:00
Wolfgang Bumiller
04b3bdb6cd proxmox-compression: update d/control
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:17:01 +01:00
Dominik Csapak
b8bf6a5c81 split out compression code into new crate 'proxmox-compression'
this removes quite a bit of dependecies of proxmox-async

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[set proxmox-lang dep to 1.1]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:10:53 +01:00