5
0
mirror of git://git.proxmox.com/git/proxmox-backup.git synced 2025-01-06 13:18:00 +03:00
Commit Graph

115 Commits

Author SHA1 Message Date
Christian Ebner
6771869cc1 client/server: use dedicated api type for all archive names
Instead of using the plain String or slices of it for archive names,
use the dedicated api type and its methods to parse and check for
archive type based on archive filename extension.

Thereby, keeping the checks and mappings in the api type and
resticting function parameters by the narrower wrapper type to reduce
potential misuse.

Further, instead of declaring and using the archive name constants
throughout the codebase, use the `BackupArchiveName` helpers to
generate the archive names for manifest, client logs and encryption
keys.

This allows for easy archive name comparisons using the same
`BackupArchiveName` type, at the cost of some extra allocations and
avoids the currently present double constant declaration of
`CATALOG_NAME`.

A positive ergonomic side effect of this is that commands now also
accept the archive type extension optionally, when passing the archive
name.

E.g.
```
proxmox-backup-client restore <snapshot> <name>.pxar.didx <target>
```
is equal to
```
proxmox-backup-client restore <snapshot> <name>.pxar <target>
```

The previously default mapping of any archive name extension to a blob
has been dropped in favor of consistent mapping by the api type
helpers.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

FG: use LazyLock for constant archive names
FG: add missing import

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-11-22 13:47:05 +01:00
Christian Ebner
0415304ca4 pxar: bin: rework and dynamically generate list test data
Commit f16c5de757 ("pxar: bin: test `pxar list` with payload-input")
introduced a regression test for listing of split pxar archives. This
test relies on a large pxar blob file, the large size (> 100M) being
overlooked when writing the test.

In order to not depend on this file any further in the future, drop
it and rewrite the test to dynamically generate the files, needed and
further extend the test thereby also cover the archive creation and
extraction for split pxar archives.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-09-11 16:04:46 +02:00
Dominik Csapak
b43845aa07 datastore: remove unused data blob writer
This is leftover code that is not currently used outside of its own
tests.

Should we need it again, we can just revert this commit.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-08-07 18:51:09 +02:00
Wolfgang Bumiller
96b7812b6a update to proxmox-log 0.2 and proxmox-rest-server 0.7
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-24 14:53:07 +02:00
Wolfgang Bumiller
c4f2bb70da bump sys and rest-server dependencies to 0.6
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-11 16:02:38 +02:00
Gabriel Goller
7e2486e800 switch from task_log! macro to tracing
Import `proxmox-log` and substitute all `task_log!`
(and task_warn!, task_error!) invocations with tracing calls (info!,
warn!, etc..). Remove worker references where it isn't necessary
anymore.

Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2024-07-11 15:56:04 +02:00
Christian Ebner
992487929a client: pxar: add archive creation with reference test
Add a basic regression test for archive creation with reference
metadata archive and index.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:42 +02:00
Christian Ebner
5a5d454083 client: chunk stream: switch payload stream chunker
Use the dedicated chunker with boundary suggestions for the payload
stream, by attaching the channel sender to the archiver and the
channel receiver to the payload stream chunker.

The archiver sends the file boundaries for the chunker to consume.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:42 +02:00
Christian Ebner
f16c5de757 pxar: bin: test pxar list with payload-input
Add a unit test to check for correct listing of pxar archives with
split payload input.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
7de35dc243 client: streams: add channels for dynamic entry injection
To reuse dynamic entries of a previous backup run and index them for
the new snapshot. Adds a non-blocking channel between the pxar
archiver and the chunk stream, as well as the chunk stream and the
backup writer.

The archiver sends forced boundary positions and the dynamic
entries to inject into the chunk stream following this boundary.

The chunk stream consumes this channel inputs as receiver whenever a
new chunk is requested by the upload stream, forcing a non-regular
chunk boundary in the pxar stream at the requested positions.

The dynamic entries to inject and the boundary are then send via the
second asynchronous channel to the backup writer's upload stream,
indexing them by inserting the dynamic entries as known chunks into
the upload stream.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Christian Ebner
0fd3bcebe7 client: pxar: combine writers into struct
Introduce a `PxarWriters` struct to bundle all writer instances
required for the pxar archive creation into a single object to limit
the number of function call parameters.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2024-06-05 16:39:41 +02:00
Wolfgang Bumiller
24146067f0 tests: add oneOf schema support
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-02-02 15:07:38 +01:00
Wolfgang Bumiller
3a45719ecf move api-types tests to api-types and drop vec![] macro
we don't need to allocate here

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-01-10 10:13:45 +01:00
Philipp Hufnagl
086cd7db33 tests: check if include/exclude behavior works correctly
This checks if including and excluding works as expected. That the
filter are added out of order is on purpose since it sould make no
difference.

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
2024-01-10 10:13:45 +01:00
Fabian Grünbichler
16f6766a68 clippy 1.65 fixes
and rustfmt

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-12-05 11:40:02 +01:00
Fabian Grünbichler
237beaed93 cargo fmt
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-10-11 09:37:16 +02:00
Fabian Grünbichler
f097eaa80f clippy fixes
and one additional API fn "allow many parameters" addition.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-10-11 09:36:12 +02:00
Wolfgang Bumiller
134779664e manager: hidden command to move datastore prune opts into jobs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-30 13:58:43 +02:00
Wolfgang Bumiller
93205cbe92 tests: switch to PruneJobOptions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-30 13:58:43 +02:00
Wolfgang Bumiller
db87d93efc make datastore BackupGroup/Dir ctors private
And use the api-types for their contents.

These are supposed to be instances for a datastore, the pure
specifications are the ones in pbs_api_types which should be
preferred in crates like clients which do not need to deal
with the datastore directly.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 11:56:23 +02:00
Thomas Lamprecht
12558e0dde tree wide: some stylistic clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-11 08:14:28 +02:00
Thomas Lamprecht
6082d75966 tests: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:49:26 +02:00
Fabian Grünbichler
aa174e8e8a tree-wide: drop redundant clones
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 15:02:07 +01:00
Dietmar Maurer
25877d05ac update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 10:32:27 +01:00
Dietmar Maurer
d5790a9f27 use new proxmox-sys crate
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-19 11:06:35 +01:00
Dominik Csapak
fe9c47ab4f tests/prune: add tests for protecteded backups
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:56 +02:00
Dominik Csapak
02db72678f add protected info of snapshots to api and task logs
adds the info that a snapshot is protected to:
* snapshot list
* manual pruning (also dry-run)
* prune jobs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:51 +02:00
Dominik Csapak
92c5cf42d1 pbs-datastore: add protection info to BackupInfo
and add necessary helper functions (protected_file/is_protected)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:44 +02:00
Wolfgang Bumiller
6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dietmar Maurer
49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Dietmar Maurer
2e44983a37 proxmox-rest-server: improve docs
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
2021-09-30 10:51:41 +02:00
Dietmar Maurer
fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately
And make ServerState private.
2021-09-30 08:41:30 +02:00
Dietmar Maurer
38da8ca1bc proxmox-rest-server: improve logging
And rename server_state_init() into catch_shutdown_and_reload_signals().
2021-09-29 14:48:46 +02:00
Dietmar Maurer
0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer
1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer
c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer
b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer
fd6d243843 move ApiConfig, FileLogger and CommandoSocket to proxmox-rest-server workspace
ApiConfig: avoid using  pbs_config::backup_user()
CommandoSocket: avoid using  pbs_config::backup_user()
FileLogger: avoid using  pbs_config::backup_user()
- use atomic_open_or_create_file()

Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
36c6e7bb82 fix tests/worker-task-abort.rs - correctly spawn command socket
And wait for the task.

Note: The test is still ignored (but works now when run a root)
2021-09-14 10:42:44 +02:00
Dietmar Maurer
67a5cf4714 fix regression tests 2021-09-10 12:45:06 +02:00
Dietmar Maurer
89725197c0 move PruneOptions to pbs_api_types workspace 2021-09-10 09:21:27 +02:00
Dietmar Maurer
8cc3760e74 move acl to pbs_config workspaces, pbs_api_types cleanups 2021-09-09 10:50:08 +02:00
Dietmar Maurer
bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Wolfgang Bumiller
42dad3abd3 fixup imports in tests and examples
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:32:21 +02:00
Thomas Lamprecht
02eae829f7 tests: move pxar test to its crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-20 18:54:23 +02:00
Wolfgang Bumiller
2b7f8dd5ea move client to pbs-client subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 12:58:43 +02:00
Dietmar Maurer
917230e4f8 verify-api: support nested AllOf schemas 2021-02-25 13:44:17 +01:00
Stefan Reiter
6afb60abf5 asyncify pxar create_archive
...to take advantage of the aio::Encoder from the pxar create.

Rather straightforward conversion, but does require getting rid of
references in the Archiver struct, and thus has to be given the Mutex
for the catalog directly. The callback is boxed.

archive_dir_contents can call itself recursively, and thus needs to
return a boxed future.

Users are adjusted, namely PxarBackupStream is converted to use an
Abortable future instead of a thread so it supports async in its handler
function, and the pxar bin create_archive is converted to an async API
function. One test case is made to just use 'block_on'.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-17 09:24:20 +01:00
Dietmar Maurer
cc2a0b12f8 test: define tape tests as submodule 2021-02-02 14:38:15 +01:00
Dietmar Maurer
f4d231e70a test: add regression tests for tape inventory 2021-02-02 12:19:28 +01:00