5
0
mirror of git://git.proxmox.com/git/proxmox-backup.git synced 2025-02-26 21:57:33 +03:00

156 Commits

Author SHA1 Message Date
Wolfgang Bumiller
5c7ff2afd2 bump sys dep to 0.4.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-10-19 14:26:00 +02:00
Stefan Sterz
a756b0c1a1 fix: datastore: make relative_group_path() return relative path
previously the BackGroup trait used the datastore's
namespace_path() method to construct a base path. this would result in
it returning an absolute path equivalent to full_group_path(). use
the namspace's path() method instead to get a relative path, in-line
with backup_dir's relative_path().

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2022-09-12 10:02:46 +02:00
Wolfgang Bumiller
135aaee2d9 bump proxmox-sys dep to 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-28 13:47:37 +02:00
Wolfgang Bumiller
1be05037c2 datastore: clippy fixes
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-27 15:26:50 +02:00
Fabian Grünbichler
e1db06705e more clippy fixes and annotations
the remaining ones are:
- type complexity
- fns with many arguments
- new() without default()
- false positives for redundant closures (where closure returns a static
  value)
- expected vs actual length check without match/cmp

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-07-26 14:05:25 +02:00
Wolfgang Bumiller
487c5be9ab datastore: add BackupGroup::exists helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-14 11:17:15 +02:00
Wolfgang Bumiller
5444b914af datastore: factor type out of ListGroups into ListGroupsType
In the API we want to iterate over all backup groups
belonging to a particular type at least once, and iterating
through *everything* and simply "skipping" over every single
entry from another type makes no sense given that the groups
are organized into subdirectories based on their type.

Let's have an `.iter_backup_type()` method which returns an
iterator over all the groups of a specific type named
ListGroupsType and factorize the type level iterator out of
ListGroups for reuse.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-14 11:17:15 +02:00
Wolfgang Bumiller
8f38e53896 datastore: simplify iter_backup_groups_ok
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-07-14 11:14:49 +02:00
Fabian Grünbichler
87b7fa0226 move to/write_canonical_json to proxmox-serde
as preparation for extracting subscription-related code into its own
crate.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-07-04 10:08:53 +02:00
Dominik Csapak
6de1899bb1 partially fix #2915: 'stat' in case ReadDirEntry does not contain type
readdir/getdents may return 'DT_UNKNOWN' for the file type
(which corresponds to 'None' in nix::dir::Entry), so stat the file and
check the type

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-29 09:45:11 +02:00
Wolfgang Bumiller
04dffc3ad7 bump proxmox-sys dep to 0.3.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-29 09:45:11 +02:00
Hannes Laimer
707e2b39f3 fixed_index: remove unused 'print_info' function
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Hannes Laimer
dce4b5403b pbs-datastore: replace print with log macro
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-06-21 10:43:19 +02:00
Thomas Lamprecht
74cad4a8bd tree wide: typo fixes through codespell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-07 14:08:09 +02:00
Thomas Lamprecht
51d900d187 datastore: swap ConfigVersionCache with digest for change detection
We got the digest available anyway, and it's only 16 bytes more to
save (compared to last_generation and the recently removed last_time,
both being 64 bit = 8 bytes each)

Side benefit, we detect config changes made manually (e.g., `vim
datacenter.cfg`) immediately.

Note that we could restructure the maintenance mode checking to only
be done after checking if there's a cached datastore, in which case
using the generation could make sense to decide if we need to re-load
it again before blindly loading the config anyway. As that's not only
some (not exactly hard but not really trivial like a typo fix either)
restructuring work but also means we'd lose the "detect manual
changes" again I'd rather keep using the digest.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-04 15:26:50 +02:00
Thomas Lamprecht
519ca9d010 datastore: make unsafe fn public again, useful for example/test
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-03 17:10:17 +02:00
Thomas Lamprecht
615a50c108 datastore: make unsafe functions only visible in their own crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-03 13:42:42 +02:00
Thomas Lamprecht
c66fa32c08 datastore: add safety doc comment for unsafe opens
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-03 10:58:33 +02:00
Thomas Lamprecht
2515ff35c2 datastore: reduce chunk store open visibility and comment pitfalls
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-03 10:15:41 +02:00
Thomas Lamprecht
33a1ef7aae datastore: rename non-telling map to datastore_cache
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-03 10:11:09 +02:00
Thomas Lamprecht
9c12e82006 datastore: drop bogus last_update stale-cache mechanism
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-03 10:04:16 +02:00
Thomas Lamprecht
c7f7236b88 datastore: more concise comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-02 17:48:08 +02:00
Dominik Csapak
0bd9c87010 datastore: lookup: reuse ChunkStore on stale datastore re-open
When re-opening a datastore due to the cached entry being stale
(config change) but also if the last re-open was >60s ago). On
datastore open the chunk store was also re-opened, which in turn
creates a new ProcessLocker, loosing any existing shared lock which
can cause conflicts between long running (24h+) backups  and GC.

To fix this, reuse the existing ChunkStore, and thus  its
ProcessLocker, when creating a up-to-date datastore instance on
lookup, since only the datastore config should be reloaded. This is
fine as the ChunkStore path is not updatable over our API.

This was always a potential issue but got exposed in practice by
commit 118deb4db8e709b02704bc66c0551bfa7e4369ed which introduced the
unconditional "re-open after 60s" mechanism.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ T: reword commit message a bit and reference commit that made the
   issue much more likely ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-02 17:00:49 +02:00
Fabian Grünbichler
11ca834317 update to nix 0.24 / rustyline 9 / proxmox-sys 0.3
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-06-02 14:33:33 +02:00
Wolfgang Bumiller
dba37e212b add prune jobs api
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-30 13:58:43 +02:00
Wolfgang Bumiller
5557af0efb api-types: add PruneJobConfig
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-30 13:58:43 +02:00
Thomas Lamprecht
b8858d5186 datastore: avoid unsafe transmute, use to_ne_bytes
which is stable since rustc 1.32 but wasn't available in out
toolchain when this was originally written in commit 7bc1d7277

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-26 11:42:13 +02:00
Thomas Lamprecht
bc001e12e2 datastore: clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-26 11:42:13 +02:00
Fabian Grünbichler
abd8248520 tree-wide: remove DatastoreWithNamespace
instead move the acl_path helper to BackupNamespace, and introduce a new
helper for printing a store+ns when logging/generating error messages.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-26 11:42:10 +02:00
Fabian Grünbichler
d22363ad08 BackupDir/BackupGroup: add ns to Debug impl
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
7784698948 BackupGroup: stop implementing Display
this shouldn't be printed/logged - use DatastoreWithNamespace /
pbs_api_types::BackupGroup instead.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
e13303fca6 tree-wide: prefer api-type BackupGroup for logging
together with DatastoreWithNamespace where needed, to not forget
namespace information.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
eefa297aa0 BackupDir: stop implementing Display
the api type implements it already, all call sites should rather use
DatastoreWithName and pbs_api_types::BackupDir for logging/..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
f2fe00f1e2 BackupDir: fix manifest_lock_path
this definitely shouldn't rely on BackupDir's Display implementation..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
1afce610c7 tree-wide: prefer api-type BackupDir for logging
in combination with DatastoreWithNamespace to not lose the namespace
information.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
f15601f1c9 BackupDir: add group/dir accessors
for getting the respective api type references for convenient
printing/logging.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Thomas Lamprecht
a7646fe42a Revert "fix #4001: datastore/catalog: add number of files to directory entry"
causes trouble with UI and is inconsistent as its still missing in
file restore (daemon)

We probably want to use a separate property to safe this to avoid
confusion with size.

This reverts commit 66ad63bac23359b8ede822496f68ebde843cf988.
2022-05-16 17:51:35 +02:00
Fabian Grünbichler
9c75e2f3e1 build: bump required log version
else logging using "{var}" in format strings doesn't work properly.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 15:02:07 +02:00
Thomas Lamprecht
456456483e datastore: ns iter: clamp depth to MAX_NAMESPACE_DEPTH from datastore root
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-16 09:37:38 +02:00
Thomas Lamprecht
3aafa61362 namespace deletion: propagate delete-groups=false but ENOTEMPTY as error
after all we couldn't delete all that got requested, ideally this
should become a task where we can log what got deleted and what
not...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 16:32:49 +02:00
Thomas Lamprecht
ca3f8757ba datastore: clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 16:01:09 +02:00
Thomas Lamprecht
118e984996 datastore: move backup dir/group/namespace iter to own module
no changes in interface for users of the crate as we re-export
anyway, so more for avoiding to crowd the datastore module to much

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 15:59:43 +02:00
Thomas Lamprecht
d1f9cceada namespace deletion: make destroying groups separate choice
And make that opt-in in the API endpoint, to avoid bad surprises by
default.

If not set we'll only prune empty namespaces.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 14:09:25 +02:00
Stefan Sterz
66ad63bac2 fix #4001: datastore/catalog: add number of files to directory entry
When listing the content of a catalog, add the number of files
contained in the directory as its size. Also removes redundant code,
the `mtime` and the `size` of a file is already set when creating the
archive entry, but we naturally need to override the size now for
directories.

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 14:08:51 +02:00
Dominik Csapak
fc99c2791b api: tape/restore: check and create target namespace
checks the privilegs for the target namespace. If that does not exist,
try to recursively create them while checking the privileges.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 14:08:32 +02:00
Thomas Lamprecht
2a088b9975 datastore: drop bogus chunk size check, can cause trouble
other sizes can happen in legitimate and illegitimate ways:
 - illegitimate: encryped chunks and bad actor client
 - legitimate: same chunk but newer zstd version (or compression
   level) can compress it better (or worse) so the

Ideally we could take the actual smaller chunk so that improved zstd
tech gets leveraged, but we could only allow to do that for
un-encrypted ones.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 15:41:20 +02:00
Thomas Lamprecht
0408f60b58 datastore: add new Lookup for operations tracking
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.

Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:36:56 +02:00
Thomas Lamprecht
9ccf933be5 datastore: move update_manifest into BackupDir impl
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht
5566099849 datastore: move cleanup_unreferenced_files to BackupDir impl and fix NS awareness
sync failed on cleanup due to always trying to do so in the root NS

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Wolfgang Bumiller
53d073ec1a datastore: minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00