IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
instead of hardcoding the default deep inside the code. This makes it
much easier to see what is the actual default
the first instance of ChunkOrder::None was only for the test case, were
the ordering doe not matter
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
fixups for DatastoreFSyncLevel:
* use derive for Default
* add some more derives (Clone, Copy)
chunk store:
* drop to_owned for chunk_dir_path
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
currently, we don't (f)sync on chunk insertion (or at any point after
that), which can lead to broken chunks in case of e.g. an unexpected
powerloss. To fix that, offer a tuning option for datastores that
controls the level of syncs it does:
* None (default): same as current state, no (f)syncs done at any point
* Filesystem: at the end of a backup, the datastore issues
a syncfs(2) to the filesystem of the datastore
* File: issues an fsync on each chunk as they get inserted
(using our 'replace_file' helper) and a fsync on the directory handle
a small benchmark showed the following (times in mm:ss):
setup: virtual pbs, 4 cores, 8GiB memory, ext4 on spinner
size none filesystem file
2GiB (fits in ram) 00:13 0:41 01:00
33GiB 05:21 05:31 13:45
so if the backup fits in memory, there is a large difference between all
of the modes (expected), but as soon as it exceeds the memory size,
the difference between not syncing and syncing the fs at the end becomes
much smaller.
i also tested on an nvme, but there the syncs basically made no difference
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
previously the BackGroup trait used the datastore's
namespace_path() method to construct a base path. this would result in
it returning an absolute path equivalent to full_group_path(). use
the namspace's path() method instead to get a relative path, in-line
with backup_dir's relative_path().
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
the remaining ones are:
- type complexity
- fns with many arguments
- new() without default()
- false positives for redundant closures (where closure returns a static
value)
- expected vs actual length check without match/cmp
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
In the API we want to iterate over all backup groups
belonging to a particular type at least once, and iterating
through *everything* and simply "skipping" over every single
entry from another type makes no sense given that the groups
are organized into subdirectories based on their type.
Let's have an `.iter_backup_type()` method which returns an
iterator over all the groups of a specific type named
ListGroupsType and factorize the type level iterator out of
ListGroups for reuse.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
readdir/getdents may return 'DT_UNKNOWN' for the file type
(which corresponds to 'None' in nix::dir::Entry), so stat the file and
check the type
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We got the digest available anyway, and it's only 16 bytes more to
save (compared to last_generation and the recently removed last_time,
both being 64 bit = 8 bytes each)
Side benefit, we detect config changes made manually (e.g., `vim
datacenter.cfg`) immediately.
Note that we could restructure the maintenance mode checking to only
be done after checking if there's a cached datastore, in which case
using the generation could make sense to decide if we need to re-load
it again before blindly loading the config anyway. As that's not only
some (not exactly hard but not really trivial like a typo fix either)
restructuring work but also means we'd lose the "detect manual
changes" again I'd rather keep using the digest.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When re-opening a datastore due to the cached entry being stale
(config change) but also if the last re-open was >60s ago). On
datastore open the chunk store was also re-opened, which in turn
creates a new ProcessLocker, loosing any existing shared lock which
can cause conflicts between long running (24h+) backups and GC.
To fix this, reuse the existing ChunkStore, and thus its
ProcessLocker, when creating a up-to-date datastore instance on
lookup, since only the datastore config should be reloaded. This is
fine as the ChunkStore path is not updatable over our API.
This was always a potential issue but got exposed in practice by
commit 118deb4db8 which introduced the
unconditional "re-open after 60s" mechanism.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: reword commit message a bit and reference commit that made the
issue much more likely ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
which is stable since rustc 1.32 but wasn't available in out
toolchain when this was originally written in commit 7bc1d7277
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead move the acl_path helper to BackupNamespace, and introduce a new
helper for printing a store+ns when logging/generating error messages.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this shouldn't be printed/logged - use DatastoreWithNamespace /
pbs_api_types::BackupGroup instead.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the api type implements it already, all call sites should rather use
DatastoreWithName and pbs_api_types::BackupDir for logging/..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
causes trouble with UI and is inconsistent as its still missing in
file restore (daemon)
We probably want to use a separate property to safe this to avoid
confusion with size.
This reverts commit 66ad63bac2.
after all we couldn't delete all that got requested, ideally this
should become a task where we can log what got deleted and what
not...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
no changes in interface for users of the crate as we re-export
anyway, so more for avoiding to crowd the datastore module to much
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
And make that opt-in in the API endpoint, to avoid bad surprises by
default.
If not set we'll only prune empty namespaces.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When listing the content of a catalog, add the number of files
contained in the directory as its size. Also removes redundant code,
the `mtime` and the `size` of a file is already set when creating the
archive entry, but we naturally need to override the size now for
directories.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
checks the privilegs for the target namespace. If that does not exist,
try to recursively create them while checking the privileges.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
other sizes can happen in legitimate and illegitimate ways:
- illegitimate: encryped chunks and bad actor client
- legitimate: same chunk but newer zstd version (or compression
level) can compress it better (or worse) so the
Ideally we could take the actual smaller chunk so that improved zstd
tech gets leveraged, but we could only allow to do that for
un-encrypted ones.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.
Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.
For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).
The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.
Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.
Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
on depth == 0 we only yield the anchor ns, this simplifies usage in
the API as for, e.g. list-snapthos the depth == 0 means only the
snapshots from the passed namespace, nothing below.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The behavior on "any snapshot was protected" isn't yet ideal, as we
then do not cleanup any (sub) namespace, even if some of them where
cleaned from groups & snapshots completely. But that isn't easy to do
with our current depth-first pre-order iterator behavior, and it's
also not completely wrong either, the user can re-do the removal on
the sub-namespaces, so leave that for later.
Should get moved to a datastore::BackupNamespace type once/if we get
one
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Make it easier by adding an helper accepting either group or
directory
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
and issue a warning. We can do this, because we know an empty chunk
cannot be valid, and we (assumedly) have a valid chunk in memory.
Having empty chunks on disk is currently possible when PBS crashes,
but the rename of the chunk was flushed to disk, when the actual data
was not.
If it's not empty but there is a size mismatch, return an error.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
data blobs can only appear in a BackupDir (snapshot) in the backup
hierachy, so makes more sense that it lives in there.
As it wasn't widely used anyway it's easy to move the single
non-package call site over to the new one directly and drop the
implementation from Datastore completely.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the manifest is owned by the backup dir (snapshot) so it should also
handle locking, makes no sense to have the implementation somewhere
higher up.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
move the check for directory before doing the OSString -> String
conversion, which should be a bit more efficient.
Also let the match return the entry in the non-skip/return case to
reduce indentation level for the inner "yield element" part, making
it slightly easier to follow.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
And drop the base_path parameter on a first bunch of
functions (more reordering will follow).
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Same as previous commits: this will be linked to a
particular DataStore and Eq/Ord is now only part of the
api types, for now.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Same as previous commit: this is supposed to be connected to
a datastore and Eq/PartialEq only make sense for the
api-type part.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
And use the api-types for their contents.
These are supposed to be instances for a datastore, the pure
specifications are the ones in pbs_api_types which should be
preferred in crates like clients which do not need to deal
with the datastore directly.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
datastore's group_path will be moved to BackupDir soon and
this is required to be able to properly distinguish them
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The type is a real enum.
All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.
Also, they are now flattened into a few structs instead of
being copied manually.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
will be more used in the future, when the upend-datastore master plan
comes in effect.
also a preparatory work for namespaces
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While currently it's still only used in a collected() way, most call
sites can be switched over to use the iterator directly, as often
they already convert the not-so-cheap, in-memory vector back in
.into_iter() anyway.
somewhat also preparatory (yak shaving) work for namespaces
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
