IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
the error message for failure to sync the whole namespace was too long, so
split it into two lines and make it a warning.
the namespace creation one lacked context (that the error was caused by the
remote side or the connection) and had too much (the datastore, which is
already logged very often) at the same time.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
add a bit more detail for the pull side, and reword some comments on the push
side to make them easier to read.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
`try_exists` will return Ok(false) if the path is or containts a dangling
symlink, treat that as hard error just like if `try_exists` has returned an
Err(..).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
one million chunks are a bit much, considering that chunks are representing
1-2MB (dynamic) to 4MB (fixed) of input data, that would mean 1-4TB of re-used
input data in a single snapshot.
64k chunks are still representing 64-256GB of input data, which should be
plenty (and for such big snapshots with lots of re-used chunks, growing the
allocation of the HashSet should not be the bottleneck), and is also the
default capacity used for pulling.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
instead of calling this three times, call it once:
retrieving the highest backup timestamp doesn't need its own request, it can
re-use the "main" result, the corresponding helper can thus be dropped.
remove_vanished can re-use the earlier result - if anybody prunes the backup
group or adds new snapshots while the sync is running, the whole group sync is
racy and might cause spurious errors anyway.
since re-syncing the last already existing snapshot is not possible at the
moment, the code can also be simplified by treating such a snapshots already
fully synced.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
a vanished namespace is one that
- exists on the target side, below the target prefix
- but within the specified max_depth
- and was not part of the synced namespaces
Co-developed-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
two parameters that only differ by a letter are not very nice for quickly
understanding semantics..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
BackupGroup is serializable as its API parameter components, like BackupDir.
move the (always present) namespace closer to the group to improve readability.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to make it easier to distinguish from missing "Prune" privs when removing
vanished groups.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to make the complex logic code shorter and easier to parse. no semantic changes
intended.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Documents the caveats of sync jobs in push direction, explicitly
recommending setting up dedicted remotes for these sync jobs.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Expose the 'prune-delete-stats' as supported feature, in order for
the sync job in pull direction to pass the optional
`error-on-protected=false` flag to the api calls when pruning backup
snapshots, groups or namespaces.
Add and optionally expose the backup group delete statistics by adding the
return type to the corresponding REST API endpoints.
Clients can opt-into the new behaviour by setting the new `error-on-protected`
flag to `false` when calling the api endpoints, which results in removal not
erroring out when encountering protected snapshots.
The default value for the flag remains `true` for now, to remain backwards
compatible with older clients expecting this behaviour.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
FG: reworded commit message slightly
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
In order to load data using the same model from different sources,
set the proxy on the store instead of the model.
This allows to use the view to display sync jobs in either pull or
push direction, by setting the `sync-direction` ont the view.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Switch the subject and labels to be shown based on the direction of
the sync job, and set the `sync-direction` parameter from the
submit values in case of push direction.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Show sync jobs in pull and in push direction in two separate grids,
visually separating them to limit possible misconfiguration.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Switch to the local datastore, used as sync source for jobs in push
direction, to get the available group filter options.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The namespace has to be set in order to get the correct groups to be
used as group filter options with a local datastore as source,
required for sync jobs in push direction.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
`list_sync_jobs` exists as api method in `api2::admin::sync` and
`api2::config::sync`.
Rename the admin api endpoint method to `list_config_sync_jobs` in
order to reduce possible confusion when searching/reviewing.
No functional change intended.
Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Exposes and switch the config type for sync job operations based
on the `sync-direction` parameter, exposed on required api endpoints.
If not set, the default config type is `sync` and the default sync
direction is `pull` for full backwards compatibility. Whenever
possible, determine the sync direction and config type from the sync
job config directly rather than requiring it as optional api
parameter.
Further, extend read and modify access checks by sync direction to
conditionally check for the required permissions in pull and push
direction.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Move the sync job owner check to its own helper function, for it to
be reused for the owner check for sync jobs in push direction.
No functional change intended.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Read access to sync jobs is not granted to users not having at least
PRIV_DATASTORE_AUDIT permissions on the datastore. However a user is
able to create or modify such jobs, without having the audit
permission.
Therefore, further restrict the modify check by also including the
audit permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Moves and refactores the sync_job_do function into the common server
sync module so that it can be reused for both sync directions, pull
and push.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Expose the sync job in push direction via a dedicated API endpoint,
analogous to the pull direction.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In order for sync jobs to be either pull or push jobs, allow to
configure the direction of the job.
Adds an additional config type `sync-push` to the sync job config, to
clearly distinguish sync jobs configured in pull and in push
direction and defines and implements the required `SyncDirection` api
type.
This approach was chosen in order to limit possible misconfiguration,
as unintentionally switching the sync direction could potentially
delete still required snapshots.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Adds the functionality required to push datastore contents from a
source to a remote target.
This includes syncing of the namespaces, backup groups and snapshots
based on the provided filters as well as removing vanished contents
from the target when requested.
While trying to mimic the pull direction of sync jobs, the
implementation is different as access to the remote must be performed
via the REST API, not needed for the pull job which can access the
local datastore via the filesystem directly.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a dedicated api type for the `version` api endpoint and helper
methods for supported feature comparison.
This will be used to detect api incompatibility of older hosts, not
supporting some features.
Use the new api type to refactor the version endpoint and set it as
return type.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
To correctly account also for the number of deleted backup groups, in
preparation to correctly return the delete statistics when removing
contents via the REST API.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Make the `BackupGroupDeleteStats` exposable via the API by implementing
the ApiTypes trait via the api macro invocation and add an additional
field to account for the number of deleted groups.
Further, add a method to add up the statistics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In preparation for the delete stats to be exposed as return type to
the backup group delete api endpoint.
Also, rename the private field `unremoved_protected` to a better
fitting `protected_snapshots` to be in line with the method names.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Adding the privileges to allow backup, namespace creation and prune
on remote targets, to be used for sync jobs in push direction.
Also adds dedicated roles setting the required privileges.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add `remote_acl_path` method which generates the acl path from the sync
job configuration. This helper allows to easily generate the acl path
from a given sync job config for privilege checks.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a `remote_acl_path` helper method for creating acl paths for
remote namespaces, to be used by the priv checks on remote datastore
namespaces for e.g. the sync job in push direction.
Factor out the common path extension into a dedicated method.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Extend the component limit for ACL paths of `remote` to include
possible namespace components.
This allows to limit the permissions for sync jobs in push direction
to a namespace subset on the remote datastore.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Combine the two if statements checking the datastores ACL path
components, which can be represented more concisely as one.
Further, extend the pre-existing comment to clarify that `datastore`
ACL paths are not limited to the datastore name, but might have
further sub-components specifying the namespace.
Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a method `upload_index_chunk_info` to be used for uploading an
existing index and the corresponding chunk stream.
Instead of taking an input stream of raw bytes as the
`upload_stream`, this takes a stream of `MergedChunkInfo` object
provided by the local chunk reader of the sync jobs source.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In preparation for implementing push support for sync jobs.
Factor out the upload stream for merged chunks, which can be reused
to upload the local chunks to a remote target datastore during a
snapshot sync operation in push direction.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In preparation for push support in sync jobs.
Extend and move `BackupStats` into `backup_stats` submodule and add
method to create them from `UploadStats`.
Further, introduce `UploadCounters` struct to hold the Arc clones of
the chunk upload statistics counters, simplifying the house keeping.
By bundling the counters into the struct, they can be passed as
single function parameter when factoring out the common stream future
in the subsequent implementation of the chunk upload for sync jobs
in push direction.
Co-developed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Allow to filter namespaces by given callback function. This will be
used to pre-filter the list of namespaces to push to a remote target
for sync jobs in push direction, based on the privs of the sync jobs
local user on the source datastore.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
`BackupGroup` implements `cmp::Ord`, so use that implementation for
comparing groups during sorting. Furtuher, only sort the list of
backup groups after filtering, thereby possibly reducing the number
of required comparisons.
No functional changes.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
To ensure the fix for avoiding printing verbose log levels to stderr,
stdout is included, as that spams the log with the full worker log
tasks.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
I plugged in a USB pen drive and the whole disk list UI became
completely unusable because smartctl fails to handle that device due
to some `Unknown USB bridge [0x090c:0x1000 (0x1100)]` error.
That itself might be improvable, but most often I do not care at all
about smart data, and certainly not enough to make failing gathering
it disallow me from viewing my disks (or the smart data from disks
where it still could be gathered, for that matter!)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
removable datastores will have a PBS-managed mountpoint as path, direct
access to the field needs to be replaced with a helper that can account
for this.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
