IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
> Transitions separate other body elements. A transition should not
> begin or end a section or document, nor should two transitions be
> immediately adjacent.
-- https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#transitions
Never sphinx output an error for these. Unify the simple/nested cases
while at it and drop the initial one, the "usage" string is defined
above and will always be empty.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Might be also a LazyCell, as this is currently mainly used in Perl,
where we cannot really use threads anyway, but never say never and it
might be used in pure rust too someday; besides, it can be changed if
needed.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This is a merge of git.proxmox.com/git/proxmox-resource-scheduling.git
but with moving the whole history to a subdirectory first using:
git filter-repo --to-subdirectory-filter proxmox-resource-scheduling
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Only the relative difference for values between different alternatives
is relevant, meaning 0.002 vs 0.004 and 0.2 vs 0.4 will influence the
scoring in the same way. This is not really desirable, because values
closer to 1.0 indicate higher load and thus should influence the
scoring more than differences that are actually tiny, but big when
viewed as a relative difference.
To avoid the issue, simply add 1.0 to all values. Like that, 1.002 vs
1.004 will also be small when viewed as a relative difference.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When nodes have different stats, the sum of percentage values will be
different for different alternatives, so the linear average is enough.
But when nodes have the same stats, this is not the case, the sum will
be the same, thus the average won't influence the scoring. If there is
an already overcommitted node, all alternatives besides the already
overcommitted node would be scored the same.
To fix it, use the squares of percentages instead, where more evenly
distributed usage across nodes will lead to a smaller value and thus
better scoring.
It's not really necessary to divide by length or take the sqrt, but it
seemed nicer to have something that would give 1.0 if all inputs are
1.0.
Reported-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this now wouldn't cause the weigh() lines to exceed 100
chars ;-)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We already use functions that take or return the full [T; N]
array, so let's just do the same here, it's a bit shorter
this way.
If we get worried about large values we should implement
versions with `&mut [T; N]` consistently for all math
functions.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This enforces the order in the Criteria and its From into
the Matrix entry.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Models usage of guests and nodes, and allows scoring nodes on which to
start a new service via TOPSIS. For this scoring, each node in turn is
considered as if the service was already running on it.
CPU and memory usage are used as criteria, with memory being weighted
much more, because it's a truly limited resource. For both, CPU and
memory, highest usage among nodes (weighted more, as ideally no node
should be overcommited) and average usage of all nodes (to still be
able to distinguish in case there already is a more highly commited
node) are considered.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Implement the TOPSIS[0] algorithm to score multi-valued alternatives
according to a given set of weighted criteria.
The number of alternatives cannot be known at compile time, but the
number of criteria should be (a given module using the topsis module
should have one (or more) fixed sets of criteria). Therefore, the
TopsisMatrix is implemented as a Vec of N_CRITERIA-sized arrays.
Compared to the description in [0] the weighing of the matrix
according to the weights of the criteria only happens during distance
calculation to the idealized alternatives. It turned out more natural
like that, because the matrix doesn't need to be mutable.
[0]: https://en.wikipedia.org/wiki/TOPSIS
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
it actually would be, it's our code through and through, but we used
the revision for meta updates or if we borked a bump in the past, so
lets keep it for now like (almost) all other librust-* source code
packages from this workspace here.
If we switch, we should switch them all at once for consistency.
And FWIW, this is really not final in any direction, we could switch
back and forth as much as we want as long as something gets bumped,
not that we should ;-)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This is not really nicely done, but this crate caused me much more
friction and time required to get it bootstrap than the rest combined,
so it really needs to continue for now, even if still being far from
ideal.
I hinted a few things in a bigger FIXME comment in the tests, but
probably not complete and to be fair, it might not be easy to improve
this substantially, but certainly also not impossible; in short, repos
should be much simpler to add centrally, without having to sprinkle
stuff all over the place, the RepoHandle type might be avoidable, and
if then it really should go, and the test system should be simpler to
edit and shouldn't need any change at all when adding a new distro, or
building for a new one, besides maybe a coverage test to ensure that
all existing distro have some tests.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While a small thing, this often points to an issue where newline
addition was handled by a component which cannot do a proper job.
Like here, the newlines where added in a function that serializes a
single repo entry, it simply has no idea in which context and as such
it really cannot decide if a separating newline after this entry is
warranted or not. So rather move that handling to the place where we
serialize whole files, as there we actually got the required context
to handle this correctly.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
PDM did this right from the beginning, the others will change this for
Trixie.
If this should be fully correct it would need to have the suite passed
as parameter, as is we can only return the default for the one for
which the crate release is currently build for, this is a bit of an
API limitation.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This was not tested in practice and might need follow ups, I enable it
nonetheless as it should better ensure that we do not forget and thus
avoid a nosiy warning by apt.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
debcargo only tests with default features due to *potential* build
conflict between optional features. While that certainly is a thing,
it's not for our crates here (currently).
For the broader ecosystem this probably could be improved in debcargo,
either looking if there's info available to improve coverage
automatically or by providing an opt-out config option for specific
feature combos–just my relatively uninformed opinion though, albeit
I'm very certain the status quo is improvable.
Anyhow, until such an improvement manifests, lets just add a simple
`cargo test` to the build.sh script, which is used for all common
makefile targets like the ${pkg}-deb ones. The downside here is that
it does not "seeps" into the source package, so if one, e.g., uses
sbuild they still won't profit from this increased coverage; but as
debcargo creates the build-dependency field of d/control with the
reduced feature set, we cannot just add the full cargo test call to
d/rules.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The `Date` header in the generated email includes a timestamp with a
time zone offset (RFC 2822 format). This causes test failures when the
host system's time zone differs from CET/CEST, as the expected and
actual outputs no longer match.
As a workaround, the `Date` header is excluded from comparisons between
the generated and reference emails.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Link: https://lore.proxmox.com/20250526134211.278900-1-l.wagner@proxmox.com
The notification system as we know it know underwent a serious overhaul
shortly before its initial release in PVE one year ago. Before the
overhaul, there were filters (the 'inverse' of a matchers as we know
them today) and groups (allowed one to send a notification to
multiple targets at once, they were replaced by being able to select
multiple targets in a matcher).
Unfortunately, this previous iteration of the notification was already
merged and packaged in the pvetest repository, resulting in an unknown
number of systems which may have already been using filters and groups.
To avoid breaking these systems when updating to the overhauled
notification system, we've kept the section config type
for both config sections. In anticipation of a future removal of these
section types, we proactively removed these entries when loading and
saving the notification config file.
With the upcoming release of PVE 9 and PBS 4 we can drop the
schema. The upgrade check scripts (e.g. pve8to9) will
include checks to make sure that no legacy config sections are present
any more.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Link: https://lore.proxmox.com/20250526114138.204228-1-l.wagner@proxmox.com
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thanks to the regression tests introduced in the previous commit we can
be certain that everything works as it should.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This allows us to write some tests for it.
No behavioral changes intended.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We generally don't want anything else in the workspace to depend on
the webauthn-rs crate directly.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Add some basic non-exhaustive guidelines–this cannot be complete
without a crystal ball–as otherwise a dev might be really unsure about
what the expected course of action here is.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This was quite definitively applied due to debcargo not running the
tests with all features enabled.
While that indeed cannot be done in general due to potential
conflicting features where that itself will result in compiler errors,
it could be enabled as opt-out, or with some settings for feature
combinations to skip, but I digress.
Fixes: 48fc0a2a ("http: use Watcher of GracefulShutdown to avoid races")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since we do not consider trixie crates to be *released* yet, but want
updates to work via apt for internal trixie development, this is a
revision bump.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We need the GracefulShutdown's `.watcher()` method so we can get rid
of the `Arc<>` around it so we can *actually* call `.shutdown()` which
*consumes* for some reason...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
since this hyper-based server is not related to the rest of the stack
and has very simple requirements, it is implemented using just
hyper(-util) and not proxmox-http.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
enabling logging while we are at it (useful when debugging things like
the graceful shutdown implementation).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and switch to proxmox-http's Body implementation.
hyper now has a special (opaque) Body implementation called Incoming
which is used for incoming request bodies on the server side, and
incoming response bodies on the client side, so our API handler's now
consume an instance of this type.
the Accept trait previously offered by hyper is gone in 1.0, and needs
to be replaced with an accept loop. our corresponding Acceptor
implementations have been dropped as well.
hyper now has its own Service and async Read/Write traits, but helpfully
provides wrappers for tower's Service and tokio's AsyncRead/AsyncWrite
variants.
graceful shutdown handling is now exposed differently on the hyper side,
and to allow usage with upgradable connections the variant form
hyper-util needs to be used, as the one straight from hyper doesn't
support it (yet).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and switch to proxmox_http's body. hyper now has a special (opaque) Body
implementation called Incoming which is used for incoming request bodies
on the server side, and incoming response bodies on the client side, so
our API handler's now consume an instance of this type.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
use the new Empty body type, since Body is now a trait and no longer a
struct containing different body types.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the main change requiring adaptations here is that hyper no longer
provides a Body struct, but switched to a trait, so we use our own Body
type introduced by the previous commit.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
hyper/http 1.0 now only have a Body trait and some implementations for
specific use cases. following reqwest's lead (and copying some parts of
its implementation), implement our own Body struct for the two common
use cases:
- a body instance containing the full body data as Bytes
- a streaming body instance
together with the most common helper methods (empty body, convert, wrap
existing stream as body) this should make the rest of the upgrade fairly
straight-forward.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[WB: merge with previously included http_1 dependency]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
by switching to tower's Service and wrapping in TokioIo as needed. hyper
now uses their own Service type to not expose tower in their public API,
and their own Async IO traits, but they provide wrappers to not require
too many changes for crates like ours here that already used hyper 0.14.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
using the legacy client from hyper_util, which is the replacement for
the pre 1.0 Client from hyper itself
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
using the legacy client from hyper_util, which is the replacement for
the pre 1.0 Client from hyper itself
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
put optional dependencies first, followed by features on external
crates, followed by internal features.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
This slightly increases IO safety.
For the connection peeking we need to keep the raw FD handling, but
move the use statements to a more narrow scope so that using those
low-level dangerous traits won't proliferate to easily.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since ureq 3 is a complete rewrite and switched to using the http
crate, we now depend on http version 1.
We still carry around the 0.2 dependency in public interfaces as long
as we use the old hyper code and need to upgrade this crate again to
use *only* http 1.0 in the future.
For now, for the client-trait impl we just convert the http 1.0
Response to a http 0.2 Response.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Mainly adapting to nix adopting more IO Safety by switching the mmap
module and set_tcp_keepalive to accept any AsFd.
Note that the mmap accepting that as value while set_tcp_keepalive
accepting that as reference is mirroring the signatures of the
underlying nix methods for consistency, but nix has no real reason for
this mix either I think.
Besides that most methods that can take a dirfd switched to wrapping
that in an Option, with the main reason being seemingly just because
some other methods already do that, i.e. nix sometimes tries for
consistency...
I also adopted NonNull for the data member in the mmap struct, as nix
switched to that and that type _should_ provide a bit of a safer API.
The newer nix version also updated it's bitflags dependency, which it
uses on lots of places. We mostly notice that in the unsafe
from_bits_unchecked method now being a safe from_bits_retain.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Was already uploaded that way in Bookworm and for these crates it
should be fine, as they _are_ native and even meta changes can do a
cargo version bump.
So continue this as sort of experiment, maybe we can switch over other
crates too depending on the experience here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Using crates not updated yet, will be done one by one.
Drop the bindgen feature, it's now always enabled and not a feature
anymore.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this should make the endpoint behave closer to how it behaved before
the HttpOnly changes. the `ticket_info` field is superfluous anyway,
as the response also includes the proper ticket in this case already.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
A big proportion of the consumers of this API use the `format!` macro to
define the base of the url, hence we use a `Cow<'_, str>` on the
constructor to prevent unnecessary allocations.
A `perl-api-path-builder` feature was added to satisfy the needs of
pve-api-types.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
These helpers exist on the untyped SectionConfigData struct, but not on
the typed one. To facilitate the migration to the typed
SectionConfigData<T> add these as macros.
These can be used to simply lookup an item with a specific type by
passing the type, e.g.:
lookup!(config, "root", UserSectionConfig::User)
or retrieve a vector with the sections that correspond to the passed
type, e.g.:
convert_to_typed_array!(config, UserSectionConfig::ApiToken)
Also add some simple unit-tests to test them.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Currently to insert a SectionConfig-section correctly we need to insert
it into the HashMap and insert the key in the order vector as well. This
is not ensure the SectionConfig gets written in the same order as it was
read. By implementing DerefMut and making the underlying HashMap `pub`
we allow to insert sections in the HashMap without inserting the key
into the the order. This caused a whole range of bugs where sections
where inserted, but never written (because we iterate over the order).
To improve this, we add our own insert and remove functions that
automatically add and remove the keys from the order. Also delete the
DerefMut impl and make the underlying Vec and HashMap private.
Note that this is a breaking change.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
The underlying function which gets called by `write_section_config` also
takes a `impl AsRef<Path>` and in some cases we do pass a path, so
taking the same parameter is better.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Without this newline running commands with missing arguments, e.g.
```
proxmox-backup-client backup
```
will print their usage string without a newline at the end, which is
costing me my sanity.
As per commit d872eb9d7e ("router: cli: rework newline handling for
doc and help output"), usage shouldn't contain trailing new lines and
this in principle should not print double new lines.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
By default, the mapping between tracing/log error levels to
syslog priorities is as follows:
error! -> Error
warn! -> Warning
info! -> Notice
debug! -> Informational
trace! -> Debug
Before using tracing-journald, we used `syslog` to write log messages
to the journal. That one has the following mapping:
error! -> Error
warn! -> Warning
info! -> Informational
debug! -> Debug
trace! -> Debug
The changed mapping of info! log messages led to many messages being
written with the Notice priority, which are formatted bold when running
journalctl to view log messages.
This commit changes the mapping so that it is the same as with the
syslog crate.
Support for custom priority mappings was only introduced in
tracing-journald 0.3.1, so we have to bump the dependency
to that version.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Allow to instantiate a `HttpsConnector` not using the default
`getaddrinfo` based `GaiResolver` for domain name resolution, but
rather a custom resolver implementing the required traits.
The usecase for this is to swap out the DNS resolver for the
statically linked proxmox-backup-client binary, where the glibc
dependency is problematic because of possible ABI incompatibility.
However, set the generic type on `HttpsConnector` to default to
`GaiResolver` to limit inconvenience for implementations using it.
Adds tower-service as cargo workspace dependency and build dependency
to debian/control.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Previously, notification templates could be modified by the user, but
these were overwritten again with installing newer package versions of
pve-manager and proxmox-backup.
Now override templates can be created cluster-wide in the path
“/etc/{pve,proxmox-backup}/notification-templates/{namespace}”, which
are used with priority. The folder structure has to be created and
populated manually (e.g. /etc/pve/notification-templates/default).
If override templates are not existing or their rendering fails, the
vendor templates in
"/usr/share/{pve-manager,proxmox-backup}/templates/default/" are used.
Sequence: [override html -> vendor html ->] override txt -> vendor txt
An error is only returned if none of the template candidates could be
used. Using an override template gets not logged.
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Link: https://lore.proxmox.com/20250321133341.151340-1-a.zeidler@proxmox.com
The old code was wrong and overestimated the memory used because it
did not take into account things like "SReclaimable", a part of slab
(in-kernel memory allocator) describing things like caches that can be
reclaimed, plus the memory for "Active(file)" and "Inactive(file)",
and other internal kernel things that even though small for each one,
can add up quickly.
Most of these metrics are exposed and could be included in the
calculation, but this will simply become obsolete in the future as the
kernel changes how it does things and how it calculates such available
memory, as it has done many times in the past.
To solve this problem for the long term, the MemAvailable field was
added to /proc/meminfo as of kernel 3.14. It describes "the amount of
memory available for a new workload without pushing the system into
swap". While it is only an estimate, it is as good as it gets, and
since it comes from the kernel, we can always assume that it is
correct for the currently booted kernel.
So, switch over to this metric for calculating the used memory by
subtracting MemAvailable from MemTotal.
Also adds a simple test case for the parser.
This commit is based on a patch from Dietmar [1].
[0]: https://git.kernel.org/torvalds/c/34e431b0ae398fc54ea69ff85ec700722c9da773
[1]: https://lore.proxmox.com/all/20250313114535.99912-2-dietmar@proxmox.com/
Originally-by: Dietmar Maurer <dietmar@proxmox.com>
[TL: rewrite comments and commit message from scratch]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Introduces two new, simple API types representing the built-in PAM and
PBS authentication realms.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
[TL: adapt doc-comment for new default field like Shannon pointed out
but with a slightly different text than she proposed]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The field indicates whether the realm should be the default realm to
select in the login dialog.
Per definition, the field should only ever be set to `true` on exactly
one realm - up to the consumer/storage to ensure that.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
[TL: adapt doc-comment for new default field like Shannon pointed out
but with a slightly different text than she proposed and change
regular comment to doc-comment.]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add the `gc-atime-cutoff` option to the datastore tuning parameters.
This allows to specify the time after which the chunks are not
considered in use anymore if their atime has not been updated since
then.
The default is to keep chunks within the 24h 5m timespan (given no
active writers).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add the `gc-atime-safety-check` flag to the datastore tuning
parameters. This flag allows to enable/disable a check to detect if
the atime update is honored by the filesystem backing the chunk store
during phase 1 of garbage collection and during datastore creation.
The default is to perform the check.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Sometimes it's useful to give the user control over how much
parallelized the job is, so introduce a worker thread option.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
To quote from RFC 9110 [1]:
A user agent SHOULD send Content-Length in a request when
the method defines a meaning for enclosed content and it
is not sending Transfer-Encoding. For example, a user agent
normally sends Content-Length in a POST request even when
the value is 0 (indicating empty content).
A user agent SHOULD NOT send a Content-Length header field
when the request message does not contain content and the
method semantics do not anticipate such data.
It seemed like our HTTP client lib did not set the header
automatically, which is why we should do it manually.
While most services seemed to have worked fine without setting
the header, some Microsoft services seem to require it
to accept the webhook request [2].
[1] https://datatracker.ietf.org/doc/html/rfc9110#name-content-length
[2] https://forum.proxmox.com/threads/158827
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Currently the anyhow error context of a given error is not included
in the error message, as `to_string` does use the default formatting
[0].
Include the error context, formatting it as single line as the
message is also shown to the users in e.g. the Proxmox Backup Severs
task state in the UI.
[0] https://docs.rs/anyhow/latest/anyhow/struct.Error.html#display-representations
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
These will produce an error in edition 2024 otherwise. The reason this
is needed is because the `unsafe` block has its own scope.
The bytes were defined inside of the let-mut block to preserve the
lifetime they had before this commit.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Prevents the following error:
```
error[E0597]: `inner` does not live long enough
--> proxmox-async/src/broadcast_future.rs:109:24
|
107 | inner: Arc<Mutex<BroadCastFutureBinding<T>>>,
| ----- binding `inner` declared here
108 | ) -> impl Future<Output = Result<T, Error>> {
109 | let mut data = inner.lock().unwrap();
| ^^^^^ borrowed value does not live long enough
...
121 | data.broadcast.listen()
| ----------------------- argument requires that `inner` is borrowed for `'static`
122 | }
| - `inner` dropped here while still borrowed
error[E0597]: `data` does not live long enough
--> proxmox-async/src/broadcast_future.rs:121:9
|
109 | let mut data = inner.lock().unwrap();
| -------- binding `data` declared here
...
121 | data.broadcast.listen()
| ^^^^-------------------
| |
| borrowed value does not live long enough
| argument requires that `data` is borrowed for `'static`
122 | }
| - `data` dropped here while still borrowed
```
The use<...> pattern was introduced in rust 1.82.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
To write result message manually, bypassing tracing.
The workertasks currently get their status from parsing the log
messages in the task-log file. The problem is that if these messages are
filtered – which is now possible using the PBS_LOG env variable – some
workertasks will end up with a "stopped: unknown" status. This is not
desirable so write the message manually to the workertask file and
bypass tracing.
This way we are guaranteed that, regardless of the max logging level the
user sets, the final message (and status) is written.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Add a layer that outputs messages to stderr in a specific format. In
PVE, stderr is rerouted to the tasklog if the we are within a
workertask. Therefore, ensure the stderr output is formatted
appropriately.
Reported-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Add a builder-like struct to compose a tracing logger using different
layers. Instead of having an init function per product/binary or
super-specific init functions that describe the logger, have a dynamic
builder. The builder improves the usability and makes the logging
initialization more self-explaining.
Suggested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
The `new_args` Vec is directly passed to the other Vec's `.extend()`,
which takes an `IntoIterator` consuming it, so just pass the
intermediate `Iterator`.
The `rest` Vec owns its strings and we don't need it afterwards, so
similarly, we can consume it via `.extend()` instead of a manual
push(s.clone()) loop.
The .truncate(0) can just be .clear() - they are equivalent according
to their documentation.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
a previous commit fixed up all intra doc links that were present on
public apis, this also fixes the links for private members.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this fixes intra document links or rephrases the documentation in a
more appropriate way to remove all `broken_intra_doc_links` warnings.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this panics when running `cargo test` otherwise, as the api macro
requires fields in `ObjectSchema`s to be sorted now.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
commit 68b13965 (router: docs: add horizontal line before nested
command docs) broke the nested command group test case. this commit
adapts the expected output accordingly.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
the `type_complexity` clippy lint [1] is intended to make the code
more legible in most cases. however, the lint triggers on a member of
a private enum, an example minimal rest server and a private static
variable here. so the benefits of declaring a new type that would
encapsulate this complexity is minimal. hence, ignore the warnings for
now.
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
the `ApiHandler`'s `PartialEq` implementation makes heavy use of
`transmute`. clippy wants the types to be explicitly stated here and
not inferred, to avoid potential undefined behaviour if an unexpected
type is inferred. however, the types that would be inferred here are
so complex, that the code would become illegible anyway, so ignore
this lint for now.
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#missing_transmute_annotations
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
while usually this would improve ergonomics, in this case it isn't
clear whether all uses of `FromStr` would be considered valid here.
renaming the function would also make the type more confusing to use
as `from_str_with_default` also exists, so keep this for consistency.
this ignores a clippy lint [1].
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#should_implement_trait
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
While *technically* breaking as it changes the method signature for
the `create_ticket` call to use a struct for its parameters, this is
only (supposed to be) used via its `CREATE_TICKET_API_METHOD` handler
to be passed to a router. Direct use of this does not make sense.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
if the client knows the auth cookie's name, it now passes it on to the
relevant parts of `proxmox-login` so that the ticket is send the
correct cookie
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this should make it possible to use the proxmox-client crate outside
of context where HttpOnly cookies are handled for us. if a cookie
name is provided to a client, it tries to find a corresponding
`Set-Cookie` header in the login response and passes tries to parse
it as a ticket. that ticket is then passed on to proxmox-login like
any regular ticket.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
previously the name in which the ticket was send was derived by the
product abbreviation in the ticket itself. the assumption was that
authentication cookies would always have a name like this:
`<PRODUCT_ABBREVIATION>AuthCookie`.
this commit adds helpers that allow specifying the cookie's name by
users of this crate.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
tickets that end in `::ticketinfo` are not properly signed and just
include information such as the timestamp when the ticket was created.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this allows client to be aware that the ticket they manage is
informational only and that the real ticket should have been set via
a HttpOnly cookie.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
depending on the context a client may or may not have access to
HttpOnly cookies. this change allows them to pass such values to
`proxmox-login` to take them into account when parsing login
responses.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
in certain context (for example, the browser), no password needs to be
provided when using HttpOnly cookies as they are handle by said
context. so make renewing ticket with password optional and add a new
helper function that does not require a password.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
tickets created through the new HttpOnly ticket endpoint won't return
a ticket in the password field. so this field will be left empty.
hence make it optional.
the endpoint does return a ticket_info parameter, though, that
includes the information when a ticket needs to be refreshed. so add
a new optional field for that too.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
adds a new endpoint that is useful when dealing with HttpOnly cookies
that cannot be removed by client-side javascript (and by extension
wasm) code. the logout handle simply removes the cookie that is used
for storing the current ticket. this works the same way as it does in
the front-end: by setting an expired cookie with the same name.
as cookies are now prefixed with `__Host-` by default, the cookie here
also needs to be `Secure` and have the same `Path` to not be rejected
by the browser before it can remove the old cookie.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this adds a new endpoint for requesting tickets. instead of returning
the ticket in the responses body, the ticket is set as a HttpOnly
cookie. this has a couple of advantages:
- the cookie cannot be stolen if an attacker downgrades the connection
to http and injects malicious javascript (`HttpOnly`)
- we don't need to rely on the client to make sure that the cookie is
only send in the appropriate context and only over https
connections (`Secure`, `SameSite`).
- the cookie cannot be overwritten by other subdomains, insecure
connections etc. (the default is to prefix them with `__Host-`)
this follows the best practice guide for secure cookies from MDN
[1]. we also set the cookies to expire when the ticket would so that
the browser removes the cookie once the ticket isn't valid anymore.
the endpoint still returns a ticket that only contains the
informational portions of the ticket but not a valid signature. this
is helpful to let clients know when to refresh the ticket by querying
this endpoint again. it still protects the cookie, though, as it
isn't a valid ticket by itself.
[1]: https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Cookies
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this makes sure that newly generated cookies that are prefixed with,
for example, `__Host-`, for security purposes, are correctly picked
up on. otherwise, the new cookies would not be able to yield proper
authentication.
currently this still falls back to insecure non-prefixed cookies. we
should deprecate them in the long-term and remove this fallback.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this adds the function `prefixed_auth_cookie_name` to the
`AuthContext` trait. said function can be used by users of this crate
to modify the expected prefix of the auth cookie. most products
should be able to use the default of `__Host-` though, so this also
adds a default implementation.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this allows us to write api handlers that have access to a request's
headers and to create a low level response while being able to also
specify the parameter in the request's body. this is useful for
endpoints that should not use url parameters, but still need to
access/set specific headers.
previously, `AsyncHttp` did not allow for parameters that were marked
as non-optional to be passed in the body of a request.
as a side-effect, the body is parsed by the rest server to check for
parameters and consumed. so it cannot be passed on by the handler.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this function does not require ownership of the parts parameter, so we
can simply borrow it. this way we can pass the parameter on to a
handler that consumes them even when using `get_request_parameter`
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
(it's on the wrong function, and also it's wrong - std's str
comparison forwards to the byte comparison anyway since this is fine
for utf-8 if we're not doing anything natural-language-aware...)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This verifies *at compile time* that the properties and variants of
ObjectSchemas and OneOfSchemas are sorted.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Pretty much all functions accepting `CreateOptions` take a value and not
a reference, so I've found myself using `.clone()` quite often in code
I've written recently.
The struct is only 24 bytes large (verified by a
`std::mem::size_of::<CreateOptions>()`), so it should be absolutely fine
to just derive Copy for it.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This fixes the reStructuredText (ReST) output for more complex formats
where we got, for example, a definition list of all properties of a
(section) configuration with one or more properties then having a
sub-format that is also rendered as a list.
Such nested lists are possible in ReST but need to be separated by a
blank line [0].
Without this we got quite a few warnings/errors from the Sphinx/ReST
buildsystem looking like:
> config/datastore/config.rst:19: ERROR: Unexpected indentation.
> config/datastore/config.rst:20: WARNING: Block quote ends without a blank line; unexpected unindent.
[0]: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks
Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Improved errors when panics occur and the panic message is a
formatted (not static) string. This worked already for &str literals,
but not for Strings.
Downcasting to both &str and String is also done by the Rust Standard
Library in the default panic handler. See:
b605c65b6e/library/std/src/panicking.rs (L777)
Signed-off-by: Laurențiu Leahu-Vlăducu <l.leahu-vladucu@proxmox.com>
Move over the whole history of pbs-api-types from the proxmox-backup
git repo to the common workspace as we want to re-use that in PDM and
the yew UI components, and thus require a real source-code package for
this crate.
Choose this repo over proxmox-api-types to avoid the need to copy this
build-system over there, rather we want to also merge pve-api-types
from that repo into ours here in the future.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Fixes a style issue introduced by b31ab119 ("client: improve api error
message (avoid duplicate status code)"), as one always should use
inline variables for format template strings even if there are some
that cannot be used, e.g. as some method needs to be called on them
like here. The reason for this is to reduce the amount of
free-standing "{}" and parameters as that reduces the need to manually
match what "{}" resolves to which expression. While it is not that bad
for only two format variables it's still not winning us anything if we
don't do it and breaks consistency with newer code style.
Noticed as that commit made the line overly long, causing rustfmt
changing the line.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The libc function mkdtemp takes a C char pointer while we previously
cast our OSString buffer as i8 pointer, but that's not valid on
platforms like AArch64 (ARM), where char is equivalent with a u8.
Fix that by using the c_char type that was explicitly made to always
get the correct, platform-independent type for C chars when doing FFI.
This was reported by OJaksch on our Arch Linux User Repo (AUR) package
[0].
https://aur.archlinux.org/packages/proxmox-backup-client#comment-1006851
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This way we can copy and modify a schema.
Eg. via the following ways:
const FOO_SCHEMA: Schema = SOME_SCHEMA
.unwrap_integer_schema_cloned()
.description("Foo")
.schema();
Note that for example there is currently no builder to set a
`default_key` for an `ObjectSchema` back to None, so one could do:
const FOO_SCHEMA: Schema = const {
let mut schema = SOME_SCHEMA.unwrap_object_schema_cloned();
schema.default_key = None;
schema.schema()
};
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Because the UI kept producing them...
And still does...
This is NOT meant to be used by anything other than generated legacy
code for PDM (pve-api-types) and only affects the serde based
deserializer. The "old" `schema.parse_property_string() -> Value`
method does not utilize this for now.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
To prepare for a breaking change in proxmox-schema.
Since new fields in the schema constitute a breaking change, using the
builder methods will let this compile later as well.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
so that we can have a property:
```
foo: Option<PropertyString<Bar>>,
```
within a struct that derives `Updater`
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We tried this unconditionally on start-up in the PDM for the priv. API
daemon, but we actually only want to clean-up on fresh bind, not on
restoring the FD on daemon reload. Otherwise the unprivileged daemon
cannot connect to the privileged one anymore after the latter got
reloaded.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For consistency with it's other repositories, we might migrate other
products also to this schema with a future major release (and nginx
rewrite config for backward compatibility)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Migrated from `log` to `tracing`. Imported `tracing` only as it has a
smaller footprint (and less dependencies) than `proxmox_log`.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
in edition 2024 references to mutable static will be disallowed [1]. the
recommended way around this is to use types with inner mutability. so
use a `OnceLock` for `ACME_ACME_CONFIG` as the directoryfor ACME
configuration purposes shouldn't change throughout the run time of an
application.
[1]:
https://doc.rust-lang.org/nightly/edition-guide/rust-2024/static-mut-references.html
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Fixes the manual_flatten clippy lint:
```
warning: unnecessary `if let` since only the `Some` variant of the iterator element is used
--> proxmox-apt/src/repositories/mod.rs:40:5
|
40 | for digest in digests.values() {
| ^ ---------------- help: try: `digests.values().copied().flatten()`
| _____|
| |
41 | | if let Some(digest) = digest {
42 | | common_raw.extend_from_slice(&digest[..]);
43 | | }
44 | | }
| |_____^
|
help: ...and remove the `if let` statement in the for loop
--> proxmox-apt/src/repositories/mod.rs:41:9
|
41 | / if let Some(digest) = digest {
42 | | common_raw.extend_from_slice(&digest[..]);
43 | | }
| |_________^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten
= note: `#[warn(clippy::manual_flatten)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
FG: use `into_values().flatten()` instead of `values().copied().flatten()
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Fixes the clippy lint:
```
warning: empty line after outer attribute
--> proxmox-sys/src/systemd.rs:7:1
|
7 | / #[allow(clippy::manual_range_contains)]
8 | |
| |_
9 | fn parse_hex_digit(d: u8) -> Result<u8, Error> {
| ---------------------------------------------- the attribute applies to this function
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#empty_line_after_outer_attr
= note: `#[warn(clippy::empty_line_after_outer_attr)]` on by default
= help: if the empty line is unintentional remove it
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy lints:
```
warning: empty line after doc comment
--> proxmox-lang/src/lib.rs:33:1
|
33 | / /// ```
34 | |
| |_
35 | #[macro_export]
36 | macro_rules! try_block {
| ---------------------- the comment documents this macro
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#empty_line_after_doc_comments
= note: `#[warn(clippy::empty_line_after_doc_comments)]` on by default
= help: if the empty line is unintentional remove it
warning: empty line after doc comment
--> proxmox-router/src/cli/mod.rs:308:5
|
308 | / /// Can be used multiple times.
309 | |
| |_
310 | /// Finish the command line interface.
311 | pub fn build(self) -> CommandLineInterface {
| ------------------------------------------ the comment documents this method
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#empty_line_after_doc_comments
= note: `#[warn(clippy::empty_line_after_doc_comments)]` on by default
= help: if the empty line is unintentional remove it
help: if the documentation should include the empty line include it in the comment
|
309 | ///
|
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the single_match clippy lint:
```
warning: you seem to be trying to use `match` for destructuring a single pattern. Consider using `if let`
--> proxmox-apt/src/repositories/mod.rs:41:9
|
41 | / match digest {
42 | | Some(digest) => common_raw.extend_from_slice(&digest[..]),
43 | | None => (),
44 | | }
| |_________^ help: try: `if let Some(digest) = digest { common_raw.extend_from_slice(&digest[..]) }`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_match
= note: `#[warn(clippy::single_match)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
```
warning: doc list item without indentation
--> proxmox-notify/src/api/webhook.rs:131:5
|
131 | /// (`400 Bad request`)
| ^^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
= note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
|
131 | /// (`400 Bad request`)
| ++
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the manual_unwrap_or_default clippy lint:
```
warning: match can be simplified with `.unwrap_or_default()`
--> proxmox-apt/src/repositories/file.rs:369:30
|
369 | let mut origin = match repo.get_cached_origin(apt_lists_dir) {
| ______________________________^
370 | | Ok(option) => option,
371 | | Err(_) => None,
372 | | };
| |_____________^ help: replace it with: `repo.get_cached_origin(apt_lists_dir).unwrap_or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
= note: `#[warn(clippy::manual_unwrap_or_default)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Factor out the open-flags to use for directories and add the CLOEXEC
flag to ensure that open FDs do not get passed to any child process.
A prominent cases where this can happen is the proxmox-daemon reload
code, which re-execs itself in a forked child-process and thus gets
all FDs passed.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[ TL: expand doc-comment and reword commit message to point at actual
thing this fixes (exec not daemon reload) ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In general we want all open files to have set CLOEXEC since our
reloading mechanism can basically fork at any moment and we don't want
newer daemons to carry around old file descriptors, especially lock
files.
Since `make_tmp_file` is called by many things (e.g. open_file_locked,
logrotate, rrd), set O_CLOEXEC with mkostemp.
This fixes issues with leftover file descriptors e.g. tape backups not
working because of lingering locks after a reload, or having deleted
rrd files open.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this lock can be quite contended, until the surrounding code is properly split
to reduce this contention it should help to give the worker task
creation/cleanup code a bit more breathing room.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this mimics how the count is updated when spawning a new task - the lock scope
needs to cover the count update itself, else there's a race when multiple
worker's log their result at the same time..
Co-developed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if setting up a new worker fails after it has been inserted into the
WORKER_TASK_LIST, we need to clean it up instead of bubbling up the error right
away, else we "leak" the worker task and it never finishes..
a worker task that never finishes will indefinitely block shutdown
of the rest server process, including the "old" process when reloading
the rest server.
this issue was found in the wild on a system with lock contention on the
file-based lock covering task index updating leading to lock acquiring
timeouts.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this moves the mail forwarding implementation from `proxmox-notify` into
`proxmox-sendmail` to cover more `sendmail` related use-cases in one
place.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
use the new `proxmox-sendmail` crate instead of the bespoke
implementation in `proxmox-notify`.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
add the `proxmox-sendmail` crate that makes it easier to send mails via
the `sendmail` utility. features include:
- multipart/alternative support for html+plain text mails
- multipart/mixed support for mails with attachments
- automatic nesting of multipart/alternative and multipart/mixed parts
- masks multiple receivers by default, can be disabled
- encoding Subject, To, From, and attachment file names correctly
- adding an `Auto-Submitted` header to avoid triggering automated mails
also includes several tests to ensure that mails are formatted
correctly. debian packaging is also provided.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
[ TL: update years in d/copyright ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Derive and implement the traits to allow comparison of two
`ApiVersion` instances for more direct and easy api version
comparisons. Further, add some basic test cases to reduce risk of
regressions.
This is useful for e.g. feature compatibility checks by comparing api
versions of remote instances.
Example comparison:
```
api_version >= ApiVersion::new(3, 3, 0)
```
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The `ApiVersion` type was introduced in commit ba850a25
("api/api-types: refactor api endpoint version, add api types")
including the `repoid`, added for completeness when converting from
a pre-existing `ApiVersionInfo` instance, as returned by the
`version` api endpoint.
Drop the additional `repoid` field, since this is currently not used,
can be obtained fro the `ApiVersionInfo` as well and only hinders the
implementation for easy api version comparison.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
and improve the variable namign while we are at it. this allows the check to be
re-used in other code paths, like when starting a garbage collection.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
This parameter causes the FileLogger to duplicate the log output to
stdout. This causes duplicate output on proxmox-backup-manager because
this is now handled by tracing. This should be removed completely in the
future.
In the worst case this will only result in missing log lines on stdout
(which is visible only on proxmox-backup-manager/client invocations
anyway).
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[ TL: add doc-comment to struct, note why it can be removed ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Extend the already present `From<TimeSpan> for f64` implementation to
allow using the reference as well. There is no need to take ownership
and consume the `TimeSpan` object for conversion.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
since the SyncJobConfig struct now contains a 'sync-direction' property, we can
omit the 'direction' property of the SyncJobStatus struct. This makes a
few adaptions in the ui necessary:
* use the correct field
* handle 'pull' as default (since we don't necessarily get a
'sync-direction' in that case)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Jobs for both sync directions are now stored using the same `sync`
config section type, so drop the outdated helpers.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Add the sync direction for the sync job as optional config parameter
and refrain from using the config section type for conditional
direction check, as they are now the same (see previous commit).
Use the configured sync job parameter instead of passing it to the
various methods as function parameter and only filter based on sync
direction if an optional api parameter to distingush/filter based on
direction is given.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
By default, the sync client from proxmox-http (powered by ureq) does not
have any request timeout. To avoid blocking the current thread for a
prolonged period of time, we now make use of the previously added
`Client::new_with_timeout` function to create a new HTTP client with a
default timeout of 10 seconds.
In the long run it would be nicer to have a higher timeout here, say
60s, to cope with flaky and high-latency networks and potentially
overloaded targets. But for that we need to change the architecture of
how notifications are send out to ensure that now thread accepting
connections can be blocked.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
[ TL: Change timeout from 5s to 10s as trade-off and expand commit
message slightly with some reasoning for that still relatively
short time value ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This commits adds the possibility to set a HTTP request timeout for the
sync client.
For now, I've opted to add this as a separate option than can be set via
a separate new_with_timeout method as compared to adding it to the HttpOptions
struct. While it of course would make a lot of sense to add it to the
latter, this would require adding support for request timeouts to the
async client as well. Some users of the async client handle request
timeouts externally via tokio::time::timeout, so these would need to
modified as well. I don't want to touch this at the moment,
so I've opted to introduce the timeout to the sync client only for now.
We can always revisit this at a later time and move the option to the
HttpOptions struct.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Commit 0d66acd3 ("api types: introduce `BackupArchiveName` type")
introduced a dedicated archive name api type to add rust type
checking and bundle helpers to the api type. Since this, the backup
archive name to server archive name mapping is handled by its parser.
This however did not cover the `.conf` extension used for VM config
files. Add the missing `.conf` to `.conf.blob` to the match statement
and the test cases.
Fixes: 0d66acd3 ("api types: introduce `BackupArchiveName` type")
Reported-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
... and deserialize with default if field is missing in data.
Reported-by: Aaron Lauterer <a.lauterer@proxmox.com>
Fixes: 35fb5d4f7f ("pbs-api-types: add mount_status field to DataStoreListItem")
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
So it is possible to reset it after a failed unmount, or abort an
unmount task by resetting it through the API.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
The proxmox-notify crate can render notification text based on two
different templates, plaintext and html. The html template is at the
moment only used for email-based notifications. If we try to render
a html-formatted message but there is no html template, we try to
fall back to the plaintext template and wrap the rendered message
in <pre> tags.
As a preparation for user-supplied/overridden templates, I added a log
message "html template not found, falling back to plaintext ..." to
educate the user about this behavior.
In Proxmox Backup Server, we only ship plaintext templates at the
moment, meaning that this log message will be shown for every single
(email) notification that is sent out. This might be a bit confusing,
because the log message can be interpreted as an error, which it isn't.
This commit removes the log message completely for now. Once we add
support for user-overridable notification templates we could consider
adding it back it, but maybe phrased a bit differently, to avoid it
being interpreted as an error.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Even if we don't have an HTML template available, we always
send an HTML part (the plain text part wrapped in <pre>) to
improve rendering in certain mail clients. This means
we can simply message formatting, since we do not have to
distinguish between single-part and multi-part messages.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
... and add MaintenanceType::Delete to it. We also want to clear any
cach entries if we are deleting the datastore, not just if it is marked
as offline.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Selective because there are quite a few more such old-style format
strings, but I had those already adapted and currently do not have
time do clean-up tree-wide, but it's fine to change this
incrementally.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
That's rather excessive and has not much value for users. So degrade
two of the messages to debug-level.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we already have two different password schemas, `PBS_PASSWORD_SCHEMA`
being the stricter one, which ensures a minimum length of new
passwords. however, this wasn't used on the change password endpoint
before, so add it there too. this is also in-line with NIST's latest
recommendations [1].
[1]: https://pages.nist.gov/800-63-4/sp800-63b.html#passwordver
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
PathPatterns is hard to distinguish from PathPattern, so would need to be
renamed anyway.. but there isn't really a reason to define a separate API type
just for this.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Introduces a dedicated api type `PathPattern` and the corresponding
format and input validation schema. Further, add a `PathPatterns`
type for collections of path patterns and implement required traits
to be able to replace currently defined api parameters.
In preparation for using this common api type for all api endpoints
exposing a match pattern parameter.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
This option allows us to "fix" corrupt snapshots (and/or their chunks)
by pulling them from another remote. When traversing the remote
snapshots, we check if it exists locally, and if it is, we check if the
last verification of it failed. If the local snapshot is broken and the
`resync-corrupt` option is turned on, we pull in the remote snapshot,
overwriting the local one.
This is very useful and has been requested a lot, as there is currently
no way to "fix" corrupt chunks/snapshots even if the user has a healthy
version of it on their offsite instance.
Originally-by: Shannon Sterz <s.sterz@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Introduces a dedicated wrapper type to be used for backup archive
names instead of plain strings and associated helper methods for
archive type checks and archive name mappings.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
FG: use LazyLock for constant archive names reduces churn, and saves some
allocations
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Moving the `ArchiveType` to avoid crate dependencies on
`pbs-datastore`.
In preparation for introducing a dedicated `BackupArchiveName` api
type, allowing to set the corresponding archive type variant when
parsing the archive name based on it's filename.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In order for sync jobs to be either pull or push jobs, allow to
configure the direction of the job.
Adds an additional config type `sync-push` to the sync job config, to
clearly distinguish sync jobs configured in pull and in push
direction and defines and implements the required `SyncDirection` api
type.
This approach was chosen in order to limit possible misconfiguration,
as unintentionally switching the sync direction could potentially
delete still required snapshots.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a dedicated api type for the `version` api endpoint and helper
methods for supported feature comparison.
This will be used to detect api incompatibility of older hosts, not
supporting some features.
Use the new api type to refactor the version endpoint and set it as
return type.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Make the `BackupGroupDeleteStats` exposable via the API by implementing
the ApiTypes trait via the api macro invocation and add an additional
field to account for the number of deleted groups.
Further, add a method to add up the statistics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In preparation for the delete stats to be exposed as return type to
the backup group delete api endpoint.
Also, rename the private field `unremoved_protected` to a better
fitting `protected_snapshots` to be in line with the method names.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Adding the privileges to allow backup, namespace creation and prune
on remote targets, to be used for sync jobs in push direction.
Also adds dedicated roles setting the required privileges.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add `remote_acl_path` method which generates the acl path from the sync
job configuration. This helper allows to easily generate the acl path
from a given sync job config for privilege checks.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a `remote_acl_path` helper method for creating acl paths for
remote namespaces, to be used by the priv checks on remote datastore
namespaces for e.g. the sync job in push direction.
Factor out the common path extension into a dedicated method.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
For how to work on the crates in this workspace while actually working
on a separate project without having to constantly reinstall `.deb`
files.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The 'Anyhow' error is not useful and meant for throw-away errors which
cannot be dealt with anyway, and we'd like to be able to tell apart
network problems from actual HTTP responses, so that we can
potentially try a different node in a cluster connection.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
removable datastores will have a PBS-managed mountpoint as path, direct
access to the field needs to be replaced with a helper that can account
for this.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
I considered keeping it as log::trace level, but IMO that's just not
worth it, as just the peek_len is not giving one much more and can
also be basically also gathered through strace.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When a connection is closed by the client before we have enough data
to determine if it contains a TLS Handshake or not, the socket stays
in a readable state.
While we setup a tokio backed timeout of 10s for the connection
build-up here, this timeout does not trigger on said early connection
abort from the client side, causing then the async_io loop to
endlessly loop around peeking into the client, which always returns
the last available bytes before the connection was closed. This in
turn causes 100% CPU usage for one of the PBS threads.
The timeout not triggering is rather odd, and does indicate some
potential for further improvement in tokio itself, but our
questionable use of the WouldBlock error does violate the API
contract, so this is not a clear cut.
Such an early connection abort is often triggered by monitoring
solutions, which use it to relatively cheaply check if TCP on a port
still works as "is service up" heuristic.
To fix this, save the amount of bytes peek returned and if they did
not change between invocations of the callback, we can assume that the
connection was closed and thus exit the connection attempt with an
error.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: reword commit message and change error to ConnectionAborted ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
All in all pretty similar to other endpoint APIs.
One thing worth noting is how secrets are handled. We never ever
return the values of previously stored secrets in get_endpoint(s)
calls, but only a list of the names of all secrets. This is needed
to build the UI, where we display all secrets that were set before in
a table.
For update calls, one is supposed to send all secrets that should be
kept and updated. If the value should be updated, the name and value
is expected, and if the current value should preseved, only the name
is sent. If a secret's name is not present in the updater, it will be
dropped. If 'secret' is present in the 'delete' array, all secrets
will be dropped, apart from those which are also set/preserved in the
same update call.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-By: Stefan Hanreich <s.hanreich@proxmox.com>
This target type allows users to perform HTTP requests to arbitrary
third party (notification) services, for instance
ntfy.sh/Discord/Slack.
The configuration for these endpoints allows one to freely configure
the URL, HTTP Method, headers and body. The URL, header values and
body support handlebars templating to inject notification text,
metadata and secrets. Secrets are stored in the protected
configuration file (e.g. /etc/pve/priv/notification.cfg) as key value
pairs, allowing users to protect sensitive tokens/passwords.
Secrets are accessible in handlebar templating via the secrets.*
namespace, e.g. if there is a secret named 'token', a body
could contain '{{ secrets.token }}' to inject the token into the
payload.
A couple of handlebars helpers are also provided:
- url-encoding (useful for templating in URLs)
- escape (escape any control characters in strings)
- json (print a property as json)
In the configuration, the body, header values and secret values
are stored in base64 encoding so that we can store any string we want.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-By: Stefan Hanreich <s.hanreich@proxmox.com>
A recent commit [1] changed the `Display` implementation of `TimeSpan` such
that minutes are now displayed as `20m` instead of `20min`.
This commit adapts the tests for the notification template renderer
accordingly.
[1] 19129960 ("time: display minute/month such that it can be parsed again")
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
To avoid openssl's unhelpful error messages when the proxy.key or
proxy.pem files have the wrong permissions, we open the files. To load
the private key, we can simply read from the file and pass it to the
`set_private_key` openssl function. Sadly such a function does not exist
for loading certificate chains, so we have to open and close the file
before calling the `set_certificate_chain_file` fn.
Motivation: https://forum.proxmox.com/threads/proxmox-backup-tailscale-proxmox-backup-proxy-service-wont-boot.153204
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
In the reference Bugzilla entry, a certificate with an IP address as a
SAN was used. rustls seems to have problems with that [1].
Also, pretty much all of our code uses native-tls at the moment, so
it makes sense to not pull in a second TLS implementation.
Tested by rebuilding libpve-rs-perl and testing a Gotify notification
target with a self-signed TLS certificate (one that is accepted by
OpenSSL but not by rusttls).
[1] https://github.com/rustls/rustls/issues/184
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
A description is required for the API schema types and we fallback to
the rust doc-comment when no explicit one is set.
But a empty string was returned if no doc-comment existed, so check
for the comment to be non-empty and throw a compile-time error
otherwise.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The wrap_text helper accepts and initial indentation, so use that as
central point to add the indentation that glues the list entry
together with its description.
Mostly a small optimization, should not matter in practice, i.e. where
all properties should have a description.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
There was basically no documentation at all, so try to document the
basic format syntax and where it comes from. The text is partially
adapted from the systemd docs.
Could be still expanded with some example code and the methods might
do good with getting some docs too, but those parts can be relatively
easily be figured out from the code itself, the basic underlying
design and format background is much harder to guess..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Previously we displayed, e.g., "4m 1h 1min", i.e. using "m" for months
and "min" for minutes but "m" was not accepted as month when parsing
a timespan string, so a 4 month timespan would be printed "4m" but if
parsed again it would result in a timespan of 4 minutes.
So switch month to an uppercase "M" and minute to the lower case "m",
which makes renderings of common timespans nicer, as in most of our
use cases they are in the range of minutes to hours, sometimes days
but seldom longer than weeks. So using single letters for all but
"min" stuck out quite a bit, e.g.: "1h 5min 2s" looks odd compared to
"1h 5m 1s"
While the duplicate letter is not 100% ideal it's still better than
the status quo, where rendering and parsing would interpret things
differently.
Also, the order still makes it quite clear, e.g.:
"7m 2w 3d 1h 5min 44s" now becomes "7M 2w 3d 1h 5m 44s"
As a side effect this also brings the display format closer to what is
used inside PVE backup job taks logs.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Seconds are not displayed when the value is smaller than 0.1s and
they are not at the start of the display output, e.g. `1h 2m`.
Drop the additional whitespace currently appended for this edge case.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
(conveted with pandoc and some minor manual fixups)
Mention that the crates should activate `doc_cfg, doc_auto_cfg` and
ideally `#[deny(unsafe_op_in_unsafe_fn)]` and `#[deny(missing_docs)]`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Object schemas can now declare a field which causes
'additional_properties' to be set to true and the field being ignored
in the schema.
This allows adding a flattened HashMap<String, Value> to gather the
additional unspecified properties.
#[api(additional_properties: "rest")]
struct Something {
#[serde(flatten)]
rest: HashMap<String, Value>,
}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
So we can get going on the wasm side where we don't yet have access to
the webauthn-rs crate.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
To ease development on new machines, this provides an easy way to just
do
# apt install $(make list-packages)
to get started.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Print error if the parsing of the env-var fails on the proxmox-backup-*
daemons as well. Output correct env-var on binaries that use different
variables.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Don't panic when the syslog is not available - which happens commonly in
containers and sbuild environments (chroot and unshare) - instead
fallback to stderr.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>
The following targets should be self-explanatory:
# make install
$ make DESTDIR=$SOME_PATH install
# make proxmox-sys-install
$ make DESTDIR=$SOME_PATH proxmox-sys-install
# make proxmox-<any other crate name>-install
Additionally, these are used as building blocks to create
systemd-sysext(8) images:
$ make proxmox-sys-sysext
builds an `extensions/proxmox-sys.raw`
This can be copied/symlinked to `/run/extensions/` and then activated.
As root:
# ln -s $REPO_DIR/extensions/proxmox-sys.raw /run/extensions/
# systemd-systext refresh
For the complete workspace, an `extensions/proxmox-workspace.raw` can
be built via
$ make sysext
This also takes a `CRATES` var to limit the crates which should be
included, and takes an optional `NOCLEAN=1` which prevents cleaning
out the previously installed to "add" new crates on the go:
Assuming there's a symlink like:
# ln -s $REPO_DIR/extensions/proxmox-workspace.raw /run/extensions/proxmox-workspace.raw
One can modify the installed crates like this:
$ make CRATES=proxmox-sys sysext
$ sudo systemd-sysext refresh
Now only the current proxmox-sys crate is overridden.
$ make NOCLEAN=1 CRATES=proxmox-time sysext
$ sudo systemd-sysext refresh
Now proxmox-sys as well as proxmox-time are installed.
To undo the changes, either just do, as root:
# systemd-sysext unmerge
or remove the files which should specifically be dropped from
/run/extensions/ and run as root:
# systemd-sysext refresh
Another way to temporarily install single crates is to just have the
extensions/ folder *be* the `/run/extensions` folder:
# rmdir /run/extensions
# ln -s $REPO_DIR/extensions /run/extensions
Then just build individual extensions:
$ make proxmox-sys-sysext
$ sudo systemd-sysext refresh
$ make proxmox-router-sysext
$ sudo systemd-sysext refresh
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The get/put/post/put_without_body/... methods now have a default
implementation forwarding to a generic `request` method as all our
implementations do the same already anyway.
Additionally, in order to allow easy access to a "streaming body", the
Body type is now exposed.
In the future, this crate may also require a wrapper to standardize
the handling of `application/json-seq` streams if we end up using
them, but for now, a simple way to expose the body is enough to get
going.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These are Iterators or Streams which continuously produce output. They
can either be formatted, in which they are serialized like the as
usually, or, if the client caccepts `application/json-seq` via an
`Accept` header, it will be streamed as a sequence directly.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This does not "stream", but rather skips the intermediate step to
serialize the entire output into a local json string.
We now reserve the "Stream*" prefix for actual *streaming*, that is,
producing an API response which gets streamed continuously as it is
asynchronously produced.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Previously when using `env_logger` all of our cli-tools logged to
stderr, make tracing do the same. Export `tracing::Level` so that we can
use the `tracing::enabled!` macro.
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Deprecate the proxmox-router init_cli_logger function used in client
binaries such as `proxmox-backup-client`, `proxmox-backup-manager`,
'pxar', etc... Add a new init_cli_logger function that uses tracing
instead of env_logger. It checks if the task is in a workertask and
prints the message either to stdout or to the tasklog (this is
neccessary for commands in proxmox-backup-manager that call api handlers
that start workerthreads from the client).
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
If we don't do this, then properties from a serde flattened struct will
be positioned at the end of the list, rather than properly sorted with
the other properties.
Since the tests also feature non-sorted properties, we have to adapt
them too.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The rate and burst parameters are integers, so the mapping from value
with `.as_str()` will always return `None` effectively never
applying any rate limit at all.
Fix it by turning them into a HumanByte instead of an integer.
To not crowd the parameter section so much, create a
ClientRateLimitConfig struct that gets flattened into the parameter list
of the backup client.
To adapt the description of the parameters, add new schemas that copy
the `HumanByte` schema but change the description.
With this, the rate limit actually works, and there is no lower limit
any more.
The old TRAFFIC_CONTROL_RATE/BURST_SCHEMAs can be deleted since the
client was the only user of them.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
While this is technically a breaking API change since the trait is
public, we don't implement it anywhere and it isn't meant to be
implemented from the outside.
Also, encode that these types are all Send + Sync via a super trait
notation.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes:
warning: unresolved link to `seccfg`
--> proxmox-section-config/src/typed.rs:18:71
|
18 | /// If the [`SectionConfig`] returned by the [`section_config()`][seccfg] method includes the
| ^^^^^^ no item named `seccfg` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
= note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default
warning: unresolved link to `seccfg`
--> proxmox-section-config/src/typed.rs:22:10
|
22 | /// [seccfg] ApiSectionDataEntry::section_config()
| ^^^^^^ no item named `seccfg` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: doc list item missing indentation
--> proxmox-access-control/src/acl.rs:518:9
|
518 | /// -- user/token is more specific than group at each level
| ^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
= note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
|
518 | /// -- user/token is more specific than group at each level
| ++
warning: doc list item missing indentation
--> proxmox-access-control/src/acl.rs:519:9
|
519 | /// -- roles lower in the tree are more specific than those higher up along the path
| ^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
help: indent this line
|
519 | /// -- roles lower in the tree are more specific than those higher up along the path
| ++
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: match can be simplified with `.unwrap_or_default()`
--> proxmox-apt/src/cache_api.rs:77:28
|
77 | let mut notified = match cache.notified {
| ____________________________^
78 | | Some(notified) => notified,
79 | | None => std::collections::HashMap::new(),
80 | | };
| |_________^ help: replace it with: `cache.notified.unwrap_or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
= note: `#[warn(clippy::manual_unwrap_or_default)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: this expression creates a reference which is immediately dereferenced by the compiler
--> proxmox-router/src/cli/completion.rs:154:25
|
154 | &completion_functions,
| ^^^^^^^^^^^^^^^^^^^^^ help: change this to: `completion_functions`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
= note: `#[warn(clippy::needless_borrow)]` on by default
warning: this expression creates a reference which is immediately dereferenced by the compiler
--> proxmox-router/src/cli/completion.rs:201:21
|
201 | &completion_functions,
| ^^^^^^^^^^^^^^^^^^^^^ help: change this to: `completion_functions`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
API tokens between rust & perl code bases are inconsistent... this
needs fixing, but for now this is faster and more compatible.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The original typed section config data would insert and remove the
type properties. With the introduction of `.with_type_key()` this is
done on the parse/write side instead, so we need to be able to opt out
of this.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Before, a call to `update` was necessary to load an existing database
into the cache. If `update` was never called, `extract_cached_data`
would simply return no data.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This function is the same as the regular `update_value`, but it
sets the `new_only` flag when updating the value in the rrd map.
This avoids "time in past" messages being logged in case a data point
happens to be added twice. This new function will just silently reject
values that have an older timestamp than the most recent one.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This crate contains a file-backed, rotating cache.
The cache should be safe to be accessed from multiple processes at
once.
The cache stores the value at the provided path.
If `keep_old` is >0, the cache will keep up to `keep_old` versions
around which can be queried via the `get_last` method.
The value and its history are stored in the same file,
each generation represented by a single line of JSON.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Under the hood, this function calls `mkdtemp` from libc. Unfortunatly
the nix crate did not provide bindings for this function, so we have
to call into libc directly.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the cargo docs warning:
warning: unresolved link to `systemd_notify`
--> proxmox-daemon/src/server.rs:314:6
|
314 | /// [systemd_notify] with [SystemdNotify::Ready](proxmox_systemd::notify::SystemdNotify) when the
| ^^^^^^^^^^^^^^ no item named `systemd_notify` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
= note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the cargo docs warning:
warning: unclosed HTML tag `username`
--> proxmox-login/src/api.rs:35:47
|
35 | /// realm is simply added to the username <username>@<relam>.
| ^^^^^^^^^^
|
= note: `#[warn(rustdoc::invalid_html_tags)]` on by default
warning: unclosed HTML tag `relam`
--> proxmox-login/src/api.rs:35:58
|
35 | /// realm is simply added to the username <username>@<relam>.
| ^^^^^^^
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the cargo doc warnings:
warning: redundant explicit link target
--> proxmox-http/src/client/mod.rs:4:19
|
4 | //! in [`Client`](crate::client::Client).
| -------- ^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant
| |
| because label contains path that resolves to same destination
|
= note: when a link's destination is not specified,
the label is used to resolve intra-doc links
= note: `#[warn(rustdoc::redundant_explicit_links)]` on by default
help: remove explicit link target
|
4 | //! in [`Client`].
| ~~~~~~~~~~
warning: redundant explicit link target
--> proxmox-http/src/client/mod.rs:7:22
|
7 | //! [`sync::Client`](crate::client::sync::Client).
| -------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant
| |
| because label contains path that resolves to same destination
|
= note: when a link's destination is not specified,
the label is used to resolve intra-doc links
help: remove explicit link target
|
7 | //! [`sync::Client`].
| ~~~~~~~~~~~~~~~~
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the clippy lint:
warning: bound is defined in more than one place
--> proxmox-schema/src/property_string.rs:352:14
|
352 | pub fn parse<T: ApiType>(value: &str) -> Result<T, Error>
| ^
353 | where
354 | T: for<'de> Deserialize<'de>,
| ^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#multiple_bound_locations
= note: `#[warn(clippy::multiple_bound_locations)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the clippy warning:
warning: doc list item missing indentation
--> proxmox-subscription/src/subscription_info.rs:179:9
|
179 | /// (this mode is used to decide whether to refresh the subscription information)
| ^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
= note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
|
179 | /// (this mode is used to decide whether to refresh the subscription information)
| +
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the clippy warnings:
warning: `proxmox-apt` (lib) generated 1 warning
warning: using `clone` on type `Option<[u8; 32]>` which implements the `Copy` trait
--> proxmox-apt/tests/repositories.rs:117:22
|
117 | let old_digest = file.digest.clone().unwrap();
| ^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `file.digest`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
= note: `#[warn(clippy::clone_on_copy)]` on by default
warning: using `clone` on type `[u8; 32]` which implements the `Copy` trait
--> proxmox-apt/tests/repositories.rs:135:24
|
135 | file.digest = Some(old_digest.clone());
| ^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `old_digest`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
For when the underlying datatype is supposed to contain the type
property and the schema does not mark it as optional.
The use case here is to support flat `Remote` type where the "type" of
pve/pmg/pbs is a property which is present in the `Remote` struct
while being derived from the section type.
This will implicitly include and strip the type of the json object
after/before de/serializing.
Alternatives would be
- to mark the type as optional and just fill it out later when loading
the data, but that is technically wrong...
- have a 2nd version of the struct with the type field removed and
From/Into implemented, but that's even more unwieldy.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
`OwnedFd`s are now (rustc 1.80+) checked for validity when dropped in a debug
build, to catch usage after closing. Unfortunately those checks don't account
for the special value `AT_FDCWD` (-100) which is not a "real" FD, but a magic
constant used by many libc functions to signify operations starting at the
current working directory.
changing our `cwd` helper to open the CWD for real, instead of just returning
the magic value that pretends to be an FD, works around those limitations with
the least API churn.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
For regular error cases, `std::io::Error` can now also box errors and
provides an `Error::other()` function since rust 1.74.
For the case where it was used directly with strings
(io_err_other("string")) -> that's what `io_format_err!()` and
`io_bail!()` are for.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These structs are returned by the public method
`Deflate{En,De}coder::builder`.
Reported-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
While technically an API break, we don't use the public API for this
anywhere and the trait we're changing is explicitly marked as
`#[doc(hidden)]`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The `_application_name` argument is not used anymore. It was previously
used to set the correct journald unit-name, which is now handled
automatically.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Used only for read/writing a pid_t (an integer) to a socket.
The standard to_ne_bytes()/from_ne_bytes() should be sufficient here,
we already have libc::pid_t which we can use to get the correct type
namespace for accessing ::from_ne_bytes().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
When reaching the final command we relied on the positional parameters
"ending" the global option parsing. This means we did not get global
options at the end, and failed with "unknown option" instead.
Fix this by simply retaining unknown options in that case and not
stopping at positional parameters.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reorder the filters for the journald layer. This sets the LevelFilter
last, which means tracing can disable all log statements lower than the
current level without evaluating the LogContext::exists function.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[ TL: note that this is just an optimization in the subject ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Deduplicate the check for whether the argument exists by trying to
fetch the schema once and use check the option instead of calling
'.contains_key()' multiple times.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Passing the &GlobalOptions through is more telling than an opaque
Iterator<&str>...
Also: actually generate the property descriptions in non-ReST mode for
global options as well, so the `help` output for a specific command
includes the property documentation instead of only showing the name.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The rules are as follows:
- An "item" by itself does neither start nor end with a newline.
- Where items are connected, the appropriate amount of newlines must
be inserted, assuming items do not have any trailing newlines.
Otherwise this just gets WAY too confusing everywhere!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This is completely wrong and make working with it extremely annoying.
Whether or not there should be separation should be decided where
multiple elements are connected, they shouldn't automatically come
with a bunch of trailing new lines for absolutely no reason.
Places using this will need to be fixed as they get noticed.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This one does *explicitly* *not* support long options with a single
dash because it is too ambiguous if we want to add support for short
options at some point.
The parsing of the command line and invoking of the command is
separated. `CommandLine::parse` returns an `Invocation` which is
called and consumed via its `call` method.
This allows updating the CLI environment between parsing and invoking
the command, in order to allow *handling* the global options in
between those two steps if desired.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The CLI environment can now contain ApiType structs which can be
accessed by their type.
The TypeId is used since the options inside must be unique anyway and
we can't have the same type specified multiple times. It also makes
for a somewhat convenient interface:
env.take_global_option::<ConnectInfo>()
where ConnectInfo is a struct containing the server, user, port, ...
since these will not be passed as *parameters* to the API functions.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The scope and sync_scope methods simply activate the context, they
don't affect the counter, the counter is initialized when creating the
context with LogContext::new().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Since hyper can spawn() more tasks, when we stop passing `WorkerTask`
references down the stack, we still need to be able to *inherit* the
current logging context. Hyper provides a way to replace its used
`spawn()` method, so we need to provide a way to reuse the logging
context.
Instead of having the `FileLogger` and warn counter separately
available with local-only access, put them behind an Arc<Mutex<>>.
Previously they already *were* behind an Arc<Mutex<>> as part of the
WorkerTaskState.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
proxmox_notify is the only user of those functions, so it makes
sense to move them here. A future commit will mark the
original functions from proxmox_sys as deprecated.
The functions were slightly modified, mostly to not
rely on anyhow for error reporting. Also they
are now private functions.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Enable the tracing-system by setting the LOGGER task local variable
to a instance of a FileLogger and initializing the WARN_COUNTER.
Removed the task_log! macros and some occurences.
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[WB: remove flog! import in doctests]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Add the `proxmox_log` crate which includes the new logging infra.
Export the `init_logger` function, which creates the `tracing` logger
that includes the default subscriber and two layer.
The first layer comes from the tracing-journald crate and logs
everything that does not come from a worker-task/thread to the syslog.
The second layer filters the exact opposite and writes the logs into the
corresponding task-log file.
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
On rare occasions, the TLS "client hello" message [1] is delayed after
a connection with the server was established, which causes HTTPS
requests to fail before TLS was even negotiated. In these cases, the
server would incorrectly respond with "HTTP/1.1 400 Bad Request"
instead of closing the connection (or similar).
The reasons for the "client hello" being delayed seem to vary; one
user noticed that the issue went away completely after they turned off
UFW [2]. Another user noticed (during private correspondence) that the
issue only appeared when connecting to their PBS instance via WAN, but
not from within their VPN. In the WAN case a firewall was also
present. The same user kindly provided tcpdumps and strace logs on
request.
The issue was finally reproduced with the following Python script:
import socket
import time
HOST: str = ...
PORT: int = ...
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT))
time.sleep(1.5) # simulate firewall / proxy / etc. delay
sock.sendall(b"\x16\x03\x01\x02\x00")
data = sock.recv(256)
print(data)
The additional delay before sending the first 5 bytes of the "client
hello" message causes the handshake checking logic to incorrectly fall
back to plain HTTP.
All of this is fixed by the following:
1. Increase the timeout duration to 10 seconds (from 1)
2. Instead of falling back to plain HTTP, refuse to accept the
connection if the TLS handshake wasn't initiated before the
timeout limit is reached
3. Only accept plain HTTP if the first 5 bytes do not correspond to
a TLS handshake fragment [3]
4. Do not take the last number of bytes that were in the buffer into
account; instead, only perform the actual handshake check if
5 bytes are in the peek buffer using some of tokio's low-level
functionality
Regarding 1.: This should be generous enough for any client to be able
to initiate a TLS handshake, despite its surrounding circumstances.
Regarding 4.: While this is not 100% related to the issue, peeking into
the buffer in this manner should ensure that our implementation here
remains correct, even if the kernel's underlying behaviour regarding
edge-triggering is changed [4]. At the same time, there's no need for
busy-waiting and continuously yielding to the event loop anymore.
[1]: https://www.rfc-editor.org/rfc/rfc8446.html#section-4.1.2
[2]: https://forum.proxmox.com/threads/disable-default-http-redirects-on-8007.142312/post-675352
[3]: https://www.rfc-editor.org/rfc/rfc8446.html#section-5.1
[4]: https://lwn.net/Articles/864947/
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
This adds the structs `AcceptState` and `AcceptFlags` and adapts
relevant method signatures of `AcceptBuilder` accordingly. This makes
it easier to add further parameters in the future.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
The Backup Server can compress the content using deflate so we teach the
client how to decode it.
If a request is sent with the `Accept-Encoding` [2] header set to
`deflate`, and the response's `Content-Encoding` [1] header is equal to
`deflate` we wrap the Body stream with a stream that can decode `zlib`
on the run.
Note that from the `Accept-Encoding` docs [2], the `deflate` encoding is
actually `zlib`.
This can be also tested against
http://eu.httpbin.org/#/Response_formats/get_deflate by adding the
following test:
```rust
#[tokio::test]
async fn test_client() {
let client = Client::new();
let headers = HashMap::from([(
hyper::header::ACCEPT_ENCODING.to_string(),
"deflate".to_string(),
)]);
let response = client
.get_string("https://eu.httpbin.org/deflate", Some(&headers))
.await;
assert!(response.is_ok());
}
```
at `proxmox-http/src/client/simple.rs` and running
```
cargo test --features=client,client-trait
```
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
Suggested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
We test the deflate encoder against the deflate decoder using (or not)
zlib and with different small buffer sizes. We also test compression and
decompression against the flate2 crate.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
Because it was only used for the test setup. Instead, we simply
add an apt_lists_dir parameter where we need it.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
clippy rightfully complains about a create() with an unspecified
truncation behavior. This file has no contents so let's just not
truncate it in case we ever want to also have data in it...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The code chooses whichever one of a multitude of functions returns
Some, switching to .map for the final else would make it less
readable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes the following clippy warning:
warning: returning the result of a `let` binding from a block
--> proxmox-access-control/src/acl.rs:687:13
|
686 | let config = TestAcmConfig { roles };
| ------------------------------------- unnecessary `let` binding
687 | config
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#let_and_return
= note: `#[warn(clippy::let_and_return)]` on by default
help: return the expression directly
|
686 ~
687 ~ TestAcmConfig { roles }
|
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: the following explicit lifetimes could be elided: 'a
--> proxmox-acme/src/async_client.rs:65:30
|
65 | pub async fn new_account<'a>(
| ^^
66 | &'a mut self,
| ^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
= note: `#[warn(clippy::needless_lifetimes)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: casting to the same type is unnecessary (`usize` -> `usize`)
--> proxmox-http/src/websocket/mod.rs:446:40
|
446 | mask.copy_from_slice(&data[mask_offset as usize..payload_offset as usize]);
| ^^^^^^^^^^^^^^^^^^^^ help: try: `mask_offset`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_cast
= note: `#[warn(clippy::unnecessary_cast)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy error:
error: redundant redefinition of a binding `data`
--> proxmox-http/src/websocket/mod.rs:375:9
|
375 | let data = data;
| ^^^^^^^^^^^^^^^^
|
help: `data` is initially defined here
--> proxmox-http/src/websocket/mod.rs:369:27
|
369 | pub fn try_from_bytes(data: &[u8]) -> Result<Option<FrameHeader>, WebSocketError> {
| ^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_locals
= note: `#[deny(clippy::redundant_locals)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: type parameter `T` goes unused in function definition
--> proxmox-shared-memory/tests/raw_shared_mutex.rs:80:19
|
80 | fn create_test_dir<T: Init>(filename: &str) -> Option<PathBuf> {
| ^^^^^^^^^ help: consider removing the parameter
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#extra_unused_type_parameters
= note: `#[warn(clippy::extra_unused_type_parameters)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: initializer for `thread_local` value can be made `const`
--> proxmox-router/src/cli/command.rs:221:71
|
221 | static HELP_CONTEXT: RefCell<Option<Arc<CommandLineInterface>>> = RefCell::new(None);
| ^^^^^^^^^^^^^^^^^^ help: replace with: `const { RefCell::new(None) }`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#thread_local_initializer_can_be_made_const
= note: `#[warn(clippy::thread_local_initializer_can_be_made_const)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: casting raw pointers to the same type and constness is unnecessary (`*mut fs::acl::libc::c_void` -> `*mut fs::acl::libc::c_void`)
--> proxmox-sys/src/fs/acl.rs:130:23
|
130 | let mut ptr = ptr::null_mut() as *mut c_void;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `ptr::null_mut()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_cast
= note: `#[warn(clippy::unnecessary_cast)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the following clippy warnings:
warning: the borrowed expression implements the required traits
--> proxmox-tfa/src/api/recovery.rs:86:24
|
86 | Ok(hex::encode(&hmac))
| ^^^^^ help: change this to: `hmac`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrows_for_generic_args
and
warning: this expression creates a reference which is immediately dereferenced by the compiler
--> proxmox-network-api/src/api_impl.rs:108:47
|
108 | interface.set_bond_slave_list(&slaves)?;
| ^^^^^^^ help: change this to: `slaves`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
= note: `#[warn(clippy::needless_borrow)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy lints:
warning: unnecessary use of `get("lo").is_none()`
--> proxmox-network-api/src/config/parser.rs:603:30
|
603 | if config.interfaces.get("lo").is_none() {
| ------------------^^^^^^^^^^^^^^^^^^^
| |
| help: replace it with: `!config.interfaces.contains_key("lo")`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_get_then_check
= note: `#[warn(clippy::unnecessary_get_then_check)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: use of `unwrap_or_else` to construct default value
--> proxmox-tfa/src/api/mod.rs:1355:43
|
1355 | |cap| cap.map(Vec::with_capacity).unwrap_or_else(Vec::new),
| ^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `unwrap_or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unwrap_or_default
= note: `#[warn(clippy::unwrap_or_default)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
the old location has been deprecated for a while, and rustc 1.78 will start to warn about it.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this way the types defined in this crate can be re-used in places
without necessarily having to use the ACL, token shadow and
(cached) user config implementations.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
since `CachedUserInfo` takes care of both, the user config and the acl
config, we need to also bump the cache generation when storing the
acl config.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this commit factors out the user config. it also add two new functions
to the `AccessControlConfig` trait to handle caching in a more
generalized way.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this commit factors out the token shadow implementation from
`proxmox-backup` so it can be used in other products.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
these types are used by the user config in `proxmox-backup` server.
this commit factors them out so we can re-use them in other products
as well as this crate.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this commit factors out the acl tree from proxmox-backup so we can
re-use it accross other products. to use it, the product needs to
implement the `AcmConfig` trait and provide this crate with a
location to safe its configuration files.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Add confirmation helper that outputs a prompt and lets the user
confirm or deny it.
Implemented to close#4763.
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
as a first step of improving our error handling story, printing context
and causes if the error contains them.
The downside to adding context is that the default Display implementation
will *just* print the context, which hides the root cause. This is why
we print the errors using the pretty-print formatter in this change.
Originally-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[WB: prefix commit message with crate]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
under autopkgtest, the rundir is not writable, but cargo gives us a tmpdir that
we can use in all cases.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Some users might want to switch to using only the newer .sources files
already, which Debian is going to switch to in the long run.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Normally root, but can be the same as the api_user if the product
does not use priviledge separation.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
This was an old version of a const-fn compatible checked c_str
implementation which was never enabled.
When we get rust 1.72, `CStr::from_bytes_with_nul` becomes usable in
const contexts.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Extract the URI creation for write and health URIs. Add unit test to
test the encoding of special characters in the organization and bucket
parameters.
Follow-up-to: bfa73aad ("metrics: encode influxdb org and bucket parameters")
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
FG: downgraded form_urlencoded version to packaged one
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
due to missing `use` statements they failed, as they should, but for
the wrong reasons. also adapt a test case that presumably was meant
to test whether `TokennameRef` can be compared, but instead
duplicated the `UsernameRef` test case.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
by using `openssl::memcmp::eq()` we can avoid potential timing side
channels as its runtime only depends on the length of the arrays, not
the contents. this requires the two arrays to have the same length, but
that should be a given since the hashes should always have the same
length.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we used `sha256scrypt` for password hashing. while this may
by safe if used with the correct parameters, we used the default
parameters which are considered unsafe. according to `man crypt(5)`:
> The default CPU time cost parameter is 5000, which is too low for
> modern hardware.
hence, we needed to adapt this code anyway. conveniently, verification
with crypt also works for older hashes as the parameters for the
hashing function are encoded in the output of crypt. so this is a drop
in replacement that will simply use yescrypt for new hashes while
old hashes will still verify properly.
this commit also adds a wrapper for `crypt_gensalt_rn` to more easily
generate correctly formatted salt strings. this is also useful for
switching the cpu time hardness parameter, as otherwise we'd need to
encode that ourselves.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we used our own hmac-like implementation for csrf token
signing that simply appended the key to the message (csrf token).
however, this is possibly insecure as an attacker that finds a
collision in the hash function can easily forge a signature. after all,
two messages would then produce the same start conditions before
hashing the key. while this is probably a theoretic attack on our csrf
implementation, it does not hurt to move to the safer standard hmac
implementation that avoids such pitfalls.
this commit re-uses the hmac key wrapper used for the keyring. it also
keeps the old construction around so we can use it for a transition
period between old and new csrf token implementations.
this is a breaking change as it changes the signature of the
`csrf_secret` method of the `AuthContext` trait to return an hmac
key.
also exposes `assemble_csrf_prevention_toke` so we can re-use this
code here instead of duplicating it in e.g. proxmox-backup's
auth_helpers.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
by using openssl's `memcmp::eq()` we can avoid potential side-channel
attack on the csrf token comparison. this comparison's runtime only
depends on the length of the two byte vectors, not their contents.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we only used asymmetric cryptographic schemes to
authenticate tickets. this is fairly costly and not necessary in every
instance. imagine a service that runs as a single daemon. this daemon
is then the only party that needs to sign and verify tickets. this
makes hmac perfectly suitable for such usecases. hmac has some
advantages over asymmetric schemes:
- much simpler and well reviewed construction
- much faster and better optimized crypto primitives (hash functions)
this commit first introduces a new hmac key wrapper that uses openssl's
hmac implementation and can easily be reused by other parts of the
code. it also refactors the keyring code to make it easier to rotate
new hmac keys into place so switching to hmac keys is easier.
hmac keys are symmetric, so the verification key is the same key as the
signing key. this breaks the previous assumption by the keyring that
these correspond to public and private keys. thus, this commit
introduces two wrapper enums to distinguish between hmac and asymmetric
signature schemes.
the verification of hmac keys is also done via `openssl::memcmp::eq()`
to avoid potential timing side-channel attacks.
below are some simple benchmarks done with criterion.rs to show how much
faster hmac is, no matter the actual hash function:
rsa 4096 + sha256 time: [2.7825 ms 2.7907 ms 2.7995 ms]
ed25519 time: [94.411 µs 94.840 µs 95.324 µs]
hmac sha256 time: [5.7202 µs 5.7412 µs 5.7645 µs]
hmac sha384 time: [6.6577 µs 6.6780 µs 6.7006 µs]
hmac sha3_256 time: [5.6930 µs 5.7114 µs 5.7322 µs]
rsa with 4096 bit keys and a sha256 digest is our current default. the
test itself consists of a single sign + verification cycle. criterion
repeats this test as it sees fit to arrive at the above numbers.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we used P-256 as the curve of our choice for ec signatures.
however, in the meantime Ed25519 has become a lot more wide-spread.
this simplifies our ec generation code significantly while keeping the
same security level. Ed25519 was also specifically designed and
reviewed to avoid implementation errors likely making it a more secure
choice
note that Ed25519 as a signature scheme always uses sha512, so signing
or verifying with a chosen digest is not supported.
as this mostly affects newly generated keys, this should not break any
existing setups.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
this commit moves the current ticket signing code into the private key
implementation. the upside is that the caller does not need to deal
with openssl's `Signer` directly. it also simplifies and unifies the
code by using the same helper for verifying a signature and creating it.
also derive `Clone` on `PrivateKey` and `PublicKey`. as they are
essentially thin wrappers around `openssl::pkey::PKey<Private>` and
`openssl::pkey::PKey<Public>`, which can be cloned, deriving `Clone`
just makes them easier to use.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Since we don't query each drives status seperately, but rely on a single
call to the drives listing parameter for that, we now add the option
to query the activity there too. This makes that data avaiable for us
to show in a seperate (by default hidden) column.
Also we show the activity in the 'State' column when the drive is idle
from our perspective. This is useful when e.g. an LTO-9 tape is loaded
the first time and is calibrating, since that happens automatically.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we use the VHF part from the DT Device Activity page for that.
This is intended to query the drive for it's current state and activity.
Currently only the activity is parsed and used.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and show them on the ui. This can help uses with seeing how much a tape
is used.
The value is updated on 'commit' and when the tape is changed during a
backup.
For drives not supporting the volume statistics, this is simply skipped.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
So that we can use the Interface struct with create and update api calls (which
currently use 'iface' instead of 'name').
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Remove the regex for influxdb organizations and buckets. Influxdb does
not place any constraints on these names and allows all characters. This
allows influxdb organization names with slashes.
Also remove a duplicate comment and add some missing ones.
This also aligns the behavior to PVE as there are no restrictions there
either.
The motivation for this patch is this forum post:
https://forum.proxmox.com/threads/influx-db-organization-doesnt-allow-slash.145402/
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
In order to remove the current limitations on the bucket and
organization names, we need to make sure that they are transmitted
correctly. In order to do this, we encode them using the url crate.
This way we support organization/bucket names that include slashes,
whitespaces, etc.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
* Add vlan_id and vlan_raw_device fields to the Interface api type
* Write to the network config the vlan specific properties for vlan
interface type
* Add several tests to verify the functionally
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Basically just a thin wrapper over the existing LDAP-based realm sync
job, which retrieves the appropriate config and sets the correct user
attributes.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Same as with datastores, this option determines whether we send
notifications the old way (send email via sendmail to a user's email
address) or the new way (emit matchable notification events to the
notification stack).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This one lets the user choose between the old notification behavior
(selecting an email address/user and always/error/never behavior per
datastore) and the new one (emit notification events to the
notification system)
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
proxmox-schema will automatically append text (e.g. 'Can be specified
more than once'), so we should end every comment with a '.'.
Also copy over some text from PVE docs, since these doc comments will
now be visible in the PBS documentation.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
PBS sends notifications for all events but successful prune jobs.
There we only care about errors.
This commit adapts the 'default-matcher' to reflect that behavior
as well.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
This is needed because we want to access CONFIG and PRIVATE_CONFIG
from the docgen helper in PBS.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
The old notification stack in proxmox-backup includes the nodename, so
we include it here as well.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
The API endpoints in Proxmox Backup Server require ApiType to be
implemented for any deserialized parameter.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
This method allows us to get a list of all notification targets.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
We need this for queuing notifications on PBS from the unprivileged
proxy process.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
It uses proxmox_sys::nodename - the dep is needed, otherwise the code
does not compile in some feature flag permutations.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Instead of passing the template strings for subject and body when
constructing a notification, we pass only the name of a template.
When rendering the template, the name of the template is used to find
corresponding template files. For PVE, they are located at
/usr/share/proxmox-ve/templates/default. The `default` part is
the 'template namespace', which is a preparation for user-customizable
and/or translatable notifications.
Previously, the same template string was used to render HTML and
plaintext notifications. This was achieved by providing some template
helpers that 'abstract away' HTML/plaintext formatting. However,
in hindsight this turned out to be pretty finicky. Since the
current changes lay the foundations for user-customizable notification
templates, I ripped these abstractions out. Now there are simply two
templates, one for plaintext, one for HTML.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
to avoid drifting definitions and reduce duplication. with the next major
release, the 'upid' field could then be renamed and aliased to be in line with
the other jobs, which all use 'last-run-upid'. doing it now would break
existing callers of the GC status endpoint (or consumers of the on-disk status
file).
the main difference is that the GC status fields are now not optional (except
for the UPID) in the job status, since flattening an optional value is not
possible. this only affects datastores that were never GCed at all, and only
direct API consumers, since the UI handles those fields correctly.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Adds an api endpoint on the datastore that reports the gc job status
such as:
- Schedule
- State (of last run)
- Duration (of last run)
- Last Run
- Next Run (if scheduled)
- Pending Chunks (of last run)
- Pending Bytes (of last run)
- Removed Chunks (of last run)
- Removed Bytes (of last run)
Adds a dedicated endpoint admin/gc that reports gc job status for all
datastores including the onces without a gc-schedule.
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
Originally-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewd-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Maintenance mode Delete locks the datastore. It must not be possible to go
back to normal modes, because the datastore may be in undefined state.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
it seems there are repositories out there that don't (always) include
it, and while it is required for the .deb packages themselves in Debian,
the repository "spec" doesn't make it mandatory.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it seems there are repositories out there that don't (always) include
it, and while it is required for the .deb packages themselves in Debian,
the repository "spec" doesn't make it mandatory.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Versions of Gotify < 2.2.0 only supported the 'X-Gotify-Key' header
for passing the API token. This comment sets this header in addition
to the regular 'Authorization' header in order to be compatible with
older Gotify servers.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
The root DSE holds common attributes about the LDAP server itself.
Needed to e.g. support Active Directory-based LDAP servers to retrieve
the base DN from the server itself, based on an valid bind.
See also RFC 4512, Section 5.1 [0] for more information about this
special object.
[0] https://www.rfc-editor.org/rfc/rfc4512#section-5.1
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
The `attrs` parameter of `Ldap::search()` is an `impl AsRef<[impl
AsRef<str>]>` anyway, so replace `vec![..]` with `&[..]`.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
We keep a DataStore cache, so ChunkStore's and lock files are kept by
the proxy process and don't have to be reopened every time. However,
for specific maintenance modes, e.g. 'offline', our process should not
keep file in that datastore open. This clears the cache entry of a
datastore if it is in a specific maintanance mode and the last task
finished, which also drops any files still open by the process.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Macro rules are not hygienic, and current rust macro visibility rules
are a nightmare. Using const_format::concatcp!() is a much cleaner
solution.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Some types were recently renamed but the examples not updated
accordingly.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 2f94283367 "rrd: spell out hard to understand abbreviations in public types"
rather than just always allowing additional properties, only return
true if any of the available schemas allows it
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Allow more complex strings for the acr-value when using openid. The
openid documentation only specifies the acr-value *should* be an URI
[0]. Implemented a regex that loosely disallows some of the reserved
URI characters specified in the RFC [1].
Currently values like:
- "urn:mace:incommon:iap:silver"
- "urn:comsolve.nl:idp:contract:rba:location"
do NOT work, although they are correct URI's and common acr tokens.
For Proxmox VE we had to actually make this more strict to align with
each other, as there we accepted any string.
[0]: https://openid.net/specs/openid-connect-core-1_0.html
[1]: https://www.rfc-editor.org/rfc/rfc2396.txt
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
A 'oneOf' schema is basically exactly what a rust `enum` is.
Exactly one of the possible values must match the data.
This should ultimately be the base to allow using the
`#[api]` macro on a newtype style enum as well as using this
schema as a configuration for our section config parser.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
so CLI tools can pre-parse out non-api parameters before
passing the remaining stuff to the router
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Adds a privileged_addr to ApiConfig, and some helpers for
hyper (both server and client)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
The proxmox-backup repo was filtered using `git filter-repo` using the
following paths:
proxmox-rrd
proxmox-rrd-api-types
src/rrd
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Nightly currently produces a different output format so this command
doesn't work right now when +system is not the default cargo.
Let's hope this is just a temporary hiccup in nightly, given that
there is an explicit `--format-version=1` parameter...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
For example, one can now use:
match-field exact:type=vzdump,replication
to match on vzdump AND replication events.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Seems like this was forgotten in the initial version. Without it,
it's not really possible to create matchers for forwarded mails.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
In the 'delete'-handler targets, we check if a
target is still referenced by a matcher - if it is, we return an
error. For built-in targets, this is actually not necessary, since
'deleting' a built-in only resets it to its default settings - it will
continue to exist after that.
The user could easily trigger this if 'mail-to-root', which is
referenced by 'default-matcher' is modified and then reset to its
defaults: An error is shown, the built-in target is not reset.
This commit disables this check if it is a built-in target.
Renamed the helper 'ensure_unused' to 'ensure_safe_to_delete' in the
process.
Also fixed the tests in api::test - they were never executed due to a
faulty #[cfg] directive.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
`Auto-Submitted` is defined in the rfc 5436 [1] and describes how
an automatic response (f.e. ooo replies, etc.) should behave on the
emails. When using `Auto-Submitted: auto-generated` (or any value
other than `none`) automatic replies won't be triggered.
[1]: https://www.rfc-editor.org/rfc/rfc3834.html
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
For mails forwarded by `proxmox-mail-forward` to an SMTP target, the
original message was nested as a 'message/rfc822' message part.
Originally this approach was chosen to avoid having to rewrite
message headers.
Good email-clients, such as Thunderbird can display these inline.
Other, more limited clients will show these messages as an attached
.eml file, which is not really a good user experience.
This patch changes the approach for message forwarding to be more like
forwarding mails in a mail client. We create a new message and
add the original message body as a body. Additionally, we also copy
over all message headers that are relevant to correctly display the
original message body (e.g. Content-Type, Content-Transfer-Encoding)
Tested with a couple of different email messages (varying in
structure, body parts, encoding, etc.) against the following SMTP
relays:
- gmail
- outlook
- our own webmail service
Originally reported in our community forum:
https://forum.proxmox.com/threads/proxmox-mail-forward-sends-mails-as-eml.137710/
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
as the previous commit: simply keep the previous Display impl and call
it from out of the new GroupFilter impl
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
After some discussion I canged the include/exclude behavior to first run
all include filter and after that all exclude filter (rather then
allowing to alternate inbetween). This is done by splitting them into 2
lists, running include first.
A lot of discussion happened how edge cases should be handled and we
came to following conclusion:
no include filter + no exclude filter => include all
some include filter + no exclude filter => filter as always
no include filter + some exclude filter => include all then exclude
Since a GroupFilter now also features an behavior, the Struct has been
renamed To GroupType (since simply type is a keyword). The new
GroupFilter now has a behaviour as a flag 'is_exclude'.
I considered calling it 'is_include' but a reader later then might not
know what the opposite of 'include' is (do not include? deactivate?). I
also considered making a new enum 'behaviour' but since there are only 2
values I considered it over engeneered.
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
`strftime`'s formatting is locale-dependent. If the system locale was
set to e.g. de_DE.UTF-8, the `Date` header became invalid
(e.g Mo instead of Mon for 'Monday'), tripping up some mail clients
(e.g. KMail).
This commit should fix this by using the new `epoch_to_rfc2822`
function from proxmox_time. Under the hood, this function uses
`strftime_l` with a fixed locale (C).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This variant of strftime can be provided with a locale_t, which
determines the locale used for time formatting.
A struct `Locale` was also introduced as a safe wrapper around
locale_t.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
instead of having it in a property string. For now this should be fine,
and if we need many more such options, we can still move them into a
property string if we want.
Also update the cli command in the docs on how to set it now.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by converting the bool into an option, otherwise having the options not
set at all will fail the unload while deserializing with
'eject-before-unload is not optional'
Also if we can automatically decide this in the future, we can now
detect if the option was explicitely set or not.
Fixes: 99f24b20 ("fix #4904: tape changer: add option to eject before unload")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
some tape libraries need the tape being ejected from the drive before
doing an unload. Since we cannot easily detect if that's the case,
introduce an 'eject_before_unload' option.
Instead of just adding a bool flag to the config, add a new 'options'
property string where we can put such niche options similar to how we
handle the datastore tuning options.
Extend the LtoTapeHandle with 'medium_present' which just uses a
TEST UNIT READY command to check for present medium, so we don't
try to eject an already ejected tape.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we'll need more info from there in the future, so derive clone for it
and save the whole config instead of adding an additional field.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Instead of returning -1 if we can't get the attributes, we use an
Option which will not be serialized on `None`.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Note that this was currently not deserialized anywhere, so this was
not an issue, but the api-macro now treats this as an error.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
separated out of FieldAttrib without the `flatten` attribute, since we
don't support this on enum variants
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
According to the rfc, the meta field contains additional fields that
weren't covered by the Meta struct. Of the additional fields, only
external_account_required will be used in the near future, but others
were added for completeness and the case that they might be used in the
future.
Signed-off-by: Folke Gleumes <f.gleumes@proxmox.com>
Functionality was added as a additional setter function, which hopefully
prevents any breakages. Since a placeholder Option an the AccountData
was already present, but has never been used, replacing the field with
an Option of a fully defined type should also be minimally intrusive.
Signed-off-by: Folke Gleumes <f.gleumes@proxmox.com>
Return a struct with all the components of the kernel version like it
has been done in pve. Also return the legacy `kversion` to keep
backwards compat.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Added field that shows the bootmode of the node. The bootmode is either
Legacy Bios, EFI, or EFI (Secure Boot). To detect the mode we use the
exact same method as in pve: We check if the `/sys/firmware/efi` folder
exists, then check if the `SecureBoot-xx...` file in the `efivars`
directory has the SecureBoot flag enabled.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Helper that return the current boot_mode and secureboot status.
Detection works the same as in pve, we use `/sys/firmware/efi` and
the `efivars/SecureBoot-xxx..` file.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
The new regex is similar to BLOCKDEVICE_NAME_REGEX but also allows
numbers at the end of the device name (also allows partitions names).
For nvme partitions it also allows the letter p and a number.
Signed-off-by: Markus Frank <m.frank@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
The new public function `accept_tls_optional()` is added, which
accepts both plain TCP streams and TCP streams running TLS. Plain TCP
streams are sent along via a separate channel in order to clearly
distinguish between "secure" and "insecure" connections.
Furthermore, instead of `AcceptBuilder` itself holding a reference to
an `SslAcceptor`, its public functions now take the acceptor as an
argument. The public functions' names are changed to distinguish
between their functionality in a more explicit manner:
* `accept()` --> `accept_tls()`
* NEW --> `accept_tls_optional()`
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
internally `SubscriptionInfo` already uses the `parse_next_due` helper
to parse the next due date to an epoch. this exposes a function that
allows us to use the epoch outside of this crate too. for example, a
user of pom may have multiple subscription for the same system. in
that case we want to apply the one with the due date that is furthest
in the future.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously this type lived inside of pom. this made it harder to
access the product type from a `SubscriptionInfo` trait in other
products. move the type here so we can check product types more
consistently across products (e. g. in pom and pbs)
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
This allows us to define a (modifiable) builtin-config, which is
at the moment hardcoded in PVEContext
The 'origin' parameter indicates whether a config entry was created by
a user, builtin or a modified builtin.
These changes require context to be set for tests, so we set
PVEContext by default if in a test context. There might be a nicer
solution for that, but for now this should work.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds a new endpoint type, namely 'smtp'. This endpoint
uses the `lettre` crate to directly send emails to SMTP relays.
The `lettre` crate was chosen since it is by far the most popular SMTP
implementation for Rust that looks like it is well maintained.
Also, it includes async support (for when we want to extend
proxmox-notify to be async).
For this new endpoint type, a new section-config type was introduced
(smtp). It has the same fields as the type for `sendmail`, with the
addition of some new options (smtp server, authentication, tls mode,
etc.).
Some of the behavior that is shared between sendmail and smtp
endpoints has been moved to a new `endpoints::common::mail` module.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit moves PVEContext from `proxmox-perl-rs` into the
`proxmox-notify` crate, since we now also need to access it from
`promxox-mail-forward`. The context is now hidden behind a feature
flag `pve-context`, ensuring that we only compile it when needed.
This commit adds PBSContext, since we now require it for
`proxmox-mail-forward`. Some of the code for PBSContext comes
from `proxmox-mail-forward`.
This commit also changes the global context from being stored in a
`once_cell` to a regular `Mutex`, since we now need to set/reset
the context in `proxmox-mail-forward`.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
As preparation for the integration of `proxmox-mail-foward` into the
notification system, this commit makes a few changes that allow us to
forward raw email messages (as passed from postfix).
For mail-based notification targets, the email will be forwarded
as-is, including all headers. The only thing that changes is the
message envelope.
For other notification targets, the mail is parsed using the
`mail-parser` crate, which allows us to extract a subject and a body.
As a body we use the plain-text version of the mail. If an email is
HTML-only, the `mail-parser` crate will automatically attempt to
transform the HTML into readable plain text.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This should be a bit more intuitive to users than the current
behavior, which is 'always match' for mode==all and 'never match' for
mode==any. The current behavior originates in the neutral element of
the underlying logical operation (and, or).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This shifts notification routing into the matcher-system. Every
notification has associated metadata (key-value fields, severity -
to be extended) that can be match with match directives in
notification matchers. Right now, there are 2 matching directives,
match-field and match-severity. The first one allows one to do a
regex match/exact match on a metadata field, the other one allows one
to match one or more severites.
Every matcher also allows 'target' directives, these decide which
target(s) will be notified if a matcher matches a notification.
Since routing now happens in matchers, the API for sending is
simplified, since we do not need to specify a target any more.
The API routes for filters and groups have been removed completely.
The parser for the configuration file will still accept filter/group
entries, but will delete them once the config is saved again. This is
needed to allow a smooth transition from the old system to the new
system, since the old system was already available on pvetest.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This will be useful later for system mail forwarding, where
the content of the mail should be forwarded unchanged.
This moves notification properties into this new type and calls them
'data'. They will exclusively used for template rendering.
`Notification` will receive a separate field for metadata, which
will be useful for notification filtering. This decouples
template rendering and filtering, which enables us to be very precise
about which metadata fields we allow to be used in filters.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
support for it got added to Proxmox repositories, so there is no need to use
custom logic and manual fetching for this anymore.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
As termproxy only used this module from the huge proxmox-sys crate,
and this module was also only used here, it makes sense to move it
over there (and dropping proxmox-sys halved build-time from 8.5 s to
4.2 s).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
If there is only one error, output in a single line, without any
excessive newlines at the end. If there are multiple errors, show them
in a bulleted list.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
We had two call sites deriving the directory "shard" where the task
log file is actually saved to, this can lead to ugly bugs and is
better done in a central single-source-of-truth way.
While at it factor out the creation of the log file (and it's shard
directory) to avoid crowding the WorkerTask new fn to much.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This mostly affected attribute parsing (due to the syn::Meta changes).
Also creating `DelimSpan`s for custom-built `syn::Attribute`s is a
bit... ugly.
Upshot: turns out we can drop some helpers in util.rs with the new
`syn::Meta` changes.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
previously we changed the internal type of the 'data' property
from Option<T> to T in the assumption the api always returns
'data:null'.
this is actually only the case when the api call succeeds. in an error
case there is no data property at all.
to fix this issue while behaving the same for 'data:null' we have to
revert to Option<T> for RawApiResponse but instead of always throwing an
error for 'data:null' in 'check' we now try there to deserialize from
Value::Null for T if there was no data. This will succeed for the Type
'()' which was the motivation for the original change.
The only downside is that the RawApiResponse now has a trait bound that
T is deserializeable, but was a requirement for using it anyway
(as there was no other way of constructing it)
Fixes: 271a55f ("client: remove option from inner RawApiResponse")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
`Auto-Submitted` is defined in the rfc 5436 [1] and describes how
an automatic response (f.e. ooo replies, etc.) should behave on the
emails. When using `Auto-Submitted: auto-generated` (or any value
other than `none`) automatic replies won't be triggered.
[1]: https://www.rfc-editor.org/rfc/rfc3834.html
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
technically an empty string is not valid json, but when sending an api
request without any parameters, treating the empty body as an empty
parameter hash instead of an error, makes the the api more robust for
clients
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
note: not complete, there's other code to check and rework, but I had
this already done so commit it, better than nothing.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This moves the clean-up to happen up-front. That way one can still
inspect the test data after, e.g., a failed test.
Originally done almost like this in a patch from Fiona [0] that I just
overlooked, but now also factored out, avoid crowding the test code to
much with duplicate code.
[0]: https://lists.proxmox.com/pipermail/pve-devel/2023-June/057136.html
Suggested-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the "main" repo only exists for Quincy to allow an easier transition
from Proxmox VE 7 to Proxmox VE 8, for when the enterprise repo got
added for ceph too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For now just duplicate the Ceph Quincy entries, as I want to avoid
using macros and we do not yet have support for enums inside enums
with the api macro.
Adapt and expand the tests slightly to have at least some simple
coverage there too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Files inside CARGO_TARGET_TMPDIR are only cleaned on `cargo clean`, so
tests that expect files to not exist need to cleanup themselves.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
on other targets we print the timestamp without fractional seconds
('.xxxZ'), so we should remove that too on wasm
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when using the client for an api call that does not return any data
(it returns '{"data":null}'), we would always get an error 'api returned
no data'. The message is technically correct, but it should not be an
error when we expect no data (e.g. most of our CRUD PUT/POST calls)
instead of having the Option<T> in the RawApiResponse type itself, move
it into to the 'nodata' function intended for api calls where we don't
expect any data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This commit has the aim of making template rendering a bit more
robust. It does so by a.) Accepting also strings for helpers that
expect a number, parsing the number if needed, and b.) Ignoring errors
if a template helper fails to render a value and showing an error in
the logs, instead of failing to render the whole template (leading
to no notification being sent).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit updates all helper functions, taking into account recent
developments regarding `tokio`.
In particular, the `block_in_place()` and `block_on()` functions now
don't panic anymore if used within the single-threaded `tokio` runtime
and instead behave as expected in both runtime flavours.
Furthermore, because `tokio` may add more runtime flavours in the
future, all helpers will now panic if used within an unsupported
runtime. This is to prevent unforeseen behavioural quirks and
interactions with `tokio` internals.
The above changes make `BlockingGuard` redundant; it is consequently
removed.
The documentation is also updated, describing the behaviour of the
helper functions and the purpose of the `runtime.rs` module in more
detail.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
lto 9 tapes have a new density code which leads to these tapes not
being recognized properly. add the new density code and TapeDensity to
improve lto 9 support. since the documentation states that we support
lto 5 and above this constitutes a bug fix for lto 9 support.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Since the WASM client cannot actually use a `http::Request` the way we
expect it to, that is, it cannot manually along cookies, we turn the
client bit inside out:
This crate mainly defines the `HttpApiClient` trait which expects the
http client to perform *authenticated* API calls, that is, the
handling of API tokens and tickets should happen at the *implementor*
side.
The product clients will require *this* trait to be implemented, and
will not themselves offer a way to login.
As for the `Client` struct, this will now instead *implement* this
trait and will *not* be used in the `wasm` ecosystem. Rather, this is
the ticket handling http client that already exists in the PWT based
ui code.
The PVE client in `pve-api-types` will not *contain* a `Client`
anymore, but rather, it will provide PVE api call implementations for
something implementing `HttpApiClient`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The environment trait was useful on the CLI, but does not really
translate well to eg. the wasm ui (or pdm for that matter), so drop it
and instead have `.login` and `.login_tfa` just take the
`proxmox_login` type and handle the updating of authentication data.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Because we ultimately also want to drop the `Environment` trait since
it is not suitable for all use cases (eg. wasm ui)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
In the uncommon circumstance that calls to `read()` end up reading any number of
bytes other than 4096, the subsequently read bytes become misaligned, causing
blocks of zeroes to be written unnecessarily.
To illustrate, imagine you have a 12KiB file:
[x][x][x][x][ ][ ][ ][ ][x][x][x][x]
└──4096──┘ └──4096──┘ └──4096──┘
The first and last block are filled with some data, whereas the middle block is
empty and will therefore result in only zeroes being read.
In order for the empty block to be skipped with `seek()`, the entire buffer has
to be filled with zeroes.
If, for example, the first `read()` ends up putting only 3KiB into the buffer,
the empty block in the middle won't be detected properly, as the buffer will
now always contain some data. What results are four misaligned reads:
[x][x][x][x][ ][ ][ ][ ][x][x][x][x]
├─────┘ ├────────┘ ├────────┘ │
1 2 3 4
This is fixed by ensuring chunks of 4KiB are always read into the buffer,
except when the last block is truncated. In order to prevent frequent small
reads, the incoming reader is also buffered via `io::BufReader`.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
this should avoid most common size limitations. the search should also
complete quicker as fewer results need to be computed. note that this
way a configuration may be accepted, but the related sync job can
fail due to and exceeded size limit warning for some ldap servers
(such as 2.5.14+dfsg-0ubuntu0.22.04.2).
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Commit 5720ba2d ("use new auth api crate") moved all auth-related code
into it's own crate inside the `proxmox` repo, including this file. Thus
drop it here, it's not even included in the compile.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
To see if it is even still necessary given that it's not a trait
object type where auto traits would need to be explicit...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This should be moved to where we actually need it, not be part of the
generic product client.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we have use cases where we have bytes, and serde_json has a from_slice
method, doing the utf-8 check unnecessarily is pointless, while going
from &str to &[u8] is free...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
since `totp_locked` is not wrapped in an `Option` we need to
explicitly tell serde about its default
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Break out proxmox-router's HttpError into it's own crate so that it can
be used without pulling in proxmox-router.
This commit also implements `Serialize` for `HttpError` so that it can
be returned from perlmod bindings, allowing Perl code to access the
status code as well as the message.
Also add some smoke-tests to make sure that the `http_bail` and
`http_err` macros actually produce valid code.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
[w.bumiller@proxmox.com: set d/changelog to UNRELEASED]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This introduces a new configuration parameter `mailto-user`.
A user's email address will be looked up in the product-specific
user database.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Since `proxmox-notify` is intended to be used by multiple products,
there needs to be a way to inject product-specific behavior.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds template rendering to the `proxmox-notify` crate, based
on the `handlebars` crate.
Title and body of a notification are rendered using any `properties`
passed along with the notification. There are also a few helpers,
allowing to render tables from `serde_json::Value`.
'Value' renderers. These can also be used in table cells using the
'renderer' property in a table schema:
- {{human-bytes val}}
Render bytes with human-readable units (base 2)
- {{duration val}}
Render a duration (based on seconds)
- {{timestamp val}}
Render a unix-epoch (based on seconds)
There are also a few 'block-level' helpers.
- {{table val}}
Render a table from given val (containing a schema for the columns,
as well as the table data)
- {{object val}}
Render a value as a pretty-printed json
- {{heading_1 val}}
Render a top-level heading
- {{heading_2 val}}
Render a not-so-top-level heading
- {{verbatim val}} or {{/verbatim}}<content>{{#verbatim}}
Do not reflow text. NOP for plain text, but for HTML output the text
will be contained in a <pre> with a regular font.
- {{verbatim-monospaced val}} or
{{/verbatim-monospaced}}<content>{{#verbatim-monospaced}}
Do not reflow text. NOP for plain text, but for HTML output the text
will be contained in a <pre> with a monospaced font.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds a way to filter notifications based on severity. The
filter module also has the necessary foundation work for more complex
filters, e.g. matching on properties or for creating arbitarily complex
filter structures using nested sub-filters.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Add an endpoint for Gotify [1], showing the how easy it is to add new
endpoint implementations.
[1] https://gotify.net/
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
StringSchema::check_constraint runs `parse_property_string` for
property strings, but when we deserialize a `PropertyString` we
immediately follow that up with deserializing it using the schema, so
there's no need to check it beforehand.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This provides `proxmox_schema::property_string::PropertyString<T>` for
a typed property-string.
To facilitate this, this introduces
`proxmox_schema:🇩🇪:SchemaDeserializer` which is a serde deserializer
for property strings given a schema.
This basically maps to one of `de::SeqAccess` (for array schemas) or
`de::MapAccess` (for object schemas).
Additionally, a `de::NoSchemaDeserializer` is added, since properties
within the strings may have string schemas with no format to it, while
the type we serialize to may ask for an array (a simple "list") via
serde.
The deserializers support borrowing, for which a helper `Cow3` needed
to be added, since property strings support quoting with escape
sequences where an intermediate string would be allocated and with an
intermediate lifetime distinct from the `'de` lifetime.
A `de::verify` module is added which uses serde infrastructure to
validate schemas without first having to deserialize a complete
`serde_json::Value`.
For serialization, `proxmox_schema::ser::PropertyStringSerializer` is
added split into similar parts `ser::SerializeStruct` and
`ser::SerializeSeq` at the top level, and the same prefixed with
`Element` for inside the actual string. This should also properly
quote the contents if required.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These were missing the new client-ip parameter in the auth
function calls which was introduced to support `PAM_RHOST`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Since this requires access to the user data, we need to add
a generic parameter to the unlock methods.
To avoid having to create another major API bump affecting
all our products this short after release, we keep the old
version around with the old behavior.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Like in PVE.
This means that /access/users is now a 'protected' call to
get access to 'tfa.cfg'.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this commit makes the ldap realm endpoints check whether a new or
updated configuration works correctly. it uses the new
`check_connection` function to make sure that a configuration can be
successfully used to connect to and query an ldap directory.
doing so allows us to remove the ldap domain regex. instead of relying
on a regex to make sure that a given distinguished name (dn) could be
correct, we simply let the ldap directory tell us whether it accepts
it. this should also aid with usability as a dn that looks correct
could still be invalid.
this also implicitly removes unauthenticated binds, since the new
`check_connection` function does not support those. it will simply
bail out of the check if a `bind_dn` but no password is configured.
therefore, this is a breaking change.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
this function checks if a given connection could work. it uses the
current config to connect to an ldap directory and perform a search
with the provided base_dn. this enables us to verify a connection
before storing it in a more meaningful way than with a regex.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
by using the default empty string if no password was provided,
unauthenticated binds were possible. to bring pbs in-line with pve,
switch to throwing an error in this case instead. however, this will
break any pre-existing setup that relied on this behavior.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
The module previously lived in `pbs-api-types`, however turned out to
be useful in other places as well (POM, proxmox-notify), so it is moved
here as its own micro-crate.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
It is very common for TOTP URIs to contain the algorithm in lowercase,
hence we convert to lowercase when doing From<&str> for Algorithm.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
if the match arms are this far away from the actual `match`
keyword, this needs to be split up...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
when zipping a directory, our intention was to skip over files that
cannot be zipped (e.g. the file can't be read/vanished/etc.), so we
ignored errors and simply logged it.
but when 'add_entry' fails, we will never actually restore, since every
error there is fatal to the point that the zip cannot be finished thats
because we take the 'target' sink out of self, and only insert it again
after all writes succeeded. so if an error occurs in between 'target' is
not put into self again (and never will be) and the zip cannot be
finished (even if we would catch all those intermediate errors and
restore 'target', we don't know in which state the output was, so we're
unable to finish a valid zip)
to fix that, split the actual 'add_entry' part there out of the async
move block and treat its errors always as fatal
without this, we generate heaps of log lines even after an error
occurred, and can never recover
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
APT doesn't mind a repository with either "/" or "./" as suite/distribution,
such as
deb https://example.com/debian ./
in that case, the 'dists' part of the URL and the trailing slash (which would
be encoded as '_') is dropped in the file name in '/var/lib/apt/lists/'.
Other suite values with a trailing or leading '/' are rejected with an error by APT:
E: Malformed entry 1 in sources file /etc/apt/sources.list.d/test.list (absolute Suite Component)
E: The list of sources could not be read.
so this should be the only special case requiring handling.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
APT will not store the InRelease file in some cases, and some repositories
might not even have one in the first place.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
A build-system that needs all build-dependencies even for a simple
make clean invocation is a PITA..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The old 'main' component stays valid, pointing to no-subscription,
which means the is_referenced_repository() check needs a special case
for it. It will eventually go away, together with the handles for
Quincy.
Alternatively, the standard repository's info() could've been changed
to return multiple possible components, similar to URLs, but as
opposed to URLs, there could be a standard repository that wants to
have multiple components and it feels a bit unnatural, because
multiple components are usually not aliases of the same. And adapting
is_referenced_repository() would be needed here too. So overall, the
above alternative just felt better.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This mostly serves as documentation for the API call to be
implemented across our products. It's otherwise already just
a oneliner on the TfaConfig.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Use a custom result type to return success/failure and the
need to save the user data to the caller, while having
logged the error messages rather than returning them.
We count general TFA failures and also TOTP specifically,
and lock the user out of their 2nd factors on too many
failures.
To this end, all errors are now treated as failures.
While technically we can have crypto errors the user might
not be able to cause, we can't always know, and not all
errors are guaranteed to be a host side configuration issue,
so instead, all errors (since they are rare) now now counted
as a regular TFA error.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
TfaUserData uses `#[serde(deny_unknown_fields)]`, so we add
this now, but using it will require explicitly enabling it.
If the TOTP count is high, the user should be locked out of
TOTP entirely until they use a recovery key to reset the
count.
If a user's TFA try count is too high, they should get rate
limited.
In both cases they should receive some kind of notification.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
without adding the entire API as well, so API clients can
actually use the types used by the api methods without
requiring the backend implementation being built in as
well...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
If WA or U2F fail to produce a challenge, the user may still
log in with other factors and the challenge will be
considered to not be empty.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This should only ever be explicitly removed.
Similarly, include an empty array of recovery keys in the
tfa challenge, so that clients know about empty recoveries
rather than getting an empty challenge when there are no
other factors available.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Specifying the transfer-last parameter limits the amount of backups
that get synced via the pull command/sync job. The parameter specifies
how many of the N latest backups should get pulled/synced. All other
backups will get skipped.
This is particularly useful in situations where the sync target has
less disk space than the source. Syncing all backups from the source
is not possible if there is not enough disk space on the target.
Additionally this can be used for limiting the amount of data
transferred, reducing load on the network.
The newest backup will always get re-synced, regardless of the setting
of the transfer-last parameter.
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
a non capturing group is '(?:)' not '(:?)' so fix that.
None of these regexes are used where would use capturing groups.
DATASTORE_MAP_REGEX and TAPE_RESTORE_SNAPSHOT_REGEX are only used
as api types and BLOCKDEVICE_NAME_REGEX is only used once outside of the
api and there we also don't look at the capturing groups.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Otherwise, a substring match is enough to fulfill the constraint.
Fixes: 3aba0d9a ("api-types: ldap: add verification regex for LDAP DNs")
Reported-by: Friedrich Weber <f.weber@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
the requests are all created via the agent that already contains the user
agent, so this internal helper isn't needed anymore.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this allows us to slim down our code, and once
https://github.com/algesten/ureq/pull/597 is merged upstream (and/or we update
to a version containing the fix) it also means the custom user agent is used
for requests to the proxy host, if one is configured.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
To the existing `.alias(item)`, add a
`.aliases(into-item-iter)` similar to how `Extend` works.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The connection submodule now allows building an "acceptor"
for hyper connections which can either take an explicit ssl
acceptor, or builds a default one with a self signed
certificate.
The rate-limited-stream feature enables a method to
lookup/update rate limiters for connections.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
rest-server can now optionally provide a PeerAddress
implementation for RateLimitedStream by activating its
'rate-limited-stream' feature
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
instead of implementing 'peer_addr' specifically for
RateLimitedStream<tokio::net::TcpStream>, just provide
.inner() and .inner_mut() so the user can reach the inner
stream directly.
This way we can drop the tokio/net feature as well
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Instead of a ServerAdapter for the index page and
authentication checking (which don't relate to each other),
provide a `.with_auth_handler` and `.with_index_handler`
builder for ApiConfig separately.
Both are optional. Without an index handler, it'll produce a
404. Without an auth handler, an `AuthError::NoData` is
returned.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
instead, allow attaching routers to path prefixes and also
add an optional non-formatting router
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This is what actually defines the API server after all.
The ApiService trait in between is a hyper impl detail.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We pulled in proxmox-http with the client feature solely to
implement the `Service` trait on
`SslStream<RateLimitedStream<TcpStream>>`.
All those `Service` impls are the same: provide a peer
address and return an `ApiService`.
Let's put the `peer_addr()` call into a trait and build from
there.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We should not use the sys crate to pull in the sortable
macro, just depend on its crate instead...
And the identity macro used to be required by the sortable
macro, but is not anymore and has been deprecated for a
while, so we can now drop it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The properties are mainly based on the ones from PVE, except:
* consistent use of kebab-cases
* `mode` replaces deprecated `secure`
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds integration tests to ensure that the crate works as intended.
The tests are executed against a real LDAP server, namely `glauth`. `glauth` was
chosen because it ships as a single, statically compiled binary and can
be configured with a single configuration file.
The tests are written as off-the-shelf unit tests. However, they are
#[ignored] by default, as they have some special requirements:
* They required the GLAUTH_BIN environment variable to be set,
pointing to the location of the `glauth` binary. `glauth` will be
started and stopped automatically by the test suite.
* Tests have to be executed sequentially (`--test-threads 1`),
otherwise multiple instances of the glauth server might bind to the
same port.
The `run_integration_tests.sh` checks whether GLAUTH_BIN is set, or if
not, attempts to find `glauth` on PATH. The script also ensures that the
tests are run sequentially.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds the search_entities function, which allows to search for
LDAP entities given certain provided criteria.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
In the LDAP world, authentication is done using the bind operation, where
users are authenticated with the tuple (dn, password). Since we only know
the user's username, it is first necessary to look up the user's
domain (dn).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
it's the version currently shipped by bookworm, so let's unify this widely-used
dependency to make bootstrapping easier.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
these two functions don't actually use the `type_name` parameter, but the
interface including custom formatters require it.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
HashMaps are not ordered, so each package build containing a section config
dump would have the documentation ordered randomly.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Mostly relevant when the config is written out again after parsing it
with unknown sections. Previously, with duplicate keys, only the last
value would be saved. Now, duplicate keys are assumed to be part of
an array schema and handled as such.
Because the unknown section parsing does not know if a certain
property does actually have an array schema, it's not possible to
detect duplicate keys for non-array-schema properties, and if a
property with array-schema shows up only once, it will not be saved as
a Value::Array, but a Value::String.
Writing, or to be precise the format_section_content methods, already
handle Value::Array, so don't need to be adapted.
Fixes: 0cd0d16 ("section config: support allowing unknown section types")
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
while these are (currently) only used by a single member each, having *all*
dependency versions specified in the top level Cargo.toml only makes the whole
process of managing them less error-prone.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
- proxmox_io::boxed::uninitialized(len) -> Box<[u8]>
same as vec::uninitialized, but as a box
- proxmox_io::boxed::zeroed(len) -> Box<[u8]>
same as vec::zeroed, but as a box
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
these are only used by a single member at the moment, but we can move them to
the workspace to have a single location for version + base feature set
specification.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
these are only used by a single member at the moment, but we can move them to
the workspace to have a single location for version + base feature set
specification.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
so that we no longer have to (or forget to) bump the version in multiple places.
notable changes:
- outdated versions have been unified
- proxmox-metrics -> proxmox-async no longer uses explicit empty features
(proxmox-async doesn't provide any anyway)
- proxmox-subscription -> proxmox-http no longer uses explicit default_features
= false (proxmox-http has an empty default feature anyway)
- missing path dependencies added (mainly proxmox-rest-server)
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
noteworthy changes:
- proxmox-http had a default_features_false dep on hyper, which is dropped (the
default feature is empty anyway)
- hyper, libc, nix, tokio and url versions are unified
- missing (cosmetic) bindgen feature on zstd enabled everywhere
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
instead of hardcoding the default deep inside the code. This makes it
much easier to see what is the actual default
the first instance of ChunkOrder::None was only for the test case, were
the ordering doe not matter
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
rationale is that it makes the backup much safer than 'none', but does not
incur a big of a performance hit as 'file'.
here some benchmark:
data to be backed up:
~14GiB semi-random test images between 12kiB and 4GiB
that results in ~11GiB chunks (more than ram available on the target)
PBS setup:
virtualized (on an idle machine), PBS itself was also idle
8 cores (kvm64 on Intel 12700k) and 8 GiB memory
all virtual disks are on LVM with discard and iothread on
the HDD is a 4TB Seagate ST4000DM000 drive, and the NVME is a 2TB
Crucial CT2000P5PSSD8
i tested each disk with ext4/xfs/zfs (default created with the gui)
with 5 runs each, inbetween the caches are flushed and the filesystem synced
i removed the biggest and smallest result and from the remaining 3
results built the average (percentage is relative to the 'none' result)
result:
test none filesystem file
hdd - ext4 125.67s 140.39s (+11.71%) 358.10s (+184.95%)
hdd - xfs 92.18s 102.64s (+11.35%) 351.58s (+281.41%)
hdd - zfs 94.82s 104.00s (+9.68%) 309.13s (+226.02%)
nvme - ext4 60.44s 60.26s (-0.30%) 60.47s (+0.05%)
nvme - xfs 60.11s 60.47s (+0.60%) 60.49s (+0.63%)
nvme - zfs 60.83s 60.85s (+0.03%) 60.80s (-0.05%)
So all in all, it does not seem to make a difference for nvme drives,
for hdds 'filesystem' increases backup time by ~10%, while
for 'file' it largely depends on the filesystem, but always
in the range of factor ~3 - ~4
Note that this does not take into account parallel actions, such as gc,
verify or other backups.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Similar to commit c9ede1c ("support unknown types in section config")
in pve-common.
Unknown sections are parsed as String-JSON String key-value pairs
without any additional checks and also written as-is.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
fixups for DatastoreFSyncLevel:
* use derive for Default
* add some more derives (Clone, Copy)
chunk store:
* drop to_owned for chunk_dir_path
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we already expose the raw sections which are sometimes
easier to use, but we don't expose the order at all this way
otherwise
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Additional properties will be parsed according to the default string
schema.
This is relevant for use cases when the full schema is not known for
some reason or another. In particular this allows support for parsing
older/newer versions of configuration files. One example of this is
the proposed proxmox-mail-forward helper binary, which currently
doesn't have access to the PBS API types for dependency reasons and
is only interested in the email field for the root user. If it can
only use a minimal schema with additional_properties set to true, it
will be robust against changes.
Writing already works, because the ObjectSchema's verify_json()
already handles additional_properties correctly and
format_section_content() handles them like all other properties
(method doesn't depend on the schema).
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
the dropped .into() is guarded by the bumped build-dependency on
proxmox-sys 0.4.1, the missing Eq is a new clippy lint.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
currently, we don't (f)sync on chunk insertion (or at any point after
that), which can lead to broken chunks in case of e.g. an unexpected
powerloss. To fix that, offer a tuning option for datastores that
controls the level of syncs it does:
* None (default): same as current state, no (f)syncs done at any point
* Filesystem: at the end of a backup, the datastore issues
a syncfs(2) to the filesystem of the datastore
* File: issues an fsync on each chunk as they get inserted
(using our 'replace_file' helper) and a fsync on the directory handle
a small benchmark showed the following (times in mm:ss):
setup: virtual pbs, 4 cores, 8GiB memory, ext4 on spinner
size none filesystem file
2GiB (fits in ram) 00:13 0:41 01:00
33GiB 05:21 05:31 13:45
so if the backup fits in memory, there is a large difference between all
of the modes (expected), but as soon as it exceeds the memory size,
the difference between not syncing and syncing the fs at the end becomes
much smaller.
i also tested on an nvme, but there the syncs basically made no difference
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
split out from proxmox-backup using `git filter-repo` including
history with the following --paths-from-file:
```
proxmox-rest-server
src/api/server.rs
src/server/command_socket.rs
src/server/config.rs
src/server/environment.rs
src/server/formatter.rs
src/server/h2service.rs
src/server/rest.rs
src/server/state.rs
src/tools/compression.rs
src/tools/daemon.rs
src/tools/file_logger.rs
src/worker_task.rs
```
we converted the prune settings of datastores to prune-jobs, but did
not actually implement the notifications for them, even though
we had the notification options in the gui (they did not work).
implement the basic ok/error notification for prune jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
intended for passing the format to the file-restore client/daemon
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to allow detection of repositories that support downloading indices via
their hash instead of their filename.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
which returns if a file reference is architecture specific, and for
which architecture it is relevant.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
no real change for PBS usage - the ApiHandler enum is marked
non_exhaustive now because it has extra values if the new (enabled by
default) "server" feature is enabled.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
signed subscription info files should always be checked to catch
attempts of invalid signatures, but the age and serverid checks only
need to apply to "active" files, else the status might switch from a
more meaningful one to "invalid" by accident.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
for use cases where the full request body is not available from the
start, or the response doesn't need to be fully read in one go.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
so we have proxmox_http::client::Client for the async, hyper-based
client and proxmox_http::client::sync::Client for the sync, ureq-based
one.
this is a breaking change.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to allow get requests with extra headers (such as `Authorization`) and a
generic `request` fn to increase flexibility even more.
this is a breaking change.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and rename it to HttpOptions, since it's not specific to the "Simple"
client at all.
this is a breaking change.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
these would cause failures when building the sub-crates directly from
their sub-directory.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to reduce the creep of optional dependencies being exposed as features.
this requires cargo 0.63 (and debcargo built against at least that
version), but greatly reduces the number of binary packages and provides
generated, while still allowing sensible selection of optional
dependencies via the explicit feature meant for pulling them in.
diff stat for running `make deb` after this change:
proxmox-http/debian/control | 226 ++++--------------------------------
proxmox-router/debian/control | 74 +-----------
proxmox-schema/debian/control | 53 ++-------
proxmox-subscription/debian/control | 17 +--
proxmox-sys/debian/control | 51 +++-----
proxmox-tfa/debian/control | 110 ++----------------
6 files changed, 72 insertions(+), 459 deletions(-)
the 'dep:' prefix marks something on the RHS inside the features section
as dependency, it's only allowed if the string after it is an optional
dependency an no explicit feature of the same name exists. if all
pointers to the optional dependency in the features section are marked
as such, the optional dependency itself will not be exposed as a feature
(either on the cargo or debian/control level).
the '?' suffix marks dependencies as "weak", which only enables the
optional dependency + its feature(s) if the optional dependency itself
is also enabled. it has no effect on d/control since such a relationship
is not encodable in Debian package relations, but it does affect cargo
dependency resolution and allows skipping the build of unneeded optional
dependencies in some cases.
with no packages/crates depending on the no longer exposed automatically
generated features/packages, so these are safe to remove even though
it's technically a breaking change.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if we encounter a file reference pointing to a component that is not
contained in the componenents list, we can just ignore it as unknown.
only treat parsing errors for references pointing to known components as
actual errors.
this currently triggers with (In)Release files for debian-updates and
debian-security, which reference (empty) files for a "non-free-firmware"
component that is not listed in the `Components` field of the release
file.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the remaining ones are:
- type complexity
- fns with many arguments
- new() without default()
- false positives for redundant closures (where closure returns a static
value)
- expected vs actual length check without match/cmp
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
drop the let binding, easier to use in const context since
CStr::from_bytes_with_nul_unchecked is const since 1.59
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
and handle signed keys differently w.r.t. age checks, since they will be
refreshed less frequently.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
taking over slighlty generified helpers and types:
- subscription info and status
- checking subscription status with shop
- reading/writing local subscription-related files
the perl-based code uses base64 with newlines for the data, and base64
without padding for the checksum. accordingly, calculate the checksum
with and without newlines, and compare the decoded checksum instead of
the encoded one.
furthermore, the perl-based code encodes the subscription status using
Capitalized values instead of lowercase, so alias those for the time
being.
PVE also stores the serverid as 'validdirectory', so add that as alias
as well.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Otherwise we have to use BackupType::iter().iter() whenever
we're not using a `for _ in iter()` construct.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
similar to 'zip_directory', this is intended to tar a local directory,
e.g. when we're in a restore vm.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by not requiring the 'anyhow::Error' type from the source, but only
that it implements 'Into<Error>'. This way, we can also accept a stream
that produces e.g. an io::Error
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when creating the documentation (e.g. `cargo doc --open`), it would
warn that `Display` is not in scope.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
it's only used there and pulls in hyper and tokio, which we don't
want/need in http-helpers.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
when using readdir/getdents the file type might be 'DT_UNKNOWN'
(depending on the filesystem). Do a fstatat in that case to
get the file type. Since maybe the callback wants to do
a stat anyway, pass it there (if done)
adds two new helpers:
'file_type_from_file_stat': uses a FileStat struct to get the file type
'get_file_type': calls fstatat to determine the file_type
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
InfluxDbUdp and InfluxDbHttp for now
introduces schemas for host:port and https urls
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
which is a bit generic and allows us to use e.g. a map result to be
passed here
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we can now use it for the error case and will further use it for the
can access namespace but not datastore case in a future patch
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead move the acl_path helper to BackupNamespace, and introduce a new
helper for printing a store+ns when logging/generating error messages.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
for usage in permission check error messages, to allow easily indicating
which privs are missing.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
syncing to a namespace only requires privileges on the namespace (and
potentially its children during execution).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the namespace is optional, but should be captured to allow ACL checks
for unprivileged non-job-owners.
also add FIXME for other job types and workers that (might) need
updating.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
We can have those in existing verify jobs configs, and that'd break
stuff. So, even while the "bad" commit got released only recently
with `2.1.6-1` (14 April 2022), we still need to cope with those that
used it, and using some serde parser magic to transform on read only
is hard here due to section config (json-value and verify currently
happen before we can do anything about it)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This reverts commit c72fe7d77c.
We could already cause the behavior by simply setting ignore-verified
to false, aas that flag is basically an on/off switch for even
considering outdated-after or not.
So avoid the extra logic and just make the gui use the previously
existing way.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by adding the 'default' serde hint and renaming 'recursion_depth' to
'max_depth' (to be in line with sync job config)
also add the logic to actually add/update the tape backup job config
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding a new parameter 'namespaces', which contains a mapping
for a namespace like this:
store=datastore,source=foo,target=bar,max-depth=2
if source or target are omitted the root namespace is used for its value
this mapping can be given several times (on the cli) or as an array (via
api) to have mappings for multiple datastores
if a specific snapshot list is given simultaneously, the given snapshots
will be restored according to this mapping, or to the source namespace
if no mapping was found.
to do this, we reutilize the restore_list_worker, but change it so that
it does not hold a lock for the duration of the restore, but fails
if the snapshot does exist at the end. also the snapshot will now
be temporarily restored into the target datastore into the
'.tmp/<media-set-uuid>' folder.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
these are helpers for the few cases where we want to print and parse
from a format that has the namespace and snapshot combined, like for
the on-tape catalog and snapshot archive.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
into the regular one (with default == MAX) and the one used for
pull/sync, where the default is 'None' which actually means the remote
end reduces the scope of sync automatically (or, if needed,
backwards-compat mode without any remote namespaces at all).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and use it when creating a sync job, and simplify the check on updating
(only check the final, resulting config instead of each intermediate
version).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
There is a race upon reload, where it can happen that:
1. systemd forks off /bin/kill -HUP $MAINPID
2. Current instance forks off new one and notifies systemd with the
new MAINPID.
3. systemd sets new MAINPID.
4. systemd receives SIGCHLD for the kill process (which is the current
control process for the service) and reads the PID of the old
instance from the PID file, resetting MAINPID to the PID of the old
instance.
5. Old instance exits.
6. systemd receives SIGCHLD for the old instance, reads the PID of the
old instance from the PID file once more. systemd sees that the
MAINPID matches the child PID and considers the service exited.
7. systemd receivese notification from the new PID and is confused.
The service won't get active, because the notification wasn't
handled.
To fix it, update the PID file before sending the MAINPID
notification, similar to what a comment in systemd's
src/core/service.c suggests:
> /* Forking services may occasionally move to a new PID.
> * As long as they update the PID file before exiting the old
> * PID, they're fine. */
but for our Type=notify "before sending the notification" rather than
"before exiting", because otherwise, the mix-up in 4. could still
happen (although it might not actually be problematic without the
mix-up in 6., it still seems better to avoid).
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.
Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Allow pulling all groups from a certain source namespace, and
possibly sub namespaces until max-depth, into a target namespace.
If any sub-namespaces get pulled, they will be mapped relatively from
the source parent namespace to the target parent namespace.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.
For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).
The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.
Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.
Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
allow to list any namespace with privileges on it and allow to create
and delete namespaces if the user has modify permissions on the parent
namespace.
Creation is only allowed if the parent NS already exists.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[ T: renamed from NAMESPACE_RECURSION_DEPTH_SCHEMA & moved to from
jobs to datastore ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
given a namespace, a source prefix and a target prefix this helper
strips the source prefix and replaces it with the target one (erroring
out if the prefix doesn't match).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Make it easier by adding an helper accepting either group or
directory
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The idea is to have namespaces in a datastore to allow grouping and
namespacing backups from different (but similar trusted) sources,
e.g., different PVE clusters, geo sites, use-cases or company
service-branches, without separating the underlying
deduplication domain and thus blowing up data and (GC/verify)
resource usage.
To avoid namespace ID clashes with anything existing or future
usecases use a intermediate `ns` level on *each* depth.
The current implementation treats that as internal and thus hides
that fact from the API, iow., the namespace path the users passes
along or gets returned won't include the `ns` level, they do not
matter there at all.
The max-depth of 8 is chosen with the following in mind:
- assume that end-users already are in a deeper level of a hierarchy,
most often they'll start at level one or two, as the higher ones
are used by the seller/admin to namespace different users/groups,
so lower than four would be very limiting for a lot of target use
cases
- all the more, a PBS could be used as huge second level archive in a
big company, so one could imagine a namespace structure like:
/<state>/<intra-state-location>/<datacenter>/<company-branch>/<workload-type>/<service-type>/
e.g.: /us/east-coast/dc12345/financial/report-storage/cassandra/
that's six levels that one can imagine for a reasonable use-case,
leave some room for the ones harder to imagine ;-)
- on the other hand, we do not want to allow unlimited levels as we
have request parameter limits and deep nesting can create other
issues as well (e.g., stack exhaustion), so doubling the minimum
level of 4 (1st point) we got room to breath even for the
more odd (or huge) use cases (2nd point)
- a per-level length of 32 (-1 due to separator) is enough to use
telling names, making lives of users and admin simpler, but not
blowing up parameter total length with the max depth of 8
- 8 * 32 = 256 which is nice buffer size
Much thanks for Wolfgang for all the great work on the type
implementation and assisting greatly with the design.
Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This allows fixing up things such as `skip_serialize_if`
calls like so:
#[derive(Updater)]
struct Foo {
#[serde(skip_serializing_if = "MyType::is_special")]
#[updater(serde(skip_serializing_if = "Option::is_none"))]
field: MyType,
}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This would allow the following components:
* all in one : &["system/network"]
* mixed: &["system/network", "dns"]
* with templates: &["datastore/{store}"]
* with the value of template being a path, e,g, with ns = "foo/bar":
&["/datastore/{store}/{ns}"]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Returning the GC status was dropped by mistake in commit fdcb2694
("datastore status: factor out api type DataStoreStatusListItem")
As this is considered a breaking change which we also felt, due to
the gc-status being used in the web interface for the datastore
overview list (not the dashboard), re add it.
Fixes: fdcb2694
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: add reference to breaking commit, reword message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
these should not be required as the use cases should all be
covered by the non-reference impls
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
so our examples can more easily access a datastore without
going over a configuration & cache
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
let BackupGroup implement Hash
let BackupGroup and BackupDir be AsRef<BackupGroup>
let BackupDir be AsRef<BackupDir>
the pbs-datastore types will implement these AsRefs as well
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The type is a real enum.
All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.
Also, they are now flattened into a few structs instead of
being copied manually.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
use where clauses where the parameter list is short enough
to become a single line
easier to read
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
similar to our DeflateEncoder, takes a Stream and implements it itself,
so that we can use it as an adapter for async api calls
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
inspired by tar::Builder, but limited to the things we need and using
AsyncRead+AsyncWrite instead of the sync variants.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
namely 'StreamingSync' and 'StreamingAsync'
in rest-server by using the new formatter function,
and in the debug binary by using 'to_value'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
that takes the data in form of a `Box<dyn SerializableReturn + Send>`
instead of a Value.
Implement it in json and extjs formatter, by starting a thread and
stream the serialized data via a `BufWriter<SenderWriter>` and use
the Receiver side as a stream for the response body.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
they should behave like their normal variants, but return a
`Box<dyn SerializableReturn + Send>` instead of a value. This is useful
since we do not have to generate the `Value` in-memory, but can
stream the serialization to the client.
We cannot simply use a `Box<dyn serde::Serialize>`, because that trait
is not object-safe and thus cannot be used as a trait-object.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this will be useful as a generic return type for api calls which
must implement Serialize.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this wraps around a tokio Sender for Vec<u8>, but implements a blocking
write. We can use thas as an adapter for something that only takes a
writer, and can read from it asynchonously
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The entries in a file go from oldest end-time in the first time to
newest end-time in the last line. So, just because the first line is
older than the cut-off time, the remaining one doesn't necessarily
have to be old enough too. What we can know for sure that older than
the current checked rotations of the task archive are definitively up
for deletion.
Another possibility would be to check the last line, but as scanning
backwards is more expensive/complex to do while only being an actual
improvement in a very specific edge case (it's more likely to have a
mixed time-cutoff vs. task-log-file boundary than that those are
aligned)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by not bubbling up most errors, and continuing on. this avoids that we
stop cleaning up because e.g. one directory was missing.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
add support for multi-line comments to node.cfg and the api, similar to
how pve handles multi-line comments
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
`FoldSeqVisitor` doesn't actually own a `T` and therefore
cannot drop a `T`, we only use it via the `Fn(&mut Out, T)`,
so use `fn(T)` in the `PhantomData` to keep `T`
contravariant.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
ureq has support for a HTTP proxy, but no support for HTTPS proxy yet.
ureq doesn't query `all_proxy` and `ALL_PROXY` environment variables by
itself, the way curl does. So set the proxy in code if any of the above
environment variables are set.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
the latest changes to this api call changed/removed some things that
were actually necessary for the gui. Readd those and document them this
time.
The change from u64 to i64 limits us to 8EiB of Datastore sizes (instead if
16EiB) but if we reach that, we must adapt most other parts to use 128bit
sizes anyway
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
with two variants:
(expr, expr) => assumes that the second is an 'Error'
(expr, (tt)+) => passes the tt through anyhow::format_err
also added tests
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order
Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option so that the admin can tune that behaviour
for each datastore.
The default stays the same (sorting by inode)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
'unwrap_' because they will panic and as `const fn` since
panic in const fn is now possible
Note that const evaluation will only be triggered when
actually used in const context, so to ensure *compile time*
checks, use something like this:
const FOO_SCHEMA: &AllOfSchema =
SomeType::API_SCHEMA.unwrap_all_of_schema();
then_use(FOO_SCHEMA);
or to use the list of enum values of an enum string type
with compile time checks:
const LIST: &'static [EnumEntry] =
AnEnumStringType::API_SCHEMA
.unwrap_string_schema()
.unwrap_format()
.unwrap_enum_format();
for values in LIST {
...
}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
and move the comment from the local io_bail in pbs-client/src/pxar/fuse.rs
to the only use
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this removes quite a bit of dependecies of proxmox-async
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[set proxmox-lang dep to 1.1]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this removes proxmox_sys as a dependecy for proxmox-async
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This allows free form text to exist within property strings,
quoted, like:
key="A value with \"quotes, also commas",key2=value2
or also:
"the value for a default_key",key2=value2
And drop ';' as a key=value separator since those are meant
for arrays inside property strings...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
for ease of use implement these traits for ParameterError:
* From<(&str, Error)>
* From<(String, Error)>
* Extend<(String, Error)>
* Extend<(&str, Error)>
* IntoIterator<Item = (String, Error)>
and add the following methods:
* fn into_inner(self) -> Vec<(String, Error)>;
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this was once a callback in an early version, but it changed to a
channel, but the comment was not updated
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
std::convert::{TryFrom, TryInto} are now part of the prelude.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Rather than going from a list of `(&str, &str)` tuples to a
`HashMap<String, String>`, add a `.tag()` builder method
and use `Cow` behind the scenes to efficiently allow the
caller to choose between a static literal and a `String`
value.
Previously the methods forced `&str` slices and then
always-copied those into `String`s even if the caller could
just *move* it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
influxdb (udp + http(s)) only for now
general architecture looks as follows:
the helper functions influxdb_http/udp start a tokio task and return
a Metrics struct, that can be used to send data and wait for the tokio
task. if the struct is dropped, the task is canceled.
so it would look like this:
let metrics = influxdb_http(..params..)?;
metrics.send_data(...).await?;
metrics.send_data(...).await?;
metrics.join?;
on join, the sending part of the channel will be dropped and thus
flushing the remaining data to the server
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[renamed proxmox_async::io::udp -> proxmox_async::net::udp]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
so that we do not have to always check the target ipaddr family manually
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
these need to be checked (and are) via libssl anyway before persisting,
and newer versions might contain new ciphers/variants/... (and things
like @STRENGTH or @SECLEVEL=n were missing).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
there isn't really a concept of 'nodes' in PBS (yet) anyway - and if
there ever is, it needs to be handled by the rest-server / specific API
endpoints (like in PVE), and not by the schema.
this allows dropping proxmox-sys from pbs-api-types (and thus nix and
some other transitive deps as well).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's the only thing requiring openssl in pbs-api-types, and it's only
used by the client to pretty-print the 'master' key, which is
client-specific.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this flag marks the file names as 'UTF-8' encoded if they are valid UTF-8.
By default, encoding of file names in zips are defined as code page 437,
but we save the filenames as bytes (like in linux fs).
For linux systems this would not be a problem since most tools
simply use the filenames as bytes, but for the zip utility under
windows it's important since NTFS uses UTF-16 for file names.
For filenames that are valid UTF-8, they are decoded as UTF-8 everywhere
correctly (Linux as UTF-8 bytes, Windows as correct UTF-16 sequence) and
for other filenames with a high bit set, it depends on the OS/Software
what exactly happens. Some cases below:
* Windows + Built-in/7zip: decoded as CP437
* Debian + zip: Bytes taken as-is
* Debian + 7z: interpreted as Windows1252, decoded as UTF-8
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the doctest code uses non-public `fold`, up for re-evaluation if this
gets moved to proxmox-serde and made public..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
same as with sys/xattr, these touch files, so should use a tmpdir
provided by cargo, which requires them being integration tests.
if the tmpdir doesn't support O_TMPFILE (like overlayfs), the test is
not run (unfortunately, there is no way to indicate this via the test
result like with other test frameworks).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
these touch files, so should use the cargo-provided tmp dir, but that is
only available to benchmarks and integration tests, not unit tests.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
without these, the generated d/control files are incomplete and builds
fail on clean systems.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Some users want to import historical backups but they run into the
original lower backuo-time limit one can pass. That original limit
was derived from the initial PBS development start in 2019, it was
assumed that no older backup can exist with PBS before it existing,
but imports of older backups is a legitimate thing.
I pondered using 683071200 (1991-08-25), aka the first time Linux was
publicly announced by Linus Torvalds as new limit but at the end I
did not wanted to risk that and backup software is IMO to serious for
such easter eggs, so I went for 1, to differ between the bogus 0 some
tools fallback too if there's something off with time.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
will break usage of the `Result::Ok()' with anyhow 1.0.49+ as that
added a new Ok helper, so a glob-import would make that shadow the
core one.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
will break usage of the `Result::Ok()' with anyhow 1.0.49+ as that
added a new Ok helper, so a glob-import would make that shadow the
core one.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
like we do in pve. this breaks the (newly added) compute_next_event
method of CalendarEvent, but we can keep compatibilty with the
deprecated function
adapt the tests so that they add a ' UTC' by default
(so they can run reliably on any machine)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to be compatible with our perl calendar events, we have to make hour optional
in that case we select every hour, so 'X' is the same as writing '*:X'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Most functionality is now in proxmox-sys. The common-regex.rs is
moved back into proxmox-backup-server, because it is only used there.
Serde code is now in new proxmox-serde crate.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
commit 8c1ec5c802 introcuded a bug by
using fp.to_string(). Replace this with fp.signature() which correctly
returns the full fingerprint instead of the short version.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
otherwise we cannot properly parse the api return value from older
versions, since that field does not exist there.
fixes sync from older versions without the protected feature
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Adapted from Dietmar's v3 on pbs-devel but some changes:
- reworked with a strip_suffix fn that does matching, way shorter and
even easier to read IMO
- make b/B byte symbol fully optional, not just for base-10
- also trim trailing whitespace for SizeUnit::Byte
- simplify the FromStr impl
- adapt parser unit tests such that we actually see the failed test's
definition line, simplifies debugging a bit
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The new SizeUnit type takes over the auto scaling logic and could be
used on its own too.
Switch the internal type of HumanByte from u64 to f64, this results
in a slight reduce of usable sizes we can represent (there's no
unsigned float type after all) but we support pebibyte now with quite
the precision and ebibytes should be also work out ok, and that
really should us have covered for a while..
Partially adapted by Dietmar's version, but split up and change so:
* there's no None type, for a SizeUnit that does not makes much sense
* print the unit for byte too, better consistency and one can still
use as_u64() or as_f64() if they do not want/need the unit rendered
* left the "From usize/u64" impls intact, just convenient to have and
avoids all over the tree changes to adapt to loosing that
* move auto-scaling into SizeUnit, good fit there and I could see
some re-use potential for non-human-byte users in the future
* impl Display for SizeUnit instead of the separate unit_str method,
better usability as it can be used directly in format (with zero
alloc/copy) and saw no real reason of not having that this way
* switch the place where we auto-scale in HumanByte's to the new_X
helpers which allows for slightly reduced code usage and simplify
implementation where possible
* use rounding for the precision limit algorithm. This is a stupid
problem as in practices there are cases for requiring every variant:
- flooring would be good for limits, better less than to much
- ceiling would be good for file sizes, to less can mean ENOSPACE
and user getting angry if their working value is messed with
- rounding can be good for rendering benchmark, closer to reality
and no real impact
So going always for rounding is really not the best solution..
Some of those changes where naturally opinionated, if there's a good
practical reason we can switch back (or to something completely
different).
The single thing I kept and am not _that_ happy with is being able to
have fractional bytes (1.1 B or even 0.01 B), which just does not
makes much sense as most of those values cannot exist at all in
reality - I say most as multiple of 1/8 Byte can exists, those are
bits.o
Note, the precission also changed from fixed 2 to max 3 (trailing
zeros stripped), while that can be nice we should see if we get
a better precision limiting algorithm, e.g., directly in the printer.
Rust sadly does not supports "limit to precision of 3 but avoid
trailing zeros" so we'd need to adapt their Grisu based algorithm our
own - way to much complexity for this though..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
switch WebauthnConfig to use Url for the origin field, via a wrapper
type to make Updater and ApiType happy.
the two new Credential fields `verified` and `registration_policy` are
always set to `false` and `Discouraged`, to get the same behaviour as
before.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Tested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this (external) struct gets new fields in webauthn-rs 0.3, so let's
properly wrap / convert it instead of just aliasing, else deserializing
will fail.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
we even use that for basically all the related schema names, "groups"
allone is just rather not so telling, i.e., "groups" what?
While due to the additive nature of `group-filter` is not the best
possible name for passing multiple arguments on the CLI (the web-ui
can present this more UX-friendly anyway) due to possible confusion
about if the filter act like AND vs OR it can be documented and even
if a user is confused they still are safe on more being synced than
less. Also, the original param name wasn't really _that_ better in
that regards
Dietmar also suggested to use singular for the CLI option, while
there can be more they're passed over repeating the option, each with
a single filter.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
and convert existing (manually created/edited) jobs to the previous
default value of 'true'. the GUI has always set this value and defaults
to 'false'.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
like for manual pulls, but persisted in the sync job config and visible
in the relevant GUI parts.
GUI is read-only for now (and defaults to no filtering on creation), as
this is a rather advanced feature that requires a complex GUI to be
user-friendly (regex-freeform, type-combobox, remote group scanning +
selector with additional freeform input).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
at the API level, this is a simple (wrapped) Vec of Strings with a
verifier function. all users should use the provided helper to get the
actual GroupFilter enum values, which can't be directly used in the API
schema because of restrictions of the api macro.
validation of the schema + parsing into the proper type uses the same fn
intentionally to avoid running out of sync, even if it means compiling
the REs twice.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the regex was already there, and we need a simple type/schema for
passing in multiple groups as Vec/Array via the API.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by storing the proxy url as string in the struct and setting it on
each invocation of `execute`, since execute calls reset on the
curl::easy::Easy object.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
previously, this was only used for the server side handling of web
sockets. by making the mask part of the WebSocket struct and making some
of the fns associated, we can re-use this for client-side connections
such as in proxmox-websocket-tunnel.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
by replacing the whole Vec.
if we ever want to support adding/removing/modifying elements of a Vec
via the Updater, we'd need to extend it anyway (or use a custom
updater).
Suggested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Without this, the tests fail in debug mode.
Also having start (u64) underflow to a value greater than end does
not really make sense
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
adds the info that a snapshot is protected to:
* snapshot list
* manual pruning (also dry-run)
* prune jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
some custom ACME endpoints do not have a TOS, and thus do not return
a meta property at all
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The fsync is required for consistency after power failure, so it should
be set when writing config files or otherwise important data.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
our ui expects a map here with 'field: "error"'. This way it can mark
the relevant field as invalid and correctly shows the complete error
message
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Because the types used inside the RRD have other requirements
than the API types:
- other serialization format
- the API may not support all RRD features
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
it's a rather low-level tool mostly useful for debugging and some of
it is rather "dumb" (by design) anyway, e.g., it does not
transparently applies journal but really only operates on the DB
files as is (which can conflict with daemon operations).
In summary, not (yet) a tool meant for end user consumption.
Move it to examples folder to avoid compilation on packaging (we do
not ship it anyway) which allows us to move the rather expensive
proxmox-router (pulls in hyper) to the dev-dependencies section.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Because the types used inside the RRD have other requirements
than the API types:
- other serialization format
- the API may not support all RRD features
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.
Apply the journal every 30 minutes and on daemon startup.
Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this is a helper that removes task log files that are not referenced
by the task archive anymore
it gets the oldest task archive file, gets the first endtime (the
oldest) and removes all files in the taskdir where the mtime is older
than that
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
else it's rather to subtle and not a nice interface considering that
we only want to have a thin wrapper for sd_notify_barrier..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
until now, we manually polled the systemd service state during a reload
so that the sd_notify messages get processed in the correct order
(RELOAD(old) -> MAINPID(old) -> READY(new))
with systemd >= 246 there is now 'sd_notify_barrier' which
blocks until systemd processed all prior messages
with that change, the daemon does not need to know the service name anymore
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
show how to generally start a daemon that serves a rest api + index page
api calls are (prefixed with either /api2/json or /api2/extjs):
/ GET listing
/ping GET returns "pong"
/items GET lists existing items
POST lets user create new items
/items/{id} GET returns the content of a single item
PUT updates an item
DELETE deletes an item
Contains a small dummy user/authinfo
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).
this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.
The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When printing unicode text, a glyph can take up more (or less) space than
a single column. To handle that, use the 'unicode-width' crate which
calculates the width by the unicode standard.
This makes the text tables correctly aligned when printing unicode
characters (e.g. in a datastore/user/syncjob comment).
'unicode-width' is used itself in the rust compiler to format errors
(see e.g. the Cargo.toml in /compiler/rustc_errors of the rust git)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we can easily go beyond that when having long datastore/remote names
also because we do 'systemd-encode' them, which means that every special
char takes up 4 bytes (e.g. '-' => '\x2d')
while we could just increase the lenght to say 256 or 512, i do not
really see the benefit to limit this at all, since users cannot create
tasks with arbitrary names, and all other fields are generated from
other valid types (username, datastore, remote, etc.)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
In order to avoid name conflicts with WorkerTaskContext
- renamed WorkerTask::log to WorkerTask::log_message
Note: Methods have different fuction signatures
Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.
Use the task_log!() and task_warn!() macros more often.
it's not immediately obvious that they can be specified more than once
otherwise.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this is only used for CLI synopsis/usage strings, the API viewer already
prints the full type text in a correct format. the old variant was also
rather misleading, since on the CLI we don't pass in an array, but each
item as its own parameter.
noticed this while working on the pull/sync filtering series, but it
affects quite a lot of stuff, among other things the Updater and
Deleteable CLI, e.g. from `man proxmox-backup-manager`:
> --delete <array>
> List of properties to delete.
vs.
> --delete disable|validation-delay
> List of properties to delete.
But some of them might only have <string> as the item type text,
which is not much nicer but also not really worse.
The whole "List of .." is confusing anyway, but not easily solvable,
since the description is used for
- API dump/viewer (where it is a list/array of ..)
- usage message/man pages (where it's a parameter that gives a single
element, but it might be passed in multiple times to construct an
array)
Also, for some common occurrences, the item description is too
generic, and it's not possible to override the description for
external types with the current api macro.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[ Thomas: Added more context that was in the diffstat of the path ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ApiConfig: avoid using pbs_config::backup_user()
CommandoSocket: avoid using pbs_config::backup_user()
FileLogger: avoid using pbs_config::backup_user()
- use atomic_open_or_create_file()
Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Handle auth logs the same way as access log.
- Configure with ApiConfig
- CommandoSocket command to reload auth-logs "api-auth-log-reopen"
Inside API calls, we now access the ApiConfig using the RestEnvironment.
The openid_login api now also logs failed logins and return http_err!(UNAUTHORIZED, ..)
on failed logins.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This need impl UserInformation for Arc<CachedUserInfo> which is implemented
with proxmox 0.13.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ApiConfig: avoid using pbs_config::backup_user()
CommandoSocket: avoid using pbs_config::backup_user()
FileLogger: avoid using pbs_config::backup_user()
- use atomic_open_or_create_file()
Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this prevents us from using it under certain conditions and it's
actually not necessary, so drop them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
* Updatable is now named UpdaterType
* UPDATER_IS_OPTION is now assumed to always be true
While an updater can be a non-optional struct, being an
updater, all its fields are also Updaters, so after
expanding all levels of nesting, the resulting list of
fields can only contain optional values.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This way we can assign `API_SCHEMA` constants to `Option`
types.
Here's why:
The api-macro generated code usese `T::API_SCHEMA` when
building ObjectSchemas.
For Updaters we replace `T` with
`<T as Updatable>::Updater`
This means for "simple" wrappers like our `Authid` or
`Userid`, the ObjectSchema will try to refer to
`<Authid as Updatable>::Updater::API_SCHEMA`
which resolves to:
`Option<Authid>::API_SCHEMA`
which does not exist, for which we cannot add a normal
`impl` block to add the schema variable, since `Option` is
not "ours".
But we now have a blanket implementation of `ApiType` for
`Option<T> where T: ApiType` which just points to the
original `T::API_SCHEMA`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
#[api]
struct Foo {
field: Bar,
}
does not require the use of
#[api(
properties: {
field: {
type: Bar,
},
},
)]
anymore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
which allows to get rid of an possible error with check_suites, and
easily detect unexpected values with get_current_release_codename.
The check_repos function needs to be adapted, since the type does
not include suite names like oldstable,experimental,etc.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
by exchanging loops in the check_suites function, which was the only
user. Exchanging loops also helps for introducing a type for Debian condenames.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Like that, a potential error is further up the stack, and it's more
consistent with what the standard_repository functions do.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
use the first appearance of '-' or '/' to detect the variant instead
of keeping a list of possible variants, which would need to include
things like "-proposed-updates-debug".
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Require that the suite matches too when detecting standard
repositories, since no or invalid updates will be obtained when the
suite is wrong. Thus, it should not be considered to be the same
repository.
Add the parameter for get_standard_repository too, so that the two
related calls have more similar parameters, and the detection of the
current release code name can be done once in the caller once.
This also will fix an issue with the front-end, where adding a
standard repository would end up just enabling an already present
repository with the wrong suite.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Some file systems don't support renameat2's RENAME_NOREPLACE
flag (eg. ZFS), at the some time, some other file systems
don't support hardlinks via link (eg. vfat, cifs), so we now
try both: first the rename (since it's more efficient), then
link+unlink for the rest.
If both fail, the file system is simply not supported for
our purposes anyway...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
pbs-datastore now ended up depending on tokio after all, but
that's fine for now
for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link
the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we are not really compatible with pbs1, needs a stable-1 branch if we
need to backport something
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stored in atomically-updated 'notes' file in backup group directory.
Available via dedicated GET/PUT API calls, as well as the first line
being included in list_groups (similar to list_snapshots).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
a match expresses the fallback slightly nicer and needs no mut,
which is always nice to avoid.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
move key_derivation to pbs-datastore
pbs-api-types should only contain "basic" types which
* are usually required by clients
* don't depend on pbs-related code directly
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We send it already to the user via the response body, but the
log_response does not has, nor wants to have FWIW, access to the
async body stream, so pass it through the ErrorMessageExtension
mechanism like we do else where.
Note that this is not only useful for PBS API proxy/daemon but also
the REST server of the file-restore daemon running inside the restore
VM, and it really is *very* helpful to debug things there..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
so that parsing CD ROM repositories with spaces in the name works too.
But it's not limited to that, and should make one-line parsing rather
similar to what APT does (stanza parsing in APT doesn't use
ParseQuoteWord at all AFAICS).
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Add a description for the handle, which can be useful to display
alongside the name. The descriptions are essentially the first
sentence from PVE's "Package Repositories" docs, but without the
product name.
Also drop the " Repository" suffix from the names, as it's not useful,
but can be ugly: e.g. for the UI when the label already is
'Repository:'.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
which is obtained from the cached InRelease file and also works for
mirrors, host aliases, direct IPs.
The has_official_uri function was replaced by origin_from_uris.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Allows obtaining the 'Origin' property from the cached InRelease file.
Used the once_cell crate for the config module, because it is already
used in proxmox-backup and seemed to be the right fit here.
For now, the config module is just used to be able to override the
path for the test environment, but those are actual APT config
variables, and in the future, it can be extended and used to actually
parse the apt.conf(.d/*) on the system.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Get handles for the available repositories along with their current
configuration status and make it possible to add them.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
- refactor the combinators,
- make it take a `&T: Serialize` instead of a Value, and
allow sending the raw string via `send_raw_command`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
basically the same as commit 5fb15b781e
Will be required once we get to use a newer rustc, at least the
client build for archlinux was broken due to this.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This allows switching the base user identification/authentication method
in the rest server. Will initially be used for single file restore VMs,
where authentication is based on a ticket file, not the PBS user
backend (PAM/local).
To avoid putting generic types into the RestServer type for this, we
merge the two calls "extract_auth_data" and "check_auth" into a single
one, which can use whatever type it wants internally.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
I made some comparision with bombardier[0], the one listed here are
30s looped requests with two concurrent clients:
[ static download of ext-all.js ]:
lvl avg / stdev / max
none 1.98 MiB 100 % 5.17ms / 1.30ms / 32.38ms
fastest 813.14 KiB 42 % 20.53ms / 2.85ms / 58.71ms
default 626.35 KiB 30 % 39.70ms / 3.98ms / 85.47ms
[ deterministic (pre-defined data), but real API call ]:
lvl avg / stdev / max
none 129.09 KiB 100 % 2.70ms / 471.58us / 26.93ms
fastest 42.12 KiB 33 % 3.47ms / 606.46us / 32.42ms
default 34.82 KiB 27 % 4.28ms / 737.99us / 33.75ms
The reduction is quite better with default, but it's also slower, but
only when testing over unconstrained network. For real world
scenarios where compression actually matters, e.g., when using a
spotty train connection, we will be faster again with better
compression.
A GPRS limited connection (Firefox developer console) requires the
following load (until the DOMContentLoaded event triggered) times:
lvl t x faster
none 9m 18.6s x 1.0
fastest 3m 20.0s x 2.8
default 2m 30.0s x 3.7
So for worst case using sligthly more CPU time on the server has a
tremendous effect on the client load time.
Using a more realistical example and limiting for "Good 2G" gives:
none 1m 1.8s x 1.0
fastest 22.6s x 2.7
default 16.6s x 3.7
16s is somewhat OK, >1m just isn't...
So, use default level to ensure we get bearable load times on
clients, and if we want to improve transmission size AND speed then
we could always use a in-memory cache, only a few MiB would be
required for the compressable static files we server.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'
if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This allows anything that can be represented as a UnixStream to be used
as transport for an API server (e.g. virtio sockets).
A tower service expects an IP address as it's peer, which we can't
reliably provide for unix socket based transports, so just fake one.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
it seems that sometimes, the child process signal gets handled
before the parent process signal. Systemd then ignores the
childs signal (finished reloading) and only after going into
reloading state because of the parent. this will never finish.
Instead, wait for the state to change to 'reloading' after sending
that signal in the parent, an only fork afterwards. This way
we ensure that systemd knows about the reloading before actually trying
to do it.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Fabian Ebner <f.ebner@proxmox.com>
when authenticating a token, and not just when authenticating a
user/ticket.
Reported-By: Dominik Jäger <d.jaeger@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
sd_notify is not synchronous, iow. it only waits until the message
reaches the queue not until it is processed by systemd
when the process that sent such a message exits before systemd could
process it, it cannot be associated to the correct pid
so in case of reloading, we send a message with 'MAINPID=<newpid>'
to signal that it will change. if now the old process exits before
systemd knows this, it will not accept the 'READY=1' message from the
child, since it rejects the MAINPID change
since there is no (AFAICS) library interface to check the unit status,
we use 'systemctl is-active <SERVICE_NAME>' to check the state until
it is not 'reloading' anymore.
on newer systemd versions, there is 'sd_notify_barrier' which would
allow us to wait for systemd to have all messages from the current
pid to be processed before acknowledging to the child, but on buster
the systemd version is to old...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of await'ing the result of 'create_service' directly,
poll it together with the shutdown_future
if we reached that, fork_restart the new daemon, and await
the open future from 'create_service'
this way the old process still handles open connections until they finish,
while we already start a new process that handles new incoming connections
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we have information here not available in the access log, especially
if the /api2/extjs formatter is used, which encapsulates errors in a
200 response.
So keep the auth log for now, but extend it use from create ticket
calls to all authentication failures for API calls, this ensures one
can also fail2ban tokens.
Do that logging in a central place, which makes it simple but means
that we do not have the user ID information available to include in
the log.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
re-use the future we already have for task log rotation to trigger
it.
Move the FileLogger in ApiConfig into an Arc, so that we can actually
update it and REST using the new one.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this is internal for now, use the comanndo socket struct
implementation, and ideally not a new one but the existing ones
created in the proxy and api daemons.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This is a preparatory step to replace the task control socket with it
and provide a "reopen log file" command for the rest server.
Kept it simple by disallowing to register new commands after the
socket gets spawned, this avoids the need for locking.
If we really need that we can always wrap it in a Arc<RWLock<..>> or
something like that, or even nicer, register at compile time.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
writing to a file can explode quite easily.
time formatting to rfc3339 should be more robust, but it has a few
conditions where it could fail, so catch that too (and only really
do it if required).
The writes to stdout are left as is, it normally is redirected to
journal which is in memory, and thus breaks later than most stuff,
and at that point we probably do not care anymore anyway.
It could make sense to actually return a result here..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in most generic places. this is accompanied by a change in
RpcEnvironment to purposefully break existing call sites.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and use that in ApiConfig to avoid that it is owned by root if the
proxmox-backup-api process creates it first.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
reuse the FileLogger module in append mode.
As it implements write, which is not thread safe (mutable self) and
we use it in a async context we need to serialize access using a
mutex.
Try to use the same format we do in pveproxy, namely the one which is
also used in apache or nginx by default.
Use the response extensions to pass up the userid, if we extract it
from a ticket.
The privileged and unprivileged dameons log both to the same file, to
have a unified view, and avoiding the need to handle more log files.
We avoid extra intra-process locking by reusing the fact that a write
smaller than PIPE_BUF (4k on linux) is atomic for files opened with
the 'O_APPEND' flag. For now the logged request path is not yet
guaranteed to be smaller than that, this will be improved in a future
patch.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add a generous limit now and return the correct error (414 URI Too
Long). Otherwise we could to pretty larger GET requests, 64 KiB and
possible bigger (at 64 KiB my simple curl test failed due to
shell/curl limitations).
For now allow a 3072 characters as combined length of URI path and
query.
This is conform with the HTTP/1.1 RFCs (e.g., RFC 7231, 6.5.12 and
RFC 2616, 3.2.1) which do not specify any limits, upper or lower, but
require that all server accessible resources mus be reachable without
getting 414, which is normally fulfilled as we have various length
limits for stuff which could be in an URI, in place, e.g.:
* user id: max. 64 chars
* datastore: max. 32 chars
The only known problematic API endpoint is the catalog one, used in
the GUI's pxar file browser:
GET /api2/json/admin/datastore/<id>/catalog?..&filepath=<path>
The <path> is the encoded archive path, and can be arbitrary long.
But, this is a flawed design, as even without this new limit one can
easily generate archives which cannot be browsed anymore, as hyper
only accepts requests with max. 64 KiB in the URI.
So rather, we should move that to a GET-as-POST call, which has no
such limitations (and would not need to base32 encode the path).
Note: This change was inspired by adding a request access log, which
profits from such limits as we can then rely on certain atomicity
guarantees when writing requests to the log.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
needs new proxmox dependency to get the RpcEnvironment changes,
adding client_ip getter and setter.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The 'Ok::<_, Self::Error>(res)' type annotation was from a time where
we could not use async, and had a combinator here which needed
explicity type information. We switched over to async in commit
df52ba5e45 and, as the type annotation
is already included in the Future type, we can safely drop it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
while we probably do not add much more to them, it still looks ugly.
If this was made so that adding a World readable API call is "hard"
and not done by accident, it rather should be done as a test on build
time. But, IMO, the API permission schema definitions are easy to
review, and not often changed/added - so any wrong World readable API
call will normally still caught.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.
Clamp the value to 0 in that case.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
- remove chrono dependency
- depend on proxmox 0.3.8
- remove epoch_now, epoch_now_u64 and epoch_now_f64
- remove tm_editor (moved to proxmox crate)
- use new helpers from proxmox 0.3.8
* epoch_i64 and epoch_f64
* parse_rfc3339
* epoch_to_rfc3339_utc
* strftime_local
- BackupDir changes:
* store epoch and rfc3339 string instead of DateTime
* backup_time_to_string now return a Result
* remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir
- DynamicIndexHeader: change ctime to i64
- FixedIndexHeader: change ctime to i64
- remove chrono dependency
- depend on proxmox 0.3.8
- remove epoch_now, epoch_now_u64 and epoch_now_f64
- remove tm_editor (moved to proxmox crate)
- use new helpers from proxmox 0.3.8
* epoch_i64 and epoch_f64
* parse_rfc3339
* epoch_to_rfc3339_utc
* strftime_local
- BackupDir changes:
* store epoch and rfc3339 string instead of DateTime
* backup_time_to_string now return a Result
* remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir
- DynamicIndexHeader: change ctime to i64
- FixedIndexHeader: change ctime to i64
Also swap the order of a couple of `.map_err().await` to
`.await.map_err()` since that's generally more efficient.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
is a helper to spawn an internal tokio task without it showing up
in the task list
it is still tracked for reload and notifies the last_worker_listeners
this enables the console to survive a reload of proxmox-backup-proxy
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of exposing handlebars itself, offer a register_template and
a render_template ourselves.
render_template checks if the template file was modified since
the last render and reloads it when necessary
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Especially helpful for requests not coming from browsers (where the
URL is normally easy to find out).
Makes it easier to detect if one triggered a request with an old
client, or so..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We get the path to our executable via a readlink() on
"/proc/self/exe", which appends a " (deleted)" during
package reloads.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
and drop the now unused extract_lists function
this also fixes a bug, where we did not add the datastore to the list at
all when there was no rrd data
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
there is now a 'extract_cached_data' which just returns
the data of the specified field, and an api function that converts
a list of fields to the correct serde value
this way we do not have to create a serde value in rrd/cache.rs
(makes for a better interface)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this is an interface to simply get the Vec<Option<f64>> out of rrd
without going through serde values
we return a list of timestamps and a HashMap with the lists we could find
(otherwise it is not in the map)
if no lists could be extracted, the time list is also empty
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
using a handlebars instance in ApiConfig, to cache the templates
as long as possible, this is currently ok, as the index template
can only change when the whole package changes
if we split this in the future, we have to trigger a reload of
the daemon on gui package upgrade (so that the template gets reloaded)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).
When we need to depend on "parts" of the crate later on
we'll just have to use features.
The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
some fitting rules copied over from PVE's ext6-pve.css file.
simply place it in the css subfolder where the proxmox-backup-gui.js
file is hosted and add a "css/" alias for that directory, the
formatter gets use the right content type with that.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
To ensure the new process' parent is pid 1, so systemd won't
complain about supervising a process it does not own.
Fixes the following log spam on reloads:
Apr 25 10:50:54 deb-dev systemd[1]: proxmox-backup.service: Supervising process 1625 which is not our child. We'll most likely not notice when it exits.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Now that we let hyper shutdown gracefully we need an owned
version of the listening socket to prevent it from closing
before running the reload preparations.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.
The struct handling the reload is a 'Reloader', not a
'ReexecStore'.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Proxy and daemon for now just want to handle reload via
`SIGHUP`, so provide a helper creating the signalfd stream
doing that - this is simply a filtered stream which passes
the remaining signals through, so it can be used exactly
like the signalfd stream could before to add more signals.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We want to run the public server as user www-data. Requests needing
root priviledges needs to be proxied to the proxmox-backup.service, which
now listens to 127.0.0.1:82.
"Group filter based on group identifier ('group:GROUP'), group type ('type:<vm|ct|host>'), or regex ('regex:RE'). Can be inverted by prepending 'exclude:'.")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
