IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
To quote from RFC 9110 [1]:
A user agent SHOULD send Content-Length in a request when
the method defines a meaning for enclosed content and it
is not sending Transfer-Encoding. For example, a user agent
normally sends Content-Length in a POST request even when
the value is 0 (indicating empty content).
A user agent SHOULD NOT send a Content-Length header field
when the request message does not contain content and the
method semantics do not anticipate such data.
It seemed like our HTTP client lib did not set the header
automatically, which is why we should do it manually.
While most services seemed to have worked fine without setting
the header, some Microsoft services seem to require it
to accept the webhook request [2].
[1] https://datatracker.ietf.org/doc/html/rfc9110#name-content-length
[2] https://forum.proxmox.com/threads/158827
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Currently the anyhow error context of a given error is not included
in the error message, as `to_string` does use the default formatting
[0].
Include the error context, formatting it as single line as the
message is also shown to the users in e.g. the Proxmox Backup Severs
task state in the UI.
[0] https://docs.rs/anyhow/latest/anyhow/struct.Error.html#display-representations
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
These will produce an error in edition 2024 otherwise. The reason this
is needed is because the `unsafe` block has its own scope.
The bytes were defined inside of the let-mut block to preserve the
lifetime they had before this commit.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Prevents the following error:
```
error[E0597]: `inner` does not live long enough
--> proxmox-async/src/broadcast_future.rs:109:24
|
107 | inner: Arc<Mutex<BroadCastFutureBinding<T>>>,
| ----- binding `inner` declared here
108 | ) -> impl Future<Output = Result<T, Error>> {
109 | let mut data = inner.lock().unwrap();
| ^^^^^ borrowed value does not live long enough
...
121 | data.broadcast.listen()
| ----------------------- argument requires that `inner` is borrowed for `'static`
122 | }
| - `inner` dropped here while still borrowed
error[E0597]: `data` does not live long enough
--> proxmox-async/src/broadcast_future.rs:121:9
|
109 | let mut data = inner.lock().unwrap();
| -------- binding `data` declared here
...
121 | data.broadcast.listen()
| ^^^^-------------------
| |
| borrowed value does not live long enough
| argument requires that `data` is borrowed for `'static`
122 | }
| - `data` dropped here while still borrowed
```
The use<...> pattern was introduced in rust 1.82.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
To write result message manually, bypassing tracing.
The workertasks currently get their status from parsing the log
messages in the task-log file. The problem is that if these messages are
filtered – which is now possible using the PBS_LOG env variable – some
workertasks will end up with a "stopped: unknown" status. This is not
desirable so write the message manually to the workertask file and
bypass tracing.
This way we are guaranteed that, regardless of the max logging level the
user sets, the final message (and status) is written.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Add a layer that outputs messages to stderr in a specific format. In
PVE, stderr is rerouted to the tasklog if the we are within a
workertask. Therefore, ensure the stderr output is formatted
appropriately.
Reported-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Add a builder-like struct to compose a tracing logger using different
layers. Instead of having an init function per product/binary or
super-specific init functions that describe the logger, have a dynamic
builder. The builder improves the usability and makes the logging
initialization more self-explaining.
Suggested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
The `new_args` Vec is directly passed to the other Vec's `.extend()`,
which takes an `IntoIterator` consuming it, so just pass the
intermediate `Iterator`.
The `rest` Vec owns its strings and we don't need it afterwards, so
similarly, we can consume it via `.extend()` instead of a manual
push(s.clone()) loop.
The .truncate(0) can just be .clear() - they are equivalent according
to their documentation.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
a previous commit fixed up all intra doc links that were present on
public apis, this also fixes the links for private members.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this fixes intra document links or rephrases the documentation in a
more appropriate way to remove all `broken_intra_doc_links` warnings.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this panics when running `cargo test` otherwise, as the api macro
requires fields in `ObjectSchema`s to be sorted now.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
commit 68b13965 (router: docs: add horizontal line before nested
command docs) broke the nested command group test case. this commit
adapts the expected output accordingly.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
while the lint is correct about how these enum members should be
capitalized, the enum is marked as `pub` and all users of it would
need to adapt. so ignore the lint for now [1].
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
while this is a reasonable convention to follow, in this case the new
function is part of a public trait and changing the signature would
force all users to adapt. so ignore the lint for now [1].
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#new_ret_no_self
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
while the function would be more useful as pointed out by the clippy
lint, it i currently `pub` and users of the function would need to
adapt to the change here. so ignore the lint for now.
[1]: https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
the `type_complexity` clippy lint [1] is intended to make the code
more legible in most cases. however, the lint triggers on a member of
a private enum, an example minimal rest server and a private static
variable here. so the benefits of declaring a new type that would
encapsulate this complexity is minimal. hence, ignore the warnings for
now.
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
the `ApiHandler`'s `PartialEq` implementation makes heavy use of
`transmute`. clippy wants the types to be explicitly stated here and
not inferred, to avoid potential undefined behaviour if an unexpected
type is inferred. however, the types that would be inferred here are
so complex, that the code would become illegible anyway, so ignore
this lint for now.
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#missing_transmute_annotations
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
while usually this would improve ergonomics, in this case it isn't
clear whether all uses of `FromStr` would be considered valid here.
renaming the function would also make the type more confusing to use
as `from_str_with_default` also exists, so keep this for consistency.
this ignores a clippy lint [1].
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#should_implement_trait
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this fixes a clippy lint that complains about map invocations followed
by any invocations that are just checking for identity as this can be
replaced by just the any invocation alone [1].
[1]:
https://rust-lang.github.io/rust-clippy/master/index.html#map_all_any_identity
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
While *technically* breaking as it changes the method signature for
the `create_ticket` call to use a struct for its parameters, this is
only (supposed to be) used via its `CREATE_TICKET_API_METHOD` handler
to be passed to a router. Direct use of this does not make sense.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
if the client knows the auth cookie's name, it now passes it on to the
relevant parts of `proxmox-login` so that the ticket is send the
correct cookie
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this should make it possible to use the proxmox-client crate outside
of context where HttpOnly cookies are handled for us. if a cookie
name is provided to a client, it tries to find a corresponding
`Set-Cookie` header in the login response and passes tries to parse
it as a ticket. that ticket is then passed on to proxmox-login like
any regular ticket.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
previously the name in which the ticket was send was derived by the
product abbreviation in the ticket itself. the assumption was that
authentication cookies would always have a name like this:
`<PRODUCT_ABBREVIATION>AuthCookie`.
this commit adds helpers that allow specifying the cookie's name by
users of this crate.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
tickets that end in `::ticketinfo` are not properly signed and just
include information such as the timestamp when the ticket was created.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this allows client to be aware that the ticket they manage is
informational only and that the real ticket should have been set via
a HttpOnly cookie.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
depending on the context a client may or may not have access to
HttpOnly cookies. this change allows them to pass such values to
`proxmox-login` to take them into account when parsing login
responses.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
in certain context (for example, the browser), no password needs to be
provided when using HttpOnly cookies as they are handle by said
context. so make renewing ticket with password optional and add a new
helper function that does not require a password.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
tickets created through the new HttpOnly ticket endpoint won't return
a ticket in the password field. so this field will be left empty.
hence make it optional.
the endpoint does return a ticket_info parameter, though, that
includes the information when a ticket needs to be refreshed. so add
a new optional field for that too.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
adds a new endpoint that is useful when dealing with HttpOnly cookies
that cannot be removed by client-side javascript (and by extension
wasm) code. the logout handle simply removes the cookie that is used
for storing the current ticket. this works the same way as it does in
the front-end: by setting an expired cookie with the same name.
as cookies are now prefixed with `__Host-` by default, the cookie here
also needs to be `Secure` and have the same `Path` to not be rejected
by the browser before it can remove the old cookie.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this adds a new endpoint for requesting tickets. instead of returning
the ticket in the responses body, the ticket is set as a HttpOnly
cookie. this has a couple of advantages:
- the cookie cannot be stolen if an attacker downgrades the connection
to http and injects malicious javascript (`HttpOnly`)
- we don't need to rely on the client to make sure that the cookie is
only send in the appropriate context and only over https
connections (`Secure`, `SameSite`).
- the cookie cannot be overwritten by other subdomains, insecure
connections etc. (the default is to prefix them with `__Host-`)
this follows the best practice guide for secure cookies from MDN
[1]. we also set the cookies to expire when the ticket would so that
the browser removes the cookie once the ticket isn't valid anymore.
the endpoint still returns a ticket that only contains the
informational portions of the ticket but not a valid signature. this
is helpful to let clients know when to refresh the ticket by querying
this endpoint again. it still protects the cookie, though, as it
isn't a valid ticket by itself.
[1]: https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Cookies
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this makes sure that newly generated cookies that are prefixed with,
for example, `__Host-`, for security purposes, are correctly picked
up on. otherwise, the new cookies would not be able to yield proper
authentication.
currently this still falls back to insecure non-prefixed cookies. we
should deprecate them in the long-term and remove this fallback.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this adds the function `prefixed_auth_cookie_name` to the
`AuthContext` trait. said function can be used by users of this crate
to modify the expected prefix of the auth cookie. most products
should be able to use the default of `__Host-` though, so this also
adds a default implementation.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this allows us to write api handlers that have access to a request's
headers and to create a low level response while being able to also
specify the parameter in the request's body. this is useful for
endpoints that should not use url parameters, but still need to
access/set specific headers.
previously, `AsyncHttp` did not allow for parameters that were marked
as non-optional to be passed in the body of a request.
as a side-effect, the body is parsed by the rest server to check for
parameters and consumed. so it cannot be passed on by the handler.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this function does not require ownership of the parts parameter, so we
can simply borrow it. this way we can pass the parameter on to a
handler that consumes them even when using `get_request_parameter`
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
(it's on the wrong function, and also it's wrong - std's str
comparison forwards to the byte comparison anyway since this is fine
for utf-8 if we're not doing anything natural-language-aware...)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This verifies *at compile time* that the properties and variants of
ObjectSchemas and OneOfSchemas are sorted.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Pretty much all functions accepting `CreateOptions` take a value and not
a reference, so I've found myself using `.clone()` quite often in code
I've written recently.
The struct is only 24 bytes large (verified by a
`std::mem::size_of::<CreateOptions>()`), so it should be absolutely fine
to just derive Copy for it.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This fixes the reStructuredText (ReST) output for more complex formats
where we got, for example, a definition list of all properties of a
(section) configuration with one or more properties then having a
sub-format that is also rendered as a list.
Such nested lists are possible in ReST but need to be separated by a
blank line [0].
Without this we got quite a few warnings/errors from the Sphinx/ReST
buildsystem looking like:
> config/datastore/config.rst:19: ERROR: Unexpected indentation.
> config/datastore/config.rst:20: WARNING: Block quote ends without a blank line; unexpected unindent.
[0]: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks
Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Improved errors when panics occur and the panic message is a
formatted (not static) string. This worked already for &str literals,
but not for Strings.
Downcasting to both &str and String is also done by the Rust Standard
Library in the default panic handler. See:
b605c65b6e/library/std/src/panicking.rs (L777)
Signed-off-by: Laurențiu Leahu-Vlăducu <l.leahu-vladucu@proxmox.com>
Move over the whole history of pbs-api-types from the proxmox-backup
git repo to the common workspace as we want to re-use that in PDM and
the yew UI components, and thus require a real source-code package for
this crate.
Choose this repo over proxmox-api-types to avoid the need to copy this
build-system over there, rather we want to also merge pve-api-types
from that repo into ours here in the future.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Fixes a style issue introduced by b31ab119 ("client: improve api error
message (avoid duplicate status code)"), as one always should use
inline variables for format template strings even if there are some
that cannot be used, e.g. as some method needs to be called on them
like here. The reason for this is to reduce the amount of
free-standing "{}" and parameters as that reduces the need to manually
match what "{}" resolves to which expression. While it is not that bad
for only two format variables it's still not winning us anything if we
don't do it and breaks consistency with newer code style.
Noticed as that commit made the line overly long, causing rustfmt
changing the line.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The libc function mkdtemp takes a C char pointer while we previously
cast our OSString buffer as i8 pointer, but that's not valid on
platforms like AArch64 (ARM), where char is equivalent with a u8.
Fix that by using the c_char type that was explicitly made to always
get the correct, platform-independent type for C chars when doing FFI.
This was reported by OJaksch on our Arch Linux User Repo (AUR) package
[0].
https://aur.archlinux.org/packages/proxmox-backup-client#comment-1006851
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This way we can copy and modify a schema.
Eg. via the following ways:
const FOO_SCHEMA: Schema = SOME_SCHEMA
.unwrap_integer_schema_cloned()
.description("Foo")
.schema();
Note that for example there is currently no builder to set a
`default_key` for an `ObjectSchema` back to None, so one could do:
const FOO_SCHEMA: Schema = const {
let mut schema = SOME_SCHEMA.unwrap_object_schema_cloned();
schema.default_key = None;
schema.schema()
};
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Because the UI kept producing them...
And still does...
This is NOT meant to be used by anything other than generated legacy
code for PDM (pve-api-types) and only affects the serde based
deserializer. The "old" `schema.parse_property_string() -> Value`
method does not utilize this for now.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
To prepare for a breaking change in proxmox-schema.
Since new fields in the schema constitute a breaking change, using the
builder methods will let this compile later as well.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
so that we can have a property:
```
foo: Option<PropertyString<Bar>>,
```
within a struct that derives `Updater`
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We tried this unconditionally on start-up in the PDM for the priv. API
daemon, but we actually only want to clean-up on fresh bind, not on
restoring the FD on daemon reload. Otherwise the unprivileged daemon
cannot connect to the privileged one anymore after the latter got
reloaded.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For consistency with it's other repositories, we might migrate other
products also to this schema with a future major release (and nginx
rewrite config for backward compatibility)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Migrated from `log` to `tracing`. Imported `tracing` only as it has a
smaller footprint (and less dependencies) than `proxmox_log`.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
with edition 2024 `static mut` references will be disallowed [1]. the
recommended way to work around this is to use inner mutability, so use a
`OnceLock` for the `PRODUCT_CONFIG` as that should not change throughout
the run time of an application.
[1]:
https://doc.rust-lang.org/nightly/edition-guide/rust-2024/static-mut-references.html
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
in edition 2024 references to mutable static will be disallowed [1]. the
recommended way around this is to use types with inner mutability. so
use a `OnceLock` for `ACME_ACME_CONFIG` as the directoryfor ACME
configuration purposes shouldn't change throughout the run time of an
application.
[1]:
https://doc.rust-lang.org/nightly/edition-guide/rust-2024/static-mut-references.html
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Fixes the manual_flatten clippy lint:
```
warning: unnecessary `if let` since only the `Some` variant of the iterator element is used
--> proxmox-apt/src/repositories/mod.rs:40:5
|
40 | for digest in digests.values() {
| ^ ---------------- help: try: `digests.values().copied().flatten()`
| _____|
| |
41 | | if let Some(digest) = digest {
42 | | common_raw.extend_from_slice(&digest[..]);
43 | | }
44 | | }
| |_____^
|
help: ...and remove the `if let` statement in the for loop
--> proxmox-apt/src/repositories/mod.rs:41:9
|
41 | / if let Some(digest) = digest {
42 | | common_raw.extend_from_slice(&digest[..]);
43 | | }
| |_________^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten
= note: `#[warn(clippy::manual_flatten)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
FG: use `into_values().flatten()` instead of `values().copied().flatten()
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Fixes the clippy lint:
```
warning: empty line after outer attribute
--> proxmox-sys/src/systemd.rs:7:1
|
7 | / #[allow(clippy::manual_range_contains)]
8 | |
| |_
9 | fn parse_hex_digit(d: u8) -> Result<u8, Error> {
| ---------------------------------------------- the attribute applies to this function
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#empty_line_after_outer_attr
= note: `#[warn(clippy::empty_line_after_outer_attr)]` on by default
= help: if the empty line is unintentional remove it
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy lints:
```
warning: empty line after doc comment
--> proxmox-lang/src/lib.rs:33:1
|
33 | / /// ```
34 | |
| |_
35 | #[macro_export]
36 | macro_rules! try_block {
| ---------------------- the comment documents this macro
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#empty_line_after_doc_comments
= note: `#[warn(clippy::empty_line_after_doc_comments)]` on by default
= help: if the empty line is unintentional remove it
warning: empty line after doc comment
--> proxmox-router/src/cli/mod.rs:308:5
|
308 | / /// Can be used multiple times.
309 | |
| |_
310 | /// Finish the command line interface.
311 | pub fn build(self) -> CommandLineInterface {
| ------------------------------------------ the comment documents this method
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#empty_line_after_doc_comments
= note: `#[warn(clippy::empty_line_after_doc_comments)]` on by default
= help: if the empty line is unintentional remove it
help: if the documentation should include the empty line include it in the comment
|
309 | ///
|
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the single_match clippy lint:
```
warning: you seem to be trying to use `match` for destructuring a single pattern. Consider using `if let`
--> proxmox-apt/src/repositories/mod.rs:41:9
|
41 | / match digest {
42 | | Some(digest) => common_raw.extend_from_slice(&digest[..]),
43 | | None => (),
44 | | }
| |_________^ help: try: `if let Some(digest) = digest { common_raw.extend_from_slice(&digest[..]) }`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#single_match
= note: `#[warn(clippy::single_match)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
```
warning: doc list item without indentation
--> proxmox-notify/src/api/webhook.rs:131:5
|
131 | /// (`400 Bad request`)
| ^^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
= note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
|
131 | /// (`400 Bad request`)
| ++
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the manual_unwrap_or_default clippy lint:
```
warning: match can be simplified with `.unwrap_or_default()`
--> proxmox-apt/src/repositories/file.rs:369:30
|
369 | let mut origin = match repo.get_cached_origin(apt_lists_dir) {
| ______________________________^
370 | | Ok(option) => option,
371 | | Err(_) => None,
372 | | };
| |_____________^ help: replace it with: `repo.get_cached_origin(apt_lists_dir).unwrap_or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
= note: `#[warn(clippy::manual_unwrap_or_default)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Factor out the open-flags to use for directories and add the CLOEXEC
flag to ensure that open FDs do not get passed to any child process.
A prominent cases where this can happen is the proxmox-daemon reload
code, which re-execs itself in a forked child-process and thus gets
all FDs passed.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[ TL: expand doc-comment and reword commit message to point at actual
thing this fixes (exec not daemon reload) ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In general we want all open files to have set CLOEXEC since our
reloading mechanism can basically fork at any moment and we don't want
newer daemons to carry around old file descriptors, especially lock
files.
Since `make_tmp_file` is called by many things (e.g. open_file_locked,
logrotate, rrd), set O_CLOEXEC with mkostemp.
This fixes issues with leftover file descriptors e.g. tape backups not
working because of lingering locks after a reload, or having deleted
rrd files open.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this lock can be quite contended, until the surrounding code is properly split
to reduce this contention it should help to give the worker task
creation/cleanup code a bit more breathing room.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this mimics how the count is updated when spawning a new task - the lock scope
needs to cover the count update itself, else there's a race when multiple
worker's log their result at the same time..
Co-developed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if setting up a new worker fails after it has been inserted into the
WORKER_TASK_LIST, we need to clean it up instead of bubbling up the error right
away, else we "leak" the worker task and it never finishes..
a worker task that never finishes will indefinitely block shutdown
of the rest server process, including the "old" process when reloading
the rest server.
this issue was found in the wild on a system with lock contention on the
file-based lock covering task index updating leading to lock acquiring
timeouts.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this moves the mail forwarding implementation from `proxmox-notify` into
`proxmox-sendmail` to cover more `sendmail` related use-cases in one
place.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
use the new `proxmox-sendmail` crate instead of the bespoke
implementation in `proxmox-notify`.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
add the `proxmox-sendmail` crate that makes it easier to send mails via
the `sendmail` utility. features include:
- multipart/alternative support for html+plain text mails
- multipart/mixed support for mails with attachments
- automatic nesting of multipart/alternative and multipart/mixed parts
- masks multiple receivers by default, can be disabled
- encoding Subject, To, From, and attachment file names correctly
- adding an `Auto-Submitted` header to avoid triggering automated mails
also includes several tests to ensure that mails are formatted
correctly. debian packaging is also provided.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
[ TL: update years in d/copyright ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Derive and implement the traits to allow comparison of two
`ApiVersion` instances for more direct and easy api version
comparisons. Further, add some basic test cases to reduce risk of
regressions.
This is useful for e.g. feature compatibility checks by comparing api
versions of remote instances.
Example comparison:
```
api_version >= ApiVersion::new(3, 3, 0)
```
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The `ApiVersion` type was introduced in commit ba850a25
("api/api-types: refactor api endpoint version, add api types")
including the `repoid`, added for completeness when converting from
a pre-existing `ApiVersionInfo` instance, as returned by the
`version` api endpoint.
Drop the additional `repoid` field, since this is currently not used,
can be obtained fro the `ApiVersionInfo` as well and only hinders the
implementation for easy api version comparison.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
and improve the variable namign while we are at it. this allows the check to be
re-used in other code paths, like when starting a garbage collection.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
This parameter causes the FileLogger to duplicate the log output to
stdout. This causes duplicate output on proxmox-backup-manager because
this is now handled by tracing. This should be removed completely in the
future.
In the worst case this will only result in missing log lines on stdout
(which is visible only on proxmox-backup-manager/client invocations
anyway).
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[ TL: add doc-comment to struct, note why it can be removed ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Extend the already present `From<TimeSpan> for f64` implementation to
allow using the reference as well. There is no need to take ownership
and consume the `TimeSpan` object for conversion.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
since the SyncJobConfig struct now contains a 'sync-direction' property, we can
omit the 'direction' property of the SyncJobStatus struct. This makes a
few adaptions in the ui necessary:
* use the correct field
* handle 'pull' as default (since we don't necessarily get a
'sync-direction' in that case)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Jobs for both sync directions are now stored using the same `sync`
config section type, so drop the outdated helpers.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Add the sync direction for the sync job as optional config parameter
and refrain from using the config section type for conditional
direction check, as they are now the same (see previous commit).
Use the configured sync job parameter instead of passing it to the
various methods as function parameter and only filter based on sync
direction if an optional api parameter to distingush/filter based on
direction is given.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
By default, the sync client from proxmox-http (powered by ureq) does not
have any request timeout. To avoid blocking the current thread for a
prolonged period of time, we now make use of the previously added
`Client::new_with_timeout` function to create a new HTTP client with a
default timeout of 10 seconds.
In the long run it would be nicer to have a higher timeout here, say
60s, to cope with flaky and high-latency networks and potentially
overloaded targets. But for that we need to change the architecture of
how notifications are send out to ensure that now thread accepting
connections can be blocked.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
[ TL: Change timeout from 5s to 10s as trade-off and expand commit
message slightly with some reasoning for that still relatively
short time value ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This commits adds the possibility to set a HTTP request timeout for the
sync client.
For now, I've opted to add this as a separate option than can be set via
a separate new_with_timeout method as compared to adding it to the HttpOptions
struct. While it of course would make a lot of sense to add it to the
latter, this would require adding support for request timeouts to the
async client as well. Some users of the async client handle request
timeouts externally via tokio::time::timeout, so these would need to
modified as well. I don't want to touch this at the moment,
so I've opted to introduce the timeout to the sync client only for now.
We can always revisit this at a later time and move the option to the
HttpOptions struct.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Commit 0d66acd3 ("api types: introduce `BackupArchiveName` type")
introduced a dedicated archive name api type to add rust type
checking and bundle helpers to the api type. Since this, the backup
archive name to server archive name mapping is handled by its parser.
This however did not cover the `.conf` extension used for VM config
files. Add the missing `.conf` to `.conf.blob` to the match statement
and the test cases.
Fixes: 0d66acd3 ("api types: introduce `BackupArchiveName` type")
Reported-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
... and deserialize with default if field is missing in data.
Reported-by: Aaron Lauterer <a.lauterer@proxmox.com>
Fixes: 35fb5d4f7f ("pbs-api-types: add mount_status field to DataStoreListItem")
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
So it is possible to reset it after a failed unmount, or abort an
unmount task by resetting it through the API.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
The proxmox-notify crate can render notification text based on two
different templates, plaintext and html. The html template is at the
moment only used for email-based notifications. If we try to render
a html-formatted message but there is no html template, we try to
fall back to the plaintext template and wrap the rendered message
in <pre> tags.
As a preparation for user-supplied/overridden templates, I added a log
message "html template not found, falling back to plaintext ..." to
educate the user about this behavior.
In Proxmox Backup Server, we only ship plaintext templates at the
moment, meaning that this log message will be shown for every single
(email) notification that is sent out. This might be a bit confusing,
because the log message can be interpreted as an error, which it isn't.
This commit removes the log message completely for now. Once we add
support for user-overridable notification templates we could consider
adding it back it, but maybe phrased a bit differently, to avoid it
being interpreted as an error.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Even if we don't have an HTML template available, we always
send an HTML part (the plain text part wrapped in <pre>) to
improve rendering in certain mail clients. This means
we can simply message formatting, since we do not have to
distinguish between single-part and multi-part messages.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
... and add MaintenanceType::Delete to it. We also want to clear any
cach entries if we are deleting the datastore, not just if it is marked
as offline.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Selective because there are quite a few more such old-style format
strings, but I had those already adapted and currently do not have
time do clean-up tree-wide, but it's fine to change this
incrementally.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
That's rather excessive and has not much value for users. So degrade
two of the messages to debug-level.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add a custom handlebars escape function. It's nearly identical to the
default `html_escape` fn [0], but it does not escape the '='. This is
needed to support base64 encoded values.
[0]: https://docs.rs/handlebars/latest/handlebars/fn.html_escape.html
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[ TL: use full width for comment ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we already have two different password schemas, `PBS_PASSWORD_SCHEMA`
being the stricter one, which ensures a minimum length of new
passwords. however, this wasn't used on the change password endpoint
before, so add it there too. this is also in-line with NIST's latest
recommendations [1].
[1]: https://pages.nist.gov/800-63-4/sp800-63b.html#passwordver
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
PathPatterns is hard to distinguish from PathPattern, so would need to be
renamed anyway.. but there isn't really a reason to define a separate API type
just for this.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Introduces a dedicated api type `PathPattern` and the corresponding
format and input validation schema. Further, add a `PathPatterns`
type for collections of path patterns and implement required traits
to be able to replace currently defined api parameters.
In preparation for using this common api type for all api endpoints
exposing a match pattern parameter.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
This option allows us to "fix" corrupt snapshots (and/or their chunks)
by pulling them from another remote. When traversing the remote
snapshots, we check if it exists locally, and if it is, we check if the
last verification of it failed. If the local snapshot is broken and the
`resync-corrupt` option is turned on, we pull in the remote snapshot,
overwriting the local one.
This is very useful and has been requested a lot, as there is currently
no way to "fix" corrupt chunks/snapshots even if the user has a healthy
version of it on their offsite instance.
Originally-by: Shannon Sterz <s.sterz@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Introduces a dedicated wrapper type to be used for backup archive
names instead of plain strings and associated helper methods for
archive type checks and archive name mappings.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
FG: use LazyLock for constant archive names reduces churn, and saves some
allocations
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Moving the `ArchiveType` to avoid crate dependencies on
`pbs-datastore`.
In preparation for introducing a dedicated `BackupArchiveName` api
type, allowing to set the corresponding archive type variant when
parsing the archive name based on it's filename.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In order for sync jobs to be either pull or push jobs, allow to
configure the direction of the job.
Adds an additional config type `sync-push` to the sync job config, to
clearly distinguish sync jobs configured in pull and in push
direction and defines and implements the required `SyncDirection` api
type.
This approach was chosen in order to limit possible misconfiguration,
as unintentionally switching the sync direction could potentially
delete still required snapshots.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a dedicated api type for the `version` api endpoint and helper
methods for supported feature comparison.
This will be used to detect api incompatibility of older hosts, not
supporting some features.
Use the new api type to refactor the version endpoint and set it as
return type.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Make the `BackupGroupDeleteStats` exposable via the API by implementing
the ApiTypes trait via the api macro invocation and add an additional
field to account for the number of deleted groups.
Further, add a method to add up the statistics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In preparation for the delete stats to be exposed as return type to
the backup group delete api endpoint.
Also, rename the private field `unremoved_protected` to a better
fitting `protected_snapshots` to be in line with the method names.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Adding the privileges to allow backup, namespace creation and prune
on remote targets, to be used for sync jobs in push direction.
Also adds dedicated roles setting the required privileges.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add `remote_acl_path` method which generates the acl path from the sync
job configuration. This helper allows to easily generate the acl path
from a given sync job config for privilege checks.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add a `remote_acl_path` helper method for creating acl paths for
remote namespaces, to be used by the priv checks on remote datastore
namespaces for e.g. the sync job in push direction.
Factor out the common path extension into a dedicated method.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
For how to work on the crates in this workspace while actually working
on a separate project without having to constantly reinstall `.deb`
files.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The 'Anyhow' error is not useful and meant for throw-away errors which
cannot be dealt with anyway, and we'd like to be able to tell apart
network problems from actual HTTP responses, so that we can
potentially try a different node in a cluster connection.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
removable datastores will have a PBS-managed mountpoint as path, direct
access to the field needs to be replaced with a helper that can account
for this.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
I considered keeping it as log::trace level, but IMO that's just not
worth it, as just the peek_len is not giving one much more and can
also be basically also gathered through strace.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When a connection is closed by the client before we have enough data
to determine if it contains a TLS Handshake or not, the socket stays
in a readable state.
While we setup a tokio backed timeout of 10s for the connection
build-up here, this timeout does not trigger on said early connection
abort from the client side, causing then the async_io loop to
endlessly loop around peeking into the client, which always returns
the last available bytes before the connection was closed. This in
turn causes 100% CPU usage for one of the PBS threads.
The timeout not triggering is rather odd, and does indicate some
potential for further improvement in tokio itself, but our
questionable use of the WouldBlock error does violate the API
contract, so this is not a clear cut.
Such an early connection abort is often triggered by monitoring
solutions, which use it to relatively cheaply check if TCP on a port
still works as "is service up" heuristic.
To fix this, save the amount of bytes peek returned and if they did
not change between invocations of the callback, we can assume that the
connection was closed and thus exit the connection attempt with an
error.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: reword commit message and change error to ConnectionAborted ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
All in all pretty similar to other endpoint APIs.
One thing worth noting is how secrets are handled. We never ever
return the values of previously stored secrets in get_endpoint(s)
calls, but only a list of the names of all secrets. This is needed
to build the UI, where we display all secrets that were set before in
a table.
For update calls, one is supposed to send all secrets that should be
kept and updated. If the value should be updated, the name and value
is expected, and if the current value should preseved, only the name
is sent. If a secret's name is not present in the updater, it will be
dropped. If 'secret' is present in the 'delete' array, all secrets
will be dropped, apart from those which are also set/preserved in the
same update call.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-By: Stefan Hanreich <s.hanreich@proxmox.com>
This target type allows users to perform HTTP requests to arbitrary
third party (notification) services, for instance
ntfy.sh/Discord/Slack.
The configuration for these endpoints allows one to freely configure
the URL, HTTP Method, headers and body. The URL, header values and
body support handlebars templating to inject notification text,
metadata and secrets. Secrets are stored in the protected
configuration file (e.g. /etc/pve/priv/notification.cfg) as key value
pairs, allowing users to protect sensitive tokens/passwords.
Secrets are accessible in handlebar templating via the secrets.*
namespace, e.g. if there is a secret named 'token', a body
could contain '{{ secrets.token }}' to inject the token into the
payload.
A couple of handlebars helpers are also provided:
- url-encoding (useful for templating in URLs)
- escape (escape any control characters in strings)
- json (print a property as json)
In the configuration, the body, header values and secret values
are stored in base64 encoding so that we can store any string we want.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-By: Stefan Hanreich <s.hanreich@proxmox.com>
A recent commit [1] changed the `Display` implementation of `TimeSpan` such
that minutes are now displayed as `20m` instead of `20min`.
This commit adapts the tests for the notification template renderer
accordingly.
[1] 19129960 ("time: display minute/month such that it can be parsed again")
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
To avoid openssl's unhelpful error messages when the proxy.key or
proxy.pem files have the wrong permissions, we open the files. To load
the private key, we can simply read from the file and pass it to the
`set_private_key` openssl function. Sadly such a function does not exist
for loading certificate chains, so we have to open and close the file
before calling the `set_certificate_chain_file` fn.
Motivation: https://forum.proxmox.com/threads/proxmox-backup-tailscale-proxmox-backup-proxy-service-wont-boot.153204
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
In the reference Bugzilla entry, a certificate with an IP address as a
SAN was used. rustls seems to have problems with that [1].
Also, pretty much all of our code uses native-tls at the moment, so
it makes sense to not pull in a second TLS implementation.
Tested by rebuilding libpve-rs-perl and testing a Gotify notification
target with a self-signed TLS certificate (one that is accepted by
OpenSSL but not by rusttls).
[1] https://github.com/rustls/rustls/issues/184
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
A description is required for the API schema types and we fallback to
the rust doc-comment when no explicit one is set.
But a empty string was returned if no doc-comment existed, so check
for the comment to be non-empty and throw a compile-time error
otherwise.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The wrap_text helper accepts and initial indentation, so use that as
central point to add the indentation that glues the list entry
together with its description.
Mostly a small optimization, should not matter in practice, i.e. where
all properties should have a description.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
There was basically no documentation at all, so try to document the
basic format syntax and where it comes from. The text is partially
adapted from the systemd docs.
Could be still expanded with some example code and the methods might
do good with getting some docs too, but those parts can be relatively
easily be figured out from the code itself, the basic underlying
design and format background is much harder to guess..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Previously we displayed, e.g., "4m 1h 1min", i.e. using "m" for months
and "min" for minutes but "m" was not accepted as month when parsing
a timespan string, so a 4 month timespan would be printed "4m" but if
parsed again it would result in a timespan of 4 minutes.
So switch month to an uppercase "M" and minute to the lower case "m",
which makes renderings of common timespans nicer, as in most of our
use cases they are in the range of minutes to hours, sometimes days
but seldom longer than weeks. So using single letters for all but
"min" stuck out quite a bit, e.g.: "1h 5min 2s" looks odd compared to
"1h 5m 1s"
While the duplicate letter is not 100% ideal it's still better than
the status quo, where rendering and parsing would interpret things
differently.
Also, the order still makes it quite clear, e.g.:
"7m 2w 3d 1h 5min 44s" now becomes "7M 2w 3d 1h 5m 44s"
As a side effect this also brings the display format closer to what is
used inside PVE backup job taks logs.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Seconds are not displayed when the value is smaller than 0.1s and
they are not at the start of the display output, e.g. `1h 2m`.
Drop the additional whitespace currently appended for this edge case.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
(conveted with pandoc and some minor manual fixups)
Mention that the crates should activate `doc_cfg, doc_auto_cfg` and
ideally `#[deny(unsafe_op_in_unsafe_fn)]` and `#[deny(missing_docs)]`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Object schemas can now declare a field which causes
'additional_properties' to be set to true and the field being ignored
in the schema.
This allows adding a flattened HashMap<String, Value> to gather the
additional unspecified properties.
#[api(additional_properties: "rest")]
struct Something {
#[serde(flatten)]
rest: HashMap<String, Value>,
}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
So we can get going on the wasm side where we don't yet have access to
the webauthn-rs crate.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
To ease development on new machines, this provides an easy way to just
do
# apt install $(make list-packages)
to get started.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Print error if the parsing of the env-var fails on the proxmox-backup-*
daemons as well. Output correct env-var on binaries that use different
variables.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Don't panic when the syslog is not available - which happens commonly in
containers and sbuild environments (chroot and unshare) - instead
fallback to stderr.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>
The following targets should be self-explanatory:
# make install
$ make DESTDIR=$SOME_PATH install
# make proxmox-sys-install
$ make DESTDIR=$SOME_PATH proxmox-sys-install
# make proxmox-<any other crate name>-install
Additionally, these are used as building blocks to create
systemd-sysext(8) images:
$ make proxmox-sys-sysext
builds an `extensions/proxmox-sys.raw`
This can be copied/symlinked to `/run/extensions/` and then activated.
As root:
# ln -s $REPO_DIR/extensions/proxmox-sys.raw /run/extensions/
# systemd-systext refresh
For the complete workspace, an `extensions/proxmox-workspace.raw` can
be built via
$ make sysext
This also takes a `CRATES` var to limit the crates which should be
included, and takes an optional `NOCLEAN=1` which prevents cleaning
out the previously installed to "add" new crates on the go:
Assuming there's a symlink like:
# ln -s $REPO_DIR/extensions/proxmox-workspace.raw /run/extensions/proxmox-workspace.raw
One can modify the installed crates like this:
$ make CRATES=proxmox-sys sysext
$ sudo systemd-sysext refresh
Now only the current proxmox-sys crate is overridden.
$ make NOCLEAN=1 CRATES=proxmox-time sysext
$ sudo systemd-sysext refresh
Now proxmox-sys as well as proxmox-time are installed.
To undo the changes, either just do, as root:
# systemd-sysext unmerge
or remove the files which should specifically be dropped from
/run/extensions/ and run as root:
# systemd-sysext refresh
Another way to temporarily install single crates is to just have the
extensions/ folder *be* the `/run/extensions` folder:
# rmdir /run/extensions
# ln -s $REPO_DIR/extensions /run/extensions
Then just build individual extensions:
$ make proxmox-sys-sysext
$ sudo systemd-sysext refresh
$ make proxmox-router-sysext
$ sudo systemd-sysext refresh
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The get/put/post/put_without_body/... methods now have a default
implementation forwarding to a generic `request` method as all our
implementations do the same already anyway.
Additionally, in order to allow easy access to a "streaming body", the
Body type is now exposed.
In the future, this crate may also require a wrapper to standardize
the handling of `application/json-seq` streams if we end up using
them, but for now, a simple way to expose the body is enough to get
going.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These are Iterators or Streams which continuously produce output. They
can either be formatted, in which they are serialized like the as
usually, or, if the client caccepts `application/json-seq` via an
`Accept` header, it will be streamed as a sequence directly.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This does not "stream", but rather skips the intermediate step to
serialize the entire output into a local json string.
We now reserve the "Stream*" prefix for actual *streaming*, that is,
producing an API response which gets streamed continuously as it is
asynchronously produced.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Previously when using `env_logger` all of our cli-tools logged to
stderr, make tracing do the same. Export `tracing::Level` so that we can
use the `tracing::enabled!` macro.
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Deprecate the proxmox-router init_cli_logger function used in client
binaries such as `proxmox-backup-client`, `proxmox-backup-manager`,
'pxar', etc... Add a new init_cli_logger function that uses tracing
instead of env_logger. It checks if the task is in a workertask and
prints the message either to stdout or to the tasklog (this is
neccessary for commands in proxmox-backup-manager that call api handlers
that start workerthreads from the client).
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
If we don't do this, then properties from a serde flattened struct will
be positioned at the end of the list, rather than properly sorted with
the other properties.
Since the tests also feature non-sorted properties, we have to adapt
them too.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The rate and burst parameters are integers, so the mapping from value
with `.as_str()` will always return `None` effectively never
applying any rate limit at all.
Fix it by turning them into a HumanByte instead of an integer.
To not crowd the parameter section so much, create a
ClientRateLimitConfig struct that gets flattened into the parameter list
of the backup client.
To adapt the description of the parameters, add new schemas that copy
the `HumanByte` schema but change the description.
With this, the rate limit actually works, and there is no lower limit
any more.
The old TRAFFIC_CONTROL_RATE/BURST_SCHEMAs can be deleted since the
client was the only user of them.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
While this is technically a breaking API change since the trait is
public, we don't implement it anywhere and it isn't meant to be
implemented from the outside.
Also, encode that these types are all Send + Sync via a super trait
notation.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes:
warning: unresolved link to `seccfg`
--> proxmox-section-config/src/typed.rs:18:71
|
18 | /// If the [`SectionConfig`] returned by the [`section_config()`][seccfg] method includes the
| ^^^^^^ no item named `seccfg` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
= note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default
warning: unresolved link to `seccfg`
--> proxmox-section-config/src/typed.rs:22:10
|
22 | /// [seccfg] ApiSectionDataEntry::section_config()
| ^^^^^^ no item named `seccfg` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: doc list item missing indentation
--> proxmox-access-control/src/acl.rs:518:9
|
518 | /// -- user/token is more specific than group at each level
| ^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
= note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
|
518 | /// -- user/token is more specific than group at each level
| ++
warning: doc list item missing indentation
--> proxmox-access-control/src/acl.rs:519:9
|
519 | /// -- roles lower in the tree are more specific than those higher up along the path
| ^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
help: indent this line
|
519 | /// -- roles lower in the tree are more specific than those higher up along the path
| ++
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: match can be simplified with `.unwrap_or_default()`
--> proxmox-apt/src/cache_api.rs:77:28
|
77 | let mut notified = match cache.notified {
| ____________________________^
78 | | Some(notified) => notified,
79 | | None => std::collections::HashMap::new(),
80 | | };
| |_________^ help: replace it with: `cache.notified.unwrap_or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
= note: `#[warn(clippy::manual_unwrap_or_default)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: this expression creates a reference which is immediately dereferenced by the compiler
--> proxmox-router/src/cli/completion.rs:154:25
|
154 | &completion_functions,
| ^^^^^^^^^^^^^^^^^^^^^ help: change this to: `completion_functions`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
= note: `#[warn(clippy::needless_borrow)]` on by default
warning: this expression creates a reference which is immediately dereferenced by the compiler
--> proxmox-router/src/cli/completion.rs:201:21
|
201 | &completion_functions,
| ^^^^^^^^^^^^^^^^^^^^^ help: change this to: `completion_functions`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes:
warning: unnecessary use of `get(rel_path).is_some()`
--> proxmox-rrd/src/cache/rrd_map.rs:107:21
|
107 | if self.map.get(rel_path).is_some() {
| ^^^^^^^^^^^^^^^^^^^^^^^ help: replace it with: `contains_key(rel_path)`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_get_then_check
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
API tokens between rust & perl code bases are inconsistent... this
needs fixing, but for now this is faster and more compatible.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The original typed section config data would insert and remove the
type properties. With the introduction of `.with_type_key()` this is
done on the parse/write side instead, so we need to be able to opt out
of this.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Before, a call to `update` was necessary to load an existing database
into the cache. If `update` was never called, `extract_cached_data`
would simply return no data.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This function is the same as the regular `update_value`, but it
sets the `new_only` flag when updating the value in the rrd map.
This avoids "time in past" messages being logged in case a data point
happens to be added twice. This new function will just silently reject
values that have an older timestamp than the most recent one.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This crate contains a file-backed, rotating cache.
The cache should be safe to be accessed from multiple processes at
once.
The cache stores the value at the provided path.
If `keep_old` is >0, the cache will keep up to `keep_old` versions
around which can be queried via the `get_last` method.
The value and its history are stored in the same file,
each generation represented by a single line of JSON.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Under the hood, this function calls `mkdtemp` from libc. Unfortunatly
the nix crate did not provide bindings for this function, so we have
to call into libc directly.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the cargo docs warning:
warning: unresolved link to `systemd_notify`
--> proxmox-daemon/src/server.rs:314:6
|
314 | /// [systemd_notify] with [SystemdNotify::Ready](proxmox_systemd::notify::SystemdNotify) when the
| ^^^^^^^^^^^^^^ no item named `systemd_notify` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
= note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the cargo docs warning:
warning: unclosed HTML tag `username`
--> proxmox-login/src/api.rs:35:47
|
35 | /// realm is simply added to the username <username>@<relam>.
| ^^^^^^^^^^
|
= note: `#[warn(rustdoc::invalid_html_tags)]` on by default
warning: unclosed HTML tag `relam`
--> proxmox-login/src/api.rs:35:58
|
35 | /// realm is simply added to the username <username>@<relam>.
| ^^^^^^^
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the following cargo doc warning:
warning: this URL is not a hyperlink
--> proxmox-ldap/src/lib.rs:199:9
|
199 | /// https://www.rfc-editor.org/rfc/rfc4512#section-5.1
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use an automatic link instead: `<https://www.rfc-editor.org/rfc/rfc4512#section-5.1>`
|
= note: bare URLs are not automatically turned into clickable links
= note: `#[warn(rustdoc::bare_urls)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the cargo doc warnings:
warning: redundant explicit link target
--> proxmox-http/src/client/mod.rs:4:19
|
4 | //! in [`Client`](crate::client::Client).
| -------- ^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant
| |
| because label contains path that resolves to same destination
|
= note: when a link's destination is not specified,
the label is used to resolve intra-doc links
= note: `#[warn(rustdoc::redundant_explicit_links)]` on by default
help: remove explicit link target
|
4 | //! in [`Client`].
| ~~~~~~~~~~
warning: redundant explicit link target
--> proxmox-http/src/client/mod.rs:7:22
|
7 | //! [`sync::Client`](crate::client::sync::Client).
| -------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant
| |
| because label contains path that resolves to same destination
|
= note: when a link's destination is not specified,
the label is used to resolve intra-doc links
help: remove explicit link target
|
7 | //! [`sync::Client`].
| ~~~~~~~~~~~~~~~~
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the clippy lint:
warning: bound is defined in more than one place
--> proxmox-schema/src/property_string.rs:352:14
|
352 | pub fn parse<T: ApiType>(value: &str) -> Result<T, Error>
| ^
353 | where
354 | T: for<'de> Deserialize<'de>,
| ^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#multiple_bound_locations
= note: `#[warn(clippy::multiple_bound_locations)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the clippy warning:
warning: doc list item missing indentation
--> proxmox-subscription/src/subscription_info.rs:179:9
|
179 | /// (this mode is used to decide whether to refresh the subscription information)
| ^
|
= help: if this is supposed to be its own paragraph, add a blank line
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
= note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
|
179 | /// (this mode is used to decide whether to refresh the subscription information)
| +
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Fixes the clippy warnings:
warning: `proxmox-apt` (lib) generated 1 warning
warning: using `clone` on type `Option<[u8; 32]>` which implements the `Copy` trait
--> proxmox-apt/tests/repositories.rs:117:22
|
117 | let old_digest = file.digest.clone().unwrap();
| ^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `file.digest`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
= note: `#[warn(clippy::clone_on_copy)]` on by default
warning: using `clone` on type `[u8; 32]` which implements the `Copy` trait
--> proxmox-apt/tests/repositories.rs:135:24
|
135 | file.digest = Some(old_digest.clone());
| ^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `old_digest`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
For when the underlying datatype is supposed to contain the type
property and the schema does not mark it as optional.
The use case here is to support flat `Remote` type where the "type" of
pve/pmg/pbs is a property which is present in the `Remote` struct
while being derived from the section type.
This will implicitly include and strip the type of the json object
after/before de/serializing.
Alternatives would be
- to mark the type as optional and just fill it out later when loading
the data, but that is technically wrong...
- have a 2nd version of the struct with the type field removed and
From/Into implemented, but that's even more unwieldy.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
`OwnedFd`s are now (rustc 1.80+) checked for validity when dropped in a debug
build, to catch usage after closing. Unfortunately those checks don't account
for the special value `AT_FDCWD` (-100) which is not a "real" FD, but a magic
constant used by many libc functions to signify operations starting at the
current working directory.
changing our `cwd` helper to open the CWD for real, instead of just returning
the magic value that pretends to be an FD, works around those limitations with
the least API churn.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
For regular error cases, `std::io::Error` can now also box errors and
provides an `Error::other()` function since rust 1.74.
For the case where it was used directly with strings
(io_err_other("string")) -> that's what `io_format_err!()` and
`io_bail!()` are for.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These structs are returned by the public method
`Deflate{En,De}coder::builder`.
Reported-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
While technically an API break, we don't use the public API for this
anywhere and the trait we're changing is explicitly marked as
`#[doc(hidden)]`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The `_application_name` argument is not used anymore. It was previously
used to set the correct journald unit-name, which is now handled
automatically.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Used only for read/writing a pid_t (an integer) to a socket.
The standard to_ne_bytes()/from_ne_bytes() should be sufficient here,
we already have libc::pid_t which we can use to get the correct type
namespace for accessing ::from_ne_bytes().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
When reaching the final command we relied on the positional parameters
"ending" the global option parsing. This means we did not get global
options at the end, and failed with "unknown option" instead.
Fix this by simply retaining unknown options in that case and not
stopping at positional parameters.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
As per [RFC9110] the Deflate encoding is a "zlib" data format. This
makes the rest-server compatible with the http-client.
[RFC9110] https://www.rfc-editor.org/rfc/rfc9110#field.content-encoding
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
Reorder the filters for the journald layer. This sets the LevelFilter
last, which means tracing can disable all log statements lower than the
current level without evaluating the LogContext::exists function.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[ TL: note that this is just an optimization in the subject ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Deduplicate the check for whether the argument exists by trying to
fetch the schema once and use check the option instead of calling
'.contains_key()' multiple times.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Passing the &GlobalOptions through is more telling than an opaque
Iterator<&str>...
Also: actually generate the property descriptions in non-ReST mode for
global options as well, so the `help` output for a specific command
includes the property documentation instead of only showing the name.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The rules are as follows:
- An "item" by itself does neither start nor end with a newline.
- Where items are connected, the appropriate amount of newlines must
be inserted, assuming items do not have any trailing newlines.
Otherwise this just gets WAY too confusing everywhere!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This is completely wrong and make working with it extremely annoying.
Whether or not there should be separation should be decided where
multiple elements are connected, they shouldn't automatically come
with a bunch of trailing new lines for absolutely no reason.
Places using this will need to be fixed as they get noticed.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This one does *explicitly* *not* support long options with a single
dash because it is too ambiguous if we want to add support for short
options at some point.
The parsing of the command line and invoking of the command is
separated. `CommandLine::parse` returns an `Invocation` which is
called and consumed via its `call` method.
This allows updating the CLI environment between parsing and invoking
the command, in order to allow *handling* the global options in
between those two steps if desired.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The CLI environment can now contain ApiType structs which can be
accessed by their type.
The TypeId is used since the options inside must be unique anyway and
we can't have the same type specified multiple times. It also makes
for a somewhat convenient interface:
env.take_global_option::<ConnectInfo>()
where ConnectInfo is a struct containing the server, user, port, ...
since these will not be passed as *parameters* to the API functions.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The scope and sync_scope methods simply activate the context, they
don't affect the counter, the counter is initialized when creating the
context with LogContext::new().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Since hyper can spawn() more tasks, when we stop passing `WorkerTask`
references down the stack, we still need to be able to *inherit* the
current logging context. Hyper provides a way to replace its used
`spawn()` method, so we need to provide a way to reuse the logging
context.
Instead of having the `FileLogger` and warn counter separately
available with local-only access, put them behind an Arc<Mutex<>>.
Previously they already *were* behind an Arc<Mutex<>> as part of the
WorkerTaskState.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
proxmox_notify is the only user of those functions, so it makes
sense to move them here. A future commit will mark the
original functions from proxmox_sys as deprecated.
The functions were slightly modified, mostly to not
rely on anyhow for error reporting. Also they
are now private functions.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Enable the tracing-system by setting the LOGGER task local variable
to a instance of a FileLogger and initializing the WARN_COUNTER.
Removed the task_log! macros and some occurences.
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[WB: remove flog! import in doctests]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Add the `proxmox_log` crate which includes the new logging infra.
Export the `init_logger` function, which creates the `tracing` logger
that includes the default subscriber and two layer.
The first layer comes from the tracing-journald crate and logs
everything that does not come from a worker-task/thread to the syslog.
The second layer filters the exact opposite and writes the logs into the
corresponding task-log file.
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
On rare occasions, the TLS "client hello" message [1] is delayed after
a connection with the server was established, which causes HTTPS
requests to fail before TLS was even negotiated. In these cases, the
server would incorrectly respond with "HTTP/1.1 400 Bad Request"
instead of closing the connection (or similar).
The reasons for the "client hello" being delayed seem to vary; one
user noticed that the issue went away completely after they turned off
UFW [2]. Another user noticed (during private correspondence) that the
issue only appeared when connecting to their PBS instance via WAN, but
not from within their VPN. In the WAN case a firewall was also
present. The same user kindly provided tcpdumps and strace logs on
request.
The issue was finally reproduced with the following Python script:
import socket
import time
HOST: str = ...
PORT: int = ...
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT))
time.sleep(1.5) # simulate firewall / proxy / etc. delay
sock.sendall(b"\x16\x03\x01\x02\x00")
data = sock.recv(256)
print(data)
The additional delay before sending the first 5 bytes of the "client
hello" message causes the handshake checking logic to incorrectly fall
back to plain HTTP.
All of this is fixed by the following:
1. Increase the timeout duration to 10 seconds (from 1)
2. Instead of falling back to plain HTTP, refuse to accept the
connection if the TLS handshake wasn't initiated before the
timeout limit is reached
3. Only accept plain HTTP if the first 5 bytes do not correspond to
a TLS handshake fragment [3]
4. Do not take the last number of bytes that were in the buffer into
account; instead, only perform the actual handshake check if
5 bytes are in the peek buffer using some of tokio's low-level
functionality
Regarding 1.: This should be generous enough for any client to be able
to initiate a TLS handshake, despite its surrounding circumstances.
Regarding 4.: While this is not 100% related to the issue, peeking into
the buffer in this manner should ensure that our implementation here
remains correct, even if the kernel's underlying behaviour regarding
edge-triggering is changed [4]. At the same time, there's no need for
busy-waiting and continuously yielding to the event loop anymore.
[1]: https://www.rfc-editor.org/rfc/rfc8446.html#section-4.1.2
[2]: https://forum.proxmox.com/threads/disable-default-http-redirects-on-8007.142312/post-675352
[3]: https://www.rfc-editor.org/rfc/rfc8446.html#section-5.1
[4]: https://lwn.net/Articles/864947/
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
This adds the structs `AcceptState` and `AcceptFlags` and adapts
relevant method signatures of `AcceptBuilder` accordingly. This makes
it easier to add further parameters in the future.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
The Backup Server can compress the content using deflate so we teach the
client how to decode it.
If a request is sent with the `Accept-Encoding` [2] header set to
`deflate`, and the response's `Content-Encoding` [1] header is equal to
`deflate` we wrap the Body stream with a stream that can decode `zlib`
on the run.
Note that from the `Accept-Encoding` docs [2], the `deflate` encoding is
actually `zlib`.
This can be also tested against
http://eu.httpbin.org/#/Response_formats/get_deflate by adding the
following test:
```rust
#[tokio::test]
async fn test_client() {
let client = Client::new();
let headers = HashMap::from([(
hyper::header::ACCEPT_ENCODING.to_string(),
"deflate".to_string(),
)]);
let response = client
.get_string("https://eu.httpbin.org/deflate", Some(&headers))
.await;
assert!(response.is_ok());
}
```
at `proxmox-http/src/client/simple.rs` and running
```
cargo test --features=client,client-trait
```
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
Suggested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
We test the deflate encoder against the deflate decoder using (or not)
zlib and with different small buffer sizes. We also test compression and
decompression against the flate2 crate.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
This allows creating a encoder in a more general way and allows to
specify whether we want to set zlib headers. This is useful to compress
HTTP traffic, as per [1].
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding#directives
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
This allows to add a decompression mod inside the deflate mod. This does
not touch the public API.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
Because it was only used for the test setup. Instead, we simply
add an apt_lists_dir parameter where we need it.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
clippy rightfully complains about a create() with an unspecified
truncation behavior. This file has no contents so let's just not
truncate it in case we ever want to also have data in it...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The code chooses whichever one of a multitude of functions returns
Some, switching to .map for the final else would make it less
readable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes the following clippy warning:
warning: returning the result of a `let` binding from a block
--> proxmox-access-control/src/acl.rs:687:13
|
686 | let config = TestAcmConfig { roles };
| ------------------------------------- unnecessary `let` binding
687 | config
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#let_and_return
= note: `#[warn(clippy::let_and_return)]` on by default
help: return the expression directly
|
686 ~
687 ~ TestAcmConfig { roles }
|
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: the following explicit lifetimes could be elided: 'a
--> proxmox-acme/src/async_client.rs:65:30
|
65 | pub async fn new_account<'a>(
| ^^
66 | &'a mut self,
| ^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
= note: `#[warn(clippy::needless_lifetimes)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: casting to the same type is unnecessary (`usize` -> `usize`)
--> proxmox-http/src/websocket/mod.rs:446:40
|
446 | mask.copy_from_slice(&data[mask_offset as usize..payload_offset as usize]);
| ^^^^^^^^^^^^^^^^^^^^ help: try: `mask_offset`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_cast
= note: `#[warn(clippy::unnecessary_cast)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy error:
error: redundant redefinition of a binding `data`
--> proxmox-http/src/websocket/mod.rs:375:9
|
375 | let data = data;
| ^^^^^^^^^^^^^^^^
|
help: `data` is initially defined here
--> proxmox-http/src/websocket/mod.rs:369:27
|
369 | pub fn try_from_bytes(data: &[u8]) -> Result<Option<FrameHeader>, WebSocketError> {
| ^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_locals
= note: `#[deny(clippy::redundant_locals)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: type parameter `T` goes unused in function definition
--> proxmox-shared-memory/tests/raw_shared_mutex.rs:80:19
|
80 | fn create_test_dir<T: Init>(filename: &str) -> Option<PathBuf> {
| ^^^^^^^^^ help: consider removing the parameter
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#extra_unused_type_parameters
= note: `#[warn(clippy::extra_unused_type_parameters)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: initializer for `thread_local` value can be made `const`
--> proxmox-router/src/cli/command.rs:221:71
|
221 | static HELP_CONTEXT: RefCell<Option<Arc<CommandLineInterface>>> = RefCell::new(None);
| ^^^^^^^^^^^^^^^^^^ help: replace with: `const { RefCell::new(None) }`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#thread_local_initializer_can_be_made_const
= note: `#[warn(clippy::thread_local_initializer_can_be_made_const)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: casting raw pointers to the same type and constness is unnecessary (`*mut fs::acl::libc::c_void` -> `*mut fs::acl::libc::c_void`)
--> proxmox-sys/src/fs/acl.rs:130:23
|
130 | let mut ptr = ptr::null_mut() as *mut c_void;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `ptr::null_mut()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_cast
= note: `#[warn(clippy::unnecessary_cast)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: redundant field names in struct initialization
--> proxmox-time-api/src/time_impl.rs:53:9
|
53 | localtime: localtime,
| ^^^^^^^^^^^^^^^^^^^^ help: replace it with: `localtime`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_field_names
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the following clippy warnings:
warning: the borrowed expression implements the required traits
--> proxmox-tfa/src/api/recovery.rs:86:24
|
86 | Ok(hex::encode(&hmac))
| ^^^^^ help: change this to: `hmac`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrows_for_generic_args
and
warning: this expression creates a reference which is immediately dereferenced by the compiler
--> proxmox-network-api/src/api_impl.rs:108:47
|
108 | interface.set_bond_slave_list(&slaves)?;
| ^^^^^^^ help: change this to: `slaves`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
= note: `#[warn(clippy::needless_borrow)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy lints:
warning: unnecessary use of `get("lo").is_none()`
--> proxmox-network-api/src/config/parser.rs:603:30
|
603 | if config.interfaces.get("lo").is_none() {
| ------------------^^^^^^^^^^^^^^^^^^^
| |
| help: replace it with: `!config.interfaces.contains_key("lo")`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_get_then_check
= note: `#[warn(clippy::unnecessary_get_then_check)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Fixes the clippy warning:
warning: use of `unwrap_or_else` to construct default value
--> proxmox-tfa/src/api/mod.rs:1355:43
|
1355 | |cap| cap.map(Vec::with_capacity).unwrap_or_else(Vec::new),
| ^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `unwrap_or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unwrap_or_default
= note: `#[warn(clippy::unwrap_or_default)]` on by default
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
the old location has been deprecated for a while, and rustc 1.78 will start to warn about it.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this way the types defined in this crate can be re-used in places
without necessarily having to use the ACL, token shadow and
(cached) user config implementations.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
since `CachedUserInfo` takes care of both, the user config and the acl
config, we need to also bump the cache generation when storing the
acl config.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this commit factors out the user config. it also add two new functions
to the `AccessControlConfig` trait to handle caching in a more
generalized way.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this commit factors out the token shadow implementation from
`proxmox-backup` so it can be used in other products.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
these types are used by the user config in `proxmox-backup` server.
this commit factors them out so we can re-use them in other products
as well as this crate.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
this commit factors out the acl tree from proxmox-backup so we can
re-use it accross other products. to use it, the product needs to
implement the `AcmConfig` trait and provide this crate with a
location to safe its configuration files.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Add confirmation helper that outputs a prompt and lets the user
confirm or deny it.
Implemented to close#4763.
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
as a first step of improving our error handling story, printing context
and causes if the error contains them.
The downside to adding context is that the default Display implementation
will *just* print the context, which hides the root cause. This is why
we print the errors using the pretty-print formatter in this change.
Originally-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[WB: prefix commit message with crate]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
under autopkgtest, the rundir is not writable, but cargo gives us a tmpdir that
we can use in all cases.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Some users might want to switch to using only the newer .sources files
already, which Debian is going to switch to in the long run.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Normally root, but can be the same as the api_user if the product
does not use priviledge separation.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
This was an old version of a const-fn compatible checked c_str
implementation which was never enabled.
When we get rust 1.72, `CStr::from_bytes_with_nul` becomes usable in
const contexts.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Extract the URI creation for write and health URIs. Add unit test to
test the encoding of special characters in the organization and bucket
parameters.
Follow-up-to: bfa73aad ("metrics: encode influxdb org and bucket parameters")
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
FG: downgraded form_urlencoded version to packaged one
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
due to missing `use` statements they failed, as they should, but for
the wrong reasons. also adapt a test case that presumably was meant
to test whether `TokennameRef` can be compared, but instead
duplicated the `UsernameRef` test case.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
by using `openssl::memcmp::eq()` we can avoid potential timing side
channels as its runtime only depends on the length of the arrays, not
the contents. this requires the two arrays to have the same length, but
that should be a given since the hashes should always have the same
length.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we used `sha256scrypt` for password hashing. while this may
by safe if used with the correct parameters, we used the default
parameters which are considered unsafe. according to `man crypt(5)`:
> The default CPU time cost parameter is 5000, which is too low for
> modern hardware.
hence, we needed to adapt this code anyway. conveniently, verification
with crypt also works for older hashes as the parameters for the
hashing function are encoded in the output of crypt. so this is a drop
in replacement that will simply use yescrypt for new hashes while
old hashes will still verify properly.
this commit also adds a wrapper for `crypt_gensalt_rn` to more easily
generate correctly formatted salt strings. this is also useful for
switching the cpu time hardness parameter, as otherwise we'd need to
encode that ourselves.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we used our own hmac-like implementation for csrf token
signing that simply appended the key to the message (csrf token).
however, this is possibly insecure as an attacker that finds a
collision in the hash function can easily forge a signature. after all,
two messages would then produce the same start conditions before
hashing the key. while this is probably a theoretic attack on our csrf
implementation, it does not hurt to move to the safer standard hmac
implementation that avoids such pitfalls.
this commit re-uses the hmac key wrapper used for the keyring. it also
keeps the old construction around so we can use it for a transition
period between old and new csrf token implementations.
this is a breaking change as it changes the signature of the
`csrf_secret` method of the `AuthContext` trait to return an hmac
key.
also exposes `assemble_csrf_prevention_toke` so we can re-use this
code here instead of duplicating it in e.g. proxmox-backup's
auth_helpers.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
by using openssl's `memcmp::eq()` we can avoid potential side-channel
attack on the csrf token comparison. this comparison's runtime only
depends on the length of the two byte vectors, not their contents.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we only used asymmetric cryptographic schemes to
authenticate tickets. this is fairly costly and not necessary in every
instance. imagine a service that runs as a single daemon. this daemon
is then the only party that needs to sign and verify tickets. this
makes hmac perfectly suitable for such usecases. hmac has some
advantages over asymmetric schemes:
- much simpler and well reviewed construction
- much faster and better optimized crypto primitives (hash functions)
this commit first introduces a new hmac key wrapper that uses openssl's
hmac implementation and can easily be reused by other parts of the
code. it also refactors the keyring code to make it easier to rotate
new hmac keys into place so switching to hmac keys is easier.
hmac keys are symmetric, so the verification key is the same key as the
signing key. this breaks the previous assumption by the keyring that
these correspond to public and private keys. thus, this commit
introduces two wrapper enums to distinguish between hmac and asymmetric
signature schemes.
the verification of hmac keys is also done via `openssl::memcmp::eq()`
to avoid potential timing side-channel attacks.
below are some simple benchmarks done with criterion.rs to show how much
faster hmac is, no matter the actual hash function:
rsa 4096 + sha256 time: [2.7825 ms 2.7907 ms 2.7995 ms]
ed25519 time: [94.411 µs 94.840 µs 95.324 µs]
hmac sha256 time: [5.7202 µs 5.7412 µs 5.7645 µs]
hmac sha384 time: [6.6577 µs 6.6780 µs 6.7006 µs]
hmac sha3_256 time: [5.6930 µs 5.7114 µs 5.7322 µs]
rsa with 4096 bit keys and a sha256 digest is our current default. the
test itself consists of a single sign + verification cycle. criterion
repeats this test as it sees fit to arrive at the above numbers.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously we used P-256 as the curve of our choice for ec signatures.
however, in the meantime Ed25519 has become a lot more wide-spread.
this simplifies our ec generation code significantly while keeping the
same security level. Ed25519 was also specifically designed and
reviewed to avoid implementation errors likely making it a more secure
choice
note that Ed25519 as a signature scheme always uses sha512, so signing
or verifying with a chosen digest is not supported.
as this mostly affects newly generated keys, this should not break any
existing setups.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
this commit moves the current ticket signing code into the private key
implementation. the upside is that the caller does not need to deal
with openssl's `Signer` directly. it also simplifies and unifies the
code by using the same helper for verifying a signature and creating it.
also derive `Clone` on `PrivateKey` and `PublicKey`. as they are
essentially thin wrappers around `openssl::pkey::PKey<Private>` and
`openssl::pkey::PKey<Public>`, which can be cloned, deriving `Clone`
just makes them easier to use.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Since we don't query each drives status seperately, but rely on a single
call to the drives listing parameter for that, we now add the option
to query the activity there too. This makes that data avaiable for us
to show in a seperate (by default hidden) column.
Also we show the activity in the 'State' column when the drive is idle
from our perspective. This is useful when e.g. an LTO-9 tape is loaded
the first time and is calibrating, since that happens automatically.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we use the VHF part from the DT Device Activity page for that.
This is intended to query the drive for it's current state and activity.
Currently only the activity is parsed and used.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and show them on the ui. This can help uses with seeing how much a tape
is used.
The value is updated on 'commit' and when the tape is changed during a
backup.
For drives not supporting the volume statistics, this is simply skipped.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
So that we can use the Interface struct with create and update api calls (which
currently use 'iface' instead of 'name').
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Remove the regex for influxdb organizations and buckets. Influxdb does
not place any constraints on these names and allows all characters. This
allows influxdb organization names with slashes.
Also remove a duplicate comment and add some missing ones.
This also aligns the behavior to PVE as there are no restrictions there
either.
The motivation for this patch is this forum post:
https://forum.proxmox.com/threads/influx-db-organization-doesnt-allow-slash.145402/
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
In order to remove the current limitations on the bucket and
organization names, we need to make sure that they are transmitted
correctly. In order to do this, we encode them using the url crate.
This way we support organization/bucket names that include slashes,
whitespaces, etc.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
* Add vlan_id and vlan_raw_device fields to the Interface api type
* Write to the network config the vlan specific properties for vlan
interface type
* Add several tests to verify the functionally
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Basically just a thin wrapper over the existing LDAP-based realm sync
job, which retrieves the appropriate config and sets the correct user
attributes.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Same as with datastores, this option determines whether we send
notifications the old way (send email via sendmail to a user's email
address) or the new way (emit matchable notification events to the
notification stack).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This one lets the user choose between the old notification behavior
(selecting an email address/user and always/error/never behavior per
datastore) and the new one (emit notification events to the
notification system)
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
proxmox-schema will automatically append text (e.g. 'Can be specified
more than once'), so we should end every comment with a '.'.
Also copy over some text from PVE docs, since these doc comments will
now be visible in the PBS documentation.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
PBS sends notifications for all events but successful prune jobs.
There we only care about errors.
This commit adapts the 'default-matcher' to reflect that behavior
as well.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
This is needed because we want to access CONFIG and PRIVATE_CONFIG
from the docgen helper in PBS.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
The old notification stack in proxmox-backup includes the nodename, so
we include it here as well.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
The API endpoints in Proxmox Backup Server require ApiType to be
implemented for any deserialized parameter.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
This method allows us to get a list of all notification targets.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
We need this for queuing notifications on PBS from the unprivileged
proxy process.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
It uses proxmox_sys::nodename - the dep is needed, otherwise the code
does not compile in some feature flag permutations.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tests now have their own context, so requiring pve-context is not
necessary any more.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
This saves us from some of the awkward cloning steps when updating.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Instead of passing the template strings for subject and body when
constructing a notification, we pass only the name of a template.
When rendering the template, the name of the template is used to find
corresponding template files. For PVE, they are located at
/usr/share/proxmox-ve/templates/default. The `default` part is
the 'template namespace', which is a preparation for user-customizable
and/or translatable notifications.
Previously, the same template string was used to render HTML and
plaintext notifications. This was achieved by providing some template
helpers that 'abstract away' HTML/plaintext formatting. However,
in hindsight this turned out to be pretty finicky. Since the
current changes lay the foundations for user-customizable notification
templates, I ripped these abstractions out. Now there are simply two
templates, one for plaintext, one for HTML.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
to avoid drifting definitions and reduce duplication. with the next major
release, the 'upid' field could then be renamed and aliased to be in line with
the other jobs, which all use 'last-run-upid'. doing it now would break
existing callers of the GC status endpoint (or consumers of the on-disk status
file).
the main difference is that the GC status fields are now not optional (except
for the UPID) in the job status, since flattening an optional value is not
possible. this only affects datastores that were never GCed at all, and only
direct API consumers, since the UI handles those fields correctly.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Adds an api endpoint on the datastore that reports the gc job status
such as:
- Schedule
- State (of last run)
- Duration (of last run)
- Last Run
- Next Run (if scheduled)
- Pending Chunks (of last run)
- Pending Bytes (of last run)
- Removed Chunks (of last run)
- Removed Bytes (of last run)
Adds a dedicated endpoint admin/gc that reports gc job status for all
datastores including the onces without a gc-schedule.
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
Originally-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewd-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Maintenance mode Delete locks the datastore. It must not be possible to go
back to normal modes, because the datastore may be in undefined state.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
it seems there are repositories out there that don't (always) include
it, and while it is required for the .deb packages themselves in Debian,
the repository "spec" doesn't make it mandatory.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it seems there are repositories out there that don't (always) include
it, and while it is required for the .deb packages themselves in Debian,
the repository "spec" doesn't make it mandatory.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Versions of Gotify < 2.2.0 only supported the 'X-Gotify-Key' header
for passing the API token. This comment sets this header in addition
to the regular 'Authorization' header in order to be compatible with
older Gotify servers.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
The root DSE holds common attributes about the LDAP server itself.
Needed to e.g. support Active Directory-based LDAP servers to retrieve
the base DN from the server itself, based on an valid bind.
See also RFC 4512, Section 5.1 [0] for more information about this
special object.
[0] https://www.rfc-editor.org/rfc/rfc4512#section-5.1
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
The `attrs` parameter of `Ldap::search()` is an `impl AsRef<[impl
AsRef<str>]>` anyway, so replace `vec![..]` with `&[..]`.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
We keep a DataStore cache, so ChunkStore's and lock files are kept by
the proxy process and don't have to be reopened every time. However,
for specific maintenance modes, e.g. 'offline', our process should not
keep file in that datastore open. This clears the cache entry of a
datastore if it is in a specific maintanance mode and the last task
finished, which also drops any files still open by the process.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Macro rules are not hygienic, and current rust macro visibility rules
are a nightmare. Using const_format::concatcp!() is a much cleaner
solution.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Some types were recently renamed but the examples not updated
accordingly.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 2f942833672d "rrd: spell out hard to understand abbreviations in public types"
rather than just always allowing additional properties, only return
true if any of the available schemas allows it
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Use the `std::io::IsTerminal` trait introduced in Rust 1.70.
Internally it calls `libc::isatty`, see [1, 2]. Note that it switches
the comparison from `== 1` to `!= 0` which shouldn't make a difference
assuming that libc::isatty upholds the promises made in its man page.
The MSRV was set on the workspace to reflect this change.
[1] https://doc.rust-lang.org/src/std/io/stdio.rs.html#1079
[2] https://doc.rust-lang.org/src/std/sys/unix/io.rs.html#79
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Allow more complex strings for the acr-value when using openid. The
openid documentation only specifies the acr-value *should* be an URI
[0]. Implemented a regex that loosely disallows some of the reserved
URI characters specified in the RFC [1].
Currently values like:
- "urn:mace:incommon:iap:silver"
- "urn:comsolve.nl:idp:contract:rba:location"
do NOT work, although they are correct URI's and common acr tokens.
For Proxmox VE we had to actually make this more strict to align with
each other, as there we accepted any string.
[0]: https://openid.net/specs/openid-connect-core-1_0.html
[1]: https://www.rfc-editor.org/rfc/rfc2396.txt
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
and reuse splitting code in no_schema's SeqAccess as well
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
A 'oneOf' schema is basically exactly what a rust `enum` is.
Exactly one of the possible values must match the data.
This should ultimately be the base to allow using the
`#[api]` macro on a newtype style enum as well as using this
schema as a configuration for our section config parser.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
so CLI tools can pre-parse out non-api parameters before
passing the remaining stuff to the router
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Adds a privileged_addr to ApiConfig, and some helpers for
hyper (both server and client)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
The proxmox-backup repo was filtered using `git filter-repo` using the
following paths:
proxmox-rrd
proxmox-rrd-api-types
src/rrd
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Nightly currently produces a different output format so this command
doesn't work right now when +system is not the default cargo.
Let's hope this is just a temporary hiccup in nightly, given that
there is an explicit `--format-version=1` parameter...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
For example, one can now use:
match-field exact:type=vzdump,replication
to match on vzdump AND replication events.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Seems like this was forgotten in the initial version. Without it,
it's not really possible to create matchers for forwarded mails.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
In the 'delete'-handler targets, we check if a
target is still referenced by a matcher - if it is, we return an
error. For built-in targets, this is actually not necessary, since
'deleting' a built-in only resets it to its default settings - it will
continue to exist after that.
The user could easily trigger this if 'mail-to-root', which is
referenced by 'default-matcher' is modified and then reset to its
defaults: An error is shown, the built-in target is not reset.
This commit disables this check if it is a built-in target.
Renamed the helper 'ensure_unused' to 'ensure_safe_to_delete' in the
process.
Also fixed the tests in api::test - they were never executed due to a
faulty #[cfg] directive.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
`Auto-Submitted` is defined in the rfc 5436 [1] and describes how
an automatic response (f.e. ooo replies, etc.) should behave on the
emails. When using `Auto-Submitted: auto-generated` (or any value
other than `none`) automatic replies won't be triggered.
[1]: https://www.rfc-editor.org/rfc/rfc3834.html
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
For mails forwarded by `proxmox-mail-forward` to an SMTP target, the
original message was nested as a 'message/rfc822' message part.
Originally this approach was chosen to avoid having to rewrite
message headers.
Good email-clients, such as Thunderbird can display these inline.
Other, more limited clients will show these messages as an attached
.eml file, which is not really a good user experience.
This patch changes the approach for message forwarding to be more like
forwarding mails in a mail client. We create a new message and
add the original message body as a body. Additionally, we also copy
over all message headers that are relevant to correctly display the
original message body (e.g. Content-Type, Content-Transfer-Encoding)
Tested with a couple of different email messages (varying in
structure, body parts, encoding, etc.) against the following SMTP
relays:
- gmail
- outlook
- our own webmail service
Originally reported in our community forum:
https://forum.proxmox.com/threads/proxmox-mail-forward-sends-mails-as-eml.137710/
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
as the previous commit: simply keep the previous Display impl and call
it from out of the new GroupFilter impl
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
After some discussion I canged the include/exclude behavior to first run
all include filter and after that all exclude filter (rather then
allowing to alternate inbetween). This is done by splitting them into 2
lists, running include first.
A lot of discussion happened how edge cases should be handled and we
came to following conclusion:
no include filter + no exclude filter => include all
some include filter + no exclude filter => filter as always
no include filter + some exclude filter => include all then exclude
Since a GroupFilter now also features an behavior, the Struct has been
renamed To GroupType (since simply type is a keyword). The new
GroupFilter now has a behaviour as a flag 'is_exclude'.
I considered calling it 'is_include' but a reader later then might not
know what the opposite of 'include' is (do not include? deactivate?). I
also considered making a new enum 'behaviour' but since there are only 2
values I considered it over engeneered.
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
`strftime`'s formatting is locale-dependent. If the system locale was
set to e.g. de_DE.UTF-8, the `Date` header became invalid
(e.g Mo instead of Mon for 'Monday'), tripping up some mail clients
(e.g. KMail).
This commit should fix this by using the new `epoch_to_rfc2822`
function from proxmox_time. Under the hood, this function uses
`strftime_l` with a fixed locale (C).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This variant of strftime can be provided with a locale_t, which
determines the locale used for time formatting.
A struct `Locale` was also introduced as a safe wrapper around
locale_t.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
instead of having it in a property string. For now this should be fine,
and if we need many more such options, we can still move them into a
property string if we want.
Also update the cli command in the docs on how to set it now.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by converting the bool into an option, otherwise having the options not
set at all will fail the unload while deserializing with
'eject-before-unload is not optional'
Also if we can automatically decide this in the future, we can now
detect if the option was explicitely set or not.
Fixes: 99f24b20 ("fix #4904: tape changer: add option to eject before unload")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
some tape libraries need the tape being ejected from the drive before
doing an unload. Since we cannot easily detect if that's the case,
introduce an 'eject_before_unload' option.
Instead of just adding a bool flag to the config, add a new 'options'
property string where we can put such niche options similar to how we
handle the datastore tuning options.
Extend the LtoTapeHandle with 'medium_present' which just uses a
TEST UNIT READY command to check for present medium, so we don't
try to eject an already ejected tape.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we'll need more info from there in the future, so derive clone for it
and save the whole config instead of adding an additional field.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Instead of returning -1 if we can't get the attributes, we use an
Option which will not be serialized on `None`.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Note that this was currently not deserialized anywhere, so this was
not an issue, but the api-macro now treats this as an error.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
separated out of FieldAttrib without the `flatten` attribute, since we
don't support this on enum variants
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
According to the rfc, the meta field contains additional fields that
weren't covered by the Meta struct. Of the additional fields, only
external_account_required will be used in the near future, but others
were added for completeness and the case that they might be used in the
future.
Signed-off-by: Folke Gleumes <f.gleumes@proxmox.com>
Functionality was added as a additional setter function, which hopefully
prevents any breakages. Since a placeholder Option an the AccountData
was already present, but has never been used, replacing the field with
an Option of a fully defined type should also be minimally intrusive.
Signed-off-by: Folke Gleumes <f.gleumes@proxmox.com>
Return a struct with all the components of the kernel version like it
has been done in pve. Also return the legacy `kversion` to keep
backwards compat.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Added field that shows the bootmode of the node. The bootmode is either
Legacy Bios, EFI, or EFI (Secure Boot). To detect the mode we use the
exact same method as in pve: We check if the `/sys/firmware/efi` folder
exists, then check if the `SecureBoot-xx...` file in the `efivars`
directory has the SecureBoot flag enabled.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Helper that return the current boot_mode and secureboot status.
Detection works the same as in pve, we use `/sys/firmware/efi` and
the `efivars/SecureBoot-xxx..` file.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
The new regex is similar to BLOCKDEVICE_NAME_REGEX but also allows
numbers at the end of the device name (also allows partitions names).
For nvme partitions it also allows the letter p and a number.
Signed-off-by: Markus Frank <m.frank@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
We want to get a 401 error at HTTP level.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The `Redirector` is a simple `Service` that redirects HTTP requests
to HTTPS and can be served by a `hyper::Server`.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The new public function `accept_tls_optional()` is added, which
accepts both plain TCP streams and TCP streams running TLS. Plain TCP
streams are sent along via a separate channel in order to clearly
distinguish between "secure" and "insecure" connections.
Furthermore, instead of `AcceptBuilder` itself holding a reference to
an `SslAcceptor`, its public functions now take the acceptor as an
argument. The public functions' names are changed to distinguish
between their functionality in a more explicit manner:
* `accept()` --> `accept_tls()`
* NEW --> `accept_tls_optional()`
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
internally `SubscriptionInfo` already uses the `parse_next_due` helper
to parse the next due date to an epoch. this exposes a function that
allows us to use the epoch outside of this crate too. for example, a
user of pom may have multiple subscription for the same system. in
that case we want to apply the one with the due date that is furthest
in the future.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
previously this type lived inside of pom. this made it harder to
access the product type from a `SubscriptionInfo` trait in other
products. move the type here so we can check product types more
consistently across products (e. g. in pom and pbs)
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
This allows us to define a (modifiable) builtin-config, which is
at the moment hardcoded in PVEContext
The 'origin' parameter indicates whether a config entry was created by
a user, builtin or a modified builtin.
These changes require context to be set for tests, so we set
PVEContext by default if in a test context. There might be a nicer
solution for that, but for now this should work.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds a new endpoint type, namely 'smtp'. This endpoint
uses the `lettre` crate to directly send emails to SMTP relays.
The `lettre` crate was chosen since it is by far the most popular SMTP
implementation for Rust that looks like it is well maintained.
Also, it includes async support (for when we want to extend
proxmox-notify to be async).
For this new endpoint type, a new section-config type was introduced
(smtp). It has the same fields as the type for `sendmail`, with the
addition of some new options (smtp server, authentication, tls mode,
etc.).
Some of the behavior that is shared between sendmail and smtp
endpoints has been moved to a new `endpoints::common::mail` module.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit moves PVEContext from `proxmox-perl-rs` into the
`proxmox-notify` crate, since we now also need to access it from
`promxox-mail-forward`. The context is now hidden behind a feature
flag `pve-context`, ensuring that we only compile it when needed.
This commit adds PBSContext, since we now require it for
`proxmox-mail-forward`. Some of the code for PBSContext comes
from `proxmox-mail-forward`.
This commit also changes the global context from being stored in a
`once_cell` to a regular `Mutex`, since we now need to set/reset
the context in `proxmox-mail-forward`.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
As preparation for the integration of `proxmox-mail-foward` into the
notification system, this commit makes a few changes that allow us to
forward raw email messages (as passed from postfix).
For mail-based notification targets, the email will be forwarded
as-is, including all headers. The only thing that changes is the
message envelope.
For other notification targets, the mail is parsed using the
`mail-parser` crate, which allows us to extract a subject and a body.
As a body we use the plain-text version of the mail. If an email is
HTML-only, the `mail-parser` crate will automatically attempt to
transform the HTML into readable plain text.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This should be a bit more intuitive to users than the current
behavior, which is 'always match' for mode==all and 'never match' for
mode==any. The current behavior originates in the neutral element of
the underlying logical operation (and, or).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This shifts notification routing into the matcher-system. Every
notification has associated metadata (key-value fields, severity -
to be extended) that can be match with match directives in
notification matchers. Right now, there are 2 matching directives,
match-field and match-severity. The first one allows one to do a
regex match/exact match on a metadata field, the other one allows one
to match one or more severites.
Every matcher also allows 'target' directives, these decide which
target(s) will be notified if a matcher matches a notification.
Since routing now happens in matchers, the API for sending is
simplified, since we do not need to specify a target any more.
The API routes for filters and groups have been removed completely.
The parser for the configuration file will still accept filter/group
entries, but will delete them once the config is saved again. This is
needed to allow a smooth transition from the old system to the new
system, since the old system was already available on pvetest.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This will be useful later for system mail forwarding, where
the content of the mail should be forwarded unchanged.
This moves notification properties into this new type and calls them
'data'. They will exclusively used for template rendering.
`Notification` will receive a separate field for metadata, which
will be useful for notification filtering. This decouples
template rendering and filtering, which enables us to be very precise
about which metadata fields we allow to be used in filters.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
support for it got added to Proxmox repositories, so there is no need to use
custom logic and manual fetching for this anymore.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
As termproxy only used this module from the huge proxmox-sys crate,
and this module was also only used here, it makes sense to move it
over there (and dropping proxmox-sys halved build-time from 8.5 s to
4.2 s).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
If there is only one error, output in a single line, without any
excessive newlines at the end. If there are multiple errors, show them
in a bulleted list.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
We had two call sites deriving the directory "shard" where the task
log file is actually saved to, this can lead to ugly bugs and is
better done in a central single-source-of-truth way.
While at it factor out the creation of the log file (and it's shard
directory) to avoid crowding the WorkerTask new fn to much.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This mostly affected attribute parsing (due to the syn::Meta changes).
Also creating `DelimSpan`s for custom-built `syn::Attribute`s is a
bit... ugly.
Upshot: turns out we can drop some helpers in util.rs with the new
`syn::Meta` changes.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
previously we changed the internal type of the 'data' property
from Option<T> to T in the assumption the api always returns
'data:null'.
this is actually only the case when the api call succeeds. in an error
case there is no data property at all.
to fix this issue while behaving the same for 'data:null' we have to
revert to Option<T> for RawApiResponse but instead of always throwing an
error for 'data:null' in 'check' we now try there to deserialize from
Value::Null for T if there was no data. This will succeed for the Type
'()' which was the motivation for the original change.
The only downside is that the RawApiResponse now has a trait bound that
T is deserializeable, but was a requirement for using it anyway
(as there was no other way of constructing it)
Fixes: 271a55f ("client: remove option from inner RawApiResponse")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
`Auto-Submitted` is defined in the rfc 5436 [1] and describes how
an automatic response (f.e. ooo replies, etc.) should behave on the
emails. When using `Auto-Submitted: auto-generated` (or any value
other than `none`) automatic replies won't be triggered.
[1]: https://www.rfc-editor.org/rfc/rfc3834.html
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
technically an empty string is not valid json, but when sending an api
request without any parameters, treating the empty body as an empty
parameter hash instead of an error, makes the the api more robust for
clients
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
note: not complete, there's other code to check and rework, but I had
this already done so commit it, better than nothing.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This moves the clean-up to happen up-front. That way one can still
inspect the test data after, e.g., a failed test.
Originally done almost like this in a patch from Fiona [0] that I just
overlooked, but now also factored out, avoid crowding the test code to
much with duplicate code.
[0]: https://lists.proxmox.com/pipermail/pve-devel/2023-June/057136.html
Suggested-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the "main" repo only exists for Quincy to allow an easier transition
from Proxmox VE 7 to Proxmox VE 8, for when the enterprise repo got
added for ceph too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For now just duplicate the Ceph Quincy entries, as I want to avoid
using macros and we do not yet have support for enums inside enums
with the api macro.
Adapt and expand the tests slightly to have at least some simple
coverage there too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Files inside CARGO_TARGET_TMPDIR are only cleaned on `cargo clean`, so
tests that expect files to not exist need to cleanup themselves.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
on other targets we print the timestamp without fractional seconds
('.xxxZ'), so we should remove that too on wasm
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when using the client for an api call that does not return any data
(it returns '{"data":null}'), we would always get an error 'api returned
no data'. The message is technically correct, but it should not be an
error when we expect no data (e.g. most of our CRUD PUT/POST calls)
instead of having the Option<T> in the RawApiResponse type itself, move
it into to the 'nodata' function intended for api calls where we don't
expect any data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This commit has the aim of making template rendering a bit more
robust. It does so by a.) Accepting also strings for helpers that
expect a number, parsing the number if needed, and b.) Ignoring errors
if a template helper fails to render a value and showing an error in
the logs, instead of failing to render the whole template (leading
to no notification being sent).
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit updates all helper functions, taking into account recent
developments regarding `tokio`.
In particular, the `block_in_place()` and `block_on()` functions now
don't panic anymore if used within the single-threaded `tokio` runtime
and instead behave as expected in both runtime flavours.
Furthermore, because `tokio` may add more runtime flavours in the
future, all helpers will now panic if used within an unsupported
runtime. This is to prevent unforeseen behavioural quirks and
interactions with `tokio` internals.
The above changes make `BlockingGuard` redundant; it is consequently
removed.
The documentation is also updated, describing the behaviour of the
helper functions and the purpose of the `runtime.rs` module in more
detail.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
lto 9 tapes have a new density code which leads to these tapes not
being recognized properly. add the new density code and TapeDensity to
improve lto 9 support. since the documentation states that we support
lto 5 and above this constitutes a bug fix for lto 9 support.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Since the WASM client cannot actually use a `http::Request` the way we
expect it to, that is, it cannot manually along cookies, we turn the
client bit inside out:
This crate mainly defines the `HttpApiClient` trait which expects the
http client to perform *authenticated* API calls, that is, the
handling of API tokens and tickets should happen at the *implementor*
side.
The product clients will require *this* trait to be implemented, and
will not themselves offer a way to login.
As for the `Client` struct, this will now instead *implement* this
trait and will *not* be used in the `wasm` ecosystem. Rather, this is
the ticket handling http client that already exists in the PWT based
ui code.
The PVE client in `pve-api-types` will not *contain* a `Client`
anymore, but rather, it will provide PVE api call implementations for
something implementing `HttpApiClient`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The environment trait was useful on the CLI, but does not really
translate well to eg. the wasm ui (or pdm for that matter), so drop it
and instead have `.login` and `.login_tfa` just take the
`proxmox_login` type and handle the updating of authentication data.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Because we ultimately also want to drop the `Environment` trait since
it is not suitable for all use cases (eg. wasm ui)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
In the uncommon circumstance that calls to `read()` end up reading any number of
bytes other than 4096, the subsequently read bytes become misaligned, causing
blocks of zeroes to be written unnecessarily.
To illustrate, imagine you have a 12KiB file:
[x][x][x][x][ ][ ][ ][ ][x][x][x][x]
└──4096──┘ └──4096──┘ └──4096──┘
The first and last block are filled with some data, whereas the middle block is
empty and will therefore result in only zeroes being read.
In order for the empty block to be skipped with `seek()`, the entire buffer has
to be filled with zeroes.
If, for example, the first `read()` ends up putting only 3KiB into the buffer,
the empty block in the middle won't be detected properly, as the buffer will
now always contain some data. What results are four misaligned reads:
[x][x][x][x][ ][ ][ ][ ][x][x][x][x]
├─────┘ ├────────┘ ├────────┘ │
1 2 3 4
This is fixed by ensuring chunks of 4KiB are always read into the buffer,
except when the last block is truncated. In order to prevent frequent small
reads, the incoming reader is also buffered via `io::BufReader`.
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
this should avoid most common size limitations. the search should also
complete quicker as fewer results need to be computed. note that this
way a configuration may be accepted, but the related sync job can
fail due to and exceeded size limit warning for some ldap servers
(such as 2.5.14+dfsg-0ubuntu0.22.04.2).
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Commit 5720ba2d ("use new auth api crate") moved all auth-related code
into it's own crate inside the `proxmox` repo, including this file. Thus
drop it here, it's not even included in the compile.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Dropping a copy leaves the original intact
See:
https://rust-lang.github.io/rust-clippy/master/index.html#drop_copy
I assume the `drop` was used to silence a 'unused variable' warning,
so I silenced it by other means.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
To see if it is even still necessary given that it's not a trait
object type where auto traits would need to be explicit...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This should be moved to where we actually need it, not be part of the
generic product client.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we have use cases where we have bytes, and serde_json has a from_slice
method, doing the utf-8 check unnecessarily is pointless, while going
from &str to &[u8] is free...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
since `totp_locked` is not wrapped in an `Option` we need to
explicitly tell serde about its default
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Break out proxmox-router's HttpError into it's own crate so that it can
be used without pulling in proxmox-router.
This commit also implements `Serialize` for `HttpError` so that it can
be returned from perlmod bindings, allowing Perl code to access the
status code as well as the message.
Also add some smoke-tests to make sure that the `http_bail` and
`http_err` macros actually produce valid code.
Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
[w.bumiller@proxmox.com: set d/changelog to UNRELEASED]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This introduces a new configuration parameter `mailto-user`.
A user's email address will be looked up in the product-specific
user database.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Since `proxmox-notify` is intended to be used by multiple products,
there needs to be a way to inject product-specific behavior.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds template rendering to the `proxmox-notify` crate, based
on the `handlebars` crate.
Title and body of a notification are rendered using any `properties`
passed along with the notification. There are also a few helpers,
allowing to render tables from `serde_json::Value`.
'Value' renderers. These can also be used in table cells using the
'renderer' property in a table schema:
- {{human-bytes val}}
Render bytes with human-readable units (base 2)
- {{duration val}}
Render a duration (based on seconds)
- {{timestamp val}}
Render a unix-epoch (based on seconds)
There are also a few 'block-level' helpers.
- {{table val}}
Render a table from given val (containing a schema for the columns,
as well as the table data)
- {{object val}}
Render a value as a pretty-printed json
- {{heading_1 val}}
Render a top-level heading
- {{heading_2 val}}
Render a not-so-top-level heading
- {{verbatim val}} or {{/verbatim}}<content>{{#verbatim}}
Do not reflow text. NOP for plain text, but for HTML output the text
will be contained in a <pre> with a regular font.
- {{verbatim-monospaced val}} or
{{/verbatim-monospaced}}<content>{{#verbatim-monospaced}}
Do not reflow text. NOP for plain text, but for HTML output the text
will be contained in a <pre> with a monospaced font.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
This commit adds a way to filter notifications based on severity. The
filter module also has the necessary foundation work for more complex
filters, e.g. matching on properties or for creating arbitarily complex
filter structures using nested sub-filters.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Add an endpoint for Gotify [1], showing the how easy it is to add new
endpoint implementations.
[1] https://gotify.net/
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
StringSchema::check_constraint runs `parse_property_string` for
property strings, but when we deserialize a `PropertyString` we
immediately follow that up with deserializing it using the schema, so
there's no need to check it beforehand.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This provides `proxmox_schema::property_string::PropertyString<T>` for
a typed property-string.
To facilitate this, this introduces
`proxmox_schema:🇩🇪:SchemaDeserializer` which is a serde deserializer
for property strings given a schema.
This basically maps to one of `de::SeqAccess` (for array schemas) or
`de::MapAccess` (for object schemas).
Additionally, a `de::NoSchemaDeserializer` is added, since properties
within the strings may have string schemas with no format to it, while
the type we serialize to may ask for an array (a simple "list") via
serde.
The deserializers support borrowing, for which a helper `Cow3` needed
to be added, since property strings support quoting with escape
sequences where an intermediate string would be allocated and with an
intermediate lifetime distinct from the `'de` lifetime.
A `de::verify` module is added which uses serde infrastructure to
validate schemas without first having to deserialize a complete
`serde_json::Value`.
For serialization, `proxmox_schema::ser::PropertyStringSerializer` is
added split into similar parts `ser::SerializeStruct` and
`ser::SerializeSeq` at the top level, and the same prefixed with
`Element` for inside the actual string. This should also properly
quote the contents if required.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These were missing the new client-ip parameter in the auth
function calls which was introduced to support `PAM_RHOST`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Since this requires access to the user data, we need to add
a generic parameter to the unlock methods.
To avoid having to create another major API bump affecting
all our products this short after release, we keep the old
version around with the old behavior.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Like in PVE.
This means that /access/users is now a 'protected' call to
get access to 'tfa.cfg'.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this commit makes the ldap realm endpoints check whether a new or
updated configuration works correctly. it uses the new
`check_connection` function to make sure that a configuration can be
successfully used to connect to and query an ldap directory.
doing so allows us to remove the ldap domain regex. instead of relying
on a regex to make sure that a given distinguished name (dn) could be
correct, we simply let the ldap directory tell us whether it accepts
it. this should also aid with usability as a dn that looks correct
could still be invalid.
this also implicitly removes unauthenticated binds, since the new
`check_connection` function does not support those. it will simply
bail out of the check if a `bind_dn` but no password is configured.
therefore, this is a breaking change.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
this function checks if a given connection could work. it uses the
current config to connect to an ldap directory and perform a search
with the provided base_dn. this enables us to verify a connection
before storing it in a more meaningful way than with a regex.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
by using the default empty string if no password was provided,
unauthenticated binds were possible. to bring pbs in-line with pve,
switch to throwing an error in this case instead. however, this will
break any pre-existing setup that relied on this behavior.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
The module previously lived in `pbs-api-types`, however turned out to
be useful in other places as well (POM, proxmox-notify), so it is moved
here as its own micro-crate.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
It is very common for TOTP URIs to contain the algorithm in lowercase,
hence we convert to lowercase when doing From<&str> for Algorithm.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
if the match arms are this far away from the actual `match`
keyword, this needs to be split up...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
when zipping a directory, our intention was to skip over files that
cannot be zipped (e.g. the file can't be read/vanished/etc.), so we
ignored errors and simply logged it.
but when 'add_entry' fails, we will never actually restore, since every
error there is fatal to the point that the zip cannot be finished thats
because we take the 'target' sink out of self, and only insert it again
after all writes succeeded. so if an error occurs in between 'target' is
not put into self again (and never will be) and the zip cannot be
finished (even if we would catch all those intermediate errors and
restore 'target', we don't know in which state the output was, so we're
unable to finish a valid zip)
to fix that, split the actual 'add_entry' part there out of the async
move block and treat its errors always as fatal
without this, we generate heaps of log lines even after an error
occurred, and can never recover
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
APT doesn't mind a repository with either "/" or "./" as suite/distribution,
such as
deb https://example.com/debian ./
in that case, the 'dists' part of the URL and the trailing slash (which would
be encoded as '_') is dropped in the file name in '/var/lib/apt/lists/'.
Other suite values with a trailing or leading '/' are rejected with an error by APT:
E: Malformed entry 1 in sources file /etc/apt/sources.list.d/test.list (absolute Suite Component)
E: The list of sources could not be read.
so this should be the only special case requiring handling.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
APT will not store the InRelease file in some cases, and some repositories
might not even have one in the first place.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
A build-system that needs all build-dependencies even for a simple
make clean invocation is a PITA..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The old 'main' component stays valid, pointing to no-subscription,
which means the is_referenced_repository() check needs a special case
for it. It will eventually go away, together with the handles for
Quincy.
Alternatively, the standard repository's info() could've been changed
to return multiple possible components, similar to URLs, but as
opposed to URLs, there could be a standard repository that wants to
have multiple components and it feels a bit unnatural, because
multiple components are usually not aliases of the same. And adapting
is_referenced_repository() would be needed here too. So overall, the
above alternative just felt better.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This mostly serves as documentation for the API call to be
implemented across our products. It's otherwise already just
a oneliner on the TfaConfig.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Specifying the transfer-last parameter limits the amount of backups
that get synced via the pull command/sync job. The parameter specifies
how many of the N latest backups should get pulled/synced. All other
backups will get skipped.
This is particularly useful in situations where the sync target has
less disk space than the source. Syncing all backups from the source
is not possible if there is not enough disk space on the target.
Additionally this can be used for limiting the amount of data
transferred, reducing load on the network.
The newest backup will always get re-synced, regardless of the setting
of the transfer-last parameter.
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
a non capturing group is '(?:)' not '(:?)' so fix that.
None of these regexes are used where would use capturing groups.
DATASTORE_MAP_REGEX and TAPE_RESTORE_SNAPSHOT_REGEX are only used
as api types and BLOCKDEVICE_NAME_REGEX is only used once outside of the
api and there we also don't look at the capturing groups.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Otherwise, a substring match is enough to fulfill the constraint.
Fixes: 3aba0d9a ("api-types: ldap: add verification regex for LDAP DNs")
Reported-by: Friedrich Weber <f.weber@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
The properties are mainly based on the ones from PVE, except:
* consistent use of kebab-cases
* `mode` replaces deprecated `secure`
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
these are only used by a single member at the moment, but we can move them to
the workspace to have a single location for version + base feature set
specification.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
these are only used by a single member at the moment, but we can move them to
the workspace to have a single location for version + base feature set
specification.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
besides harmonizing versions, the only global change is that the tokio-io
feature of pxar is now implied since its default anyway, instead of being
spelled out.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pbs-buildcfg is the only one that needs to inherit the version as well, since
it stores it in the compiled crate.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
instead of hardcoding the default deep inside the code. This makes it
much easier to see what is the actual default
the first instance of ChunkOrder::None was only for the test case, were
the ordering doe not matter
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
rationale is that it makes the backup much safer than 'none', but does not
incur a big of a performance hit as 'file'.
here some benchmark:
data to be backed up:
~14GiB semi-random test images between 12kiB and 4GiB
that results in ~11GiB chunks (more than ram available on the target)
PBS setup:
virtualized (on an idle machine), PBS itself was also idle
8 cores (kvm64 on Intel 12700k) and 8 GiB memory
all virtual disks are on LVM with discard and iothread on
the HDD is a 4TB Seagate ST4000DM000 drive, and the NVME is a 2TB
Crucial CT2000P5PSSD8
i tested each disk with ext4/xfs/zfs (default created with the gui)
with 5 runs each, inbetween the caches are flushed and the filesystem synced
i removed the biggest and smallest result and from the remaining 3
results built the average (percentage is relative to the 'none' result)
result:
test none filesystem file
hdd - ext4 125.67s 140.39s (+11.71%) 358.10s (+184.95%)
hdd - xfs 92.18s 102.64s (+11.35%) 351.58s (+281.41%)
hdd - zfs 94.82s 104.00s (+9.68%) 309.13s (+226.02%)
nvme - ext4 60.44s 60.26s (-0.30%) 60.47s (+0.05%)
nvme - xfs 60.11s 60.47s (+0.60%) 60.49s (+0.63%)
nvme - zfs 60.83s 60.85s (+0.03%) 60.80s (-0.05%)
So all in all, it does not seem to make a difference for nvme drives,
for hdds 'filesystem' increases backup time by ~10%, while
for 'file' it largely depends on the filesystem, but always
in the range of factor ~3 - ~4
Note that this does not take into account parallel actions, such as gc,
verify or other backups.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
fixups for DatastoreFSyncLevel:
* use derive for Default
* add some more derives (Clone, Copy)
chunk store:
* drop to_owned for chunk_dir_path
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the dropped .into() is guarded by the bumped build-dependency on
proxmox-sys 0.4.1, the missing Eq is a new clippy lint.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
currently, we don't (f)sync on chunk insertion (or at any point after
that), which can lead to broken chunks in case of e.g. an unexpected
powerloss. To fix that, offer a tuning option for datastores that
controls the level of syncs it does:
* None (default): same as current state, no (f)syncs done at any point
* Filesystem: at the end of a backup, the datastore issues
a syncfs(2) to the filesystem of the datastore
* File: issues an fsync on each chunk as they get inserted
(using our 'replace_file' helper) and a fsync on the directory handle
a small benchmark showed the following (times in mm:ss):
setup: virtual pbs, 4 cores, 8GiB memory, ext4 on spinner
size none filesystem file
2GiB (fits in ram) 00:13 0:41 01:00
33GiB 05:21 05:31 13:45
so if the backup fits in memory, there is a large difference between all
of the modes (expected), but as soon as it exceeds the memory size,
the difference between not syncing and syncing the fs at the end becomes
much smaller.
i also tested on an nvme, but there the syncs basically made no difference
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[ T: commit Sources for test & fix white space errors ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we converted the prune settings of datastores to prune-jobs, but did
not actually implement the notifications for them, even though
we had the notification options in the gui (they did not work).
implement the basic ok/error notification for prune jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
intended for passing the format to the file-restore client/daemon
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to allow detection of repositories that support downloading indices via
their hash instead of their filename.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
which returns if a file reference is architecture specific, and for
which architecture it is relevant.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if we encounter a file reference pointing to a component that is not
contained in the componenents list, we can just ignore it as unknown.
only treat parsing errors for references pointing to known components as
actual errors.
this currently triggers with (In)Release files for debian-updates and
debian-security, which reference (empty) files for a "non-free-firmware"
component that is not listed in the `Components` field of the release
file.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the remaining ones are:
- type complexity
- fns with many arguments
- new() without default()
- false positives for redundant closures (where closure returns a static
value)
- expected vs actual length check without match/cmp
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to allow usage with in-memory contents instead of actual files.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Otherwise we have to use BackupType::iter().iter() whenever
we're not using a `for _ in iter()` construct.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
when creating the documentation (e.g. `cargo doc --open`), it would
warn that `Display` is not in scope.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
InfluxDbUdp and InfluxDbHttp for now
introduces schemas for host:port and https urls
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we can now use it for the error case and will further use it for the
can access namespace but not datastore case in a future patch
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead move the acl_path helper to BackupNamespace, and introduce a new
helper for printing a store+ns when logging/generating error messages.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
for usage in permission check error messages, to allow easily indicating
which privs are missing.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
syncing to a namespace only requires privileges on the namespace (and
potentially its children during execution).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the namespace is optional, but should be captured to allow ACL checks
for unprivileged non-job-owners.
also add FIXME for other job types and workers that (might) need
updating.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
We can have those in existing verify jobs configs, and that'd break
stuff. So, even while the "bad" commit got released only recently
with `2.1.6-1` (14 April 2022), we still need to cope with those that
used it, and using some serde parser magic to transform on read only
is hard here due to section config (json-value and verify currently
happen before we can do anything about it)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This reverts commit c72fe7d77c560ad7699dd43d80dbfb98bded4d51.
We could already cause the behavior by simply setting ignore-verified
to false, aas that flag is basically an on/off switch for even
considering outdated-after or not.
So avoid the extra logic and just make the gui use the previously
existing way.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by adding the 'default' serde hint and renaming 'recursion_depth' to
'max_depth' (to be in line with sync job config)
also add the logic to actually add/update the tape backup job config
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding a new parameter 'namespaces', which contains a mapping
for a namespace like this:
store=datastore,source=foo,target=bar,max-depth=2
if source or target are omitted the root namespace is used for its value
this mapping can be given several times (on the cli) or as an array (via
api) to have mappings for multiple datastores
if a specific snapshot list is given simultaneously, the given snapshots
will be restored according to this mapping, or to the source namespace
if no mapping was found.
to do this, we reutilize the restore_list_worker, but change it so that
it does not hold a lock for the duration of the restore, but fails
if the snapshot does exist at the end. also the snapshot will now
be temporarily restored into the target datastore into the
'.tmp/<media-set-uuid>' folder.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
these are helpers for the few cases where we want to print and parse
from a format that has the namespace and snapshot combined, like for
the on-tape catalog and snapshot archive.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
into the regular one (with default == MAX) and the one used for
pull/sync, where the default is 'None' which actually means the remote
end reduces the scope of sync automatically (or, if needed,
backwards-compat mode without any remote namespaces at all).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and use it when creating a sync job, and simplify the check on updating
(only check the final, resulting config instead of each intermediate
version).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.
Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Allow pulling all groups from a certain source namespace, and
possibly sub namespaces until max-depth, into a target namespace.
If any sub-namespaces get pulled, they will be mapped relatively from
the source parent namespace to the target parent namespace.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to use slice::strip_prefix() from std
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
helpful for places where namespaces need to be (re)created
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.
For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).
The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.
Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.
Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
allow to list any namespace with privileges on it and allow to create
and delete namespaces if the user has modify permissions on the parent
namespace.
Creation is only allowed if the parent NS already exists.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
[ T: renamed from NAMESPACE_RECURSION_DEPTH_SCHEMA & moved to from
jobs to datastore ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
given a namespace, a source prefix and a target prefix this helper
strips the source prefix and replaces it with the target one (erroring
out if the prefix doesn't match).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Make it easier by adding an helper accepting either group or
directory
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The idea is to have namespaces in a datastore to allow grouping and
namespacing backups from different (but similar trusted) sources,
e.g., different PVE clusters, geo sites, use-cases or company
service-branches, without separating the underlying
deduplication domain and thus blowing up data and (GC/verify)
resource usage.
To avoid namespace ID clashes with anything existing or future
usecases use a intermediate `ns` level on *each* depth.
The current implementation treats that as internal and thus hides
that fact from the API, iow., the namespace path the users passes
along or gets returned won't include the `ns` level, they do not
matter there at all.
The max-depth of 8 is chosen with the following in mind:
- assume that end-users already are in a deeper level of a hierarchy,
most often they'll start at level one or two, as the higher ones
are used by the seller/admin to namespace different users/groups,
so lower than four would be very limiting for a lot of target use
cases
- all the more, a PBS could be used as huge second level archive in a
big company, so one could imagine a namespace structure like:
/<state>/<intra-state-location>/<datacenter>/<company-branch>/<workload-type>/<service-type>/
e.g.: /us/east-coast/dc12345/financial/report-storage/cassandra/
that's six levels that one can imagine for a reasonable use-case,
leave some room for the ones harder to imagine ;-)
- on the other hand, we do not want to allow unlimited levels as we
have request parameter limits and deep nesting can create other
issues as well (e.g., stack exhaustion), so doubling the minimum
level of 4 (1st point) we got room to breath even for the
more odd (or huge) use cases (2nd point)
- a per-level length of 32 (-1 due to separator) is enough to use
telling names, making lives of users and admin simpler, but not
blowing up parameter total length with the max depth of 8
- 8 * 32 = 256 which is nice buffer size
Much thanks for Wolfgang for all the great work on the type
implementation and assisting greatly with the design.
Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Returning the GC status was dropped by mistake in commit fdcb2694
("datastore status: factor out api type DataStoreStatusListItem")
As this is considered a breaking change which we also felt, due to
the gc-status being used in the web interface for the datastore
overview list (not the dashboard), re add it.
Fixes: fdcb2694
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: add reference to breaking commit, reword message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
so our examples can more easily access a datastore without
going over a configuration & cache
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
let BackupGroup implement Hash
let BackupGroup and BackupDir be AsRef<BackupGroup>
let BackupDir be AsRef<BackupDir>
the pbs-datastore types will implement these AsRefs as well
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The type is a real enum.
All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.
Also, they are now flattened into a few structs instead of
being copied manually.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
add support for multi-line comments to node.cfg and the api, similar to
how pve handles multi-line comments
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
ureq has support for a HTTP proxy, but no support for HTTPS proxy yet.
ureq doesn't query `all_proxy` and `ALL_PROXY` environment variables by
itself, the way curl does. So set the proxy in code if any of the above
environment variables are set.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
the latest changes to this api call changed/removed some things that
were actually necessary for the gui. Readd those and document them this
time.
The change from u64 to i64 limits us to 8EiB of Datastore sizes (instead if
16EiB) but if we reach that, we must adapt most other parts to use 128bit
sizes anyway
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order
Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option so that the admin can tune that behaviour
for each datastore.
The default stays the same (sorting by inode)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the suffix was changed with Debian Bullseye.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
std::convert::{TryFrom, TryInto} are now part of the prelude.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
these need to be checked (and are) via libssl anyway before persisting,
and newer versions might contain new ciphers/variants/... (and things
like @STRENGTH or @SECLEVEL=n were missing).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
there isn't really a concept of 'nodes' in PBS (yet) anyway - and if
there ever is, it needs to be handled by the rest-server / specific API
endpoints (like in PVE), and not by the schema.
this allows dropping proxmox-sys from pbs-api-types (and thus nix and
some other transitive deps as well).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's the only thing requiring openssl in pbs-api-types, and it's only
used by the client to pretty-print the 'master' key, which is
client-specific.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Some users want to import historical backups but they run into the
original lower backuo-time limit one can pass. That original limit
was derived from the initial PBS development start in 2019, it was
assumed that no older backup can exist with PBS before it existing,
but imports of older backups is a legitimate thing.
I pondered using 683071200 (1991-08-25), aka the first time Linux was
publicly announced by Linus Torvalds as new limit but at the end I
did not wanted to risk that and backup software is IMO to serious for
such easter eggs, so I went for 1, to differ between the bogus 0 some
tools fallback too if there's something off with time.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
commit 8c1ec5c8021b11d4ef657a55c67b060045e6ebdc introcuded a bug by
using fp.to_string(). Replace this with fp.signature() which correctly
returns the full fingerprint instead of the short version.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
otherwise we cannot properly parse the api return value from older
versions, since that field does not exist there.
fixes sync from older versions without the protected feature
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Adapted from Dietmar's v3 on pbs-devel but some changes:
- reworked with a strip_suffix fn that does matching, way shorter and
even easier to read IMO
- make b/B byte symbol fully optional, not just for base-10
- also trim trailing whitespace for SizeUnit::Byte
- simplify the FromStr impl
- adapt parser unit tests such that we actually see the failed test's
definition line, simplifies debugging a bit
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The new SizeUnit type takes over the auto scaling logic and could be
used on its own too.
Switch the internal type of HumanByte from u64 to f64, this results
in a slight reduce of usable sizes we can represent (there's no
unsigned float type after all) but we support pebibyte now with quite
the precision and ebibytes should be also work out ok, and that
really should us have covered for a while..
Partially adapted by Dietmar's version, but split up and change so:
* there's no None type, for a SizeUnit that does not makes much sense
* print the unit for byte too, better consistency and one can still
use as_u64() or as_f64() if they do not want/need the unit rendered
* left the "From usize/u64" impls intact, just convenient to have and
avoids all over the tree changes to adapt to loosing that
* move auto-scaling into SizeUnit, good fit there and I could see
some re-use potential for non-human-byte users in the future
* impl Display for SizeUnit instead of the separate unit_str method,
better usability as it can be used directly in format (with zero
alloc/copy) and saw no real reason of not having that this way
* switch the place where we auto-scale in HumanByte's to the new_X
helpers which allows for slightly reduced code usage and simplify
implementation where possible
* use rounding for the precision limit algorithm. This is a stupid
problem as in practices there are cases for requiring every variant:
- flooring would be good for limits, better less than to much
- ceiling would be good for file sizes, to less can mean ENOSPACE
and user getting angry if their working value is messed with
- rounding can be good for rendering benchmark, closer to reality
and no real impact
So going always for rounding is really not the best solution..
Some of those changes where naturally opinionated, if there's a good
practical reason we can switch back (or to something completely
different).
The single thing I kept and am not _that_ happy with is being able to
have fractional bytes (1.1 B or even 0.01 B), which just does not
makes much sense as most of those values cannot exist at all in
reality - I say most as multiple of 1/8 Byte can exists, those are
bits.o
Note, the precission also changed from fixed 2 to max 3 (trailing
zeros stripped), while that can be nice we should see if we get
a better precision limiting algorithm, e.g., directly in the printer.
Rust sadly does not supports "limit to precision of 3 but avoid
trailing zeros" so we'd need to adapt their Grisu based algorithm our
own - way to much complexity for this though..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we even use that for basically all the related schema names, "groups"
allone is just rather not so telling, i.e., "groups" what?
While due to the additive nature of `group-filter` is not the best
possible name for passing multiple arguments on the CLI (the web-ui
can present this more UX-friendly anyway) due to possible confusion
about if the filter act like AND vs OR it can be documented and even
if a user is confused they still are safe on more being synced than
less. Also, the original param name wasn't really _that_ better in
that regards
Dietmar also suggested to use singular for the CLI option, while
there can be more they're passed over repeating the option, each with
a single filter.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
and convert existing (manually created/edited) jobs to the previous
default value of 'true'. the GUI has always set this value and defaults
to 'false'.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
like for manual pulls, but persisted in the sync job config and visible
in the relevant GUI parts.
GUI is read-only for now (and defaults to no filtering on creation), as
this is a rather advanced feature that requires a complex GUI to be
user-friendly (regex-freeform, type-combobox, remote group scanning +
selector with additional freeform input).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
at the API level, this is a simple (wrapped) Vec of Strings with a
verifier function. all users should use the provided helper to get the
actual GroupFilter enum values, which can't be directly used in the API
schema because of restrictions of the api macro.
validation of the schema + parsing into the proper type uses the same fn
intentionally to avoid running out of sync, even if it means compiling
the REs twice.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the regex was already there, and we need a simple type/schema for
passing in multiple groups as Vec/Array via the API.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by storing the proxy url as string in the struct and setting it on
each invocation of `execute`, since execute calls reset on the
curl::easy::Easy object.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Without this, the tests fail in debug mode.
Also having start (u64) underflow to a value greater than end does
not really make sense
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
adds the info that a snapshot is protected to:
* snapshot list
* manual pruning (also dry-run)
* prune jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
some custom ACME endpoints do not have a TOS, and thus do not return
a meta property at all
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Because the types used inside the RRD have other requirements
than the API types:
- other serialization format
- the API may not support all RRD features
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
it's a rather low-level tool mostly useful for debugging and some of
it is rather "dumb" (by design) anyway, e.g., it does not
transparently applies journal but really only operates on the DB
files as is (which can conflict with daemon operations).
In summary, not (yet) a tool meant for end user consumption.
Move it to examples folder to avoid compilation on packaging (we do
not ship it anyway) which allows us to move the rather expensive
proxmox-router (pulls in hyper) to the dev-dependencies section.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Modulo is very slow, so we try to avoid it inside loops.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Because the types used inside the RRD have other requirements
than the API types:
- other serialization format
- the API may not support all RRD features
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Storing much more data points now got get better graphs.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.
Apply the journal every 30 minutes and on daemon startup.
Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ApiConfig: avoid using pbs_config::backup_user()
CommandoSocket: avoid using pbs_config::backup_user()
FileLogger: avoid using pbs_config::backup_user()
- use atomic_open_or_create_file()
Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
which allows to get rid of an possible error with check_suites, and
easily detect unexpected values with get_current_release_codename.
The check_repos function needs to be adapted, since the type does
not include suite names like oldstable,experimental,etc.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
by exchanging loops in the check_suites function, which was the only
user. Exchanging loops also helps for introducing a type for Debian condenames.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Like that, a potential error is further up the stack, and it's more
consistent with what the standard_repository functions do.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
use the first appearance of '-' or '/' to detect the variant instead
of keeping a list of possible variants, which would need to include
things like "-proposed-updates-debug".
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Require that the suite matches too when detecting standard
repositories, since no or invalid updates will be obtained when the
suite is wrong. Thus, it should not be considered to be the same
repository.
Add the parameter for get_standard_repository too, so that the two
related calls have more similar parameters, and the detection of the
current release code name can be done once in the caller once.
This also will fix an issue with the front-end, where adding a
standard repository would end up just enabling an already present
repository with the wrong suite.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
pbs-datastore now ended up depending on tokio after all, but
that's fine for now
for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link
the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
For PVE, URIs without the final "/pve" are also valid.
Make the single URL response a vector and iterate over it, lower
index is preferred.
Reported in the community forum:
https://forum.proxmox.com/threads/pve-7-0-9-no-proxmox-ve-repository-enabled.92427/
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
[ Thomas: extend commit message slightly ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stored in atomically-updated 'notes' file in backup group directory.
Available via dedicated GET/PUT API calls, as well as the first line
being included in list_groups (similar to list_snapshots).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
move key_derivation to pbs-datastore
pbs-api-types should only contain "basic" types which
* are usually required by clients
* don't depend on pbs-related code directly
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
so that parsing CD ROM repositories with spaces in the name works too.
But it's not limited to that, and should make one-line parsing rather
similar to what APT does (stanza parsing in APT doesn't use
ParseQuoteWord at all AFAICS).
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Add a description for the handle, which can be useful to display
alongside the name. The descriptions are essentially the first
sentence from PVE's "Package Repositories" docs, but without the
product name.
Also drop the " Repository" suffix from the names, as it's not useful,
but can be ugly: e.g. for the UI when the label already is
'Repository:'.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
which is obtained from the cached InRelease file and also works for
mirrors, host aliases, direct IPs.
The has_official_uri function was replaced by origin_from_uris.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Allows obtaining the 'Origin' property from the cached InRelease file.
Used the once_cell crate for the config module, because it is already
used in proxmox-backup and seemed to be the right fit here.
For now, the config module is just used to be able to override the
path for the test environment, but those are actual APT config
variables, and in the future, it can be extended and used to actually
parse the apt.conf(.d/*) on the system.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Get handles for the available repositories along with their current
configuration status and make it possible to add them.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.
Clamp the value to 0 in that case.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
- remove chrono dependency
- depend on proxmox 0.3.8
- remove epoch_now, epoch_now_u64 and epoch_now_f64
- remove tm_editor (moved to proxmox crate)
- use new helpers from proxmox 0.3.8
* epoch_i64 and epoch_f64
* parse_rfc3339
* epoch_to_rfc3339_utc
* strftime_local
- BackupDir changes:
* store epoch and rfc3339 string instead of DateTime
* backup_time_to_string now return a Result
* remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir
- DynamicIndexHeader: change ctime to i64
- FixedIndexHeader: change ctime to i64
and drop the now unused extract_lists function
this also fixes a bug, where we did not add the datastore to the list at
all when there was no rrd data
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
there is now a 'extract_cached_data' which just returns
the data of the specified field, and an api function that converts
a list of fields to the correct serde value
this way we do not have to create a serde value in rrd/cache.rs
(makes for a better interface)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this is an interface to simply get the Vec<Option<f64>> out of rrd
without going through serde values
we return a list of timestamps and a HashMap with the lists we could find
(otherwise it is not in the map)
if no lists could be extracted, the time list is also empty
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
"Group filter based on group identifier ('group:GROUP'), group type ('type:<vm|ct|host>'), or regex ('regex:RE'). Can be inverted by prepending 'exclude:'.")
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
