5
0
mirror of git://git.proxmox.com/git/pve-common.git synced 2025-01-09 05:17:35 +03:00
Commit Graph

1548 Commits

Author SHA1 Message Date
Dominik Csapak
dc023d60c5 SysFSTools: handle new nvidia syfsapi as mdev
with kernel 6.8 NVIDIAs vGPU driver has a different api than the
previous 'mediated devices'. Adapt our sysfcode to also recognize this
for the 'mdev' paths and add another 'nvidia' property so we can detect
this.

Also parse the new api when they exist instead of the mediated devices.

The biggest difference to the existing mdev api for our use is that the
devices don't report all generally available devices, only the
createable ones. So if a user wants to configure a VM, the selection is
restricted by what may currently run on the GPU (depending ont the exact
settings, e.g. mixed mode gpus where different models can be mixed on a
single GPU; not the default though)

We could overcome this, when we'd parse the general info from the
'nvidia-smi' tool, though I'm currently unsure if that interface is
stable and intended to be parsed (there is no json output or similar
AFAIK)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
2024-10-24 18:33:46 +02:00
Dominik Csapak
eff59571be sysfstools: file_write: properly catch errors
since `print` is doing buffered IO, we don't always get an error there,
even if the underlying write does not work.

To properly catch that, do an unbuffered `syswrite` which circumvents
all buffers and writes directly to the file handle.

We aren't actually interested in the specific error here, but only if
the write was successful or not.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
2024-10-24 18:31:04 +02:00
Thomas Lamprecht
5b0106ebee bump version to 8.2.5
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-19 16:47:23 +02:00
Thomas Lamprecht
c4945bf96b tools: load PerlIO explicitly to avoid odd failures
Since commit ef0bcc9 ("tools: file_set_contents: use syswrite instead
of print") we're using PerlIO's scalar layer to ensure we encode any
potential unicode before passing the data to syswrite, which does not
support writing code points above 255.

Add an explicit use statement for PerlIO::scalar to avoid some odd
failures.

Some more background on why this seems odd:

After the pve-common version that included this change got moved to
public repos we got some reports in our Forum about CTs failing to
start [0] due to the lxc-pve-prestart-hook failing with an error
message like:

> Can't locate PerlIO.pm in @INC [...] at /usr/share/perl5/PVE/Tools.pm line 293.

Which points to the recently added `open(my $data_fh, '>', \$data)`
line in file_set_contents. The call chain from there upwards was
$lxc_setup->ct_file_set_contents <- $lxc_setup->pre_start_hook <-
closure <- lxc_hook <- lxc-pve-prestart-hook.
This seemed especially odd as we use `file_set_contents` in a lot of
other places and there was no all to obvious breakage from the change
on our test systems.

During evaluation I noticed some additional strange behavior, if one
called `file_set_contents` inside the closure before the call to
`pre_start_hook`, the error just goes away and one can observer that
%INC, which contains all loaded modules, suddenly does have a entry
for the PerlIO module, or well, it's scalar layer, which it did not
have without that call. So why the PerlIO can get automatically loaded
just fine most of the time but not inside the `pre_start_hook` is not
really clear yet; still loading PerlIO explicitly makes the issue go
away and seems sensible, so go for that and keep a comment to remind
more explicitly of this oddity. Once it's explained it can be removed
with a commit that mentions the explanation.

Further, if the PerlIO scalar layer cannot be loaded, the result is
that the passed reference is used as filename, which is far from
ideal, see the report in the perl GH [1] and the PR that fixes this
[2] by moving PerlIO::scalar into perl core proper, which will be
available in Perl v5.40 and thus our next Debian Trixie based major
release. This might have well to do with the original symptom that
embarked me on this odd (and not 100% finished) quest..

[0]: https://forum.proxmox.com/threads/156188/
[1]: https://github.com/Perl/perl5/issues/21275
[2]: https://github.com/Perl/perl5/pull/21282

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-19 16:47:23 +02:00
Thomas Lamprecht
e4394777d2 tools: code-style cleanup file_set_contents
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-19 16:42:43 +02:00
Thomas Lamprecht
f291dcd493 bump version to 8.2.4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-16 19:02:23 +02:00
Thomas Lamprecht
f1fe7a0733 file set contents: fix error handling with or-operator precedence
In perl the `or` and the `||` operator do mostly the same thing but
with a different precedence level [0].

A statement like:
`$foo += bar() or die "error"`
is basically equivalent to:
`($foo += bar()) or die "error"`

That means as long as bar only returns zero or positive integers the
`or die` can only happen the first time, as otherwise $foo is bigger
than zero and thus will never evaluate to false. This can be
reproduced by perl -we 'my $foo = 1; $foo += 0 or die "wont happen";'

While one could switch to the `||` operator, this is a bit to subtle,
so to fix this, separate tracking the total bytes written from getting
the bytes written by the current call, this avoids the error potential
completely.

[0]: https://perldoc.perl.org/perlop#Logical-or-and-Exclusive-Or

Reported-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-14 11:14:23 +02:00
Filip Schauer
ef0bcc98ad tools: file_set_contents: use syswrite instead of print
The use of `print` can be inefficient for writing larger files due to
its default buffering in 8 KiB blocks.

This is especially problematic on `pmxcfs` where files are written in
4 KiB blocks due to the defaults of `libfuse2`. This leads to
significant write amplification on files larger than 4 KiB.

Patch (fix #5728: pmxcfs: allow bigger writes than 4k for fuse) [1]
addresses this by enabling `big_writes`, allowing up to 128 KiB blocks.
But due to the use of `print` in `file_set_contents`, writes are still
only buffered in 8 KiB blocks.

To further address this, this commit switches to using `syswrite`
instead of `print` to mitigate the block size limit imposed by `print`.
Combined with patch [1], file writes to `/etc/pve/` are now buffered in
128 KiB blocks.

The table below illustrates the drastic reduction in write
amplification when writing files of different sizes to `/etc/pve/` using
`file_set_contents`:

           print                big_writes+print     big_writes+syswrite
file size  written     amplif.  written     amplif.  written    amplif.
    1 KiB      48 KiB     48.0      45 KiB     45.0     41 KiB     41.0
    2 KiB      48 KiB     24.0      45 KiB     22.5     62 KiB     31.0
    4 KiB      82 KiB     20.5      80 KiB     20.0     73 KiB     18.3
    8 KiB     121 KiB     15.1      90 KiB     11.3     89 KiB     11.1
   16 KiB     217 KiB     13.6     146 KiB      9.1    113 KiB      7.1
   32 KiB     506 KiB     15.8     314 KiB      9.8    158 KiB      4.9
   64 KiB    1472 KiB     23.0     826 KiB     12.9    259 KiB      4.0
  128 KiB    5585 KiB     43.6    3765 KiB     29.4    452 KiB      3.5
  256 KiB   20424 KiB     79.8   10743 KiB     42.0   2351 KiB      9.2
  512 KiB   86715 KiB    169.4   43650 KiB     85.3   3204 KiB      6.3
 1024 KiB  369568 KiB    360.9  187496 KiB    183.1  15845 KiB     15.5

Since `file_set_contents` also performs a `rename` after writing, the
following table shows the results when the file is written without
renaming it afterwards:

           print                big_writes+print     big_writes+syswrite
file size  written     amplif.  written     amplif.  written     amplif.
    1 KiB      29 KiB     29.0      29 KiB     29.0     25 KiB      25.0
    2 KiB      29 KiB     14.5      30 KiB     15.0     25 KiB      12.5
    4 KiB      37 KiB      9.3      44 KiB     11.0     41 KiB      10.3
    8 KiB      61 KiB      7.6      45 KiB      5.6     45 KiB       5.6
   16 KiB     143 KiB      8.9      86 KiB      5.4     57 KiB       3.6
   32 KiB     396 KiB     12.4     225 KiB      7.0     69 KiB       2.2
   64 KiB    1281 KiB     20.0     673 KiB     10.5    105 KiB       1.6
  128 KiB    4789 KiB     37.4    3478 KiB     27.2    169 KiB       1.3
  256 KiB   18868 KiB     73.7    9976 KiB     39.0    572 KiB       2.2
  512 KiB   79304 KiB    154.9   42714 KiB     83.4   2150 KiB       4.2
 1024 KiB  347929 KiB    339.8  182483 KiB    178.2  11133 KiB      10.9

[1] https://lists.proxmox.com/pipermail/pve-devel/2024-September/065396.html

Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
2024-10-14 10:23:50 +02:00
Thomas Lamprecht
aeff857f77 bump version to 8.2.3
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-09-23 10:24:00 +02:00
Fabian Grünbichler
e83d895594 method schema: rename 'download' parameter to 'download_allowed'
to make it more obvious what it does, but keep old variant for now.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 10:19:20 +02:00
Fabian Grünbichler
c1ed9e3ee6 download handling: adapt cycle check to check value
instead of whether the key exists in the schema instance, just in case somebody
wants to set "download => 0".

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 10:19:20 +02:00
Fabian Grünbichler
6dbb3e0e78 schema: adapt description of method's download key
with the newly introduced strict checks in the API handler, the download key
actually marks which endpoints/methods are allowed to use the download
functionality of the REST server, and the "directly return filename to be
downloaded" part is no longer supported as a result.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 10:19:20 +02:00
Thomas Lamprecht
e068c3d30a cli handler: print correct command prefix for alias in asciidoc output
Otherwise only the base exename is printed, even if the alias is for
sub-command.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-07-31 20:24:04 +02:00
Thomas Lamprecht
7f3dc635d8 bump version to 8.2.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-07-30 21:42:16 +02:00
Fabian Grünbichler
48da50c2be Revert "section config: document package and its methods with POD"
This reverts commit d41bd420f8.
2024-07-24 13:44:45 +02:00
Fabian Grünbichler
b3ed595212 Revert "section config: update code style"
This reverts commit 29292d2a5d.
2024-07-24 13:44:40 +02:00
Fabian Grünbichler
50f0b6d0be Revert "section config: clean up parser logic"
This reverts commit 6cba8d7660.
2024-07-24 13:44:37 +02:00
Thomas Lamprecht
4386716c5a REST handler: use double-hyphen as argument prefix for doc and CLI help output
Use "modern" double-hyphen style, we're not some odd BSD derived
tool..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-07-22 19:10:28 +02:00
Wolfgang Bumiller
b64a40a506 fix #5529: cgroup: correctly handle change_cpu_quota without a quota
The function can be called with
- neither quota nor period
- only a period (quota will be 'max')
- both

$quota was therefore defaulted to 'max' and the check for whether
values were provided should use $period instead of $quota.
Also move the defaulting-assignment into the condition for clarity.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-22 19:07:36 +02:00
Dominik Csapak
06f436f126 fix #5486: tools: encode_text: add '%' to list of encoded characters
all text that is going through encode_text will at a later point be
decoded by 'decode_text'. The latter is decoding all percent encoded
characters, even those not originally encoded by 'encode_text'.

This means, to preserve the original data, we first have to at least
percent encode the '%' itself, otherwise it's impossible to properly
store e.g. '%20' there.

It would get saved as '%20' directly, but on the next read, it gets
decoded to ' ', which is not the original data. instead we have to save
it as '%2520', which gets then correctly decoded to '%20' again

This is especially important for the vm/ct/node description, as there
users can store external links, which already include percent encoded
characters.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-07-04 10:54:48 +02:00
Max Carrara
6cba8d7660 section config: clean up parser logic
In order to make the parser somewhat more maintainable in the future,
this commit cleans up its logic and makes its control flow easier to
follow.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-07-04 10:54:48 +02:00
Max Carrara
29292d2a5d section config: update code style
Replace `foreach` with `for` and use postfix deref instead of block
(circumfix) dereference (`$foo->%*` instead of `%$foo`).

Furthermore, make `format_config_line` a private sub instead of
unnecessarily declaring it as an anonymous subroutine, which avoids
the `&$sub_ref(...)` syntax altogether.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-07-04 10:54:48 +02:00
Max Carrara
d41bd420f8 section config: document package and its methods with POD
Apart from the obvious benefits that documentation has, this also
allows LSPs to provide docstrings e.g. via 'textDocument/hover' [0].

Tested with Perl Navigator [1].

[0]: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_hover
[1]: https://github.com/bscan/PerlNavigator

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-07-04 10:54:48 +02:00
Jing Luo via pve-devel
6dc7a73bd5 tools: fix syscall mknod()
b792e8df81 introduced a bug that can cause this:

Undefined subroutine &PVE::Syscall::SYS_mknod called at /usr/share/perl5/PVE/Syscall.pm line 11

It should be mknod, not SYS_mknod. This caused other pve perl lib failing
to build. I couldn't reproduce this on amd64 build, but I could reproduce this
on arm64 build; however this didn't seem to fix the issue, unless I revert
b792e8df81.

cf: b792e8df81
Signed-off-by: Jing Luo <jing@jing.rocks>
2024-07-03 10:25:21 +02:00
Thomas Lamprecht
1a6005ad23 bump version to 8.2.1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 15:43:01 +02:00
Wolfgang Bumiller
b518bbd5f9 interfaces: support stanzas without types/methods
This is allowed in ifupdown2 and previously interfaces named
'vmbr\d+' were recognized as bridges even if they used this mode.
With commit e68ebda4f1 this is no longer the case.

Fixes: e68ebda4f1 ("fix #545: interfaces: allow arbitrary bridge names in network config")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-04-23 15:19:43 +02:00
Thomas Lamprecht
a061517cdd bump version to 8.2.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-21 11:51:17 +02:00
Stefan Hanreich
e68ebda4f1 fix #545: interfaces: allow arbitrary bridge names in network config
Similar to other interface types, we can detect a bridge by the
presence of its bridge_ports attribute, rather than solely relying on
the "vmbr" ifname prefix heuristic. For OVS bridges we need to examine
the OVS type instead.

The check needs to be moved up since other prefixes could
theoretically be included in a bridge name and then would otherwise
get picked up wrongly.

Also added a warning for interfaces named vmbrX that are not bridges
to catch possible misconfigurations.

Originally-by: Jillian Morgan <jillian.morgan@primordial.ca>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
2024-04-21 11:22:10 +02:00
Thomas Lamprecht
55ba8536f0 bump version to 8.1.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-17 21:10:39 +02:00
Fiona Ebner
c302a28a21 json schema: add format description for pve-storage-id standard option
so that the option can be used as part of a property string.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-04-11 17:19:52 +02:00
Folke Gleumes
a656f674c2 docs: add missing prefix
include 'PVEAPIToken=' prefix in the example for target-endpoint which
is mainly used for remote migrations.

Signed-off-by: Folke Gleumes <f.gleumes@proxmox.com>
2024-04-08 18:01:35 +02:00
Wolfgang Bumiller
10349969cc bump version to 8.1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-06 12:03:12 +01:00
Max Carrara
ff39327769 ticket: remove fallback for SHA1-base64 CSRF prevention tokens
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-03-06 12:03:10 +01:00
Wolfgang Bumiller
68b234f35e add PVE::Systemd::is_unit_active
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-06 11:57:22 +01:00
Wolfgang Bumiller
c6ec71d846 schema: fixup description vs format_description in remote_format
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-01-22 12:52:41 +01:00
Fabian Grünbichler
f869c96a94 network tests: test #5141
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:18:26 +01:00
Fabian Grünbichler
80ed66dc17 fix #5141: network parser: fix accidental RE result re-use
$1 and friends are not cleared if a RE fails to match, in which case they will
contain the captured values from a previous successful match in the same scope.

deduplicate the two branches to avoid accidental re-introduction.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:16:14 +01:00
Fabian Grünbichler
fcc97ec96d network parser: iterate deterministically
makes the behaviour easier to analyze, and also helps when testing since it
allows constructing test cases that trigger certain order of parsing.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:15:37 +01:00
Fabian Grünbichler
e8c1bbfca2 network tests: switch to ifupdown2
adapt allow-* to auto, and drop the one test where behaviour is not testable
anymore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:14:43 +01:00
Wolfgang Bumiller
63b74c5089 expose SYS_prctl
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-12-14 10:09:48 +01:00
Thomas Lamprecht
17c45baaa4 bump version to 8.1.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-21 13:04:26 +01:00
Thomas Lamprecht
7ccdc805e3 section config: avoid unamed boolean parameter use hash
Even with just one param it's extra work to check what it refers too,
with named ones in a hash one hasn't that issue even with many params.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-17 09:58:01 +01:00
Dominik Csapak
7d6cc4f14f section config: add tests for separated property lists
more or less a copy from the normal section config test, but now with
properties defined multiple times as well as conflicting options

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ TL: improve consistency with property-isolation terminology ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-17 09:57:39 +01:00
Dominik Csapak
7887b1cb0e section config: allow full property-isolation for plugins
When using 'init(property_isolation => 1)', the code saves the
property lists per type instead of a big one, and using
create/updateSchema creates a new schema with the options as 'oneOf'
and/or 'instance-types' (depending if the schemas match).

With that, we change how we work with the options hash:

It's not needed anymore to specify options that are specified in the
type specific propertyList, except if it's 'fixed => 1' (since that does
not exist in the schema)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ TL: ensure consistency with new property-isolation terminology ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-17 09:57:34 +01:00
Dominik Csapak
942583468f json schema: implement 'oneOf' schema
a schema can now have the 'oneOf' property which is an array of regular
schemas. In the default case any of that has to match. If the
'type-property'/'instance-types' are given, only the schema for the specific
type will be checked (and handles as 'additionalProperties' if there is
no matching type)

the field found in 'type-property' has to be on the same level
(so for oneOf the nested schemas should not include that).

Documentation is adapted so that options are grouped per `type-property=value`
after the regular options (with their individual descriptions/types/etc.)

oneOfs without 'type-property'/'instance-tyeps' simply show up twice for
now with an 'or' line in between.

command line parsing is a bit weird for now since Getopt::Long
can't have multiple variants for the same property (but works fine with
pvesh for our current use cases). it gets shown as '--foo <multiple' if
they are not optional.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-17 09:24:09 +01:00
Dominik Csapak
15645af168 tools: add is_deeply
to compare nested hashes/lists and scalar values recursively.
Also includes some tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-17 09:24:09 +01:00
Dominik Csapak
e42f1dd6a1 section config: add test for the schemas
by simply doing an 'is_deeply' on the generated schema with
the current generated schema

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-17 09:24:09 +01:00
Filip Schauer
fe468fad74 tools: Add mount flag constants
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
2023-11-13 15:08:58 +01:00
Filip Schauer
b792e8df81 tools: Add mknod syscall
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
2023-11-13 15:08:58 +01:00
Thomas Lamprecht
118ca13a88 bump version to 8.0.10
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-07 08:58:34 +01:00