1790 Commits

Author SHA1 Message Date
Andrey Smirnov
6be5a13d5d
feat: implement machine config documents for event and log streaming
Fixes #7228

Add some changes to make Talos accept partial machine configuration
without main v1alpha1 config.

With this change, it's possible to connect a machine already running
with machine configuration (v1alpha1), the following patch will connect
to a local SideroLink endpoint:

```yaml
apiVersion: v1alpha1
kind: SideroLinkConfig
apiUrl: grpc://172.20.0.1:4000/?jointoken=foo
---
apiVersion: v1alpha1
kind: KmsgLogConfig
name: apiSink
url: tcp://[fdae:41e4:649b:9303::1]:4001/
---
apiVersion: v1alpha1
kind: EventSinkConfig
endpoint: "[fdae:41e4:649b:9303::1]:8080"
```

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-01 00:22:44 +04:00
James Callahan
c02ada7d95
fix: capabilities including ALL should be uppercase
Pod security standard requires that ALL is in caps

Signed-off-by: James Callahan <james@wavesquid.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-29 12:58:30 +05:30
Noel Georgi
cbdf96d461
feat: support environment file for extensions
Supports setting `environmentFile` for Talos System Extension Services.

Fixes: #7316

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-28 00:21:13 +05:30
Andrey Smirnov
35d6adcb9a
fix: provide stashed META values before installation
Previously, if META values were supplied to the Talos ISO via
environment variable, they will be written down and available after the
install. With this fix, values are also readable and available before
the installation runs (in maintenance mode).

Most of the PR is refactoring `meta.Value(s)` to be a shared library
which is used by the installer/imager and (now) Talos.

Also fixes an issue with not returning properly `NotExist` error when
META is not yet available as a partition on disk.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-27 20:57:43 +04:00
Noel Georgi
bc371ecfda
chore: add /sbin/shutdown
Some tools like qemu-guest-agent when ran as a extension service calls
`/sbin/shutdown` instead of `/sbin/poweroff`. This adds handling for the
same.

Ref: https://github.com/siderolabs/extensions/pull/173

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-27 16:10:51 +05:30
Utku Ozdemir
0d313b9733
feat: add reboot-mode flag to talosctl upgrade
Allow specifying the reboot mode during upgrades by introducing `--reboot-mode` flag, similar to the `--mode` flag of the reboot command.

Closes siderolabs/talos#7302.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
2023-06-26 17:37:19 +02:00
Markus Reiter
7ce87f20c3
fix: compare only basename of os.Args[0] in machined
This makes handling of `exec` more flexible.

Signed-off-by: Markus Reiter <me@reitermark.us>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-26 17:42:30 +04:00
Noel Georgi
8daf432b29
chore: bump deps
Bump deps.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-22 22:41:08 +05:30
Noel Georgi
e3f3f5794d
feat: implement revert for sd-boot
Implement revert for sd-boot.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-22 20:20:31 +05:30
Andrey Smirnov
fe0f46980f
feat: implement secure boot from disk
This includes sd-boot handling, EFI variables, etc.

There are some TODOs which need to be addressed to make things smooth.

Install to disk, upgrades work.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-16 20:15:16 +05:30
Dmitriy Matrenichev
445f5ad542
feat: support API server load balancer
This commit adds support for API load balancer. Quick way to enable it is during cluster creation using new `api-server-balancer-port` flag (0 by default - disabled). When enabled all API request will be routed across
cluster control plane endpoints.

Closes #7191

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-06-16 10:09:20 -04:00
Andrey Smirnov
19bc223de8
refactor: bootloader interface, labels
Move labels out of the bootloader interface, while moving copying assets
into the bootloader interface. GRUB is using one set of assets,
`sd-boot` will be using another one.

Fix the problem with `bootloader.Probe()` finding boot partition on the
host when it runs in a priv container, fixing issues with image creation
in the CI.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-14 17:33:11 +04:00
Dmitriy Matrenichev
665702ddd3
chore: fix cilium e2e tests
`WITH_CONFIG_PATCH_WORKER` check result was overriding any value set in `CONFIG_PATCH_FLAG` variable.
Move it to the different variable.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-14 15:08:31 +04:00
Noel Georgi
71a548d180
chore: generic boootloader implementation
This changes the bootloader code to be generic to support
multiple bootloader implementations.

Signed-off-by: Noel Georgi <git@frezbo.dev>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-13 23:36:20 +04:00
Andrey Smirnov
e9dbc9311b
test: bump versions for upgrade tests
As we're getting to 1.5.0, bump versions for upgrade tests.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-13 23:17:22 +04:00
Andrey Smirnov
0a99965efb
refactor: replace uncordonNode with controllers
Fixes #7233

Waiting for node readiness now happens in the `MachineStatus` controller
which won't mark the node as ready until Kubernetes `Node` is ready.

Handling cordoning/uncordining happens with help of additional resource
in `NodeApplyController`.

New controller provides reactive `NodeStatus` resource to see current
status of Kubernetes `Node`.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-13 21:48:42 +04:00
Dmitriy Matrenichev
c74d937280
chore: bump github.com/cosi-project/runtime
Replace resource.Resource with meta.ResourceWithRD where possible.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-06-12 09:49:08 -04:00
Andrey Smirnov
dbaf5c6997
refactor: task labelControlPlane into controllers
See #7233

The controlplane label is simply injected into existing controller-based
node label flow.

For controlplane taint default NoScheduleTaint, additional controller &
resource was implemented to handle node taints.

This also fixes a problem with `allowSchedulingOnControlPlanes` not
being reactive to config changes - now it is.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-12 15:25:13 +04:00
Dmitriy Matrenichev
3816318b9e
chore: wrap config.Provider in atomic wrapper
Because `SetConfig` can be called concurrently with `Config` there is risk of data race, if something goes wrong. Since `config.Provider` is an interface type, it means its size is two machine words. And so in very unpleasant situations it can lead to arbitrary RCE, because interface variable can be in partially updated state.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-06-09 15:05:39 -04:00
Andrey Smirnov
f5e3272fce
refactor: task 'updateBootLoader' as controller
Fixes #7232

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-09 15:27:48 +04:00
Andrey Smirnov
e7be6ee7c3
refactor: make event log streaming fully reactive
I ended up completely rewriting the controller, simplifying the flow
(somewhat) so that there's just a single control flow in the controller,
while reading from v1alpha1 events is converted to reading from a
channel.

Fixes #7227

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-08 23:13:33 +04:00
Andrey Smirnov
c719aa2316
fix: allow http:// for discovery service URL
Fixes #7333

Also fixed the discovery service controller to reconnect the client on
config changes (previously it wasn't reactive on e.g. URL changes).

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-08 20:28:12 +04:00
Andrey Smirnov
aac441f618
chore: update Go to 1.20.5, bump dependencies
Go dependencies, new pkgs, extras, etc.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-07 23:40:59 +04:00
Noel Georgi
1c0c7933df
chore: cleanup partition code
Cleanup partition code to be explicit about `Format` and `Partition`
options.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-08 00:35:09 +05:30
Noel Georgi
e912c0dfcf
chore: use go-blockdevice for zeroing partitions
Use the `go-blockdevice` library to zero partitions.

Also added a test that writes `ones` to the partition and verifies its
zeroes after zeroing it.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-07 01:12:11 +05:30
Noel Georgi
47986cb79e
chore: unify kexec phase
This changes the mounting/unmounting of `BOOT` partiton code into
`kexecPrepare` phase. Also skips if `BOOT` partition cannot be found.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-06 20:30:59 +05:30
Andrey Smirnov
5dab45e869
refactor: allow kmsg log streaming to be reconfigured on the fly
Fixes #7226

This follows same flow as other similar changes - split out logging
configuration as a separate resource, source it for now in the cmdline.

Rewrite the controller to allow multiple log outputs, add send retries.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-06 15:56:24 +04:00
Dmitriy Matrenichev
8a02ecd4cb
chore: add endpoints balancer controller
This PR adds support for creating a list of API endpoints (each is pair of host and port).

It gets them from
- Machine config cluster endpoint.
- Localhost with LocalAPIServerPort if machine is control panel.
- netip.Addr[0] and port from affiliates if they are control panels.

For #7191

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-06-05 20:47:52 -04:00
Andrey Smirnov
bab484a405
feat: use stable network interface names
Use `udevd` rules to create stable interface names.

Link controllers should wait for `udevd` to settle down, otherwise link
rename will fail (interface should not be UP).

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-01 21:29:12 +04:00
Utku Ozdemir
196dfb99b0
fix: do not probe kernel args in dashboard if not needed
If the dashboard is run without the "Config URL" screen, do not initialize it, and do not probe the kernel args for the code parameter.

Refactor the dashboard to do not construct the unused screens at all.

Closes siderolabs/talos#7300.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
2023-06-01 10:32:43 +02:00
Andrey Smirnov
8c071b5796
fix: skip DHCP RENEW if server IP in the lease is all zeroes
RENEW packets are sent unicast, so Talos needs the address of the DHCP
server to send RENEW packets to.

Fixes #7211

Fixes #7263

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-31 22:51:05 +04:00
Andrey Smirnov
badbc51e63
refactor: rewrite code to include preliminary support for multi-doc
`config.Container` implements a multi-doc container which implements
both `Container` interface (encoding, validation, etc.), and `Conifg`
interface (accessing parts of the config).

Refactor `generate` and `bundle` packages to support multi-doc, and
provide backwards compatibility.

Implement a first (mostly example) machine config document for
SideroLink API URL.

Many places don't properly support multi-doc yet (e.g. config patches).

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-31 18:38:05 +04:00
Andrey Smirnov
a0773f783c
chore: add ukify Go script
This is a port of ukify.py and systemd-measure from systemd.

This requires no actual TPM to be present to calculate the PCR
signatures.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-05-30 23:33:26 +05:30
Andrey Smirnov
dc6764871c
refactor: move around config interfaces, make RawV1Alpha1 typed
See #7230

Refactor more config interfaces, move config accessor interfaces
to different package to break the dependency loop.

Make `.RawV1Alpha1()` method typed to avoid type assertions everywhere.

No functional changes.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-23 22:08:58 +04:00
Andrey Smirnov
0bb7e8a5cf
refactor: split config.Provider into Config & Container
See #7230

This is a step towards preparing for multi-doc config.

Split the `config.Provider` interface into parts which have different
implementation:

* `config.Config` accesses the config itself, it might be implemented by
  `v1alpha1.Config` for example
* `config.Container` will be a set of config documents, which implement
  validation, encoding, etc.

`Version()` method dropped, as it makes little sense and it was almost
not used.

`Raw()` method renamed to `RawV1Alpha1()` to support legacy direct
access to `v1alpha1.Config`, next PR will refactor more to make it
return proper type.

There will be many more changes coming up.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-23 16:05:16 +04:00
Andrey Smirnov
ff11fd39c7
fix: race with udevd and mountUserDisks
Fixes #7246

The problem was that `udevd` watches via `inotify` any attempts to open
blockdevices with 'write' access.

Talos was opening with write access, but actually accessing as
read-only, so the fix is to open as read-only.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-19 22:02:48 +04:00
Andrey Smirnov
10155c390e
feat: enable xfs project quota support, kubelet feature
This is controlled with a feature flag which gets enabled automatically
for Talos 1.5+.

Fixes #7181

If enabled, configures kubelet to use project quotas to track xfs volume
usage, which is much more efficient than doing `du` periodically.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-19 20:33:39 +04:00
Andrey Smirnov
dd8336c9ee
fix: refresh kubelet self-issued serving certificates
Kubelet doesn't refresh self-issued serving certificates, so force it by
removing the cert on each restart.

Fix the code which was forcing rejoin when the nodename changes, it was
broken, as it was checking serving certificate instead of client
certificate. It worked by accident when not using controlplane-issued
serving certificates.

Fixes #7235

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-18 22:19:34 +04:00
Andrey Smirnov
bb02dd263c
chore: drop deprecated stuff for Talos 1.5
* drop old resources API, which was deprecated long time ago
* use bootstrapped event in `talosctl get --watch` to better align
  columns in the table output

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-18 19:46:37 +04:00
Dmitriy Matrenichev
61cad86731
chore: bump deps
- github.com/containerd/typeurl to v2.1.1
- github.com/aws/aws-sdk-go to v1.44.264
- alpine to 3.18.0
- node to 20.2.0-alpine
- github.com/containernetworking/plugins to v1.3.0
- github.com/docker/docker to v23.0.6+incompatible
- github.com/hetznercloud/hcloud-go to v1.45.1
- github.com/insomniacslk/dhcp to v0.0.0-20230516061539-49801966e6cb
- github.com/rivo/tview to v0.0.0-20230511053024-822bd067b165
- tools to v1.5.0-alpha.0-7-gd2dde48
- pkgs to v1.5.0-alpha.0-16-g7958db1

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-05-18 01:07:36 -04:00
Dmitriy Matrenichev
97fffaf78a
chore: use ctest.UpdateWithConflicts instead of plain UpdateWithConflicts
More type-safety.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-05-12 20:39:32 -04:00
Dmitriy Matrenichev
45e6e27af7
chore: bump runtime
Use new functions and methods from runtime module.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-05-11 17:18:08 -04:00
Noel Georgi
4f720d4653
fix: revert: set rlimit explicitly in wrapperd
This reverts commit a2565f67416e9b9bc22f2d5506df9ea7771c0c8c.

The fix done in `a2565f67`, was actually a no-op caused by the
misunderstanding the fix done in Go and backported to [Go 1.20.4](ecf7e00db8).
The fix gave a false confidence that it was working when it was tested
against Talos `main` branch since the PR #7190 bumped `x/sys` package
from [v0.7.0 -> v0.8.0](ecf7e00db8), the actual change in `x/sys` can be found here at ff18efa0a3 which meant that when updating Go to 1.20.4 the `x/sys` package should been updated too. The `x/sys` package changed how the syscall to set the rlimit was called, it got moved into the Go stdlib instead of calling rlimit syscall in the `x/sys` package, which meant a combination of using Go 1.20.4 and an older `x/sys` package means `RLIMIT_NOFILE` value would not be set back to the original value.

The Talos 1.4 release branch currently have  `x/sys`
at [v0.7.0(https://github.com/siderolabs/talos/blob/v1.4.3/go.mod#L133),
so the backport would consist of this change along another commit bumping `x/sys` package to `v0.8.0`.

Fixes: #7198
Fixes: #7206

Co-authored-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-05-11 23:38:20 +05:30
Noel Georgi
a2565f6741
fix: set rlimit explicitly in wrapperd
Now Go only sets the rlimit for the parent and any fork/exec'ed process
gets the rlimit that was the default before fork/exec. Ref: https://github.com/golang/go/issues/46279

This fix got backported to [Go 1.20.4](ecf7e00db8) breaking Talos.
Talos used to set rlimit in the [`SetRLimit`](https://github.com/siderolabs/talos/blob/v1.4.2/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L302) sequencer task.
This means any process started by `wrapperd` gets the default Rlimit
(1024). Fix this by explicitly setting `rlimit` in `wrapperd` before we
drop any capabilities.

Fixes: #7198

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-05-10 00:10:17 +05:30
Andrey Smirnov
55ae59a0ad
fix: properly skip/cleanup controlplane configs for workers
This bug is pretty cosmetic, but it shows up as a wrong check when
performing worker upgrade - Talos pretends it checks e.g. kube-apiserver
version which doesn't make sense for workers.

There were two bugs in the code:

* check for machine type was done against `TypeWorker`, while
  `MachineType` resource is initially created as `TypeUnknown`
* the cleanup code was not implemented

As I touched the code, I updated controller and tests to use modern
conventions.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-05 23:17:27 +04:00
Utku Ozdemir
62c6e9655c
feat: introduce siderolink config resource & reconnect
Introduce a new resource, `SiderolinkConfig`, to store SideroLink connection configuration (api endpoint for now).

Introduce a controller for this resource which populates it from the Kernel cmdline.

Rework the SideroLink `ManagerController` to take this new resource as input and reconfigure the link on changes.

Additionally, if the siderolink connection is lost, reconnect to it and reconfigure the links/addresses.

Closes siderolabs/talos#7142, siderolabs/talos#7143.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
2023-05-05 17:04:34 +02:00
Andrey Smirnov
860002c735
fix: don't reload control plane pods on cert SANs changes
Fixes #7159

The change looks big, but it's actually pretty simple inside: the static
pods had an annotation which tracks a version of the secrets which
forced control plane pods to reload on a change. At the same time
`kube-apiserver` can reload certificate inputs automatically from files
without restart.

So the inputs were split: the dynamic (for kube-apiserver) inputs don't
need to be reloaded, so its version is not tracked in static pod
annotation, so they don't cause a reload. The previous non-dynamic
resource still causes a reload, but it doesn't get updated when e.g.
node addresses change.

There might be many more refactoring done, the resource chain is a bit
of a mess there, but I wanted to keep number of changes minimal to keep
this backportable.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-05 16:59:09 +04:00
Andrey Smirnov
d43c61e80f
fix: enforce nolock option for all NFS mounts by default
Talos doesn't have `rpc.statsd` running, so mounting without locking is
the only option. Some places in Kubernetes don't allow to set mount
options for NFS, so setting defaults is the only way.

Fixes #6582

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-04 17:26:36 +04:00
Niklas Wik
339986db9d
fix: inhibit timer to follow kubelet timer
Ensure to wait as long as possibly given to kubelet shutdown timers.
Related to fix of siderolabs#7138

Signed-off-by: Niklas Wik <niklas.wik@nokia.com>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-04 15:08:56 +04:00
Andrey Smirnov
cbf6dc1009
fix: set timeout for unmount calls
Fixes #7137

The `umount` syscall might hang "forever" if the underlying network
filesystem endpoint is down.

To be on the safe side, add a timeout around unmount operations, and try
to umount with force as a last resort.

Sample log:

```
14795.458779] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/dbe8d7f58e21d06cbef1ae0849317661eba4e82776722e7db5c65194ad73e916/globalmount/0001-0009-rook-ceph-0000000000000001-1051beb3-8d7a-4291-bf45-5711c13523d1
[14795.459797] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount
[14795.460555] EXT4-fs (rbd0): unmounting filesystem.
[14813.461319] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount is taking longer than expected, still waiting for 1m11.999162834s
[14831.460813] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount is taking longer than expected, still waiting for 53.999567033s
[14849.461336] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount is taking longer than expected, still waiting for 35.998979117s
[14867.460748] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount is taking longer than expected, still waiting for 17.999502128s
[14885.461123] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount with force
[14885.462395] [talos] ignoring unmount error /var/lib/kubelet/pods/f3f4d789-7f48-4dd9-9ef5-649b002c8f9c/volumes/kubernetes.io~csi/pvc-a4e72749-a8a1-43d9-9152-5bc1f757c924/mount: invalid argument
[14885.463529] [talos] task unmountPodMounts (2/2): unmounting /var/run/netns/cni-0888dc71-ba9e-af8a-d322-074f654561e5
[14885.464267] [talos] task unmountPodMounts (2/2): done, 1m30.028862262s
```

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-03 23:32:23 +04:00