222 Commits

Author SHA1 Message Date
Andrey Smirnov
05cbe250c8 chore: bump dependencies via dependabot
PRs #3503 #3504 #3505

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-19 06:44:47 -07:00
Jorik Jonker
8b8542e3b5 feat: add support for reading OVF data on VMWare
The OVF environment is a way to supply guestinfo to guests. It is
a datastructure (XML) put in `extraConfig` (commonly referred to as
`guestinfo`) under the key `ovfenv`.

This OVF env is said to be the proper way to supply customization data
to guests (ie, not through `extraConfig`), and on some platforms (eg,
vCD), it is even the only option.

This change also enables the actual OVF transport in the OVA.

Signed-off-by: Jorik Jonker <jorik.jonker@eu.equinix.com>
2021-04-13 16:16:44 +03:00
Andrey Smirnov
d24df8f844 chore: re-import talos-systems/os-runtime as cosi-project/runtime
No changes, just import path change (as project got moved).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 07:44:24 -07:00
Andrey Smirnov
ef24fd6a01 chore: bump dependencies via dependabot
See #3464 #3465 #3466

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 06:32:17 -07:00
Alexey Palazhchenko
37a5edf04a feat: update Kubernetes to 1.21.0 release
See CHANGELOG:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md

Closes #3329.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-09 20:08:20 +03:00
Andrey Smirnov
e0650218a6 feat: support etcd recovery from snapshot on bootstrap
When Talos `controlplane` node is waiting for a bootstrap, `etcd`
contents can be recovered from a snapshot created with
`talosctl etcd snapshot` on a healthy cluster.

Bootstrap process goes same way as before, but the etcd data directory
is recovered from the snapshot.

This flow enables disaster recovery for the control plane: given that
periodic backups are available, destroy control plane nodes, re-create
them with the same config, and bootstrap one node with the saved
snapshot to recover etcd state at the time of the snapshot.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-08 10:15:37 -07:00
Andrey Smirnov
33035901ff fix: revert mark PMBR EFI partition as bootable
See talos-systems/go-blockdevice#34 talos-systems/talos#3440

That change broke UEFI boot.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-07 07:24:58 -07:00
Andrey Smirnov
fbfd1eb2b1 refactor: pull new version of os-runtime, update code
This is mostly refactoring to adapt to the new APIs.

There are some small changes which are not user-visible immediately (but
visible when using `talosctl get` to inspect low-level details):

* `extras` namespace is removed, it was a hack to distinguish extra and
system manifests
* `Manifests` are managed by two controllers as shared outputs, stored
in the `controlplane` namespace now
* `talosctl inspect dependencies` output got slightly changed
* resources now have `md.owner` set to the controller name which manages
the resource

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-07 06:55:09 -07:00
Andrey Smirnov
690eb20e97 chore: update blockdevice library for PMBR bootable fix
See https://github.com/talos-systems/go-blockdevice/pull/33

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-06 06:14:56 -07:00
Andrey Smirnov
39ae0415e9 chore: bump dependencies via dependabot
See #3431 #3432 #3433 #3434

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-05 06:16:24 -07:00
Alexey Palazhchenko
a9451f5712 feat: update Kubernetes to 1.21.0-beta.1
See CHANGELOG:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md

Refs #3329.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-30 03:07:03 -07:00
Andrey Smirnov
a6106815b7 chore: bump dependencies via dependabot
See #3386 #3387 #3388

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-29 06:38:55 -07:00
Andrey Smirnov
966caf7a67 chore: remove unused module replace directives
They were required long time ago, doesn't look like we need it now.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-25 08:16:25 -07:00
Alexey Palazhchenko
ed272e604e feat: update Kubernetes to 1.21.0-beta.0
See CHANGELOG:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md

Refs #3329.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-24 07:36:54 -07:00
Artem Chernyshev
6ffabe5169 feat: add ability to find disk by disk properties
Fixes: https://github.com/talos-systems/talos/issues/3323

Not exactly matching with udevd generated `by-<id>` symlinks, but should
provide sufficient amount of property selectors to be able to pick
specific disks for any kind of disk: sd card, hdd, ssd, nvme.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-03-23 14:23:02 -07:00
Andrey Smirnov
dc294db16c chore: bump dependencies via dependabot
PRs #3336 #3337 #3338 #3339

Also bump proto tools via talos-systems/tools#133

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-22 13:58:08 -07:00
Andrey Smirnov
f0512dfce9 feat: update Kubernetes to 1.20.5
See CHANGELOG:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#changelog-since-v1204

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-19 03:14:46 -07:00
Alexey Palazhchenko
08271ba931 chore: use Go 1.16 language version
It affects some languages features and go subcommands.
https://golang.org/ref/mod#go-mod-file-go

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-17 06:20:39 -07:00
Artem Chernyshev
e31790f6f5 fix: properly format spec comments in the resources
`os-runtime` now writes `yaml` block as raw yaml bytes instead of
decoding it into `yaml.Node` and encoding that `yaml.Node` back to YAML.
The reason is that `go-yaml` comments decoder can't really handle
comment placement properly, so it messes up indents here and there.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-03-15 14:21:57 -07:00
Andrey Smirnov
d4d77882e3 chore: update dependencies via dependabot
See #3301 #3302

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-15 06:12:03 -07:00
Andrey Smirnov
f4ca6e9a6e feat: update containerd to version 1.4.4
See https://github.com/containerd/containerd/releases/tag/v1.4.4

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-05 11:00:21 -08:00
Andrey Smirnov
db3785b930 fix: align partition start to the physical sector size
See https://github.com/talos-systems/go-blockdevice/pull/31

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-05 06:54:12 -08:00
Andrey Smirnov
49a23bbde8 chore: bump Go module dependencies
This bumps all the dependencies that can be bumped with minor fixups in
the code.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-03 18:45:12 +03:00
Andrey Smirnov
40a2e4d4fa feat: support JSON output in talosctl get, event types
This adds support for `-o json` (easier to use `jq` to query additional
data), and prints event name in `--watch` mode.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-03 06:18:14 -08:00
Andrey Smirnov
60aa011c7a feat: rename namespaces, resources, types etc
See https://github.com/talos-systems/os-runtime/pull/12 for new mnaming
conventions.

No functional changes.

Additionally implements printing extra columns in `talosctl get xyz`.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-02 13:34:15 -08:00
Andrey Smirnov
d7cdc8cc15 feat: implement simple layer 2 shared IP for CP
This adds a VIP (virtual IP) option to the network configuration of an
interface, which will allow a set of nodes to share a floating IP
address among them.  For now, this is restricted to control plane use
and only a single shared IP is supported.

Fixes #3111

Signed-off-by: Seán C McCord <ulexus@gmail.com>
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-26 14:14:34 -08:00
Artem Chernyshev
041620c852 feat: implement talosctl edit and patch config commands
Fixes: https://github.com/talos-systems/talos/issues/3209

Using parts of `kubectl` package to run the editor.
Also using the same approach as in `kubectl edit` command:
- add commented section to the top of the file with the description.
- if the config has errors, display validation errors in the commented
section at the top of the file.
- retry apply config until it succeeds.
- abort if no changes were detected or if the edited file is empty.

Patch currently supports jsonpatch only and can read it either from the
file or from the inline argument.

https://asciinema.org/a/wPawpctjoCFbJZKo2z2ATDXeC

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-26 02:00:20 +03:00
Andrey Smirnov
953ce643ab feat: bump etcd client library to 3.5.0-alpha.0
This version is finally using working `go.mod` files and tags, so no
more hacks with imports, and allows us to bump `grpc` library to the
latest version (I also did for this PR).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-25 10:36:15 -08:00
Andrey Smirnov
779ac74a08 fix: improve the drain function
Critical bug (I believe) was that drain code entered the loop to evict
the pod after wait for pod to be deleted returned success effectively
evicting pod once again once it got rescheduled to a different node.

Add a global timeout to prevent draining code from running forever.

Filter more pod types which shouldn't be ever drained.

Fixes #3124

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-25 07:02:24 -08:00
Andrey Smirnov
4e19b597ab test: add integration test with Canal CNI and reset API
Canal CNI is known to be trying to reach out to k8s control plane on pod
teardown.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-24 11:34:02 -08:00
Andrey Smirnov
e9fc54f6e3 feat: update Kubernetes to 1.20.3
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#changelog-since-v1202

Also updater pkgs for:

* talos-systems/pkgs#238 (raspberrypi-firmware update)
* talos-systems/pkgs#242 (Linux 5.10.17 + init_on_free=0)

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-19 05:22:34 -08:00
Artem Chernyshev
54d6a45217 feat: add state encryption support
State partition encryption support adds a new section to the machine config.
And a new step to the sequencer flow which saves encryption
configuration object as json serialized value in the META partition.

Everything else is the same as is for the ephemeral partition.
Additionally enabled state partition encryption in the disk encryption
integration tests.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-18 06:55:22 -08:00
Andrey Smirnov
e5bd35ae3c feat: add resource watch API + CLI
This uses API in `os-runtime` to pull the initial list of resources +
updates for resource by type.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-17 13:24:47 -08:00
Artem Chernyshev
02b3719df9 feat: skip filesystem for state and ephemeral partitions in the installer
Filesystem creation step is moved on the later stage: when Talos mounts
the partition for the first time.
Now it checks if the partition doesn't have any filesystem and formats
it right before mounting.

Additionally refactored mount options a bit:
- replaced separate options with a set of binary flags.
- implemented pre-mount and post-unmount hooks.

And fixed typos in couple of places and increased timeout for `apid ready`.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-17 09:37:21 -08:00
Artem Chernyshev
f96548e165 refactor: extract go-cmd into a separate library
To be used in the `go-blockdevice` library.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-16 10:31:20 -08:00
Andrey Smirnov
7f3dca8e4c test: add support for IPv6 in talosctl cluster create
Modify provision library to support multiple IPs, CIDRs, gateways, which
can be IPv4/IPv6. Based on IP types, enable services in the cluster to
run DHCPv4/DHCPv6 in the test environment.

There's outstanding bug left with routes not being properly set up in
the cluster so, IPs are not properly routable, but DHCPv6 works and IPs
are allocated (validates DHCPv6 client).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-09 13:28:53 -08:00
Andrey Smirnov
2277ce8abe feat: move to ECDSA keys for all Kubernetes/etcd certs and keys
ECDSA keys are smaller which decreases Talos config size, they are more
efficient in terms of key generation, signing, etc., so it makes boot
performance better (and config generation as well).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-02 13:25:00 -08:00
Andrey Smirnov
8974b529af chore: bump dependencies (via dependabot)
See #3072, #3073, #3074, #3076, #3077, #3078

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-01 05:48:34 -08:00
Andrey Smirnov
18db20dbc2 fix: open blockdevices with exclusive flock for partitioning
This fixes spurious race conditions when user disks are partitioned
and formatted in `mountUserDisks` task. While this task runs, `udevd` is
running to allow various `/dev/` symlinks to be used for user disks.
At the same time `udevd` might trigger syscall `BLKRRPART` at any time
concurrently with Talos which leads to a race on kernel side when Talos
tries to update kernel partition table while kernel does it on its own
as a result of `udevd` call.

As part of the fix, `RereadPartitionTable()` calls were removed (they
trigger `BLKRRPART` and they're not needed as Talos updates partition
table on its own).

Some cleanups to make sure blockdevice is open/closed just in matching
pairs (no lingering open blockdevice instances). This is import for
`WithExclusiveLock()` calls, as it would lead to a deadlock if previous
blockdevice instance is not closed.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-28 09:11:39 -08:00
Artem Chernyshev
a83af03730 refactor: update go-blockdevice and restructure disk interaction code
This refactoring is required to simplify the work to be done to support
disk encryption.

Tried to minimize amount of queries done by `blockdevice` `probe`
methods.
Instead, where we have `runtime.Runtime` we get all required blockdevices
there from blockdevice cache stored in `State().Machine().Disk()`.
This opens a way to store encryption settings in the `Partition`
objects.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-01-28 17:42:09 +03:00
Andrey Smirnov
0aaf8fa968 feat: replace bootkube with Talos-managed control plane
Control plane components are running as static pods managed by the
kubelets.

Whole subsystem is managed via resources/controllers from os-runtime.

Many supporting changes/refactoring to enable new code paths.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-26 14:22:35 -08:00
Andrey Smirnov
11863dd74d feat: implement resource API in Talos
This brings in `os-runtime` package and exposes resources with first
iteration of read-only API.

Two Talos resources (and one controller) are implemented:

* legacy.Service resource tracks Talos 'service' `RUNNING` state
* config.V1Alpha1 stores current runtime config

Glue point between existing runtime and new os-runtime based runtime is
in `v1alpha2` implementation and `V1Alpha2()` sub-interfaces of existing
`Runtime`, `State`, `Controller` interfaces.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-19 11:45:46 -08:00
Andrey Smirnov
d71ac4c4ff feat: update Kubernetes to 1.20.2
Minor point release, official changelog:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-15 09:06:18 -08:00
Artem Chernyshev
9883d0af19 feat: support Wireguard networking
This the first iteration of Wireguard network support.
What was done:
- kernel was updated to enable Wireguard kernel module.
- changed networkd to support creating Wireguard device type.
- used wgctrl to configure wireguard.
- updated `talosctl cluster create` to support generating Wireguard
network configuration automatically by just specifying the network cidr.
- added docs about Wireguard support/how to use it.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-01-14 15:51:14 +03:00
Alexey Palazhchenko
275ca76c5b chore: update protobuf, grpc-go, prototool
To stay current.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-01-11 08:52:58 -08:00
Andrey Smirnov
d19486afaa fix: allow 'console' argument in kernel args to be always overridden
Fixes #3011

See also https://github.com/talos-systems/go-procfs/pull/8

We don't want to allow all the kernel args to be overridden, as this
might compromise KSPP, but we would rather allow some args to be
overridden explicitly.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-08 08:08:34 -08:00
Andrey Smirnov
a8dd2ff30d fix: checkpoint controller-manager and scheduler
Default manifests created by bootkube so far were only enabling
pod-checkpointer for kube-apiserver. This seems to have issues with
single-node control plane scenario, when without scheduler and
controller-manager node might fall into `NodeAffinity` state.

See https://github.com/talos-systems/bootkube-plugin/pull/23

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-28 11:53:17 -08:00
Artem Chernyshev
7b6c4bcb1f refactor: define default kernel flags in machinery instead of procfs
That change should make Talos updates more straightforward in any
projects that depend on Talos.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-24 06:50:53 -08:00
Artem Chernyshev
47fb7d26e0 fix: use SetAll instead of AppendAll when building kernel args
SBC should always overwrite default kernel params.
Otherwise we will always get duplicate values for some of them.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-23 09:09:13 -08:00
Andrey Smirnov
b1d4814308 feat: update Kubernetes to 1.20.1
See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-21 23:52:29 +03:00