Commit Graph

189 Commits

Author SHA1 Message Date
Artem Chernyshev
02b3719df9 feat: skip filesystem for state and ephemeral partitions in the installer
Filesystem creation step is moved on the later stage: when Talos mounts
the partition for the first time.
Now it checks if the partition doesn't have any filesystem and formats
it right before mounting.

Additionally refactored mount options a bit:
- replaced separate options with a set of binary flags.
- implemented pre-mount and post-unmount hooks.

And fixed typos in couple of places and increased timeout for `apid ready`.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-17 09:37:21 -08:00
Artem Chernyshev
f96548e165 refactor: extract go-cmd into a separate library
To be used in the `go-blockdevice` library.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-16 10:31:20 -08:00
Andrey Smirnov
7f3dca8e4c test: add support for IPv6 in talosctl cluster create
Modify provision library to support multiple IPs, CIDRs, gateways, which
can be IPv4/IPv6. Based on IP types, enable services in the cluster to
run DHCPv4/DHCPv6 in the test environment.

There's outstanding bug left with routes not being properly set up in
the cluster so, IPs are not properly routable, but DHCPv6 works and IPs
are allocated (validates DHCPv6 client).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-09 13:28:53 -08:00
Andrey Smirnov
2277ce8abe feat: move to ECDSA keys for all Kubernetes/etcd certs and keys
ECDSA keys are smaller which decreases Talos config size, they are more
efficient in terms of key generation, signing, etc., so it makes boot
performance better (and config generation as well).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-02 13:25:00 -08:00
Andrey Smirnov
8974b529af chore: bump dependencies (via dependabot)
See #3072, #3073, #3074, #3076, #3077, #3078

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-01 05:48:34 -08:00
Andrey Smirnov
18db20dbc2 fix: open blockdevices with exclusive flock for partitioning
This fixes spurious race conditions when user disks are partitioned
and formatted in `mountUserDisks` task. While this task runs, `udevd` is
running to allow various `/dev/` symlinks to be used for user disks.
At the same time `udevd` might trigger syscall `BLKRRPART` at any time
concurrently with Talos which leads to a race on kernel side when Talos
tries to update kernel partition table while kernel does it on its own
as a result of `udevd` call.

As part of the fix, `RereadPartitionTable()` calls were removed (they
trigger `BLKRRPART` and they're not needed as Talos updates partition
table on its own).

Some cleanups to make sure blockdevice is open/closed just in matching
pairs (no lingering open blockdevice instances). This is import for
`WithExclusiveLock()` calls, as it would lead to a deadlock if previous
blockdevice instance is not closed.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-28 09:11:39 -08:00
Artem Chernyshev
a83af03730 refactor: update go-blockdevice and restructure disk interaction code
This refactoring is required to simplify the work to be done to support
disk encryption.

Tried to minimize amount of queries done by `blockdevice` `probe`
methods.
Instead, where we have `runtime.Runtime` we get all required blockdevices
there from blockdevice cache stored in `State().Machine().Disk()`.
This opens a way to store encryption settings in the `Partition`
objects.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-01-28 17:42:09 +03:00
Andrey Smirnov
0aaf8fa968 feat: replace bootkube with Talos-managed control plane
Control plane components are running as static pods managed by the
kubelets.

Whole subsystem is managed via resources/controllers from os-runtime.

Many supporting changes/refactoring to enable new code paths.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-26 14:22:35 -08:00
Andrey Smirnov
11863dd74d feat: implement resource API in Talos
This brings in `os-runtime` package and exposes resources with first
iteration of read-only API.

Two Talos resources (and one controller) are implemented:

* legacy.Service resource tracks Talos 'service' `RUNNING` state
* config.V1Alpha1 stores current runtime config

Glue point between existing runtime and new os-runtime based runtime is
in `v1alpha2` implementation and `V1Alpha2()` sub-interfaces of existing
`Runtime`, `State`, `Controller` interfaces.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-19 11:45:46 -08:00
Andrey Smirnov
d71ac4c4ff feat: update Kubernetes to 1.20.2
Minor point release, official changelog:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-15 09:06:18 -08:00
Artem Chernyshev
9883d0af19 feat: support Wireguard networking
This the first iteration of Wireguard network support.
What was done:
- kernel was updated to enable Wireguard kernel module.
- changed networkd to support creating Wireguard device type.
- used wgctrl to configure wireguard.
- updated `talosctl cluster create` to support generating Wireguard
network configuration automatically by just specifying the network cidr.
- added docs about Wireguard support/how to use it.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-01-14 15:51:14 +03:00
Alexey Palazhchenko
275ca76c5b chore: update protobuf, grpc-go, prototool
To stay current.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-01-11 08:52:58 -08:00
Andrey Smirnov
d19486afaa fix: allow 'console' argument in kernel args to be always overridden
Fixes #3011

See also https://github.com/talos-systems/go-procfs/pull/8

We don't want to allow all the kernel args to be overridden, as this
might compromise KSPP, but we would rather allow some args to be
overridden explicitly.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-08 08:08:34 -08:00
Andrey Smirnov
a8dd2ff30d fix: checkpoint controller-manager and scheduler
Default manifests created by bootkube so far were only enabling
pod-checkpointer for kube-apiserver. This seems to have issues with
single-node control plane scenario, when without scheduler and
controller-manager node might fall into `NodeAffinity` state.

See https://github.com/talos-systems/bootkube-plugin/pull/23

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-28 11:53:17 -08:00
Artem Chernyshev
7b6c4bcb1f refactor: define default kernel flags in machinery instead of procfs
That change should make Talos updates more straightforward in any
projects that depend on Talos.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-24 06:50:53 -08:00
Artem Chernyshev
47fb7d26e0 fix: use SetAll instead of AppendAll when building kernel args
SBC should always overwrite default kernel params.
Otherwise we will always get duplicate values for some of them.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-23 09:09:13 -08:00
Andrey Smirnov
b1d4814308 feat: update Kubernetes to 1.20.1
See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-21 23:52:29 +03:00
Andrey Smirnov
f90aa613ac fix: don't overwrite PMBR
See https://github.com/talos-systems/go-blockdevice/pull/24

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-18 10:21:37 -08:00
Andrey Smirnov
ed42d4a42a fix: bump blockdevice library for 2nd partitione entries copy fix
See https://github.com/talos-systems/go-blockdevice/pull/23

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-18 09:15:21 -08:00
Andrey Smirnov
80184393bc feat: update kernel to 5.9.13, new KSPP requirements
Pulls in following changes:

* https://github.com/talos-systems/toolchain/pull/20
* https://github.com/talos-systems/tools/pull/116
* https://github.com/talos-systems/pkgs/pull/214
* https://github.com/talos-systems/pkgs/pull/215
* https://github.com/talos-systems/pkgs/pull/216
* https://github.com/talos-systems/pkgs/pull/217
* https://github.com/talos-systems/go-procfs/pull/4

New empty amd64 images for u-boot & rpi-firmware reduce the size of
amd64 installer image.

For backwards compatibility QEMU provisioner still injects "legacy" KSPP
kernel args into initial boot environment.

Installer correctly upgrades KSPP options when moving from one version
of Talos to another.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-10 12:41:58 -08:00
Andrey Smirnov
872e792dbc feat: update Kubernetes to 1.20.0
Official K8s release matching Talos 0.8.0.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-09 06:11:48 -08:00
Andrey Smirnov
28d2270067 fix: make reset work again
This bumps go-blockdevice library for the fix
https://github.com/talos-systems/go-blockdevice/pull/22

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-04 19:27:41 +03:00
Artem Chernyshev
c7062e3f4d feat: make GenerateConfiguration accept current time as a parameter
If the node time is out of sync, it can generate incorrect
configuration. And maintenance mode does not allow us starting ntp,
because there is no containerd.

By providing current UTC time of the machine where talosctl client is
running, it is possible to force GenerateConfiguration use correct time.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-03 08:28:11 -08:00
Spencer Smith
ed31056d91 feat: introduce configpatcher package in machinery
This PR moves the configpatcher as a package under machinery. It also
reworks the existing function to specify that it's explicitly for JSON
6902 patching so we can add more patch types if desired later on.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-12-02 20:49:38 -05:00
Andrey Smirnov
621968977e feat: update kubernetes to 1.20.0-rc.0
Talos 0.8 is going to ship with K8s 1.20.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-02 10:50:58 -08:00
Andrey Smirnov
3951898c76 chore: update module dependencies
Generic bump of dependencies.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-01 10:04:02 -08:00
Andrey Smirnov
08c84fe678 feat: stop including K8s version by default in talosctl gen config
Default image versions are kept as commented out examples.

This allows better experience for generating config on amd64 for arm64
servers. (e.g. for RPi).

Without embedded values in the config, Talos is going to use the
defaults which work better.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-30 12:52:53 -08:00
Andrey Smirnov
07f4ed7fb4 feat: upgrade etcd to 3.4.14
No major fixes, just keeping version up to date.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-26 09:14:41 -08:00
Andrey Smirnov
1eac88e470 feat: add support for installing to SBCs
This introduces the notion of a "board" in Talos. A board is an interface that is capable
of modifying the installation in specific ways for a given SBC. This also adds support for the
libretech_all_h3_cc_h5.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-26 07:18:25 -08:00
Andrey Smirnov
28ba6e416e feat: update Kubernetes to v1.20.0-beta.2
Talos 0.8 is going to ship with K8s 1.20.x.

Changes to support new `control-plane` label,
upgrade-k8s supports automated fixups for 1.20.

See also: https://github.com/talos-systems/bootkube-plugin/pull/22

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-25 06:39:14 -08:00
Andrey Smirnov
b5173477c8 fix: bump blockdevice library for mmcblk part name fix
See https://github.com/talos-systems/go-blockdevice/pull/17

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-20 05:36:14 -08:00
Artem Chernyshev
b6874ee82a feat: add TUI based talos interactive installer
This is initial commit of the installer.
What's done:
- verifying node availability before starting any operations.
- gathering information about disks on the machine.
- allows setting: install disk, hostname, machine type, installer image,
  kubernetes version, dns domain, cluster-name.
- dumps/merges talosconfig to a file after applying configuration.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-11-18 12:34:15 -08:00
Andrey Smirnov
83bb1afcb6 feat: drop to maintenance mode in cloud platforms if userdata is missing
On first boot of Talos, if userdata is missing, Talos is going to drop
into maintenance mode which allows to upload config to the server via
`talosctl apply-config` command.

See also: https://github.com/talos-systems/go-retry/pull/4

Fixes #2780

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-16 11:03:26 -08:00
Andrey Smirnov
df6ad3fa80 feat: upgrade Kubernetes default version to 1.19.4
k8s.io modules don't have 1.19.4 tag yet :(

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-12 08:51:04 -08:00
Andrey Smirnov
58df555580 feat: add example command in maintenance, enforce cert fingerprint
Server in maintenance mode now prints certficate fingerprint and
provides sample talosctl command to upload config to the node.

`talosctl` can optionally enforce server certificate fingerprint.

See also https://github.com/talos-systems/crypto/pull/4

Fixes #2753

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-12 07:36:18 -08:00
Andrew Rynhard
71321214a1 feat: add storage API
This is the initial implementation of a storage API.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-11 10:12:25 -08:00
Andrey Smirnov
4c42e22dbf chore: bump version of x/net module
Fixes #2746

This should cover the `x/net/http2` fixes which were backported in Go
1.15.4.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-10 07:09:19 -08:00
Andrew Rynhard
8e40ea46e0 fix: address issues in webconfig
Fixes TLS and invalide redirects.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-01 12:03:55 -08:00
Andrey Smirnov
350d75eb46 feat: build talosctl-cni-bundle, use it in talosctl for QEMU
This builds a bundle with CNI plugins for talosctl which is
automatically downloaded by `talosctl` if CNI plugins are missing.

CNI directories are moved by default to the `~/.talos/cni` path.

Also add a bunch of pre-flight checks to the QEMU provisioner to make it
easier to bootstrap the Talos QEMU cluster.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-30 16:30:37 -07:00
Andrey Smirnov
d9b74f0cc6 feat: skip resizing ephemeral partition if not required
This skips writing partition table if partition doesn't have to be
resized (already resized or max size from the beginning).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-30 14:31:10 -07:00
Andrey Smirnov
a32c0a78f7 docs: improve the config reference documentation
Lots of small changes, changing layout, adding back references,
propagating examples, etc.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-29 18:41:46 -07:00
Andrey Smirnov
bc9e0c0dba fix: re-implement upgrade (install) with preserve
For 0.6 -> 0.7 upgrade, in any case config.yaml is preserved and moved
from `/boot` to `/system/state`.

For single node upgrade, `EPHEMERAL` partition is not touched and other
partitions are re-created as needed.

Bump provision tests to 0.6/0.7 upgrades as we get closer to the new
release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-28 07:25:26 -07:00
Andrey Smirnov
41f92b5d35 feat: wipe disks faster in the installer
See https://github.com/talos-systems/talos/pull/2663 for details.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-22 08:32:29 -07:00
Spencer Smith
8b5406c889 chore: move to newer release of rtnetlink with fn args
This PR makes use of a new merge into the upstream rtnetlink library
that introduces functional args for adding routes.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-10-22 06:56:22 -07:00
Andrey Smirnov
16b6d344de chore: bump module dependencies in go.mod
This covers most of the packages except for those we have to keep on
hold (etcd and grpc because of etcd).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-20 08:09:42 -07:00
Andrey Smirnov
56f1ee37fd feat: upgrade Kubernetes to 1.19.3
Just minor release bump.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-20 05:12:32 -07:00
Andrey Smirnov
4adb613f66 refactor: bring more control to install.Manifest execution
This unifies more code paths under the control of `install.Manifest` vs.
being split across the installer and manifest code.

There should be no functional changes now.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-20 01:08:14 -07:00
Andrey Smirnov
018086d1fa refactor: extract blockdevice library
Library `blockdevice` was extracted as `talos-systems/go-blockdevice`,
this PR finalizes the move by removing Talos copy of it.

Some functions around `mkfs`/`growfs` were extracted as `makefs`
package, as they depend on `cmd` package.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-05 11:18:43 -07:00
Andrey Smirnov
16eb47a1a3 feat: use kubeconfig merge in talosctl kubeconfig by default
Kubeconfig merge was completely rewritten to be "smarter":

* automatically apply renames done at previous stages to avoid asking
over and over again (in general should ask just once)

* skip checks if parts of the config match exactly

* allow overwrite as an option

* flexible way to control the output

* activating context in the end

* custom merged context name

Fixes #2578

Fixes #2587

Fixes #2577

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-03 05:36:15 -07:00
Seán C McCord
ff92d2a14b feat: add ApplyConfiguration API
Adds the ability to apply (replace) an existing node configuration with
a new one via the Machine API.

Fixes #2345

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2020-09-29 14:44:06 -07:00