2364 Commits

Author SHA1 Message Date
Andrey Smirnov
a8f1e526bf chore: build talosctl for Darwin / Apple Silicon
This adds a new target `darwin/arm64` which is supported since Go 1.16.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-20 08:05:40 -07:00
Andrey Smirnov
eb0b64d313 chore: list specifically for enabled regions
AWS now has opt-in regions which are not enabled by default, so we need
to ignore such regions to avoid failures.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-20 05:47:30 -07:00
Jorik Jonker
669a0cbdc4 fix: check if OVF env is empty
It seems when you do not supply OVF env vars, the demarshalled XML turns
out with nil values. This commit adds checks for that.

Fixes #3513

Signed-off-by: Jorik Jonker <jorik.jonker@eu.equinix.com>
2021-04-20 05:45:38 -07:00
Alexey Palazhchenko
da92049c0b chore: use codecov from the build container
That's the best we can do now with https://about.codecov.io/security-update/

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-19 10:56:05 -07:00
Alexey Palazhchenko
9996d4b028 chore: use REGISTRY_MIRROR_FLAGS if defined
REGISTRY_MIRROR_FLAGS can be already defined in the environment with .env or .envrc file.
Use it for easier and faster local e2e tests.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-19 09:13:14 -07:00
Andrey Smirnov
05cbe250c8 chore: bump dependencies via dependabot
PRs #3503 #3504 #3505

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-19 06:44:47 -07:00
Artem Chernyshev
9a91142a38 feat: print complete member info in etcd members
Fixes: https://github.com/talos-systems/talos/issues/3487

Example output:

```
NODE       ID                 HOSTNAME                 PEERS                   CLIENTS
10.5.0.2   c3d3020cf75b8728   talos-default-master-1   https://10.5.0.2:2380   https://10.5.0.2:2379
```

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-04-17 11:07:59 -07:00
Spencer Smith
bb40d6dd06 feat: update pkgs version
This updates the pkgs version to pull in a fix for CPU scaling on
raspberry pi. It will be backported to release-0.10 upon merge.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2021-04-16 09:50:30 -07:00
Andrey Smirnov
e7a9164b1e test: implement talosctl conformance command to run e2e tests
Command implements two modes:

* `fast`: conformance suite is run at maximum speed
* `certified`: conformance suite is run in serial mode, results
  are capture to produce artifacts ready for CNCF submission process

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-16 09:17:51 -07:00
Andrey Smirnov
6cb266e74e fix: update etcd client errors, print etcd join failures
Better error message to understand where the error is coming from, also
print errors to console when etcd is trying to join - this is invaluable
to understand why etcd doesn't join the cluster.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-15 11:54:25 -07:00
Andrey Smirnov
0bd8b0e800 feat: provide an option to recover etcd from data directory copy
Sometimes `talosctl etcd snapshot` might not be available, for example
when etcd is not healthy. In that case it's possible to copy raw etcd
data directory with `talosctl cp /var/lib/etcd .` and use
`member/snap/db` to recover the cluster. But such copy won't pass
integrity checks, so they should be disabled explicitly.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-14 08:25:32 -07:00
Andrey Smirnov
f98185408d chore: fix conform with scopes
In recent versions of conform, scope is regexp so `*` is not a valid
regexp.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-13 15:00:36 -07:00
Andrey Smirnov
21018f28c7 chore: bump website node.js dependencies
Getting rid of security warning from Dependabot, just result of plain
`npm update`.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-13 08:56:21 -07:00
Andrey Smirnov
e69732ed59 feat: provide Talos machine config with field placeholders and docs
For config patching, it's beneficial to have some fields to be present
in the config, even with the default (empty) value. At the same time not
all fields should be present in all versions of the config, e.g. some
config value don't apply to worker node configuration.

Empty value and nil value are treated equal by `yaml` library, but Talos
encoder can be made more smart to still output empty (non-nil) value to
the config, while skipping completely nil fields.

This PR implements that via new `talos:"omitonlyifnil"` tag and plus
moves docs for such fields into comments under the value.

GC'ed pod checkpointer config, so it doesn't get generated even as
comments (it was empty by default even for 0.8, so this just removes
comments about it).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-13 06:51:35 -07:00
Andrey Smirnov
f0970ea7ff fix: zero out manifest contents before setting new value
This might lead to manifests being appended forever on changes.

Fixes #3476

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-13 06:29:32 -07:00
Jorik Jonker
8b8542e3b5 feat: add support for reading OVF data on VMWare
The OVF environment is a way to supply guestinfo to guests. It is
a datastructure (XML) put in `extraConfig` (commonly referred to as
`guestinfo`) under the key `ovfenv`.

This OVF env is said to be the proper way to supply customization data
to guests (ie, not through `extraConfig`), and on some platforms (eg,
vCD), it is even the only option.

This change also enables the actual OVF transport in the OVA.

Signed-off-by: Jorik Jonker <jorik.jonker@eu.equinix.com>
2021-04-13 16:16:44 +03:00
Andrey Smirnov
04dbafca72 feat: update Linux to 5.10.29
Also brings tools/pkgs/extras re-tagged from a stable branch for Talos
0.10 release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 14:09:45 -07:00
Andrey Smirnov
3dc7b8a8a2 chore: fix import path mismerge
Two PRs got merged, one which had old import paths and another one which
had import paths updated, and some paths didn't get updated properly.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 11:48:38 -07:00
Andrey Smirnov
2402f20c20 feat: implement inline manifests in the machine configuration
Inline manifests work exactly same way as extra manifests, but the
manifest itself can be stored in the config body.

Example config patch:

```
--config-patch '[{"op": "replace", "path": "/cluster/inlineManifests", "value": [{"name": "foo", "contents": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: ci\n"}]}]'
```

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 10:32:31 -07:00
Andrey Smirnov
e26c977d85 fix: check retryable network errors by interface
Looks like tls errors implement the interface, but they are not derived
from the `*net.OpError`, so this check should catch more errors.

Fixes #3457

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 09:56:17 -07:00
Alexey Palazhchenko
767f3b95ef chore: apply coverage analysis to all packages
This allows us to see test coverage of packages that can't be unit-tested directly.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-12 09:29:07 -07:00
Alexey Palazhchenko
7b0ac4682d test: extend unit-tests for config contract parsing
Test that `v0.8.` is not a typo.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-12 08:22:13 -07:00
Andrey Smirnov
d24df8f844 chore: re-import talos-systems/os-runtime as cosi-project/runtime
No changes, just import path change (as project got moved).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 07:44:24 -07:00
Alexey Palazhchenko
cae25909f3 chore: use Go 1.16 global install command
Plus add stringer tool.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-12 07:40:29 -07:00
Andrey Smirnov
d5e2a45db3 feat: validate the machine configuration in the installer container
Talos validates machine configuration at boot time, and refuses to boot
if machine configuration is invalid.

As machine configuration validation rules might change over time, we
need to prevent a scenario when after an upgrade machine configuration
becomes invalid, as there's no way to roll back properly.

Machine configuration is submitted over stdin to the installer
container, and installer container validates it using the new version of
Talos (which is going to be installed).

If the config is not sent over stdin, installer assumes old version of
Talos and proceeds.

This should be backported to 0.9 to allow config validation on upgrade
to 0.10.

Fixes #3419

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 06:47:28 -07:00
Andrey Smirnov
ef24fd6a01 chore: bump dependencies via dependabot
See #3464 #3465 #3466

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 06:32:17 -07:00
Brandon McNama
94520b03f6 docs: fixup typo in encryption guide command
Fixes a typo in a documented command for going from an unencrypted
EPHEMERAL partition to an encrypted one.

Signed-off-by: Brandon McNama <brandonmcnama@outlook.com>
2021-04-11 02:29:33 -07:00
Andrey Smirnov
e3585f24bc chore: update Linux to 5.10.28, u-boot to final 2021.04 release
See talos-systems/pkgs#261 talos-systems/pkgs#262

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-10 07:30:10 -07:00
Andrey Smirnov
daf2208749 test: update upgrade tests to 0.10 release
In preparation for going 0.10 beta, start testing upgrades to 0.10, drop
0.8 and self-hosted control plane handling in the tests.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-09 12:57:04 -07:00
Alexey Palazhchenko
1fcf38f9d6 feat: add support for "none" CNI type
Closes #3411.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-09 12:53:00 -07:00
Alexey Palazhchenko
37a5edf04a feat: update Kubernetes to 1.21.0 release
See CHANGELOG:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md

Closes #3329.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-09 20:08:20 +03:00
Alexey Palazhchenko
30f687b417 fix: document HDMI problem on RPi 4
Closes #3414.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-08 14:06:12 -07:00
Alexey Palazhchenko
29da22d063 feat: add config validation warnings
Closes #3412.
Refs #3413.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-08 13:49:58 -07:00
Andrey Smirnov
eee7ad13aa release(v0.10.0-alpha.2): prepare release
This is the official v0.10.0-alpha.2 release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-08 13:03:50 -07:00
Andrey Smirnov
e0650218a6 feat: support etcd recovery from snapshot on bootstrap
When Talos `controlplane` node is waiting for a bootstrap, `etcd`
contents can be recovered from a snapshot created with
`talosctl etcd snapshot` on a healthy cluster.

Bootstrap process goes same way as before, but the etcd data directory
is recovered from the snapshot.

This flow enables disaster recovery for the control plane: given that
periodic backups are available, destroy control plane nodes, re-create
them with the same config, and bootstrap one node with the saved
snapshot to recover etcd state at the time of the snapshot.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-08 10:15:37 -07:00
Artem Chernyshev
247bd50e05 docs: describe steps to install and boot Talos from the SSD on rockpi4
Describe that gross flow while I still remember it.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-04-07 13:06:58 -07:00
Spencer Smith
e6b4e524ff test: update CAPA to 0.6.4
This PR pulls in an updated cluster api aws version, ensuring the CRDs
are closer to what's expected when we patch the CAPA image later in the
setup. We will eventually move to 0.6.5 as soon as it's cut.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2021-04-07 14:37:20 -04:00
Andrey Smirnov
28753f6dcb fix: trim endpoints/nodes from arguments in talosctl config
When copy-pasting extra space might be added around an argument to the
`talosctl config endpoints/nodes`, which breaks the config as the
endpoint doesn't parse anymore as IP address.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-07 11:37:02 -07:00
Alexey Palazhchenko
aca63b8829 docs: fix "DigitalOcean" spelling
Refs #3427.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-07 09:13:24 -07:00
Andrey Smirnov
33035901ff fix: revert mark PMBR EFI partition as bootable
See talos-systems/go-blockdevice#34 talos-systems/talos#3440

That change broke UEFI boot.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-07 07:24:58 -07:00
Andrey Smirnov
fbfd1eb2b1 refactor: pull new version of os-runtime, update code
This is mostly refactoring to adapt to the new APIs.

There are some small changes which are not user-visible immediately (but
visible when using `talosctl get` to inspect low-level details):

* `extras` namespace is removed, it was a hack to distinguish extra and
system manifests
* `Manifests` are managed by two controllers as shared outputs, stored
in the `controlplane` namespace now
* `talosctl inspect dependencies` output got slightly changed
* resources now have `md.owner` set to the controller name which manages
the resource

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-07 06:55:09 -07:00
Alexey Palazhchenko
8737ea716a feat: allow external cloud provides configration
Closes #3312.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-06 22:54:24 -07:00
Andrey Smirnov
3909e2d011 chore: update Go to 1.16.3
See talos-systems/tools#134 talos-systems/pkgs#260
talos-systems/extras#16

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-06 13:53:53 -07:00
Andrey Smirnov
690eb20e97 chore: update blockdevice library for PMBR bootable fix
See https://github.com/talos-systems/go-blockdevice/pull/33

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-06 06:14:56 -07:00
Andrey Smirnov
a8761b8e1e fix: require leader on etcd member operations
This fix is not obvious on whether we need it actually or not, but what
I've seen in the tests seems to be around the fact that added member is
not visible in the member list fetched after the add command succeeds.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-06 05:36:45 -07:00
Alexey Palazhchenko
3dc84625cb fix: make both HDMI ports work on RPi 4
Closes #3414.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-05 15:25:39 -07:00
Andrey Smirnov
bd5ae1e0b5 fix: add a check for overlay mounts in installer pre-flight checks
Overlay mount in `mountinfo` don't show up as mounts for any particular
block device, so the existing check doesn't catch them.

This was discovered as our current master can't upgrade because of
overlay mount for `/opt` and `apid` image in `/opt/apid` (which will be
fixed in a separate PR).

Without the check, installer fails on resetting partition table for the
disk effectively wiping the node (`device or resource busy` error).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-05 14:29:46 -07:00
Andrey Smirnov
df8649cbe6 refactor: download modules before go generate
This moves things around a bit so that `go generate` is called after
modules are generated, as `go generate` downloads modules as well.
This fixes a race condition which might show up randomly.

Spotted by: @AlekSi

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-05 11:38:40 -07:00
Andrey Smirnov
39ae0415e9 chore: bump dependencies via dependabot
See #3431 #3432 #3433 #3434

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-05 06:16:24 -07:00
Artem Chernyshev
e16d6d3468 fix: publish rockpi4 image to release artifacts
Attempt #2. Forgot to add it to .drone.jsonnet also 🤦

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-04-03 18:20:54 -07:00