Commit Graph

98 Commits

Author SHA1 Message Date
Artem Chernyshev
54d6a45217 feat: add state encryption support
State partition encryption support adds a new section to the machine config.
And a new step to the sequencer flow which saves encryption
configuration object as json serialized value in the META partition.

Everything else is the same as is for the ephemeral partition.
Additionally enabled state partition encryption in the disk encryption
integration tests.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-18 06:55:22 -08:00
Andrey Smirnov
7751920dba feat: add a tool and package to convert self-hosted CP to static pods
This is required to upgrade from Talos 0.8.x to 0.9.x. After the cluster
is fully upgraded, control plane is still self-hosted (as it was
bootstrapped with bootkube).

Tool `talosctl convert-k8s` (and library behind it) performs the upgrade
to self-hosted version.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-17 23:26:57 -08:00
Spencer Smith
3a78bfcecd test: trigger e2e on thrice daily
This PR adds a trigger to aws and gcp e2e tasks to make sure they get
triggered during thrice-daily.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2021-02-17 15:53:34 -08:00
Artem Chernyshev
58ff2c9808 feat: implement ephemeral partition encryption
This PR introduces the first part of disk encryption support.
New config section `systemDiskEncryption` was added into MachineConfig.
For now it contains only Ephemeral partition encryption.

Encryption itself supports two kinds of keys for now:
- node id deterministic key.
- static key which is hardcoded in the config and mainly used for test
purposes.

Talosctl cluster create can now be told to encrypt ephemeral partition
by using `--encrypt-ephemeral` flag.

Additionally:
- updated pkgs library version.
- changed Dockefile to copy cryptsetup deps from pkgs.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-17 13:39:04 -08:00
Andrey Smirnov
254e0e91e1 fix: correctly unwrap responses for etcd commands
This uses wrappers which helps to unwrap errors from proxied apid
responses.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-17 11:33:54 -08:00
Andrey Smirnov
292bc39681 chore(ci): fix schedules in Drone pipelines
That removes actually e2e-* from the `thrice-daily`.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-17 10:09:56 -08:00
Andrey Smirnov
edbaa0bc72 chore: update artifacts bucket name in Drone
This make sure release builds and branch builds after merge save
artifacts to different buckets (`$DRONE_TAG` is only set for tagged
builds).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-17 08:01:09 -08:00
Andrey Smirnov
b315a7e1f8 chore: rework Drone pipelines
Collapse builds steps, split integration tests into multiple pipelines,
save and load build artifacts.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-17 17:16:02 +03:00
Artem Chernyshev
6540e9bf70 feat: support disk image in talosctl cluster create
Fixes: https://github.com/talos-systems/talos/issues/2973

Can now supply disk image using `--disk-image-path` flag.
May need to enable `--with-apply-config` if it's necessary to bootstrap
nodes properly.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-22 17:06:00 +03:00
Andrey Smirnov
9d1ac81be5 chore: lower MTU to 1450 for the tests in the CI
This should help with the CNI encapsulation in the cluster.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-17 17:14:07 +03:00
Andrey Smirnov
221f37d46e chore: build ISOs earlier to launch e2e-iso as soon as possible
This should cut overall build time.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-17 05:42:20 -08:00
Andrey Smirnov
803d566b85 chore: add drone pipeline to upload cloud images
At the moment only AMIs for AWS, and upload results should be pushed
back to the docs as separate PR.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-16 20:50:35 +03:00
Andrey Smirnov
d961163d9a test: remove provision tests with Cilium CNI
Self-hosted control plane upgrade with Cilium looks to be really flaky,
and it often fails the test run. We still have e2e-test with Cilium for
the test coverage.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-14 05:15:51 -08:00
Andrew Rynhard
6f979d463c test: add ISO test
Adds a simple test for the ISO. Boots the ISO, and then uses the `apply-config` command
in `talosctl` to create a cluster.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-12-10 13:04:29 -08:00
Andrew Rynhard
c30374f129 chore: publish Rock64 image
This publishes the Pine64 Rock64 image.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-12-04 12:53:33 -08:00
Spencer Smith
0cb7a6495d chore: enable thrice daily pipeline
This PR actually enables the integration pipeline for running thrice
daily.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-12-04 11:27:58 -05:00
Spencer Smith
799e0236dd chore: run integration test thrice daily
This PR makes sure that we run integration more often. I did this in
UTC, with Eastern time zones in mind. In Eastern, it'll do 6AM, noon,
and 6PM. That should give us pretty good coverage across all of our
timezones so we can all have a chance to tackle new issues.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-12-04 07:40:43 -08:00
Andrew Rynhard
03094861c2 chore: output SBC images as compressed raw images
By publishing SBC images as compressed raw images, tools like etcher can flash SD cards
by using URLs to the release asset. It is also common in this community to publish compressed
images instead of tarballs.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-12-01 20:49:51 -08:00
Andrew Rynhard
db905c6f74 chore: build SBC images
This will build SBC images and output them to the artifacts directory. These images
will be published on releases.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-12-01 12:33:53 -08:00
Andrey Smirnov
1c0114b21d feat: bump package for kernel with CIFS support
See https://github.com/talos-systems/pkgs/pull/199

Also serializes push/push-latest.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-26 13:30:32 -08:00
Andrey Smirnov
1eac88e470 feat: add support for installing to SBCs
This introduces the notion of a "board" in Talos. A board is an interface that is capable
of modifying the installation in specific ways for a given SBC. This also adds support for the
libretech_all_h3_cc_h5.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-26 07:18:25 -08:00
Spencer Smith
79057f93c5 feat: support openstack platform
This PR adds the ability for us to deploy Talos in openstack. Tested in
local devstack with a supplied userdata file. It also adds support to
the Makefile for building the openstack image so it'll be published with
next release.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-11-25 07:12:57 -08:00
Andrey Smirnov
61facf700a chore: build arm64 images in CI
This changes installer image/iso output to be tar via stdout
(optionally), so that we can copy back artifacts back from remote docker
daemon.

Fixes #2776

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-13 12:34:48 -08:00
Andrew Rynhard
bf1e81b1f4 release(v0.8.0-alpha.0): prepare release
This is the official v0.8.0-alpha.0 release.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-10 12:23:02 -08:00
Andrew Rynhard
1ca61ddce7 feat: add ISO support
This reverts commit 3515f4e0f8.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-02 10:21:40 -08:00
Andrey Smirnov
350d75eb46 feat: build talosctl-cni-bundle, use it in talosctl for QEMU
This builds a bundle with CNI plugins for talosctl which is
automatically downloaded by `talosctl` if CNI plugins are missing.

CNI directories are moved by default to the `~/.talos/cni` path.

Also add a bunch of pre-flight checks to the QEMU provisioner to make it
easier to bootstrap the Talos QEMU cluster.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-30 16:30:37 -07:00
Andrey Smirnov
66829b14d5 test: bump Talos version for upgrade tests, bump Cilium version
Use 0.6.3 as upgrade source version, use latest Cilium release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-29 22:22:21 +03:00
Andrey Smirnov
ab73fb24da chore: reduce numer of steps/parallelism of Drone build
Hope is that will make Drone more stable.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-21 08:15:38 -07:00
Andrey Smirnov
c3d7277c05 chore: attempt to fix image pushing for GitHub
See https://github.com/docker/buildx/issues/327

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-02 09:39:06 -07:00
Andrey Smirnov
32746259eb chore: fix 'push' targets
These targets don't inherit from the common step definition, so they
were missing correct platform list which was in turn making them fail.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-28 06:56:56 -07:00
Andrey Smirnov
ff0d4b305a feat: build Talos images/artifacts for amd64/arm64
By default, build outside of Drone works the same and builds only amd64
version, loads images back into dockerd, etc.

If multiple platforms are used, multi-arch images are built which can't
be exported to docker or to `.tar` image, they're always pushed to the
registry (even for PR builds to our internal CI registry).

Artifacts as files (initramfs, kernel) now have `-arch` suffix:
`vmlinuz-amd64`, `initramfs-amd64.xz`. "Magic" script normalizes output
paths depending on whether single platform or multiple platforms were
given.

VM provisioners accept magic `${ARCH}` in initramfs/kernel paths which
gets replaced by cluster architecture.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-27 10:32:07 -07:00
Andrew Rynhard
7453711c4f chore: fix edge push
Fix the docker login for edge.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-23 18:04:21 -07:00
Andrew Rynhard
d8c494e1a8 chore: fix docker login
Updates the drone config to setup the GHCR credentials.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-23 16:22:55 -07:00
Spencer Smith
c6a35ba0ed chore: push edge releases on successful nightly integration
This PR makes sure we push an edge tag any time the nightly integration
tests pass. This also removes the dependency on cloud conformance to
pass in order for edge tags to be pushed.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-09-18 15:15:05 -07:00
Andrey Smirnov
f50e60937c test: re-enable Cilium e2e upgrade test
I can't reproduce the failures with Cilium anymore.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-14 11:45:58 -07:00
Andrey Smirnov
f6e075ea55 test: verify kubernetes control plane upgrade in provision tests
Add Kubernetes upgrade as part of the provisioning (upgrade tests):
first K8s control plane is upgraded, then Talos is upgraded (with
kubelet), and e2e test is run last.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-11 10:53:33 -07:00
Andrey Smirnov
788cd15c29 test: add e2e test to the provision (upgrade) tests
Add sonobuoy runner code with log fetching on failure. Use hand-picked
set of e2e tests to run: verify basic pod functionality, verify service
connectivity.

Add option `--run-e2e` to the `talosctl health` to run quick e2e test to
verify cluster health.

Add option to run provision tests with custom CNI, run one track of
provision tests with Cilium.

Bump Cilium to 1.8.2.

Talos 0.6 won't uncordon node automatically after upgrade from 0.5, as
0.5 doesn't put annotation. Workaround that in upgrade tests.

Bump upgrade test version to 0.6.0 release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-08 13:26:31 -07:00
Andrew Rynhard
3515f4e0f8 feat: remove ISO support
This feature has long been broken. It is time to remove it.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-04 01:00:40 -07:00
Andrew Rynhard
1a4059a553 feat: add grub bootloader
This moves to using grub instead of syslinux.

BREAKING CHANGE: Single node upgrades will fail in this change. This
will also break the A/B fallback setup since this version introduces
an entirely new partition scheme, that any fallback will not know about.
We plan on addressing these issues in a follow up change.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-01 12:06:43 -07:00
Andrey Smirnov
2f99f551e7 chore: drop vmlinux from assets
`vmlinux` (uncompressed kernel) shouldn't be much useful to the people
as it can't be booted directly with most of the boot tools (it's too
huge). We had it enabled for firecracker, but now firecracker has
support for decompressing `vmlinuz` on the fly.

So, to reduce confusion, make assets smaller, remove `vmlinux`.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-01 07:57:56 -07:00
Andrey Smirnov
59adf7315d feat: provide option to run Talos under UEFI in QEMU
This also adds integration pipeline tests for UEFI.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-08-28 12:51:10 -07:00
Andrey Smirnov
a48c1dbe89 chore: use qemu instead of firecracker in CI
qemu opens up a bunch of possibilities, including the bootloader
testing.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-07-30 22:43:16 +03:00
Andrey Smirnov
26317071b6 chore: really mount /tmp in CI as tmpfs
This seems to be the way to do it for kube-runner.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-07-30 17:38:08 +03:00
Andrey Smirnov
b33c6b7b74 chore: mount /tmp in CI to the build steps
This should help with I/O contention during the build as our disks are
not that performant.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-07-30 06:06:35 -07:00
Andrew Rynhard
6f5d24cc3d chore: add release notes
This ensures that releases have notes.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-07-28 14:14:33 -07:00
Andrey Smirnov
76c44ac468 test: remove apid load balancer for firecracker
We're not using load balancer for `apid` (always using client-side load
balancing), so we can remove this safely.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-07-28 20:21:21 +03:00
Andrey Smirnov
564111d9d5 chore: use outer docker as buildkit instance
This should provide caching for the builds.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-07-27 13:55:29 -07:00
Andrew Rynhard
1f31d24e55 chore: use Kubernetes pipelines
This moves to using Kubernetes pipelines.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-07-27 12:09:53 -07:00