Commit Graph

139 Commits

Author SHA1 Message Date
Andrey Smirnov
c9a9f95611
refactor: extract secure boot certificate generation
Fixes #7412

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-03 16:55:02 +04:00
Noel Georgi
e5306ef263
chore: format and cleanup test scripts
This formats and cleanups the test scripts.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-27 16:53:40 +05:30
Noel Georgi
e3f3f5794d
feat: implement revert for sd-boot
Implement revert for sd-boot.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-22 20:20:31 +05:30
Andrey Smirnov
fe0f46980f
feat: implement secure boot from disk
This includes sd-boot handling, EFI variables, etc.

There are some TODOs which need to be addressed to make things smooth.

Install to disk, upgrades work.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-16 20:15:16 +05:30
Andrey Smirnov
0797b0d168
chore: add a pipeline to test cloud-images step without a release
Also uncomment Azure uploader.

Add the Azure environment variables to the Makefile cloud-images step.

Change disk size and tier to 16GiB and tier: P3

Add boolean value to drone pipeline and the cloud images hack will check the value to determine which Azure Compute Gallery to push images to.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>
2023-06-12 19:42:27 +04:00
Andrey Smirnov
39134d8d53
chore: fix cron pipeline
The step was added to integration pipeline, but I forgot to add it to
cron pipeline.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-08 17:34:19 +04:00
Christian Rolland
e6dde8ffc5
feat: add network chaos to qemu development environment
Add flags for configuring the qemu bridge interface with chaos options:
- network-chaos-enabled
- network-jitter
- network-latency
- network-packet-loss
- network-packet-reorder
- network-packet-corrupt
- network-bandwidth

These flags are used in /pkg/provision/providers/vm/network.go at the end of the CreateNetwork function to first see if the network-chaos-enabled flag is set, and then check if bandwidth is set.  This will allow developers to simulate clusters having a degraded WAN connection in the development environment and testing pipelines.

If bandwidth is not set, it will then enable the other options.
- Note that if bandwidth is set, the other options such as jitter, latency, packet loss, reordering and corruption will not be used.  This is for two reasons:
	- Restriction the bandwidth can often intoduce many of the other issues being set by the other options.
	- Setting the bandwidth uses a separate queuing discipline (Token Bucket Filter) from the other options (Network Emulator) and requires a much more complex configuration using a Heirarchial Token Bucket Filter which cannot be configured at a granular enough level using the vishvananda/netlink library.

Adding both queuing disciplines to the same interface may be an option to look into in the future, but would take more extensive testing and control over many more variables which I believe is out of the scope of this PR.  It is also possible to add custom profiles, but will also take more research to develop common scenarios which combine different options in a realistic manner.

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-06 20:15:26 +04:00
Noel Georgi
3f68485e44
feat: add uki iso generation
This adds code to generate a UKI ISO (UEFI only).

Fixes: #7261

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-02 22:44:27 +05:30
Andrey Smirnov
bab484a405
feat: use stable network interface names
Use `udevd` rules to create stable interface names.

Link controllers should wait for `udevd` to settle down, otherwise link
rename will fail (interface should not be UP).

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-01 21:29:12 +04:00
Andrey Smirnov
3c64a5ffba
chore: optimize image generation time
Use `pigz` and `--sparse` to handle more efficiently compression of the
assets.

Also move tasks out of `setup-ci` step, as it runs always, including for
the promoted pipelines.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-05-31 17:47:23 +04:00
Noel Georgi
51d931c470
chore: faster dev cycle
This cleans up `Dockerfile` and `Makefile` targets to be in similar
parity with `kres` auto-generated targets.

Now `make talosctl` would only build the one for the specific local
machine making development easier. Also added a `iso` docker target
that builds iso for local development without having to push and pull
the imager. (`make local-iso DEST=_out`)

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-05-27 19:10:11 +05:30
Christian Rolland
e0c1585d30
feat: create azure community gallery image version on release
Create Azure Community Gallery Image Version on release:
- Add /hack/cloud-image-uploader/azure.go
	- Upload vhd file to container for all architectures
	- Create managed disk from vhd file for all architectures
	- Create image version from managed disk for all architectures
- Modify /hack/cloud-image-uploader/main.go
	- Start Community Gallery processes concurently with AWS upload
- Modify /hack/cloud-image-uploader/options.go
	- Add additional Options for Community Gallery processes
- Modify .drone.jsonnet to use secrets for environment variables
	- The following secrets need to be created for this to work:
		- azure_subscription_id
		- azure_client_id
		- azure_client_secret
		- azure_tenant_id

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>

chore: fix linting errors in readme

Fix linting errors in readme

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>

chore: fix markdown linting errors

Fix markdown linting errors in readme

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>

chore: fix markdown linting errors

Fix markdown linting errors in readme

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>

chore: change disk size to match new 10GB cloud image size

Change disk size to match 10GB cloud image size

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>
2023-05-18 17:49:55 -04:00
Andrey Smirnov
ec8c8dbafc
chore: fix container image reproducibility
Also provide make targets to sign the images.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-04-24 22:48:28 +04:00
Noel Georgi
5e9d836c3d
chore: add kernel module signtaure verification
Add kernel module signature verification for out of tree kernel modules.

Fixes: #7049

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-04-10 20:05:07 +05:30
Noel Georgi
df9b851fba
chore: load all external artifacts earlier
Load all external artifacts early in the build process so that the
binaries are available for e2e tests.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-03-20 12:29:24 +05:30
Noel Georgi
9933ebb6aa
chore: fix loaded artifacts file permission
Azure skips the file permissions when upload/downloaded from the object
store. Make sure all binaries under `_out` have executable permissions.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-03-17 17:44:59 +05:30
Spencer Smith
e9962bc3ea
chore: update CI to tag azure buckets
This PR updates CI to remove the immutability policy and tags the azure
"containers" (aka buckets) with a ci=true tag. This will allow us to
handle the deletion of buckets with the cloud-cleaner app.

Signed-off-by: Spencer Smith <spencer.smith@talos-systems.com>
2023-03-13 14:09:06 -04:00
Spencer Smith
05b0b721c9
chore: move blob storage to azure for builds
This PR moves blob storage to azure.

Signed-off-by: Spencer Smith <spencer.smith@talos-systems.com>
2023-03-04 15:50:04 -05:00
Noel Georgi
a78281214d
feat: add cilium e2e tests
Add cilium e2e tests. The existing cilium check was very old, update to
latest cilium version and also add a test for KPR strict mode.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-03-03 20:03:25 +05:30
Noel Georgi
5a01d5fd47
chore: run extension build as downstream
Run extensions build as downstream

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-02-27 20:11:10 +05:30
Andrey Smirnov
3d7566ec74
test: update Canal CNI manifest URL
With recent changes to Calico website, old URL returns 404.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-02-08 23:20:56 +04:00
Noel Georgi
0ba5e59f69
fix: drone config for renovate PR's
Fix drone config to exclude renovate pushes.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-01-30 19:01:13 +05:30
Andrey Smirnov
006449e464
test: build integration test early in the pipeline
If we don't pre-build, it's getting built each time the `e2e-*` step
runs, and we have some running in parallel.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-01-24 16:20:51 +04:00
Noel Georgi
812a2877cd
chore: bump deps + renovate cleanup
Bump dependencies.
Disable renovate for PR's and skip un-needed update checks.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-01-24 00:42:58 +05:30
Andrey Smirnov
260684a930
chore: use build-container image for s3cmd
Looks like s3cmd image is broken now.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-11-24 16:32:08 +04:00
Andrey Smirnov
1cfb6188bc
feat: implement support for cgroupsv1
Use boot kernel arg `talos.unified_cgroup_hierarchy=0` to force Talos to
use cgroups v1. Talos still defaults to cgroupsv2.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-11-11 15:49:25 +04:00
Andrey Smirnov
96aa9638f7
chore: rename talos-systems/talos to siderolabs/talos
There's a cyclic dependency on siderolink library which imports talos
machinery back. We will fix that after we get talos pushed under a new
name.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-11-03 16:50:32 +04:00
Serge Logvinov
29f2195e13
feat: support exoscale cloud
Add Exoscale cloud-init support.

Signed-off-by: Serge Logvinov <serge.logvinov@sinextra.dev>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-10-28 17:52:55 +04:00
Kris Reeves
a0151aa13e
feat: add generic rpi u-boot support
This commit adds support for building Talos for the
Compute Module 4 and other generic Raspberry Pi
hardware.

Fixes: #6273

Signed-off-by: Kris Reeves <kris@pressbuttonllc.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
2022-09-26 21:04:07 +05:30
Andrey Smirnov
276d4175bb
chore: bump extension versions in testing
Test with recent versions.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-09-21 17:16:31 +04:00
Andrey Smirnov
13fdfaffc4
test: fix up default branch name
master -> main

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-09-19 15:35:44 +04:00
Andrey Smirnov
f424e53404
fix: stop containers more thoroughly
Don't skip pods which are not ready, try still to stop containers inside
not ready pod sandboxes.

Re-enable the test with Canal CNI (upstream Calico got fixed).

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-09-12 20:16:40 +04:00
Marvin Drees
cdb6bb2cc7
feat: add Nano Pi R4S support
This commit adds initial support for the Nano Pi
R4S from Friendlyelec. This device is a networking focused
rk3399 based SBC with two 1G ethernet interfaces,
making it perfect for edge or SOHO deployments.

Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
2022-09-02 23:37:07 +05:30
Andrey Smirnov
2f2d97b6b5
fix: don't wait for the hostname in maintenance mode
Fixes #6119

With new stable default hostname feature, any default hostname is
disabled until the machine config is available.

Talos enters maintenance mode when the default config source is empty,
so it doesn't have any machine config available at the moment
maintenance service is started.

Hostname might be set via different sources, e.g. kernel args or via
DHCP before the machine config is available, but if all these sources
are not available, hostname won't be set at all.

This stops waiting for the hostname, and skips setting any DNS names in
the maintenance mode certificate SANs if the hostname is not available.

Also adds a regression test via new `--disable-dhcp-hostname` flag to
`talosctl cluster create`.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-08-23 17:52:20 +04:00
Andrey Smirnov
5dd1b40020
feat: disable Kubernetes discovery backend by default
Fixes #5827

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-08-10 22:25:31 +04:00
Larry Rosenman
c92c90655a
feat: build talosctl for FreeBSD
This PR adds support for building talosctl for FreeBSD on
amd64 and arm64 architectures.

It also pulls in the latest go-blockdevice library to ensure the FreeBSD
stubs are there for proper build.

Signed-off-by: Spencer Smith <spencer.smith@talos-systems.com>
2022-07-14 20:21:14 -04:00
RyanSquared
63caa281ae
fix: create native image format for DigitalOcean
This change replaces the gzipped tar with a gzipped image, which
DigitalOcean natively supports uploading, and means that it can be
linked directly to the image when using, for example, Terraform.

Signed-off-by: Ryan Heywood <me@ryansquared.pub>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-06-23 16:24:00 +04:00
Andrey Smirnov
da2985fe1b
fix: respect local API server port
It wasn't used when building an endpoint to the local API server, so
Talos couldn't talk to the local API server when port was changed from
the default one.

Fixes #5706

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-06-09 00:33:49 +04:00
Tim Jones
0c91c89f4f
chore: revert day-two tests for csi tests
This reverts commit d1294d014f.

Signed-off-by: Tim Jones <tim.jones@siderolabs.com>
2022-06-08 11:04:00 +02:00
Noel Georgi
1e4320b64e
chore: add support for rockpi 4A and 4B
Add support for RockPi 4A and 4B

Signed-off-by: Noel Georgi <git@frezbo.dev>
2022-04-06 23:35:16 +05:30
Andrey Smirnov
c26fa4ccc1
test: push GITHUB_TOKEN to the e2e-aws/gcp steps
This should avoid constant GitHub API rate limiting errors.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-03-31 16:55:55 +03:00
Andrey Smirnov
883d401f9f
chore: rename github organization to siderolabs
Go module import paths still use talos-systems, packages use new
siderolabs name.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-03-23 21:07:46 +03:00
Tim Jones
d1294d014f
chore: add day-two tests for e2e-qemu
Uses Sidero Labs d2ctl to install a small sample of
common cluster services to ensure the Talos cluster
is capable.

Signed-off-by: Tim Jones <tim.jones@siderolabs.com>
2022-03-22 20:53:07 +01:00
Andrey Smirnov
18a21b5f24
chore: add dependency images-essential -> images
This is needed to make sure that those steps don't run in parallel on
release builds, as they write same output files.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-03-14 18:32:03 +03:00
Andrey Smirnov
09efa62f68
chore: re-enable kexec and default to UEFI booting in tests
Fixes #4947

It turns out there's something related to boot process in BIOS mode
which leads to initramfs corruption on later `kexec`.

Booting via GRUB is always successful.

Problem with kexec was confirmed with:

* direct boot via QEMU
* QEMU boot via iPXE (bundled with QEMU)

The root cause is not known, but the only visible difference is the
placement of RAMDISK with UEFI and BIOS boots:

```
[    0.005508] RAMDISK: [mem 0x312dd000-0x34965fff]
```

or:

```
[    0.003821] RAMDISK: [mem 0x711aa000-0x747a7fff]
```

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-03-02 21:52:18 +03:00
Andrey Smirnov
b2bf3117ff
feat: implement extension services
Fixes #4694

User services run alongside with Talos system services.
Every user service container root filesystem should be already present
in the Talos root filesystem.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-02-22 23:11:20 +03:00
Andrey Smirnov
063a9e1657
test: pre-fetch CLI tools
As `_out` is a shared volume across Drone steps, we should prefetch CLI
dependencies early, as fetching them concurrently from multiple steps
might lead to surpising results.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-02-22 22:17:11 +03:00
Andrey Smirnov
df0e388a4f
feat: extract firmware part of system extensions into initramfs
Fixes #4816

This changes the way system extensions are packaged into the squashfs
images: `/lib/firmware` is now moved out of the future squashfs images
and becomes part of `initramfs` to make firmware available in the early
boot.

Talos will bind-mount `/lib/firmware` into rootfs as well, so it will be
available in the rootfs as well.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-02-09 22:58:45 +03:00
Noel Georgi
151c9df091
chore: add CSI tests for e2e-qemu
Add tests for using rook as CSI for e2e-qemu
Allow specifying cpu/memory for workers

Signed-off-by: Noel Georgi <git@frezbo.dev>
2022-01-27 20:06:10 +05:30
Andrey Smirnov
0bf161dffb
test: add integration test for system extensions
This verifies system extensions via the gVisor system extension.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2022-01-26 23:29:15 +03:00