931 Commits

Author SHA1 Message Date
Andrey Smirnov
86c94eff8d
refactor: docgen and config examples
Short version is: move from global variables/`init()` function into
explicit functions.

`docgen` was updated to skip creating any top-level global variables,
now `Doc` information is generated on the fly when it is accessed.
Talos itself doesn't marshal the configuration often, so in general it
should never be accessed for Talos (but will be accessed e.g. for
`talosctl`).

Machine config examples were changed manually from variables to
functions returning a value and moved to a separate file.

There are no changes to the output of `talosctl gen config`.

There is a small change to the generated documentation, which I believe
is a correct one, as previously due to value reuse it was clobbered with
other data.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2023-08-10 14:56:01 +04:00
Dmitriy Matrenichev
80238a05a6
chore: unify semver under github.com/blang/semver/v4
Currently, we use `github.com/coreos/go-semver/semver` and `github.com/hashicorp/go-version`
for version parsing. As we use `github.com/blang/semver/v4` in our other projects, and it
has more features, it makes sense to use it across the projects. It also doesn't allocate
like crazy in `KubernetesVersion.SupportedWith`.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-08-04 00:29:52 +03:00
Christian Rolland
ac2aff5cc5
fix: fix azure portion of cloud uploader
Correctly propagate errors back. Drop ARM templates and use native APIs.
Correctly handle restarted runs for creating image versions. fixes #7512.

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>
2023-08-03 09:38:16 -04:00
Noel Georgi
76fa45afba
docs: update cilium instructions
Update cilium instructions to skip mounting `bpffs`.

Also fix the TPM example in release notes.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-08-03 14:47:23 +05:30
Andrey Smirnov
87fe8f1a2a
feat: implement image generation profiles
Support full configuration for image generation, including image
outputs, support most features (where applicable) for all image output
types, unify image generation process.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-08-02 19:13:44 +04:00
Andrey Smirnov
e685208ce5
chore: update go 1.20.7
Some final bumps for the go.mod before going beta.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-08-02 17:11:51 +04:00
Andrey Smirnov
018e7f5871
chore: bump dependencies
Linux: 6.1.42
containerd: 1.6.22
Flannel: 0.22.1

And some other Go module bumps, new pkgs/tools/extras.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-31 22:33:22 +04:00
Noel Georgi
68e6b98f7d
feat: add security state resource
Add security state resource that describes the state of Talos SecureBoot
and PCR signing key fingerprints.

The UKI fingerprint is currently not populated.

Fixes: #7514

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-07-31 22:02:08 +05:30
Noel Georgi
209c34801e
chore: drop with-secureboot talosctl flag
The code picks up firmware files in the order it's defined. The
secureboot QEMU firmware files are defined first, so this flag is a
no-op. This was leftover from when `ovmfctl` was used.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-07-31 17:33:12 +04:00
Spencer Smith
dbb9f2bc7a
chore: add dm_multipath module
This PR pulls in the latest pkgs commit to enable dm_multipath as a module.

Signed-off-by: Spencer Smith <spencer.smith@talos-systems.com>
2023-07-27 19:05:10 -04:00
Andrey Smirnov
9b533e27cf
feat: update Kubernetes to 1.28.0-rc.0
See https://github.com/kubernetes/kubernetes/releases/tag/v1.28.0-rc.0

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-27 20:39:58 +04:00
Andrey Smirnov
9b0bc3e931
chore: split kernel modules out of the tree
Also update Linux 6.1.41 (Zenbleed workaround).

See https://github.com/siderolabs/pkgs/pull/768

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-26 21:42:29 +04:00
Andrey Smirnov
ffa48ac803
chore: workaround AWS AMI failures, disable Azure uploader
Fixes #7513

AWS image uploads recently consistently fail in some regions, which
blocks the release process. Allow to skip some AMIs if they fail to
upload.

Disable Azure until #7512 is resolved.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-26 17:14:31 +04:00
Spencer Smith
4cd7623cf7
chore: add alx drivers
This PR adds the alx drivers from pkgs to talos

Signed-off-by: Spencer Smith <spencer.smith@talos-systems.com>
2023-07-25 11:00:12 -04:00
Andrey Smirnov
d2f64af863
chore: disable cloud-images, pull in new kernel and gre module
Disable cloud-images step due to the issue with AWS & Azure atm.

Pull in https://github.com/siderolabs/pkgs/pull/761

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-25 15:15:54 +04:00
Andrey Smirnov
4d96d642fd
feat: update default Kubernetes version to 1.28.0-beta.0
See https://github.com/kubernetes/kubernetes/releases/tag/v1.28.0-beta.0

Go modules are not tagged yet, so skipped updating them.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-21 22:04:19 +04:00
Noel Georgi
170a73e161
chore: support creating qemu guest socket
Support creating a qemu guest agent socket so we can test
`qemu-guest-agent` extension in CI.

Ref: https://github.com/siderolabs/extensions/pull/173#issuecomment-1611911106

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-07-21 22:46:13 +05:30
Andrey Smirnov
60c304126f
chore: bump dependencies
* go.mod dependencies
* Linux 6.1.39
* runc 1.1.8
* dm-raid kernel module

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-20 18:25:41 +04:00
Dmitriy Matrenichev
5f34f5b41f
chore: rename api load balancer to KubePrism
Closes #7432

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-07-14 15:23:53 +03:00
Dmitriy Matrenichev
078aac92ee
chore: bump deps
Bump:
- REVERT cilium/cilium-cli to v0.14.7
- github.com/Azure/azure-sdk-for-go/sdk/azcore to v1.7.0
- github.com/Azure/azure-sdk-for-go/sdk/storage/azblob to v1.1.0
- github.com/aws/aws-sdk-go to v1.44.300
- github.com/beevik/ntp to v1.2.0
- github.com/docker/docker to v24.0.4+incompatible
- github.com/gomarkdown/markdown to v0.0.0-20230711084535-11b03c0ae6d6
- github.com/hetznercloud/hcloud-go to v1.48.0
- github.com/iancoleman/orderedmap to v0.3.0
- github.com/jsimonetti/rtnetlink to v1.3.4
- github.com/siderolabs/go-debug to v0.2.3
- golang.org/x/net to v0.12.0
- golang.org/x/tools to v0.11.0
- google.golang.org/genproto/googleapis/rpc to v0.0.0-20230711160842-782d3b101e98
- google.golang.org/grpc to v1.56.2
- google.golang.org/protobuf to v1.31.0

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-07-14 12:44:58 +03:00
Andrey Smirnov
53873b8444
refactor: move ukify into Talos code
This is intemediate step to move parts of the `ukify` down to the main
Talos source tree, and call it from `talosctl` binary.

The next step will be to integrate it into the imager and move `.uki`
build out of the Dockerfile.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-13 19:14:32 +04:00
Noel Georgi
79365d9bac
feat: tpm2 based disk encryption
Support disk encryption using tpm2 and pre-calculated signed PCR values.

Fixes: #7266

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-07-12 20:41:28 +05:30
Andrey Smirnov
d32dd3a820
chore: update Go to 1.20.6
See https://go.dev/doc/devel/release#go1.20.6

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-12 15:21:26 +04:00
Andrey Smirnov
8017afb107
feat: implement CRI image management and pre-pull on K8s upgrade
Fixes #6391

Implement a set of APIs and commands to manage images in the CRI, and
pre-pull images on Kubernetes upgrades.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-11 19:25:10 +04:00
Andrey Smirnov
1c2f19b367
feat: update Kubernetes to 1.28.0-alpha.4
The Go modules were not tagged for alpha.4, so using alpha.3 tag.

Talos 1.5 will ship with Kubernetes 1.28.0.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-11 15:40:24 +04:00
Noel Georgi
94e9891c1b
chore: bump sd-boot to v254-rc1
Bump sd-boot.
Fix parsing PE executable offsets.
Set the PE file alignment to be 512 bytes.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-07-11 15:52:57 +05:30
Noel Georgi
3206db5289
feat: drop tpm simulator for ukify measure
We do not need a tpm simulator for ukify measure. We can pre-calculate
the values. This also means we can build ukify as a static binary.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-07-10 19:21:32 +05:30
Artem Chernyshev
ce63abb219
feat: add KMS assisted encryption key handler
Talos now supports new type of encryption keys which rely on Sealing/Unsealing randomly generated bytes with a KMS server:

```
systemDiskEncryption:
  ephemeral:
    keys:
      - kms:
          endpoint: https://1.2.3.4:443
        slot: 0
```
gRPC API definitions and a simple reference implementation of the KMS server can be found in this
[repository](https://github.com/siderolabs/kms-client/blob/main/cmd/kms-server/main.go).

Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
2023-07-07 19:02:39 +03:00
Andrey Smirnov
2fec8388fc
chore: bump dependencies
Go modules, pkgs, Cilium CLI, CAPI base version.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-05 18:30:54 +04:00
Andrey Smirnov
c9a9f95611
refactor: extract secure boot certificate generation
Fixes #7412

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-07-03 16:55:02 +04:00
Noel Georgi
cbdf96d461
feat: support environment file for extensions
Supports setting `environmentFile` for Talos System Extension Services.

Fixes: #7316

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-28 00:21:13 +05:30
Noel Georgi
258f074490
fix: ukify cert generation
Fix ukify cert generation by setting the proper Subj/CN.

Fixes: #7383

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-27 20:31:50 +05:30
Andrey Smirnov
fbebc17f8b
fix: disable LVM backups/archive
Fixes #3129

Talos does not have a good location to keep LVM metadata backups.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-27 17:28:18 +04:00
Noel Georgi
e5306ef263
chore: format and cleanup test scripts
This formats and cleanups the test scripts.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-27 16:53:40 +05:30
Tim Jones
53389b1e72
feat: auto-enroll secure boot keys
Uses the auto-enrollment feature of sd-boot to enroll required UEFI Secure
Boot keys.

Fixes: #7373

Signed-off-by: Tim Jones <tim.jones@siderolabs.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-24 00:44:56 +05:30
Noel Georgi
e1b150a110
release(v1.5.0-alpha.1): prepare release
This is the official v1.5.0-alpha.1 release.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-23 00:40:12 +05:30
Noel Georgi
8daf432b29
chore: bump deps
Bump deps.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-22 22:41:08 +05:30
Andrey Smirnov
fe0f46980f
feat: implement secure boot from disk
This includes sd-boot handling, EFI variables, etc.

There are some TODOs which need to be addressed to make things smooth.

Install to disk, upgrades work.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-16 20:15:16 +05:30
Dmitriy Matrenichev
445f5ad542
feat: support API server load balancer
This commit adds support for API load balancer. Quick way to enable it is during cluster creation using new `api-server-balancer-port` flag (0 by default - disabled). When enabled all API request will be routed across
cluster control plane endpoints.

Closes #7191

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2023-06-16 10:09:20 -04:00
Dmitriy Matrenichev
665702ddd3
chore: fix cilium e2e tests
`WITH_CONFIG_PATCH_WORKER` check result was overriding any value set in `CONFIG_PATCH_FLAG` variable.
Move it to the different variable.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-14 15:08:31 +04:00
Andrey Smirnov
e858bca3a2
test: fix cilium integration tests
Due to a bug (?) cilium tests don't clean up all the deployments & pods,
leaving one pod in 'Pending' state.

Kubernetes e2e tests check for !Running pods in `kube-system` namespace.

Fix by moving cilium tests to a separate namespace.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-13 20:41:15 +04:00
Andrey Smirnov
0797b0d168
chore: add a pipeline to test cloud-images step without a release
Also uncomment Azure uploader.

Add the Azure environment variables to the Makefile cloud-images step.

Change disk size and tier to 16GiB and tier: P3

Add boolean value to drone pipeline and the cloud images hack will check the value to determine which Azure Compute Gallery to push images to.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>
2023-06-12 19:42:27 +04:00
Noel Georgi
a34a948985
fix: copy missing modules.* files
Copy missing `modules.order`, `modules.builtin` and
`modules.builtin.modinfo` files so tools can read them.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-09 17:25:28 +05:30
Noel Georgi
aef2192a65
chore: use fixed module list
Use a fixed list of modules to copy into Talos initramfs.

This makes sure we can still enable thing in Talos kernel as modules but
not ship it as default in Talos (extra modules could be extensions).

Also fixes: #7341

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-09 00:08:42 +05:30
Andrey Smirnov
a61dcdbbd5
fix: don't load RDMA over Ethernet driver by default
Document the change for upgrade notes.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-08 16:28:32 +04:00
Andrey Smirnov
aac441f618
chore: update Go to 1.20.5, bump dependencies
Go dependencies, new pkgs, extras, etc.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-07 23:40:59 +04:00
Christian Rolland
e6dde8ffc5
feat: add network chaos to qemu development environment
Add flags for configuring the qemu bridge interface with chaos options:
- network-chaos-enabled
- network-jitter
- network-latency
- network-packet-loss
- network-packet-reorder
- network-packet-corrupt
- network-bandwidth

These flags are used in /pkg/provision/providers/vm/network.go at the end of the CreateNetwork function to first see if the network-chaos-enabled flag is set, and then check if bandwidth is set.  This will allow developers to simulate clusters having a degraded WAN connection in the development environment and testing pipelines.

If bandwidth is not set, it will then enable the other options.
- Note that if bandwidth is set, the other options such as jitter, latency, packet loss, reordering and corruption will not be used.  This is for two reasons:
	- Restriction the bandwidth can often intoduce many of the other issues being set by the other options.
	- Setting the bandwidth uses a separate queuing discipline (Token Bucket Filter) from the other options (Network Emulator) and requires a much more complex configuration using a Heirarchial Token Bucket Filter which cannot be configured at a granular enough level using the vishvananda/netlink library.

Adding both queuing disciplines to the same interface may be an option to look into in the future, but would take more extensive testing and control over many more variables which I believe is out of the scope of this PR.  It is also possible to add custom profiles, but will also take more research to develop common scenarios which combine different options in a realistic manner.

Signed-off-by: Christian Rolland <christian.rolland@siderolabs.com>
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
2023-06-06 20:15:26 +04:00
Noel Georgi
3a865370f5
feat: qemu secureboot
Add qemu support for secureboot testing via `talosctl cluster create`.

Can be tested via:

```bash
sudo -E _out/talosctl-linux-amd64 cluster create --provisioner=qemu $REGISTRY_MIRROR_FLAGS --controlplanes=1 --workers=1 --iso-path=_out/talos-uki-amd64.iso --with-secureboot=true --with-tpm2=true --skip-injecting-config --with-apply-config
```

This currently only supports just booting Talos in SecureBoot mode.
Installation and Upgrade comes as extra PRs.

Fixes: #7324

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-06 19:20:07 +05:30
Noel Georgi
423a31ac9d
chore: deprectae bootloader installer option
Deprecate the `bootloader` installer option. This has not been used in a
long while.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-05 23:21:03 +05:30
Noel Georgi
bfc3419376
chore: add default console args
Add default console args for UKI iso.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2023-06-05 19:43:20 +05:30