440 Commits

Author SHA1 Message Date
Andrey Smirnov
59cfd312c1 chore: bump dependencies via dependabot
There were some upstream code changes in etcd, some code got moved
around.

PRs #3651 #3652 #3653 #3654 #3655 #3655 #3656 #3657 #3658
    #3659 #3660 #3661 #3662 #3663

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-05-24 12:15:15 -07:00
Alexey Palazhchenko
49c7276b16 chore: fix markdown linting
* without `-print0` / `-0`, paths with spaces were skipped sometimes;
* `--stdin-filename` flag incorrectly consumed the first file.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-05-19 06:08:14 -07:00
Andrey Smirnov
db9c35b570 feat: implement AddressStatusController
This controller queries addresses of all the interfaces in the system
and presents them as resources. The idea is that can be a source for
many decisions - e.g. whether network is ready (physical interface has
scope global address assigned).

This is also good for debugging purposes.

Examples:

```
$ talosctl -n 172.20.0.2 get addresses
NODE         NAMESPACE   TYPE            ID                                          VERSION
172.20.0.2   network     AddressStatus   cni0/10.244.0.1/24                          1
172.20.0.2   network     AddressStatus   cni0/fe80::9c87:cdff:fe8e:5fdc/64           2
172.20.0.2   network     AddressStatus   eth0/172.20.0.2/24                          1
172.20.0.2   network     AddressStatus   eth0/fe80::ac1b:9cff:fe19:6b47/64           2
172.20.0.2   network     AddressStatus   flannel.1/10.244.0.0/32                     1
172.20.0.2   network     AddressStatus   flannel.1/fe80::440b:67ff:fe99:c18f/64      2
172.20.0.2   network     AddressStatus   lo/127.0.0.1/8                              1
172.20.0.2   network     AddressStatus   lo/::1/128                                  1
172.20.0.2   network     AddressStatus   veth178e9b31/fe80::6040:1dff:fe5b:ae1a/64   2
172.20.0.2   network     AddressStatus   vethb0b96a94/fe80::2473:86ff:fece:1954/64   2
```

```
$ talosctl -n 172.20.0.2 get addresses -o yaml eth0/172.20.0.2/24
node: 172.20.0.2
metadata:
    namespace: network
    type: AddressStatuses.net.talos.dev
    id: eth0/172.20.0.2/24
    version: 1
    owner: network.AddressStatusController
    phase: running
spec:
    address: 172.20.0.2/24
    local: 172.20.0.2
    broadcast: 172.20.0.255
    linkIndex: 4
    linkName: eth0
    family: inet4
    scope: global
    flags: permanent
```

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-05-11 13:32:17 -07:00
Andrey Smirnov
1cf011a809 chore: bump dependencies via dependabot
See PRs #3596 #3593 #3592

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-05-11 11:20:23 -07:00
Andrey Smirnov
3c1213596c feat: implement LinkStatusController
This is the first PR of many which implement new COSI network
configuration. This controller provides low-level status of the network
interfaces (links) not touching on the addresses of the interface.

The information gathered resembles output of `ip link show` command.

Examples:

```
$ talosctl -n 172.20.0.2 get links
NODE         NAMESPACE   TYPE         ID             VERSION   TYPE       KIND     HW ADDR                                           OPER STATE   LINK STATE
172.20.0.2   net         LinkStatus   bond0          1         ether      bond     fe:c4:d6:4c:04:05                                 down         false
172.20.0.2   net         LinkStatus   cni0           5         ether      bridge   22:cc:25:7e:64:19                                 up           true
172.20.0.2   net         LinkStatus   dummy0         1         ether      dummy    0e:f6:f3:ef:53:29                                 down         false
172.20.0.2   net         LinkStatus   eth0           4         ether               ae:1b:9c:19:6b:47                                 up           true
172.20.0.2   net         LinkStatus   flannel.1      2         ether      vxlan    be:c5:4f:eb:da:5c                                 unknown      true
172.20.0.2   net         LinkStatus   ip6tnl0        1         tunnel6    ip6tnl   00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00   down         false
172.20.0.2   net         LinkStatus   lo             4         loopback            00:00:00:00:00:00                                 unknown      true
172.20.0.2   net         LinkStatus   sit0           1         sit        sit      00:00:00:00                                       down         false
172.20.0.2   net         LinkStatus   teql0          1         void                                                                  down         false
172.20.0.2   net         LinkStatus   tunl0          1         ipip       ipip     00:00:00:00                                       down         false
172.20.0.2   net         LinkStatus   veth1c1422df   2         ether      veth     6a:2d:68:be:8e:8f                                 up           true
172.20.0.2   net         LinkStatus   veth2ce7ce8d   1         ether      veth     52:fc:98:82:f7:29                                 up           true
```

```
$ talosctl -n 172.20.0.2 get links eth0 -o yaml
node: 172.20.0.2
metadata:
    namespace: net
    type: LinkStatuses.net.talos.dev
    id: eth0
    version: 4
    owner: network.LinkStatusController
    phase: running
spec:
    index: 4
    type: ether
    linkIndex: 0
    flags: UP,BROADCAST,RUNNING,MULTICAST,LOWER_UP
    hardwareAddr: ae:1b:9c:19:6b:47
    broadcastAddr: ff:ff:ff:ff:ff:ff
    mtu: 1500
    queueDisc: pfifo_fast
    operationalState: up
    kind: ""
    slaveKind: ""
    linkState: true
    speedMbit: 4294967295
    port: Other
    duplex: Unknown
```

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-05-07 10:08:21 -07:00
Andrey Smirnov
e54b6b7a3d chore: update dependencies via dependabot
PRs #3568 #3567 #3566

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-05-04 14:06:35 -07:00
Lance R. Vick
a2bb390e1d
feat: deterministic builds
Use SOURCE_DATE_EPOCH, explicit timestamp setting, and the cpio
--reproducible flag in order to get hash-for-hash identical builds
across multiple systems.

Signed-off-by: Lance R. Vick <lance@lrvick.net>
2021-04-29 18:56:34 -07:00
Andrey Smirnov
d9ba0fd016 docs: create v0.11 docs, promote v0.10 docs, add v0.10 AMIs
This makes v0.10 docs latest, v0.11 docs we generate to.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-26 12:43:48 -07:00
Andrey Smirnov
5fb38d3e5f chore: refactor Dockerfile for cross-compilation
This has two big visible changes:

* `installer` image now contains assets for both `amd64` and `arm64`, so
it can be used to generate any Talos image (including RPi on amd64 host)
* Talos is using cross-compilation instead of emulation to build
non-native architectures: on amd64, Go amd64 compiler produces binaries
for both arm64 and amd64
(before this change: Go arm64 compiler via QEMU produces arm64 binaries on amd64)

CI implications: we no longer require arm64 nodes.

Changes walkthrough:

* `installer` container now keeps assets under `/usr/install/<arch>`
* Dockerfile build starts forcing toolchain/base image to use the build
host native architecture, not target architecture
* lots of duplication for amd64/arm64 as we want to combine assets for
both arches in a single image (e.g. we have multi-arch amd64/arm64
installer image, each arch has native installer binary, but both arches
contain full set of amd64/arm64 assets)
* fixed a small bug preventing arm64 on amd64 talosctl cluster create

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-20 13:11:38 -07:00
Andrey Smirnov
a8f1e526bf chore: build talosctl for Darwin / Apple Silicon
This adds a new target `darwin/arm64` which is supported since Go 1.16.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-20 08:05:40 -07:00
Andrey Smirnov
05cbe250c8 chore: bump dependencies via dependabot
PRs #3503 #3504 #3505

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-19 06:44:47 -07:00
Alexey Palazhchenko
767f3b95ef chore: apply coverage analysis to all packages
This allows us to see test coverage of packages that can't be unit-tested directly.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-12 09:29:07 -07:00
Alexey Palazhchenko
cae25909f3 chore: use Go 1.16 global install command
Plus add stringer tool.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-12 07:40:29 -07:00
Andrey Smirnov
ef24fd6a01 chore: bump dependencies via dependabot
See #3464 #3465 #3466

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-12 06:32:17 -07:00
Alexey Palazhchenko
8737ea716a feat: allow external cloud provides configration
Closes #3312.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-04-06 22:54:24 -07:00
Andrey Smirnov
df8649cbe6 refactor: download modules before go generate
This moves things around a bit so that `go generate` is called after
modules are generated, as `go generate` downloads modules as well.
This fixes a race condition which might show up randomly.

Spotted by: @AlekSi

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-05 11:38:40 -07:00
Andrey Smirnov
39ae0415e9 chore: bump dependencies via dependabot
See #3431 #3432 #3433 #3434

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-05 06:16:24 -07:00
Andrey Smirnov
61b694b948 fix: create rootfs for system services via /system tmpfs
Container rootfs should be writeable as containerd mounts standard
filesystems `/proc` et al.

When `/opt` was used as a root of container filesystem this results in a
problem: Talos overlay mounts `/opt` on `/var/system` which means that
as long as `apid` running `/var` can't be unmounted which breaks
upgrades.

So instead use `/system/libexec` as rootfs for the containers, `/system`
is `tmpfs`, and bind-mount actually executable (`/sbin/init`, machined)
into rootfs.

This fixes upgrades for 0.10.

See also #3425

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-04-02 06:37:29 -07:00
Andrey Smirnov
8309312a3d chore: build components with race detector enabled in dev mode
This provides a variable to build core Talos components with race
detector enabled: `make initramfs WITH_RACE=yes`.

Also refactored and DRYed up the build code exposing common build/link
flags via the Makefile.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-31 10:55:50 -07:00
Andrey Smirnov
2ea20f598a feat: replace timed with time sync controller
This is a complete rewrite of time sync process.

Now the time sync process starts early at boot time, and it adapts to
configuration changes:

* before config is available, `pool.ntp.org` is used
* once config is available, configured time servers are used

Controller updates same time sync resource as other controllers had
dependency on, so they have a chance to wait for the time sync event.

Talos services which depend on time now wait on same resource instead of
waiting on timed health.

New features:

* time sync now sticks to the particular time server unless there's an
error from that server, and server is changed in that case, this
improves time sync accuracy

* time sync acts on config changes immediately, so it's possible to
reconfigure time sync at any time

* there's a new 'epoch' field in time sync resources which allows
time-dependent controllers to regenerate certs when there's a big enough
jump in time

Features to implement later:

* apid shouldn't depend on timed, it should be started early and it
should regenerate certs on time jump

* trustd should be updated in same way

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-29 09:29:43 -07:00
Andrey Smirnov
a6106815b7 chore: bump dependencies via dependabot
See #3386 #3387 #3388

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-29 06:38:55 -07:00
Alexey Palazhchenko
d7e9f6d6a8 chore: build integration tests with -race
Refs https://github.com/talos-systems/talos/issues/3378.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-26 10:08:12 -07:00
Alexey Palazhchenko
9f7d67ac71 chore: fix typo
Actually share golangci-lint cache.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-25 15:14:30 -07:00
Andrey Smirnov
51139d54d4 chore: cache go modules in the build
This does proper caching for Go modules so that when go.mod/go.sum are
changed, only updated modules are downloaded vs. all of them.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-25 06:54:57 -07:00
Andrey Smirnov
b0209fd29d refactor: move networkd, timed APIs to machined, remove routerd
This moves implementation of the user-facing APIs to the machined, and
as now all the APIs are implemented by machined, remove routerd and
adjust apid to proxy to machined.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-24 00:00:28 -07:00
Andrey Smirnov
ac8764702f refactor: move apid, routerd, timed and trustd to single executable
This removes container images for the aforementioned services, they are
now built into `machined` executable which launches one or another
service based on `argv[0]`.

Containers are started with rootfs directory which contains only a
single executable file for the service.

This creates rootfs on squashfs for each container in
`/opt/<container>`.

Service `networkd` is not touched as it's handled in #3350.

This removes all the image imports, snapshots and other things which
were associated with the existing way to run containers.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-23 09:48:11 -07:00
Andrey Smirnov
89a4b09fe8 refactor: run networkd as a goroutine in machined
This removes networkd as a separate container and image.

Reasons:

* `machined` becomes more and more bound into the core flow - now it
interacts with `etcd` for VIPs, so container has more and more
mounts/permissions
* it should be easier to COSIfy machined piece by piece if we have it
running in the same process
* initramfs size

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-23 06:22:49 -07:00
Andrey Smirnov
dc294db16c chore: bump dependencies via dependabot
PRs #3336 #3337 #3338 #3339

Also bump proto tools via talos-systems/tools#133

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-22 13:58:08 -07:00
Andrey Smirnov
a5b62f4dc2 docs: add documentation for Talos 0.10
Move default docs generation to 0.10 folder.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-22 06:24:39 -07:00
bzub
24cd0a2067 feat: publish talosctl container image
Creates a new container image and corresponding Makefile target.

Signed-off-by: bzub <Bryan.Zubrod@target.com>
2021-03-18 13:25:32 -07:00
Alexey Palazhchenko
08271ba931 chore: use Go 1.16 language version
It affects some languages features and go subcommands.
https://golang.org/ref/mod#go-mod-file-go

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-17 06:20:39 -07:00
Alexey Palazhchenko
0dbaeb9e65 chore: update tools, use new generators
To stay current.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-03-16 11:17:15 -07:00
Artem Chernyshev
22f375300c chore: update golanci-lint to 1.38.0
Fix all discovered issues.
Detected couple bugs, fixed them as well.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-03-12 06:50:02 -08:00
Andrey Smirnov
56c95eace3 chore: bump dependencies via dependabot
See #3267 #3268

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-09 06:03:55 -08:00
Andrey Smirnov
c8ae00937e chore: bump dependencies via dependabot
See #3226, #3227, #3228

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-01 06:16:53 -08:00
Andrey Smirnov
85d1669fb0 chore: bump dependencies via dependabot
PRs: #3202, #3201, #3200

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-24 07:05:55 -08:00
Artem Chernyshev
58ff2c9808 feat: implement ephemeral partition encryption
This PR introduces the first part of disk encryption support.
New config section `systemDiskEncryption` was added into MachineConfig.
For now it contains only Ephemeral partition encryption.

Encryption itself supports two kinds of keys for now:
- node id deterministic key.
- static key which is hardcoded in the config and mainly used for test
purposes.

Talosctl cluster create can now be told to encrypt ephemeral partition
by using `--encrypt-ephemeral` flag.

Additionally:
- updated pkgs library version.
- changed Dockefile to copy cryptsetup deps from pkgs.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-17 13:39:04 -08:00
Andrey Smirnov
d29a56b0c0 chore: update dependencies via dependabot
See #3116, #3117

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-08 07:20:51 -08:00
Artem Chernyshev
c36e4a9355 feat: copy cryptsetup executable from pkgs
Fixes: https://github.com/talos-systems/talos/issues/3080

`cryptsetup` is necessary to enable disk encryption.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-01 05:48:55 -08:00
Andrey Smirnov
8974b529af chore: bump dependencies (via dependabot)
See #3072, #3073, #3074, #3076, #3077, #3078

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-01 05:48:34 -08:00
Andrey Smirnov
0aaf8fa968 feat: replace bootkube with Talos-managed control plane
Control plane components are running as static pods managed by the
kubelets.

Whole subsystem is managed via resources/controllers from os-runtime.

Many supporting changes/refactoring to enable new code paths.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-26 14:22:35 -08:00
Andrey Smirnov
11863dd74d feat: implement resource API in Talos
This brings in `os-runtime` package and exposes resources with first
iteration of read-only API.

Two Talos resources (and one controller) are implemented:

* legacy.Service resource tracks Talos 'service' `RUNNING` state
* config.V1Alpha1 stores current runtime config

Glue point between existing runtime and new os-runtime based runtime is
in `v1alpha2` implementation and `V1Alpha2()` sub-interfaces of existing
`Runtime`, `State`, `Controller` interfaces.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-19 11:45:46 -08:00
Andrew Rynhard
00d345fd3a docs: add v0.9 docs
Adds documentation for v0.9, copied from v0.8.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2021-01-13 15:42:25 +03:00
Alexey Palazhchenko
275ca76c5b chore: update protobuf, grpc-go, prototool
To stay current.

Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2021-01-11 08:52:58 -08:00
Andrey Smirnov
6a0e652f0c fix: correctly transport gRPC errors from apid
Before these changes, errors were always sent as strings, so if original
error was gRPC error (which is almost always the case for apid), it is
formatted as string and original fields (like code) are lost in the
formatted string.

With this change, apid sends errors as official `grpc.Status` protobuf
structure, and client decodes that into Go grpc.Status based error.

This change is backwards and forwards compatible.

This should fix more cases when integration tests were not able to
ignore grpc `transport is closing` errors when they were sent as strings
from the apid endpoint.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-23 11:08:51 -08:00
Andrey Smirnov
4a50007bdf chore: bump dockerfile frontend version
Hoping it might fix spurious buildkit failures.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-07 05:54:54 -08:00
Andrew Rynhard
88f15b1254 fix: use the dtb from kernel pkg for libretech_all_h3_cc_h5
This adds sun50i-h5-libretech-all-h3-cc.dtb to the EFI partition.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-30 17:54:07 -08:00
Andrew Rynhard
99aa3cdba5 feat: add support for the Raspberry Pi 4 Model B
This adds support for the Raspberry Pi 4 Model B.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-30 09:29:48 -08:00
Andrey Smirnov
1eac88e470 feat: add support for installing to SBCs
This introduces the notion of a "board" in Talos. A board is an interface that is capable
of modifying the installation in specific ways for a given SBC. This also adds support for the
libretech_all_h3_cc_h5.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-11-26 07:18:25 -08:00
Andrey Smirnov
dd810d0514 docs: add 0.8 docs for the upcoming release
I reverted some reference docs changes in 0.7 which are related to changes
which went only into 0.8.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-24 06:02:40 -08:00