153 Commits

Author SHA1 Message Date
Brad Beam
88df1b50b8 feat(networkd): Add health api
This introduces a health/ready api for networkd. This
will allow us to better determine the state of networkd
and allow for some level of monitoring.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2020-01-29 09:09:27 -06:00
Brad Beam
4e22295324 chore: Clean up generated path for protoc
Came across an issue that mentioned `paths=source_relative` to address
the full go import path generation.

ref: https://github.com/golang/protobuf/issues/515#issuecomment-487988591
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2020-01-23 13:57:17 -08:00
Andrew Rynhard
4f39907b6e feat: update kernel
This brings in a kernel with VLAN filtering enabled.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2020-01-23 07:06:58 -08:00
Andrew Rynhard
a0d8656ca0 chore: use v0.1.0 tools and pkgs
This brings in the official v0.1.0 releases of tools and pkgs.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2020-01-20 07:53:08 -08:00
Andrey Smirnov
2bf8540855 test: provision Talos clusters via Firecracker VMs
This is initial PR to push the initial code, it has several known
problems which are going to be addressed in follow-up PRs:

1. there's no "cluster destroy", so the only way to stop the VMs is to
`pkill firecracker`

2. provisioner creates state in `/tmp` and never deletes it, that is
required to keep cluster running when `osctl cluster create` finishes

3. doesn't run any controller process around firecracker to support
reboots/CNI cleanup (vethxyz interfaces are lingering on the host as
they're never cleaned up)

The plan is to create some structure in `~/.talos` to manage cluster
state, e.g. `~/.talos/clusters/<name>` which will contain all the
required files (disk images, file sockets, VM logs, etc.). This
directory structure will also work as a way to detect running clusters
and clean them up.

For point number 3, `osctl cluster create` is going to exec lightweight
process to control the firecracker VM process and to simulate VM reboots
if firecracker finishes cleanly (when VM reboots).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-01-16 00:27:08 +03:00
Andrey Smirnov
810e9b418b chore: bump tools/pkgs for Go 1.13.6
Ref: https://github.com/talos-systems/tools/pull/85,
https://github.com/talos-systems/pkgs/pull/87

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-01-13 20:55:17 +03:00
Andrew Rynhard
e66ac62877 feat: upgrade Linux to v5.4.11
This brings in the latest stable linux.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2020-01-12 09:09:57 -08:00
Andrew Rynhard
7edd96947a feat: upgrade Linux to v5.4.10
This brings in the latest stable Linux.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2020-01-10 20:51:07 -08:00
Andrew Rynhard
4242acd085 feat: upgrade linux to v5.4.8
This brings in the latest 5.4 kernel.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2020-01-08 11:59:05 -06:00
Andrew Rynhard
794d9e6066 chore: update all target in Makefile
We should build the most common things by default.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2020-01-06 11:08:27 -08:00
Andrew Rynhard
93ba252e42 fix: use the correct TLD for the container version label
This fixes the container label to use the correct top level domain. It
was mistakenly reverted in a previous commit.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-29 20:15:33 -08:00
Andrew Rynhard
5a7eb631b2 feat: add installer command to installer container
This replaces the entrypoint.sh shell script with a go binary.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-26 06:41:25 -08:00
Andrew Rynhard
e4a1bc3cf9 chore: add help menu to the Makefile
This adds a help  menu to the Makefile. It documents all build
dependencies, and how to get started.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-25 11:11:41 -08:00
Andrew Rynhard
831f5524a1 chore: refactor Makefile to be more DRY
This PR aims to make the Makefile more DRY.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-24 10:48:32 -08:00
Andrew Rynhard
907f87d8e0 feat: upgrade Linux to v5.4.5
This brings in the latest stable version of Linux.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-19 17:43:34 -08:00
Andrew Rynhard
0347286511 feat: upgrade Linux to v5.3.15
This brings in the latest 5.3 version of Linux.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-04 20:15:42 -08:00
Andrew Rynhard
d4c202438c refactor: set CRI config to /etc/cri/containerd.toml
This changes the CRI specific containerd instance's config to a
different path.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-04 19:32:00 -08:00
Andrew Rynhard
1f4c17269d feat: add universal TUN/TAP device driver support
This is required when doing anything with KVM.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-04 15:25:39 -08:00
Andrew Rynhard
1d3cc0038b feat: use containerd-shim-runc-v2
This configures the CRI containerd to use containerd-shim-runc-v2.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-04 14:36:18 -08:00
Andrew Rynhard
43e6703b8b feat: upgrade containerd to v1.3.2
This brings in the latest version of Containerd.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-04 10:19:51 -08:00
Andrew Rynhard
9c0b0a1c46 chore: upgrade packages
This brings in the latest set of packages with the following changes:

- Linux v5.3.14
- Pinned ca-certificates (2019-11-27)

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-12-01 21:10:19 -08:00
Andrey Smirnov
5b7bea2471 feat: use grpc-proxy in apid
This replaces codegen version of apid proxying with
talos-systems/grpc-proxy based version. Proxying is transparent, it
doesn't require exact information about methods and response types. It
requires some common layout response to enhance it properly with node
metadata or errors.

There should be no signifcant changes to the API with the previous
version, but it's worth mentioning a few changes:

1. grpc.ClientConn is established just once per upstream (either local
service or remote apid instance).

2. When called without `-t` (`targets`), apid proxies immediately down
to local service skipping proxying to itself (as before), which results
in empty node metadata in response (before it had local node IP). Might
revert this later to proxy to itself (?).

3. Streaming APIs are now fully supported with multiple targets, but
message definition doesn't contain `ResponseMetadata`, so streaming APIs
are broken now with targets (needs a fix).

4. Errors are now returned as responses with `Error` field set in
`ResponseMetadata`, this requires client library update and `osctl` to
handle it properly.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-11-29 22:57:25 +03:00
Andrew Rynhard
9ea041c7d9 feat: upgrade packages
This brings in fix patches for:

- ca-certificates
- eudev
- libressl
- libseccomp

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-27 13:24:25 -08:00
Andrew Rynhard
3f49a15c06 feat: enable IMA measurement and appraisal
This updates the kernel to make use of a version that has IMA
measurement and appraisal enabled. It is not yet enforced. Additionally,
this adds the securityfs mount at /sys/kernel/security.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-26 03:15:58 -08:00
Andrew Rynhard
e78e1655f1 feat: upgrade packages
This brings in the following changes:

- Linux 5.3.13
- Containerd 1.3.1

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-25 10:41:47 -08:00
Andrew Rynhard
b2ec56ca92 refactor: rename version label
Since we are standardizing on talos.dev everywhere else, we should align
version label with this.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-23 06:54:51 -08:00
Andrew Rynhard
2dd90a226d refactor: remove CNI bundle
The common pattern is for CNIs to install everything required. I don't
think we need to do this beforehand anymore. If we end up finding that
we do we can always add it back.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-23 06:54:19 -08:00
Andrew Rynhard
4680f66bc5 docs: add autogenerated config reference
This adds a small program to parse our config structs and generate
markdown from them. This will allow us to enforce a standard and require
documentation for fields as they get added.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-11 08:38:39 -08:00
Andrey Smirnov
6231b7db3c chore: run gofumports after protoc-gen
This fixes import order and guarantess clean diff after `make generate`.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-11-09 00:20:45 +03:00
Brad Beam
7897374ff1 feat: Add support for streaming apis in apid
This brings in the recent updates to protoc-gen-proxy to allow support
for proxying streaming api requests. We artificially limit it to only the first
target specified in the list while we work through what multi target stream
support looks like.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-11-08 14:22:30 -06:00
Andrey Smirnov
cdda81df66 test: add k8s integration tests
Once again, mostly groundwork and one simple test for node versions.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-11-06 17:08:44 -08:00
Andrey Smirnov
551fa45d33 test: add CLI integration test
This starts with a very simple test for `osctl version` using regexps as
output of the command depends a lot on current version.

We might use more of 'gold' matches for other commands potentially.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-11-05 17:59:23 -08:00
Andrey Smirnov
b0aef2cf22 test: add integration test framework
This is just first steps and core foundation.

It can be used like:

```
make integration.test
osctl cluster create
build/integration.test -test.v
```

This should run the test against the Docker instance.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-11-05 17:21:38 +03:00
Andrew Rynhard
f43e42d845 chore: install customization requirements with ONBUILD
There is no need for these packages to be in the base image. This moves
to installing them using ONBUILD.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-11-03 22:51:05 -08:00
Andrew Rynhard
2cad745292 docs: add customization guide
This adds a section on customizing Talos.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-10-31 14:47:17 -07:00
Brad Beam
ca76ccd4af feat: Add support for creating VMware images
This PR adds support for generating VMware compatible images as an ova.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-10-31 13:39:54 -05:00
Andrew Rynhard
f26a4ce040 chore: update pkgs SHA
This brings in pkgs that have been built using a prefix of /usr.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-10-29 07:10:25 -07:00
Andrey Smirnov
c3e353aa45 chore: bump tools/pkgs for toolchain refactor
This also pulls in Go 1.13.3

See  talos-systems/toolchain#8, talos-systems/tools#82,
https://github.com/talos-systems/pkgs/pull/69

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-10-25 21:59:41 +03:00
Brad Beam
573cce8d18 feat: Add APId
This PR introduces APId. This service replaces the frontend functionality
previously provided by OSD. The main driver for this is two fold:

1. Create a single purpose application to expose the talos api

2. Make use of code generation to DRY api changes

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-10-25 13:02:33 -05:00
Brad Beam
251ab16e07 feat: Add node metadata wrapper to machine api
- Added common.proto to host NodeMetadata
- go_package names were fixed up so imports are generated with the proper
  package names
- fixed up build work (dockerfile) to prevent copying the previously
  generated go proto files. This fixes a bug where we could incorrectly
  copy the previously generated protobuf instead of a new one generated
  at an incorrect location/name/etc.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-10-22 14:42:34 -05:00
Brad Beam
5b58d8b7df chore: Rename proto files into more appropriate names
This is partially driven by the upcoming api changes, but when we tell protoc to look for api.proto,
itll find the first match in the includes(`-I`) directive.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-10-21 14:08:41 -05:00
Andrey Smirnov
0d09e6f530 chore: bump tools & pkgs for Go 1.13.2
Ref:

* https://github.com/talos-systems/pkgs/pull/68
* https://github.com/talos-systems/tools/pull/80

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-10-18 00:20:34 +03:00
Brad Beam
e6bf92ce31 feat(osd): Enable hitting multiple OSD endpoints
This enables the ability to specify additional <talos> endpoints to connect to
to pull back data.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-10-16 15:30:25 -05:00
Andrey Smirnov
e6f980463f chore: bump golangci-lint to 1.21
This release comes with support for caching lint results. It should
allow for speed ups when some packages don't change between the runs, so
cached information about packages might be used instead of running
linter collect phase on the package.

In my non-scientific test:

1. Fresh: 40s
2. One file change: 15s

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-10-15 14:01:58 -07:00
Andrew Rynhard
10b6202c4f refactor: improve metal platform
This brings in a few minor improvements to the metal platform. The first
is to use talos.config=metal-iso to indicate that the machine's config
can be found in an ISO image. The second is a fix to ensure that /mnt
exists.

This adds support for creating more than one node using the qemu-boot.sh
script.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-10-14 22:05:56 -07:00
Andrew Rynhard
80e3876df5 feat: remove proxyd
We have decided that proxyd is not the best architectue for HA
Kubernetes. Our recommendation to users will be to create a load
balancer instead.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-10-14 08:11:00 -07:00
Andrew Rynhard
fef151748b feat: use the unified pkgs repo artifacts
This moves to using a single revision of pkgs. It includes a few
changes:

- kernel with KVM host support
- containerd v1.3.0

This change brings in a kernel with host KVM support. This will allow us
to use VMs within Talos for things like integrations tests. This also
allows users to do things with KVM as they see fit.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-10-14 07:18:17 -07:00
Andrey Smirnov
bb5f5cc754 chore: bump golangci-lint to 1.20
Memory usage reduced around 8-10x: now it stays stable at 1GB.

I disabled some of the new linters, and one rule which is violated a
lot.

I might make sense to go back and enable `wsl` fixing all the issues
(leaving that for another PR).

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-10-09 22:21:08 +03:00
Andrew Rynhard
b29391f0be feat: use bootkube for cluster creation
This replaces kubeadm with bootkube.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-10-07 17:17:57 -07:00
Spencer Smith
dfe429a8b3 chore: bump kernel to 5.2.18
This PR pulls in the latest kernel container, containing v5.2.18.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-10-01 17:00:52 -04:00