docs: add NVIDIA docs
Add NVIDIA docs Signed-off-by: Noel Georgi <git@frezbo.dev>
This commit is contained in:
Noel Georgi
parent
efa3f28985
commit
bd0035f6a2
1
.gitignore
vendored
1
.gitignore
vendored
@ -37,3 +37,4 @@ package-lock.json
|
||||
node_modules
|
||||
website/resources
|
||||
website/public
|
||||
website/.hugo_build.lock
|
||||
|
||||
4
Makefile
4
Makefile
@ -192,6 +192,10 @@ docs: ## Generates the documentation for machine config, and talosctl.
|
||||
@rm -rf docs/talosctl/*
|
||||
@$(MAKE) local-$@ DEST=./ PLATFORM=linux/amd64
|
||||
|
||||
.PHONY: docs-preview
|
||||
docs-preview: ## Starts a local preview of the documentation using Hugo
|
||||
@hugo server -s website
|
||||
|
||||
# Local Artifacts
|
||||
|
||||
.PHONY: kernel
|
||||
|
||||
@ -15,6 +15,13 @@ preface = """\
|
||||
|
||||
[notes]
|
||||
|
||||
[notes.gpu]
|
||||
title = "NVIDIA GPU alpha support"
|
||||
description = """\
|
||||
Talos now has alpha support for NVIDIA GPU based workloads.
|
||||
Check the NVIDA GPU support guide at https://www.talos.dev/v1.0/guides/nvidia-gpu/
|
||||
"""
|
||||
|
||||
[notes.1orgrename]
|
||||
title = "GitHub Organization Change"
|
||||
description = """\
|
||||
|
||||
203
website/content/v1.0/guides/nvidia-gpu.md
Normal file
203
website/content/v1.0/guides/nvidia-gpu.md
Normal file
@ -0,0 +1,203 @@
|
||||
---
|
||||
title: "NVIDIA GPU"
|
||||
description: "In this guide we'll follow the procedure to support NVIDIA GPU on Talos."
|
||||
---
|
||||
|
||||
> Enabling NVIDIA GPU support on Talos is bound by [NVIDIA EULA](https://www.nvidia.com/en-us/drivers/nvidia-license/)
|
||||
> Talos GPU support is an **alpha** feature.
|
||||
|
||||
These are the steps to enabling NVIDIA support in Talos.
|
||||
|
||||
- Talos pre-installed on a node with NVIDIA GPU installed.
|
||||
- Building a custom Talos installer image with NVIDIA modules
|
||||
- Building NVIDIA container toolkit system extension which allows to register a custom runtime with containerd
|
||||
- Upgrading Talos with the custom installer and enabling NVIDIA modules and the system extension
|
||||
|
||||
Both these components require that the user build and maintain their own Talos installer image and the NVIDIA container toolkit [Talos System Extension](../system-extensions).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This guide assumes the user has access to a container registry with `push` permissions, docker installed on the build machine and the Talos host has `pull` access to the container registry.
|
||||
|
||||
Set the local registry and username environment variables:
|
||||
|
||||
```bash
|
||||
export USERNAME=<username>
|
||||
export REGISTRY=<registry>
|
||||
```
|
||||
|
||||
For eg:
|
||||
|
||||
```bash
|
||||
export USERNAME=talos-user
|
||||
export REGISTRY=ghcr.io
|
||||
```
|
||||
|
||||
> The examples below will use the sample variables set above.
|
||||
Modify accordingly for your environment.
|
||||
|
||||
## Building the installer image
|
||||
|
||||
Start by cloning the [pkgs](https://github.com/siderolabs/pkgs) repository.
|
||||
|
||||
Now run the following command to build and push custom Talos kernel image and the NVIDIA image with the NVIDIA kernel modules signed by the kernel built along with it.
|
||||
|
||||
```bash
|
||||
make kernel nonfree-kmod-nvidia PLATFORM=linux/amd64 PUSH=true
|
||||
```
|
||||
|
||||
> Replace the platform with `linux/arm64` if building for ARM64
|
||||
|
||||
Now we need to create a custom Talos installer image.
|
||||
|
||||
Start by creating a `Dockerfile` with the following content:
|
||||
|
||||
```Dockerfile
|
||||
FROM scratch as customization
|
||||
COPY --from=ghcr.io/talos-user/nonfree-kmod-nvidia:{{< release >}}-nvidia /lib/modules /lib/modules
|
||||
|
||||
FROM ghcr.io/siderolabs/installer:{{< release >}}
|
||||
COPY --from=ghcr.io/talos-user/kernel:{{< release >}}-nvidia /boot/vmlinuz /usr/install/${TARGETARCH}/vmlinuz
|
||||
```
|
||||
|
||||
Now build the image and push it to the registry.
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=0 docker build --squash --build-arg RM="/lib/modules" -t ghcr.io/talos-user/installer:{{< release >}}-nvidia .
|
||||
docker push ghcr.io/talos-user/installer:{{< release >}}-nvidia
|
||||
```
|
||||
|
||||
> Note: buildkit has a bug [#816](https://github.com/moby/buildkit/issues/816), to disable it use DOCKER_BUILDKIT=0
|
||||
|
||||
## Building the system extension
|
||||
|
||||
Start by cloning the [extensions](https://github.com/siderolabs/extensions) repository.
|
||||
|
||||
Now run the following command to build and push the system extension.
|
||||
|
||||
```bash
|
||||
make nvidia-container-toolkit PLATFORM=linux/amd64 PUSH=true TAG=510.60.02-v1.9.0
|
||||
```
|
||||
|
||||
> Replace the platform with `linux/arm64` if building for ARM64
|
||||
|
||||
## Upgrading Talos and enabling the NVIDIA modules and the system extension
|
||||
|
||||
> Make sure to use `talosctl` version {{< release >}} or later
|
||||
|
||||
First create a patch yaml `gpu-worker-patch.yaml` to update the machine config similar to below:
|
||||
|
||||
```yaml
|
||||
- op: add
|
||||
path: /machine/install/extensions
|
||||
value:
|
||||
- image: ghcr.io/talos-user/nvidia-container-toolkit:510.60.02-v1.9.0
|
||||
- op: add
|
||||
path: /machine/kernel
|
||||
value:
|
||||
modules:
|
||||
- name: nvidia
|
||||
- name: nvidia_uvm
|
||||
- name: nvidia_drm
|
||||
- name: nvidia_modeset
|
||||
- op: add
|
||||
path: /machine/sysctls
|
||||
value:
|
||||
net.core.bpf_jit_harden: 1
|
||||
```
|
||||
|
||||
Now apply the patch to all Talos nodes in the cluster having NVIDIA GPU's installed:
|
||||
|
||||
```bash
|
||||
talosctl patch mc --patch @gpu-worker-patch.yaml
|
||||
```
|
||||
|
||||
Now we can proceed to upgrading Talos with the installer built previously:
|
||||
|
||||
```bash
|
||||
talosctl upgrade --image=ghcr.io/talos-user/installer:{{< release >}}-nvidia
|
||||
```
|
||||
|
||||
Once the node reboots, the NVIDIA modules should be loaded and the system extension should be installed.
|
||||
|
||||
This can be confirmed by running:
|
||||
|
||||
```bash
|
||||
❯ talosctl read /proc/modules
|
||||
nvidia_uvm 1146880 - - Live 0xffffffffc2733000 (PO)
|
||||
nvidia_drm 69632 - - Live 0xffffffffc2721000 (PO)
|
||||
nvidia_modeset 1142784 - - Live 0xffffffffc25ea000 (PO)
|
||||
nvidia 39047168 - - Live 0xffffffffc00ac000 (PO)
|
||||
```
|
||||
|
||||
```bash
|
||||
❯ talosctl get extensions
|
||||
NODE NAMESPACE TYPE ID VERSION NAME VERSION
|
||||
172.31.41.27 runtime ExtensionStatus 000.ghcr.io-frezbo-nvidia-container-toolkit-510.60.02-v1.9.0 1 nvidia-container-toolkit 510.60.02-v1.9.0
|
||||
```
|
||||
|
||||
```bash
|
||||
❯ talosctl read /proc/driver/nvidia/version
|
||||
NVRM version: NVIDIA UNIX x86_64 Kernel Module 510.60.02 Wed Mar 16 11:24:05 UTC 2022
|
||||
GCC version: gcc version 11.2.0 (GCC)
|
||||
```
|
||||
|
||||
## Deploying NVIDIA device plugin
|
||||
|
||||
First we need to create the `RuntimeClass`
|
||||
|
||||
Apply the following manifest to create a runtime class that uses the extension:
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: node.k8s.io/v1
|
||||
kind: RuntimeClass
|
||||
metadata:
|
||||
name: nvidia
|
||||
handler: nvidia
|
||||
```
|
||||
|
||||
Install the NVIDIA device plugin:
|
||||
|
||||
```bash
|
||||
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
|
||||
helm repo update
|
||||
helm install nvidia-device-plugin nvdp/nvidia-device-plugin --version=0.11.0 --set=runtimeClassName=nvidia
|
||||
```
|
||||
|
||||
Apply the following manifest to run CUDA pod via nvidia runtime:
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: gpu-operator-test
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
runtimeClassName: nvidia
|
||||
containers:
|
||||
- name: cuda-vector-add
|
||||
image: "nvidia/samples:vectoradd-cuda11.6.0"
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
```
|
||||
|
||||
The status can be viewed by running:
|
||||
|
||||
```bash
|
||||
❯ kubectl get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
gpu-operator-test 0/1 Completed 0 13s
|
||||
```
|
||||
|
||||
```bash
|
||||
❯ kubectl logs gpu-operator-test
|
||||
[Vector addition of 50000 elements]
|
||||
Copy input data from the host memory to the CUDA device
|
||||
CUDA kernel launch with 196 blocks of 256 threads
|
||||
Copy output data from the CUDA device to the host memory
|
||||
Test PASSED
|
||||
Done
|
||||
```
|
||||
Loading…
x
Reference in New Issue
Block a user