alt-orchestra/talos
1
0
Fork 0
Code Issues Pull Requests Actions 27 Packages Projects Releases Wiki Activity

fix: always shutdown maintenance API service

Browse Source 
The problem was that `GracefulStop()` will hang forever if there is a
running API call. So if there is a running streaming call, the
maintenance service might hang until it is finished.

The problem shows up with 'Upgrade' API in the maintenance mode if there
is a concurrent streaming API call, e.g.:

1. Watch API is running against maintenance mode.
2. Upgrade API is issued, it tries to run the MaintenanceUpgrade
   sequence, which tries to take over the Initialize sequence. The
   Initialize sequence is canceled, maintenance API service context is
   canceled, but the service doesn't terminate, as it's stuck in
   `GracefulStop`. The sequence take over times out, as even the
   sequence is canceled, it hasn't terminated yet.

Sample log:

```
[talos] upgrade request received: "ghcr.io/siderolabs/installer:v1.3.3"
[talos] upgrade failed: failed to acquire lock: timeout
[talos] task loadConfig (1/1): failed: failed to receive config via maintenance service: maintenance service failed: context canceled
[talos] phase config (6/7): failed
[talos] initialize sequence: failed
<stuck here>
```

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
This commit is contained in:
Andrey Smirnov 2023-03-23 16:59:30 +04:00
parent a0a5db590d
commit cf2ccc521f
No known key found for this signature in database
GPG Key ID: 7B26396447AB6DFD
1 changed files with 7 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
internal/app/maintenance/main.go
View File

@ -120,7 +120,12 @@ func Run(ctx context.Context, logger *log.Logger) ([]byte, error) {
return nil, err
}
defer server.GracefulStop()
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(ctx, 5*time.Second)
defer shutdownCancel()
factory.ServerGracefulStop(server, shutdownCtx)
}()
go func() {
//nolint:errcheck
@ -156,12 +161,7 @@ func Run(ctx context.Context, logger *log.Logger) ([]byte, error) {
select {
case cfg := <-cfgCh:
shutdownCtx, shutdownCancel := context.WithTimeout(ctx, 5*time.Second)
defer shutdownCancel()
factory.ServerGracefulStop(server, shutdownCtx)
return cfg, err
return cfg, nil
case <-ctx.Done():
return nil, ctx.Err()
}