fix: Fixes to server shutdown stream
This commit is contained in:
@@ -195,6 +195,25 @@ marker is written and the rest is dropped. Without that cap a `yes` in a step
|
||||
is a database incident. **Nothing writes to `/data` any more**, which is why
|
||||
`server.persistence` now defaults to off and `VANTAGE_WORKFLOW_LOG_DIR` is gone.
|
||||
|
||||
**Shutdown order is load-bearing.** `main` traps SIGTERM, stops gRPC
|
||||
(`GracefulStop`, 10s cap) and only then drains HTTP. Each `CommandStream`
|
||||
handler releases its agent's presence claim on return, so a killed process
|
||||
leaves `vantage:agent:<server_id>` behind for the rest of its 30s TTL — during
|
||||
which other replicas dispatch to a pod that has exited and the caller sees
|
||||
`agent offline` for a perfectly healthy agent. Draining HTTP first would hold
|
||||
those claims for the length of the drain, which is why gRPC goes first. The
|
||||
chart's `server.terminationGracePeriodSeconds` (30s) must stay above the
|
||||
10s + 10s the stop sequence needs, or the kubelet SIGKILLs mid-shutdown and the
|
||||
handling buys nothing.
|
||||
|
||||
The agent side of the same failure: `runCommandStream` resets its backoff only
|
||||
after a stream that survived `streamHealthyAfter`. `connectAndHandleStream`
|
||||
returns an error on *every* stream end, healthy ones included, so without that
|
||||
reset the backoff only ever climbed — an agent pinned itself at the ceiling
|
||||
after a handful of ordinary deploys and stayed there. The ceiling is 30s, not
|
||||
minutes, because while the stream is down the agent still polls `SyncKeys` and
|
||||
still reads as `active` in the fleet list while answering no commands at all.
|
||||
|
||||
**The leader lock is not an optimisation.** N replicas each running the monitor
|
||||
scheduler means each check fires N times, each incident notification reaches the
|
||||
customer N times, and each hourly rollup is written N times; N reapers race to
|
||||
|
||||
Reference in New Issue
Block a user