From d1ac3e98ce8b81e6e95cad47c71809106cd41809 Mon Sep 17 00:00:00 2001 From: mrhid6 Date: Thu, 6 Aug 2026 11:11:26 +0100 Subject: [PATCH] docs: design for the workload registry Agents enumerate Docker containers, compose stacks and systemd services; start/stop/restart and bounded log snapshots from the UI. Sub-project B, Linux only. Live log following stays in the console. --- .../2026-08-06-workload-registry-design.md | 434 ++++++++++++++++++ 1 file changed, 434 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-06-workload-registry-design.md diff --git a/docs/superpowers/specs/2026-08-06-workload-registry-design.md b/docs/superpowers/specs/2026-08-06-workload-registry-design.md new file mode 100644 index 0000000..8d31adf --- /dev/null +++ b/docs/superpowers/specs/2026-08-06-workload-registry-design.md @@ -0,0 +1,434 @@ +# Workload registry + +Date: 2026-08-06 + +Agents enumerate what each server actually runs — Docker containers, the +compose stacks grouping them, and systemd services — and report it to the +control plane. Containers and units can be started, stopped and restarted from +the UI, and a bounded snapshot of their logs can be read without opening a +console. + +This is **sub-project B** of the four sketched in +`2026-08-06-package-inventory-and-cve-findings-design.md`: + +| # | Sub-project | Depends on | +| - | ----------- | ---------- | +| A | Package inventory + CVE findings — its own spec | nothing | +| B | **Workload registry** — this spec | nothing | +| C | Container image scanning | A and B | +| D | Compliance profiles | shares A's findings UI only | + +A and B are independent. C is the joiner and must not be designed before both +exist: it needs B's image list and A's findings model. + +**Workload** is the domain word throughout: one container or one systemd unit. +It gives the collection, the commands and the page a single honest name rather +than saying "container or service" in every identifier. + +Scope is **Linux only**, matching sub-project A and the existing position that +Windows agents are second-class by design. Docker runs on Windows; systemd does +not, and half a feature per platform is worse than a clear line. + +--- + +## What this is for + +The control plane can manage a fleet's keys, run workflows across it and watch +its endpoints, but it has no idea what any of those servers actually *runs*. +"Restart nginx on that box" means opening a console. "Which of these 80 servers +is still on the old image" is unanswerable. + +--- + +## Reporting and refresh are one path + +The agent reports on its own 60-second ticker through a `ReportWorkloads` RPC, +using the same hash short-circuit as the package report: it offers a SHA-256 of +the sorted workload list, and sends the body only when the server does not +already hold that hash. An unchanged list costs one small message, which on a +60-second cadence is the common case by a wide margin. + +The on-demand refresh **does not return data**. `RefreshWorkloadsCmd` carries +no payload back; it makes the agent report immediately through the normal RPC, +and the UI refetches the stored document. + +That is deliberate. A refresh that returned workloads inline would be a second +writer for the same collection, arriving by a different route, with its own +serialisation and its own opportunity to disagree with the periodic one. One +writer, one shape; the refresh is a nudge, not a channel. + +Opening a server's Workloads tab dispatches a refresh, so what is on screen is +live rather than up to a minute stale. That matters because the page has a +Restart button on it: a stale list is not merely a wrong impression, it is a +wrong action aimed at a container that already died. + +## What does answer back + +Two operations genuinely return something: + +| Command | Answers with | +| ------- | ------------ | +| `ControlWorkloadCmd{kind, id, action}` | the existing `CommandResult` — ok or error | +| `WorkloadLogsCmd{kind, id, tail}` | a new `WorkloadLogsResult{command_id, text, truncated}` | + +Both ride the proven path: `commandDispatcher.send()` for request and ack, and +a `WorkloadResults` registry mirroring `StepResults.Await`/`Deliver` over the +bus. **`Await` must subscribe before the command is dispatched** — the pod +driving the request is usually not the pod holding the agent's stream, and a +fast agent otherwise answers into a channel nobody has joined. This is not a +new hazard; it is the one `stepresults.go` already documents. + +```protobuf +rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse); + +message ReportWorkloadsRequest { + string server_id = 1; + string agent_token = 2; + string hash = 3; + bool docker_ok = 4; + string docker_error = 5; + bool systemd_ok = 6; + string systemd_error = 7; + repeated Workload workloads = 8; // empty on the offer call +} + +message ReportWorkloadsResponse { + bool need_full = 1; +} + +// ServerCommand gains three variants. +message RefreshWorkloadsCmd {} + +message ControlWorkloadCmd { + string kind = 1; // "container" | "unit" + string id = 2; + string action = 3; // "start" | "stop" | "restart" +} + +message WorkloadLogsCmd { + string kind = 1; + string id = 2; + int32 tail = 3; +} + +// AgentMessage gains one variant. +message WorkloadLogsResult { + string command_id = 1; + string text = 2; + bool truncated = 3; + string error = 4; +} +``` + +The offer-then-send handshake is the package report's, unchanged: the agent +calls once with `workloads` empty, and resends with the body only if the +response sets `need_full`. + +An agent whose stream no pod holds gets a 503 from the dispatcher, as +everything else does. Commands are not queued: a command whose owner died must +fail loudly rather than be delivered to nobody while the operator is told it +worked. + +--- + +## Not gated by licence + +Unlike CVE scanning, this reads as core fleet management rather than a premium +add-on, so v1 ships to every instance with no entitlement check. + +If that changes it is a one-line `HasFeature` check at `ReportWorkloads`, +gating collection rather than display — the same placement and the same +reasoning as sub-project A, where gating the UI alone would still pay every +write cost. + +--- + +## Data model + +One new collection, `server_workloads`, one document per server, mirroring +`server_packages`. + +```go +type ServerWorkloads struct { + ID bson.ObjectID `bson:"_id,omitempty" json:"-"` + InstanceID string `bson:"instance_id" json:"-"` + ServerID string `bson:"server_id" json:"server_id"` + Hash string `bson:"hash" json:"hash"` + Workloads []Workload `bson:"workloads" json:"workloads"` + CollectedAt time.Time `bson:"collected_at" json:"collected_at"` + + DockerOK bool `bson:"docker_ok" json:"docker_ok"` + DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"` + SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"` + SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"` +} + +type Workload struct { + Kind string `bson:"kind" json:"kind"` // "container" | "unit" + ID string `bson:"id" json:"id"` // container id, or unit name + Name string `bson:"name" json:"name"` + State string `bson:"state" json:"state"` + Health string `bson:"health,omitempty" json:"health,omitempty"` + Image string `bson:"image,omitempty" json:"image,omitempty"` + Stack string `bson:"stack,omitempty" json:"stack,omitempty"` + Ports []string `bson:"ports,omitempty" json:"ports,omitempty"` + Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"` + StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"` + Protected bool `bson:"protected" json:"protected"` +} +``` + +`State` is normalised across the two kinds: containers report `running`, +`exited`, `paused`, `restarting`, `created`; units report `active`, `inactive`, +`failed`, `activating`. They are deliberately **not** collapsed into a shared +vocabulary — a failed unit and an exited container mean different things, and +flattening them would lose the distinction the operator needs. + +Indexes: `{instance_id, server_id}` unique, plus multikey +`{instance_id, "workloads.image"}` for the fleet-wide "which servers run image +X" query. + +### Why the OK/Error pairs exist + +A host with no Docker installed and a host where Docker is installed and +running nothing both produce an empty list. One should read "not in use here", +the other "nothing running", and only the second deserves any alarm. + +The error strings separate a third case the booleans alone cannot: Docker +installed with the daemon down. "Not installed" and "installed but not +responding" are different problems with different fixes, and collapsing them +into one false boolean throws away the only thing that tells them apart. + +### Why `Protected` is reported rather than derived + +The agent already knows which unit and container it is. Sending that up lets +the UI render the action disabled with a reason instead of offering a button +whose refusal is already known. + +The field is the courtesy; the agent's own check is the boundary. See the +control section. + +### No history + +A workload list is state, not a record. Nobody asks what containers ran last +Tuesday, and keeping it would grow a collection per server per minute in +exchange for a question nobody has. + +--- + +## Collectors + +### Docker: two commands, no English parsing + +``` +docker ps -aq +docker inspect --format '{{json .}}' +``` + +Not `docker ps --format '{{json .}}'` alone. That reports health and uptime +inside a human `Status` string — `"Up 2 hours (healthy)"` — and anything built +on it is parsing English that is localised, reworded between releases, and +silently different for a paused or restarting container. `inspect` returns +`State.Health.Status`, `State.StartedAt` and `RestartCount` as typed fields. +Two execs instead of one, and no parser to be wrong. + +`RestartCount` justifies the second call by itself: a container cycling is the +single thing this page most needs to show, and it is invisible in a list that +only ever says "Up". + +Compose stacks come from the `com.docker.compose.project` label. **No YAML is +read from disk** — the label is what Docker itself treats as authoritative, and +a compose file on disk may not be what is actually running. + +Docker absent, or a socket that cannot be reached, sets `DockerOK: false`. It +is not an error and produces no log line: most servers in a fleet built around +SSH key management will not have Docker, and treating the normal case as a +fault makes the feature look broken on the majority of the estate. + +### systemd: filtered on purpose + +``` +systemctl list-units --type=service --state=running,failed --no-legend --plain --no-pager +systemctl list-unit-files --type=service --state=enabled --no-legend --plain --no-pager +``` + +Two calls because "running or failed" and "enabled but stopped" are different +questions, and an enabled unit that is not running is exactly the one worth +seeing. + +Excluded by prefix: `systemd-`, `user@`, `session-`, `init.scope`. A typical +host carries 300+ units, the platform's own accounting for most of them. +Listing all of them buries the ten anyone cares about — the same failure mode +as an unfiltered vulnerability report, and the same fix. + +Column output rather than `--output=json`: the JSON flag requires systemd 246+, +and this fleet includes older stable distributions. The column format has been +stable considerably longer than the JSON one has existed. + +--- + +## Control actions + +``` +container: docker {start|stop|restart} +unit: systemctl {start|stop|restart} +``` + +Owner or admin only. Every action writes an audit event naming the actor, the +server and the target. + +### The protected set + +Computed agent-side: `vantage-agent.service`, plus the container ID read from +`/proc/self/cgroup` should the agent ever be run inside a container. + +The agent refuses those before doing anything. As with the console relay +hardcoding `127.0.0.1` agent-side, **the control plane may name a target, but +the agent decides what it will do to itself**. A server-side denylist alone +would be bypassed by the next dispatch path someone adds, and the failure is +unrecoverable from the UI: a server that stops its own agent goes offline, and +the way back is SSH or physical access — precisely what this feature exists to +avoid needing. + +### Timeouts + +`docker stop` waits on a container that may ignore SIGTERM. `systemctl stop` +on a unit with a long `TimeoutStopSec` blocks for exactly as long as that says. +Both run under a 90-second context, and a timeout returns a real error rather +than an ack implying success. + +--- + +## Logs + +``` +container: docker logs --tail 500 --timestamps +unit: journalctl -u -n 500 --no-pager --output=short-iso +``` + +Capped at **500 lines and 256KB, whichever binds first**, with `truncated` set +so the UI can say so. Two caps because 500 lines of a container emitting 4KB +JSON blobs is 2MB, and a line count alone does not stop it — the same reasoning +that gave workflow logs both a per-line and a per-run cap. + +Live following is deliberately absent. The browser console already offers a +real terminal on the same server, where `docker logs -f` works properly with +its own scrollback and cancellation. Building a second streaming path — a +relay listener, proxy bus keys, a WebSocket upgrade and a cancellation story +for a follow nobody closed — to duplicate that would be a large amount of +machinery aimed at a capability already shipped. A bounded snapshot answers +"why did this restart", which is the question that sends people to the console +in the first place. + +### Log reads are owner or admin only, and audited + +Unlike workflow logs, these cannot be masked. A workflow's logs can be masked +because the run injected the secrets and therefore knows their values. A +container's stdout is arbitrary and may contain credentials nobody declared — +a connection string in a startup banner, a token in a stack trace. + +So log reads sit behind the same role check as control actions and are audited. +A member who can see the fleet cannot read its logs. This is a deliberate +access decision, not an oversight, and it is why log reading is not simply +folded in with the read-only snapshot endpoints. + +--- + +## REST API + +``` +GET /api/servers/:id/workloads # stored snapshot +POST /api/servers/:id/workloads/refresh # dispatch, then refetch +POST /api/servers/:id/workloads/:wid/action # {"action":"start|stop|restart"} (owner|admin) +GET /api/servers/:id/workloads/:wid/logs?tail= # (owner|admin) +GET /api/workloads?image=&stack=&state= # fleet-wide +``` + +`:wid` is a container ID or a unit name, URL-encoded. Unit names carry dots and +`@`, which are legal in a path segment but not worth relying on unencoded. + +`tail` is clamped to the 500-line cap server-side; a client asking for more +gets 500, not an error. + +--- + +## UI + +Server detail gains a **Workloads** tab, ordered compose stacks first — grouped +under the stack name — then loose containers, then units. + +That ordering is not cosmetic. A stack is one thing to an operator even when it +is six containers, and a flat list turns one decision into six rows. It is the +same argument that groups the vulnerabilities board by CVE rather than by +finding. + +A `/workloads` fleet view answers "which servers run image X", which is the +reason the snapshot is stored at all rather than fetched on demand and +discarded. + +Three rules that follow directly from the model: + +- **Protected rows render their actions disabled, with the reason**, rather + than offering a button whose refusal is already known. +- **`DockerOK: false` reads "Docker not in use on this server"**, never an + empty list, and `DockerError` when present is shown as a distinct problem. +- State never reads by colour alone: every pill carries a distinct shape and a + text label, matching the existing monitor and severity pills. + +--- + +## Testing + +Every test is pure, driven by captured fixtures, with no daemon, no database +and no network — the repository has no Go tests today and these must run under +plain `go test ./...`. + +- `docker inspect` JSON fixture → `[]Workload`, asserting `RestartCount`, + health, and that the compose label becomes `Stack`. +- `systemctl` column fixtures → `[]Workload`, including the exclusion filter + dropping `systemd-*` and `user@*` while keeping `nginx.service`. +- Protected-set computation: `vantage-agent.service` marked, `nginx.service` + not. +- Hash order-independence, matching the package report's test. +- **Log capping in both directions**: 600 lines in → 500 out with `truncated` + set; a 300KB blob of fewer than 500 lines → capped, `truncated` set. The + second is the case a line-count-only implementation silently fails. + +--- + +## Failure modes + +| Failure | Behaviour | +| ------- | --------- | +| Docker not installed | `DockerOK: false`, no error, UI reads "not in use" | +| Docker installed, daemon down | `DockerOK: false` **plus** `DockerError` — different message, different fix | +| Agent offline | 503 from the existing dispatcher. No queueing: a command whose owner died must fail loudly | +| Action on a protected workload | Agent refuses; API answers 409 naming the reason | +| `stop` exceeds its timeout | Real error surfaced, never a hopeful ack. Snapshot refreshed afterwards | +| Container removed between snapshot and action | Docker's "No such container" surfaced and a refresh dispatched — this is what on-demand refresh is for | +| Log exceeds either cap | Truncated, flagged, and stated in the UI | +| Instance deleted | **`server_workloads` must be added to the control plane's instance-deletion collection list**, alongside sub-project A's two collections | + +--- + +## Deliberately out of scope + +- **Live log following.** The console already does it. See the logs section. +- **Creating, deleting or updating containers and units.** This is a control + and visibility surface, not a deployment tool — workflows already exist for + changing what a server runs, with snapshots, audit and rollback. +- **`docker exec` into a container.** The console reaches the host; exec from + the control plane is a second remote-execution path with its own audit and + authorisation story, and it belongs in its own spec if anywhere. +- **Kubernetes and containerd.** The Docker collector shells to the `docker` + CLI, so a node whose runtime is containerd or CRI-O reports nothing from it — + `DockerOK: false`, correctly, since Docker genuinely is not in use. Covering + those runtimes means a `crictl`/`nerdctl` collector, and talking to a + Kubernetes API server is a different subsystem again. Neither is v1. +- **Podman as a supported runtime.** Its `docker`-compatible CLI means an + aliased install will largely work, and that is a happy accident rather than a + claim: nothing here is tested against Podman and its `RestartCount` and + compose-label behaviour are not verified. +- **Windows.** No systemd, and a different container story. +- **Image vulnerability scanning.** Sub-project C, which needs this spec's + image list and sub-project A's findings model.