mrhid6/vantage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity

docs: design for the workload registry

Browse Source 
Agents enumerate Docker containers, compose stacks and systemd services;
start/stop/restart and bounded log snapshots from the UI.

Sub-project B, Linux only. Live log following stays in the console.
This commit is contained in:
mrhid6
2026-08-06 11:11:26 +01:00
parent 5bba54f3e5
commit d1ac3e98ce
1 changed files with 434 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-08-06-workload-registry-design.md
+434
View File
@@ -0,0 +1,434 @@
# Workload registry
Date: 2026-08-06
Agents enumerate what each server actually runs — Docker containers, the
compose stacks grouping them, and systemd services — and report it to the
control plane. Containers and units can be started, stopped and restarted from
the UI, and a bounded snapshot of their logs can be read without opening a
console.
This is **sub-project B** of the four sketched in
`2026-08-06-package-inventory-and-cve-findings-design.md`:
| # | Sub-project | Depends on |
| - | ----------- | ---------- |
| A | Package inventory + CVE findings — its own spec | nothing |
| B | **Workload registry** — this spec | nothing |
| C | Container image scanning | A and B |
| D | Compliance profiles | shares A's findings UI only |
A and B are independent. C is the joiner and must not be designed before both
exist: it needs B's image list and A's findings model.
**Workload** is the domain word throughout: one container or one systemd unit.
It gives the collection, the commands and the page a single honest name rather
than saying "container or service" in every identifier.
Scope is **Linux only**, matching sub-project A and the existing position that
Windows agents are second-class by design. Docker runs on Windows; systemd does
not, and half a feature per platform is worse than a clear line.
---
## What this is for
The control plane can manage a fleet's keys, run workflows across it and watch
its endpoints, but it has no idea what any of those servers actually *runs*.
"Restart nginx on that box" means opening a console. "Which of these 80 servers
is still on the old image" is unanswerable.
---
## Reporting and refresh are one path
The agent reports on its own 60-second ticker through a `ReportWorkloads` RPC,
using the same hash short-circuit as the package report: it offers a SHA-256 of
the sorted workload list, and sends the body only when the server does not
already hold that hash. An unchanged list costs one small message, which on a
60-second cadence is the common case by a wide margin.
The on-demand refresh **does not return data**. `RefreshWorkloadsCmd` carries
no payload back; it makes the agent report immediately through the normal RPC,
and the UI refetches the stored document.
That is deliberate. A refresh that returned workloads inline would be a second
writer for the same collection, arriving by a different route, with its own
serialisation and its own opportunity to disagree with the periodic one. One
writer, one shape; the refresh is a nudge, not a channel.
Opening a server's Workloads tab dispatches a refresh, so what is on screen is
live rather than up to a minute stale. That matters because the page has a
Restart button on it: a stale list is not merely a wrong impression, it is a
wrong action aimed at a container that already died.
## What does answer back
Two operations genuinely return something:
| Command | Answers with |
| ------- | ------------ |
| `ControlWorkloadCmd{kind, id, action}` | the existing `CommandResult` — ok or error |
| `WorkloadLogsCmd{kind, id, tail}` | a new `WorkloadLogsResult{command_id, text, truncated}` |
Both ride the proven path: `commandDispatcher.send()` for request and ack, and
a `WorkloadResults` registry mirroring `StepResults.Await`/`Deliver` over the
bus. **`Await` must subscribe before the command is dispatched** — the pod
driving the request is usually not the pod holding the agent's stream, and a
fast agent otherwise answers into a channel nobody has joined. This is not a
new hazard; it is the one `stepresults.go` already documents.
```protobuf
rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse);
message ReportWorkloadsRequest {
string server_id = 1;
string agent_token = 2;
string hash = 3;
bool docker_ok = 4;
string docker_error = 5;
bool systemd_ok = 6;
string systemd_error = 7;
repeated Workload workloads = 8; // empty on the offer call
}
message ReportWorkloadsResponse {
bool need_full = 1;
}
// ServerCommand gains three variants.
message RefreshWorkloadsCmd {}
message ControlWorkloadCmd {
string kind = 1; // "container" | "unit"
string id = 2;
string action = 3; // "start" | "stop" | "restart"
}
message WorkloadLogsCmd {
string kind = 1;
string id = 2;
int32 tail = 3;
}
// AgentMessage gains one variant.
message WorkloadLogsResult {
string command_id = 1;
string text = 2;
bool truncated = 3;
string error = 4;
}
```
The offer-then-send handshake is the package report's, unchanged: the agent
calls once with `workloads` empty, and resends with the body only if the
response sets `need_full`.
An agent whose stream no pod holds gets a 503 from the dispatcher, as
everything else does. Commands are not queued: a command whose owner died must
fail loudly rather than be delivered to nobody while the operator is told it
worked.
---
## Not gated by licence
Unlike CVE scanning, this reads as core fleet management rather than a premium
add-on, so v1 ships to every instance with no entitlement check.
If that changes it is a one-line `HasFeature` check at `ReportWorkloads`,
gating collection rather than display — the same placement and the same
reasoning as sub-project A, where gating the UI alone would still pay every
write cost.
---
## Data model
One new collection, `server_workloads`, one document per server, mirroring
`server_packages`.
```go
type ServerWorkloads struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"-"`
ServerID string `bson:"server_id" json:"server_id"`
Hash string `bson:"hash" json:"hash"`
Workloads []Workload `bson:"workloads" json:"workloads"`
CollectedAt time.Time `bson:"collected_at" json:"collected_at"`
DockerOK bool `bson:"docker_ok" json:"docker_ok"`
DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"`
SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"`
SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"`
}
type Workload struct {
Kind string `bson:"kind" json:"kind"` // "container" | "unit"
ID string `bson:"id" json:"id"` // container id, or unit name
Name string `bson:"name" json:"name"`
State string `bson:"state" json:"state"`
Health string `bson:"health,omitempty" json:"health,omitempty"`
Image string `bson:"image,omitempty" json:"image,omitempty"`
Stack string `bson:"stack,omitempty" json:"stack,omitempty"`
Ports []string `bson:"ports,omitempty" json:"ports,omitempty"`
Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"`
StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
Protected bool `bson:"protected" json:"protected"`
}
```
`State` is normalised across the two kinds: containers report `running`,
`exited`, `paused`, `restarting`, `created`; units report `active`, `inactive`,
`failed`, `activating`. They are deliberately **not** collapsed into a shared
vocabulary — a failed unit and an exited container mean different things, and
flattening them would lose the distinction the operator needs.
Indexes: `{instance_id, server_id}` unique, plus multikey
`{instance_id, "workloads.image"}` for the fleet-wide "which servers run image
X" query.
### Why the OK/Error pairs exist
A host with no Docker installed and a host where Docker is installed and
running nothing both produce an empty list. One should read "not in use here",
the other "nothing running", and only the second deserves any alarm.
The error strings separate a third case the booleans alone cannot: Docker
installed with the daemon down. "Not installed" and "installed but not
responding" are different problems with different fixes, and collapsing them
into one false boolean throws away the only thing that tells them apart.
### Why `Protected` is reported rather than derived
The agent already knows which unit and container it is. Sending that up lets
the UI render the action disabled with a reason instead of offering a button
whose refusal is already known.
The field is the courtesy; the agent's own check is the boundary. See the
control section.
### No history
A workload list is state, not a record. Nobody asks what containers ran last
Tuesday, and keeping it would grow a collection per server per minute in
exchange for a question nobody has.
---
## Collectors
### Docker: two commands, no English parsing
```
docker ps -aq
docker inspect --format '{{json .}}' <ids…>
```
Not `docker ps --format '{{json .}}'` alone. That reports health and uptime
inside a human `Status` string — `"Up 2 hours (healthy)"` — and anything built
on it is parsing English that is localised, reworded between releases, and
silently different for a paused or restarting container. `inspect` returns
`State.Health.Status`, `State.StartedAt` and `RestartCount` as typed fields.
Two execs instead of one, and no parser to be wrong.
`RestartCount` justifies the second call by itself: a container cycling is the
single thing this page most needs to show, and it is invisible in a list that
only ever says "Up".
Compose stacks come from the `com.docker.compose.project` label. **No YAML is
read from disk** — the label is what Docker itself treats as authoritative, and
a compose file on disk may not be what is actually running.
Docker absent, or a socket that cannot be reached, sets `DockerOK: false`. It
is not an error and produces no log line: most servers in a fleet built around
SSH key management will not have Docker, and treating the normal case as a
fault makes the feature look broken on the majority of the estate.
### systemd: filtered on purpose
```
systemctl list-units --type=service --state=running,failed --no-legend --plain --no-pager
systemctl list-unit-files --type=service --state=enabled --no-legend --plain --no-pager
```
Two calls because "running or failed" and "enabled but stopped" are different
questions, and an enabled unit that is not running is exactly the one worth
seeing.
Excluded by prefix: `systemd-`, `user@`, `session-`, `init.scope`. A typical
host carries 300+ units, the platform's own accounting for most of them.
Listing all of them buries the ten anyone cares about — the same failure mode
as an unfiltered vulnerability report, and the same fix.
Column output rather than `--output=json`: the JSON flag requires systemd 246+,
and this fleet includes older stable distributions. The column format has been
stable considerably longer than the JSON one has existed.
---
## Control actions
```
container: docker {start|stop|restart} <id>
unit: systemctl {start|stop|restart} <unit>
```
Owner or admin only. Every action writes an audit event naming the actor, the
server and the target.
### The protected set
Computed agent-side: `vantage-agent.service`, plus the container ID read from
`/proc/self/cgroup` should the agent ever be run inside a container.
The agent refuses those before doing anything. As with the console relay
hardcoding `127.0.0.1` agent-side, **the control plane may name a target, but
the agent decides what it will do to itself**. A server-side denylist alone
would be bypassed by the next dispatch path someone adds, and the failure is
unrecoverable from the UI: a server that stops its own agent goes offline, and
the way back is SSH or physical access — precisely what this feature exists to
avoid needing.
### Timeouts
`docker stop` waits on a container that may ignore SIGTERM. `systemctl stop`
on a unit with a long `TimeoutStopSec` blocks for exactly as long as that says.
Both run under a 90-second context, and a timeout returns a real error rather
than an ack implying success.
---
## Logs
```
container: docker logs --tail 500 --timestamps <id>
unit: journalctl -u <unit> -n 500 --no-pager --output=short-iso
```
Capped at **500 lines and 256KB, whichever binds first**, with `truncated` set
so the UI can say so. Two caps because 500 lines of a container emitting 4KB
JSON blobs is 2MB, and a line count alone does not stop it — the same reasoning
that gave workflow logs both a per-line and a per-run cap.
Live following is deliberately absent. The browser console already offers a
real terminal on the same server, where `docker logs -f` works properly with
its own scrollback and cancellation. Building a second streaming path — a
relay listener, proxy bus keys, a WebSocket upgrade and a cancellation story
for a follow nobody closed — to duplicate that would be a large amount of
machinery aimed at a capability already shipped. A bounded snapshot answers
"why did this restart", which is the question that sends people to the console
in the first place.
### Log reads are owner or admin only, and audited
Unlike workflow logs, these cannot be masked. A workflow's logs can be masked
because the run injected the secrets and therefore knows their values. A
container's stdout is arbitrary and may contain credentials nobody declared —
a connection string in a startup banner, a token in a stack trace.
So log reads sit behind the same role check as control actions and are audited.
A member who can see the fleet cannot read its logs. This is a deliberate
access decision, not an oversight, and it is why log reading is not simply
folded in with the read-only snapshot endpoints.
---
## REST API
```
GET /api/servers/:id/workloads # stored snapshot
POST /api/servers/:id/workloads/refresh # dispatch, then refetch
POST /api/servers/:id/workloads/:wid/action # {"action":"start|stop|restart"} (owner|admin)
GET /api/servers/:id/workloads/:wid/logs?tail= # (owner|admin)
GET /api/workloads?image=&stack=&state= # fleet-wide
```
`:wid` is a container ID or a unit name, URL-encoded. Unit names carry dots and
`@`, which are legal in a path segment but not worth relying on unencoded.
`tail` is clamped to the 500-line cap server-side; a client asking for more
gets 500, not an error.
---
## UI
Server detail gains a **Workloads** tab, ordered compose stacks first — grouped
under the stack name — then loose containers, then units.
That ordering is not cosmetic. A stack is one thing to an operator even when it
is six containers, and a flat list turns one decision into six rows. It is the
same argument that groups the vulnerabilities board by CVE rather than by
finding.
A `/workloads` fleet view answers "which servers run image X", which is the
reason the snapshot is stored at all rather than fetched on demand and
discarded.
Three rules that follow directly from the model:
- **Protected rows render their actions disabled, with the reason**, rather
than offering a button whose refusal is already known.
- **`DockerOK: false` reads "Docker not in use on this server"**, never an
empty list, and `DockerError` when present is shown as a distinct problem.
- State never reads by colour alone: every pill carries a distinct shape and a
text label, matching the existing monitor and severity pills.
---
## Testing
Every test is pure, driven by captured fixtures, with no daemon, no database
and no network — the repository has no Go tests today and these must run under
plain `go test ./...`.
- `docker inspect` JSON fixture → `[]Workload`, asserting `RestartCount`,
health, and that the compose label becomes `Stack`.
- `systemctl` column fixtures → `[]Workload`, including the exclusion filter
dropping `systemd-*` and `user@*` while keeping `nginx.service`.
- Protected-set computation: `vantage-agent.service` marked, `nginx.service`
not.
- Hash order-independence, matching the package report's test.
- **Log capping in both directions**: 600 lines in → 500 out with `truncated`
set; a 300KB blob of fewer than 500 lines → capped, `truncated` set. The
second is the case a line-count-only implementation silently fails.
---
## Failure modes
| Failure | Behaviour |
| ------- | --------- |
| Docker not installed | `DockerOK: false`, no error, UI reads "not in use" |
| Docker installed, daemon down | `DockerOK: false` **plus** `DockerError` — different message, different fix |
| Agent offline | 503 from the existing dispatcher. No queueing: a command whose owner died must fail loudly |
| Action on a protected workload | Agent refuses; API answers 409 naming the reason |
| `stop` exceeds its timeout | Real error surfaced, never a hopeful ack. Snapshot refreshed afterwards |
| Container removed between snapshot and action | Docker's "No such container" surfaced and a refresh dispatched — this is what on-demand refresh is for |
| Log exceeds either cap | Truncated, flagged, and stated in the UI |
| Instance deleted | **`server_workloads` must be added to the control plane's instance-deletion collection list**, alongside sub-project A's two collections |
---
## Deliberately out of scope
- **Live log following.** The console already does it. See the logs section.
- **Creating, deleting or updating containers and units.** This is a control
and visibility surface, not a deployment tool — workflows already exist for
changing what a server runs, with snapshots, audit and rollback.
- **`docker exec` into a container.** The console reaches the host; exec from
the control plane is a second remote-execution path with its own audit and
authorisation story, and it belongs in its own spec if anywhere.
- **Kubernetes and containerd.** The Docker collector shells to the `docker`
CLI, so a node whose runtime is containerd or CRI-O reports nothing from it —
`DockerOK: false`, correctly, since Docker genuinely is not in use. Covering
those runtimes means a `crictl`/`nerdctl` collector, and talking to a
Kubernetes API server is a different subsystem again. Neither is v1.
- **Podman as a supported runtime.** Its `docker`-compatible CLI means an
aliased install will largely work, and that is a happy accident rather than a
claim: nothing here is tested against Podman and its `RestartCount` and
compose-label behaviour are not verified.
- **Windows.** No systemd, and a different container story.
- **Image vulnerability scanning.** Sub-project C, which needs this spec's
image list and sub-project A's findings model.