# Vantage

A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.

A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.

---

## Architecture Overview

```
┌──────────────────────────────────────────────┐
│  Next.js 16 Frontend (web, :3000)            │
│  servers · keys · workflows · monitors       │
│  secrets · audit · console · settings        │
└───────────────┬──────────────────────────────┘
                │ REST + cookie session
┌───────────────▼──────────────────────────────┐
│  Go Backend (server)                         │
│  :8080 REST (gin)   :9090 gRPC (agents)      │
│  MongoDB (state) · Redis (sessions)          │
│  monitor scheduler · workflow runner         │
│  guacd tunnel proxy for browser console      │
└───────────────┬──────────────────────────────┘
                │ gRPC (TLS) — outbound from agent only
┌───────────────▼──────────────────────────────┐
│  Go Agent (per server, Linux + Windows)      │
│  polls SyncKeys · CommandStream              │
│  rewrites authorized_keys (Linux only)       │
│  runs workflow steps · monitors · inventory  │
└──────────────────────────────────────────────┘
```

Multi-tenancy: every domain document carries `org_id`, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (`<slug>.vantage.<tld>`).

---

## Repository Structure

```
vantage/
├── server/
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # REST handlers
│       ├── auth/                    # local, OIDC, session, middleware, orghost
│       ├── checker/                 # server-run monitor checks
│       ├── db/                      # mongo connect + Col()
│       ├── grpc/                    # gRPC server + generated pb
│       ├── models/                  # MongoDB documents
│       ├── monitorsched/            # server-side monitor scheduler
│       ├── notify/                  # channel dispatch: http, discord, slack, telegram, smtp
│       └── services/                # business logic + migrations
├── web/                             # the application UI (authenticated)
│   ├── app/(app)/                   # authed routes
│   ├── app/login, app/setup         # unauthed routes
│   ├── components/                  # ui/, workflows/, monitors/, Sidebar
│   └── lib/                         # api client, guac console, query client
├── installer/                       # Windows: setup.ps1, nssm.exe, WiX .wxs
├── deploy/                          # docker-compose.yml, Helm chart
└── .gitea/workflows/                # server-deploy.yml, chart-release.yml
```

**Five repositories carry parts of Vantage that this one does not.** What is
left here is the control plane and its UI, and nothing else.

| Repository       | What it holds                                                                                     |
| ---------------- | ------------------------------------------------------------------------------------------------- |
| `vantage-shared` | the private Go module below — `mail`, `license`, `models`, `provision`, `backup`, `grpc/pb`, …    |
| `vantage-admin`  | Vantage HQ: the licensing authority (`server/`, was `admin/`) and its console (`web/`, was `adminsite/`) |
| `vantage-site`   | the marketing site (`web/`, was `site/`) and its contact-form service (`server/`, was `sitesvc/`)        |
| `vantage-docs`   | the user documentation, at the repository root (was `docsite/`)                                          |
| `vantage-agent`  | the agent, at the repository root (was `agent/`), and the Windows `installer/`                           |
| `vantage-ctl`    | `vantagectl`, the backup and restore CLI, at the repository root (was `vantagectl/`)                      |

**None of the three is a build dependency of anything here**, and nothing here
is a dependency of them. `vantage-site` and `vantage-docs` are wholly
independent — the contact-form service stores nothing and reads no database, so
the split cost nothing. `vantage-admin` is the only one with a live coupling,
and there is still no import in either direction, deliberately (see "Grants project, they do not
federate"). It reaches this codebase two ways at runtime, both by writing
directly into the control plane's MongoDB: `inject` for three licence fields
and `cloudprov` for instances and their owners. The parts of that contract this
repository must honour are documented where they bite — `users.auth_source ==
"hq"` and `services.ErrHQManaged`, `POST /license` answering 409
`cloud_managed`, and `FREE_INSTANCE_REAP_AFTER` needing to match admin's value.
The rest lives in that repository's own CLAUDE.md.

**`shared/` is not in this repository either.** It is the private module
`gitea.hostxtra.co.uk/vantage/vantage-shared`, and it holds `mail/` (the one
email system: transport plus templates), `license/` (payload, sign, verify,
trusted keys, plans), `models/` (Instance, User, Settings), `provision/`,
`backup/`, `cryptobox/`, `indexes/`, `grpc/pb` + `grpc/codec`, and
`cmd/lkctl/`, and `proto/vantage/v1/vantage.proto`, which documents `grpc/pb`
and moved there to sit beside it. **One** module here depends on it — `server` —
pinning a version in its own `go.mod`, as do `vantage-admin`, `vantage-site`,
`vantage-agent` and `vantage-ctl`. It was a
directory in this repository until it was extracted with its history; the
`replace ../shared` directives and the `./shared` entry in `go.work` are gone
with it.

**A version pin is now the coupling, and that is the point.** While it was a
directory, every service in a given commit built against exactly one `shared/`,
and a change there rebuilt several images at once whether or not they were ready
for it. Now a service moves when somebody bumps its pin, which is a commit under
that service's own directory — so the existing per-directory rebuild triggers
already cover it, and there is no longer any way to ship a service against a
`shared/` it was never built against. The cost is the obvious one: a fix in
`vantage-shared` is live nowhere until each consumer's pin is bumped, and
nothing in this repository will remind you.

Every Go build now needs a credential for it — `GOPRIVATE=gitea.hostxtra.co.uk/*`
plus a netrc. CI writes one per job from `REGISTRY_USER` + `RELEASE_TOKEN`
(**that token needs read access to the `vantage` org, not only `mrhid6`**), and
the four Go Dockerfiles take it as a **BuildKit secret** rather than a build
arg, because an arg survives in the builder layer's history and this one is a
Gitea token. Locally, either a netrc or
`git config --global url."git@gitea.hostxtra.co.uk:".insteadOf https://gitea.hostxtra.co.uk/`.

**`server` still builds from the repository root**, and only because its runtime
stage copies `default_steps/`. It is the only image this repository builds from
a context wider than one directory.

`go.work` survives with a single `use ./server` entry. That looks pointless and
is not: without it, a `go.work` further up the developer's filesystem is picked
up instead and the build fails with `directory prefix . does not contain modules
listed in go.work`.

---

## Subsystems

### SSH keys

Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites `/root/.ssh/authorized_keys` atomically. Keys can also be generated _on_ a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.

### Workflows

A library of reusable **steps** (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into **workflows** targeting a set of servers. Running one snapshots the resolved steps into a `WorkflowRun`, then dispatches `RunStepCmd` over the agent command stream. Step stdout/stderr streams back as `StepOutputChunk` and is written to MongoDB (`workflow_log_lines`, one document per line); the UI streams it live. Steps support `on_failure: stop|continue|retry`, per-run env passed between steps via `output_env`, and a per-run workspace directory the agent cleans up at the end.

Default steps are seeded per org at boot (`SeedDefaultSteps`) from `VANTAGE_DEFAULT_STEPS_DIR`, which `server/Dockerfile` bakes to `/opt/default-steps` from the repo's `default_steps/`. Deliberately **not** under `/data` — that is a bind mount, so the library would be editable from the host. Adding a step there means committing a file and rebuilding, which is why `default_steps/` is in the `server` rebuild trigger. **Steps with `source: "default"` are read-only**: `UpdateStep`/`DeleteStep` refuse with `ErrDefaultStep` (409), because seeding rewrites them on every boot, so an edit would silently revert and a delete would come back. `web/` mirrors this — the step modal opens read-only, Delete is hidden, and the designer's per-step script override is `readOnly` for a default library step — but as elsewhere, the API is the boundary and the UI is the courtesy. Seeding writes straight to the collection rather than through `UpdateStep`, so the guard does not lock out the seeder. Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever).

### Scheduled workflows

A workflow may carry `schedule{enabled, cron, tz}` — standard **5-field** cron
and an IANA zone name, both validated at save time. `next_run_at` is
**persisted on the document, not held in memory**: a leader handover between
computing an occurrence and firing it would otherwise lose it or fire it twice,
the same argument that put `workflow_log_seq` in MongoDB.

`server/internal/workflowsched` ticks every 30s inside the **existing**
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched` and the sweepers —
one role, one lock. **The atomic claim, not the lock, is what prevents a double
fire**: the `UpdateOne` matches on the document *and* its current `next_run_at`
while setting the recomputed one, so a second process reaching the same workflow
matches nothing and does nothing. The lock only makes it cheap.

`workflowsched` **must not import `services`** — `services` already imports it
for `SetSchedule`'s call to `NextOccurrence`, and Go has no cycles.
`TriggerWorkflow` and `LogEvent` are therefore injected as `workflowsched.Deps`
from `main.go`. Firing goes through the same `TriggerWorkflow` a person uses,
with `"schedule"` as the actor, so there is no second dispatch path and the run
detail page needed no changes.

`main.go` imports `_ "time/tzdata"`, and it is load-bearing: `server/Dockerfile`
runs on `scratch`, which ships no zone database, so without it
`time.LoadLocation("Europe/London")` fails and every schedule silently falls
back to UTC — an hour wrong for half the year, in the direction nobody notices
until a maintenance window lands in business hours. It works on a developer
machine either way, which is exactly why it gets forgotten.

Skips are recorded and surfaced, not just logged: past the 1h grace window is
`missed`, an active run is `already_running`, and a schedule that no longer
parses is disabled rather than left spinning the loop every 30 seconds forever.

### Server tags and workflow targeting

A server carries `tags map[string]string` — lowercase `[a-z0-9_-]`, key ≤32,
value ≤64, 20 per server, `sys:` reserved. **There is no `tags` collection**: a
tag is a property of a server, not an entity, so `KnownTags` aggregates over
`servers` rather than reading a registry that would need reference counting to
know when a tag stopped existing. `PUT /api/servers/:id/tags` replaces the whole
map — last-write-wins over a small map beats merge semantics between two people
editing one server. The index is `{instance_id: 1, "tags.$**": 1}`, wildcard
because the queried key is chosen by the user at request time and cannot be named
in advance; `EnsureServerIndexes` warns rather than being fatal, since a missing
index degrades tag filtering to a scan of a small collection and is no reason to
refuse to serve the fleet list.

`services.ResolveTargets` is the **single** answer to which servers a workflow
touches — the run path and validation both go through it, so the readout and the
dispatch cannot disagree. It is the distinct union of `target_server_ids` and
`target_tags` (AND across keys), ordered by the fleet rather than by the
arguments, so two runs naming the same servers differently are still comparable
line by line. **An empty selector matches nothing** on purpose: "matches
everything" turns a cleared field in the designer into a fleet-wide run. Both
empty is `ErrNoTargets` (400), not a success over zero servers. Offline servers
are **not** filtered out — the dispatcher already answers 503 per server, and a
patch run that silently omits an unreachable machine is worse than one that
visibly fails on it.

**Both halves of the selector are edited in `EditWorkflowModal`** — the named
servers in a `DualListBox`, the tag rows directly beneath it — and saved
together by one `updateWorkflow`. The designer's Targets panel is **read-only**:
it reports the count and the tags and links to Edit. Splitting the two halves
across two screens meant a workflow's reach was decided in two places with no
one view showing both.

`web/lib/targets.ts` **duplicates the match logic in TypeScript** to draw the
resolved count without a round trip, since the browser already holds the fleet.
It is a second implementation of `UnionTargets` / `MatchesTags` and must change
in the same commit as the Go one — the same shape of hazard as the mirrored
token blocks. It is a shared module rather than inline in a component because
the logic had already been written twice, and the second copy — the workflows
list — counted `target_server_ids` alone, so a **tag-only workflow reported zero
targets** while running fine.

The server picker is a hand-built two-pane list, not `<select multiple>`: a
native multi-select paints its selected rows with the platform highlight colour,
which cannot be restyled across browsers and lands outside the token palette on
a dark ground.

### Monitors

HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.

### Notification channels

Per-org outbound destinations: `webhook`, `smtp`, `discord`, `slack`, `telegram`. Monitors reference channels by ID. Channels are testable from the UI.

### Secrets vault

Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via `secret_refs` (injected as env at execution), and read by Kubernetes External Secrets Operator via `GET /api/secrets/:group/values` using a bearer token whose SHA-256 hash is stored in settings.

### Browser console

`POST /api/console/connect` mints a one-time session token; `GET /api/console/tunnel`
upgrades to a WebSocket and proxies to **guacd** using `github.com/wwt/guac`.

guacd never dials the managed server. The server binds a single-use ephemeral
listener, pushes `OpenProxyCmd` down the agent's command stream, and the agent
opens a `ProxyStream` and relays the connection from its own **`127.0.0.1`** —
the host is hardcoded agent-side, so the control plane can name only a port.
This is what makes the console work on Vantage Cloud, where the customer's
server is behind NAT on a private address. It also means the console now
**requires a live agent** on every deployment: `consoleConnect` answers 409
`agent_offline` rather than hanging.

**guacd's Service is headless on purpose.** The server resolves `GUACD_ADDR` to
build the allow-list of sources permitted to claim a relay listener; a ClusterIP
resolves to the Service's virtual address while guacd connects from its *pod*
IP, so every relay connection is rejected and every session dies with
`waiting for guacd: i/o timeout`. Compose is immune — there the name resolves to
the address that connects.

SSH connections authenticate with a stored private key; RDP/VNC credentials are
encrypted, single-use, and consumed when the tunnel opens. None of them reach
the agent — the session is negotiated end-to-end between guacd and the target
daemon, so the agent relays bytes it cannot read.

### Running more than one server replica

An agent's `CommandStream` terminates on exactly **one** server process. Every
piece of coordination below exists because of that single fact: with several
replicas, the process asked to do something to an agent is almost never the
process holding that agent's stream.

`server/internal/bus` is the Redis message bus that closes the gap. It adds no
infrastructure — Redis was already required for sessions — and it is **not
optional on a single-replica deployment**: dispatch takes the bus path always,
so the code running in production is the code running everywhere, rather than a
rare cross-pod branch that only fails under load.

| Concern                | How it crosses replicas                                                                                                                                                                       |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Which pod owns an agent | `vantage:agent:<server_id>` holds the owner's node ID with a 30s TTL, renewed every 10s. `Dispatcher.IsConnected` is an `EXISTS` on it                                                        |
| Sending a command       | published to `vantage:cmd:<server_id>`; the owner pod acks on `vantage:ack:<command_id>`. **Request/ack, not a queue** — a command whose owner died must fail loudly (503) rather than queue. The envelope carries `node`, the presence holder resolved at publish time, and a pod ignores envelopes addressed elsewhere: the channel is a fan-out, and during a reconnect a half-open stream's pod is still subscribed. Unaddressed, it could ack first and queue the command onto a dead stream — the operator told it worked, the agent never seeing it. Presence renewal is owner-only (`RenewPresence`) for the same reason: a blind `SET` let the stale pod steal the key back every 10s |
| Step results            | the owner pod publishes to `vantage:res:<command_id>`; the pod driving the run subscribes **before** dispatching, or a fast agent answers into a channel nobody has joined                    |
| Step output             | never crosses. The dispatch envelope carries the secret mask list, so the owner pod masks and writes lines itself — unmasked bytes stay off the bus                                          |
| Console relay           | **not routed to the owner pod at all.** A `ProxyStream` is its own HTTP/2 request and an L7 proxy balances requests, not connections, so it does not follow the command stream — the listener therefore cannot be bound in advance. Whichever pod receives the stream binds it and announces **its own** address on `vantage:proxyaddr:<proxy_id>`; `vantage:proxypending:<proxy_id>` (30s, consumed atomically) is what authorises the claim, and the failure reason comes back on `vantage:proxyend:<proxy_id>` |
| Background jobs         | `bus.RunAsLeader` — one Redis lock named `housekeeping`                                                                                                                                       |

**Workflow logs are in MongoDB** (`workflow_log_lines`, one document per line,
with a `workflow_log_seq` counter document per run/server). Two pods write the
same log concurrently — the run's pod emits markers, the agent's pod emits
output — so ordering only means anything if both draw sequence numbers from the
same counter. `StepRun.log_offset` is that sequence number now, not a byte
offset. Writes are batched (128 lines or 250ms) and capped: 8 KB per line,
200k lines per server-run, after which one final `[vantage] log truncated`
marker is written and the rest is dropped. Without that cap a `yes` in a step
is a database incident. **Nothing writes to `/data` any more**, which is why
`server.persistence` now defaults to off and `VANTAGE_WORKFLOW_LOG_DIR` is gone.

**Shutdown order is load-bearing.** `main` traps SIGTERM, stops gRPC
(`GracefulStop`, 10s cap) and only then drains HTTP. Each `CommandStream`
handler releases its agent's presence claim on return, so a killed process
leaves `vantage:agent:<server_id>` behind for the rest of its 30s TTL — during
which other replicas dispatch to a pod that has exited and the caller sees
`agent offline` for a perfectly healthy agent. Draining HTTP first would hold
those claims for the length of the drain, which is why gRPC goes first. The
chart's `server.terminationGracePeriodSeconds` (30s) must stay above the
10s + 10s the stop sequence needs, or the kubelet SIGKILLs mid-shutdown and the
handling buys nothing.

The agent side of the same failure: `runCommandStream` resets its backoff only
after a stream that survived `streamHealthyAfter`. `connectAndHandleStream`
returns an error on *every* stream end, healthy ones included, so without that
reset the backoff only ever climbed — an agent pinned itself at the ceiling
after a handful of ordinary deploys and stayed there. The ceiling is 30s, not
minutes, because while the stream is down the agent still polls `SyncKeys` and
still reads as `active` in the fleet list while answering no commands at all.

**The leader lock is not an optimisation.** N replicas each running the monitor
scheduler means each check fires N times, each incident notification reaches the
customer N times, and each hourly rollup is written N times; N reapers race to
purge the same Free instance. `monitorsched`, `StartReaper`, `StartLogSweeper`,
`StartAuditSweeper` and the offline sweep therefore all run inside one
`RunAsLeader("housekeeping", …)` — one role, one lock. Each takes a context
cancelled the instant leadership is lost, and must return when it is.

Redis rather than a Kubernetes `Lease` so Compose takes the identical path: one
implementation to reason about, not two.

Two deployment requirements come with `replicaCount > 1`: every replica must
share **one** Redis (a per-pod Redis partitions the bus and every agent looks
offline to two thirds of the fleet), and `POD_IP` must be set — the chart does
it from the downward API — because `PROXY_ADVERTISE_HOST` names the Service, and
a Service cannot address the one pod holding a console listener.

### Inventory and OS updates

Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`).

Windows update checking and applying go through the Windows Update COM API
(`Microsoft.Update.Session`) rather than the PSWindowsUpdate module, which would
need a PowerShell Gallery install on every host and fails on an air-gapped
fleet. `CurrentVersion` is empty on Windows and `NewVersion` carries the KB
article ID: a Windows update is not a version bump of a named package.

**The agent never reboots a host.** `ApplyUpdatesCmd` installs and stops there;
`inventory.reboot_required` reports that one is owed, set on the static snapshot
every 15 minutes. Linux fills it too, from `/var/run/reboot-required` or
`dnf needs-restarting -r`.

### Package inventory and CVE findings

Agents report their installed packages hourly; the control plane matches them
against distribution security feeds and raises findings that link to the
existing `ApplyUpdatesCmd` patching path. Gated by the `vuln_scanning` licence
feature, **checked at collection rather than display** — an ungated instance
stores no inventory, and storage is the expensive half.

**Matching uses distribution feeds, never NVD version ranges.** Distributions
backport security fixes without changing the upstream version: Ubuntu's
`openssl 3.0.2-0ubuntu1.15` is patched against CVE-2023-0286 while NVD still
calls 3.0.2 vulnerable. Matching on NVD would report a fully patched fleet as
critical, and once the first report is mostly wrong nobody reads the second.
`trivy-db` is those feeds pre-merged; `server/internal/vulndb` pulls it as an
OCI artifact to an ephemeral directory. Version comparison is bought from
`go-deb-version`/`go-rpm-version`/`go-apk-version` because dpkg epochs, `~`
sorting before the empty string, and `rpmvercmp` are each a silent false
negative waiting to happen.

**Only the leader matches.** `ReportPackages` upserts the list and sets
`scan_pending`; it does not scan. `vulnsched` runs inside the existing
`bus.RunAsLeader("housekeeping", …)` and does the matching, because otherwise
every replica needs the ~50MB database resident and a database refresh has N
replicas rescanning the same fleet and sending N digests. The tick is also the
digest's batch boundary, which is what makes "one message, not five hundred"
structural rather than a debounce someone maintains.

Findings are **never deleted when a package is patched** — the state moves to
`fixed`, so "what did we remediate last quarter" stays answerable. Acceptance
requires a reason and an expiry, and reopens automatically: permanent dismissal
is where risk goes to be forgotten. An unsupported distribution reports
`status: unsupported`, never "0 findings"; claiming clean when the truth is
unknown is the same lie as a silently stale database, which is why
`vulndb_meta.pulled_at` is on screen rather than only in a log.

**`server/Dockerfile`'s runtime stage is `scratch`, so it carries an explicitly
copied `/tmp`.** The scheduler unpacks the database to a temporary directory,
and a scratch image has none — the failure is `vulnsched: temp dir: stat /tmp:
no such file or directory`, logged once at boot while every other subsystem
runs normally, so the only symptom is a fleet that never reports a finding.

Two environment variables: `VANTAGE_TRIVY_DB_REF` mirrors the artifact for
air-gapped installs, and `VANTAGE_VULNDB_DISABLED` switches the puller and
scheduler off entirely.

### Workload registry

A **workload** is one Docker container or one systemd unit — one word for the
page, the collection and the commands, rather than saying "container or
service" in every identifier.

On Windows a workload is a Docker container or a Windows **service**, reported
under the same `unit` kind and the same `systemd_ok` / `systemd_error` fields —
one wire shape, worded per platform in the UI, which is the only layer that
knows the host's OS. The platform split lives entirely in the agent, as build
tags (`systemd_linux.go` / `services_windows.go` and the matching `control_`
and `logs_` pairs); the control plane is OS-blind and needed no changes.
Windows collection runs PowerShell through the agent's `internal/winexec`. Every
script that reports data emits JSON that a build-tag-free parser reads, so
those parsers are tested on Linux — the agent module has no Windows CI. The
control verbs and `serviceDisplayName` emit no JSON and have no parser; they
are exercised only by running the agent on Windows.

**Not gated by licence**: this reads as core fleet management, so v1 ships
everywhere with no `HasFeature` check. If that changes the check belongs at
`ReportWorkloads`, gating collection rather than display, exactly as sub-project
A does.

Agents collect on a 60-second ticker and report through `ReportWorkloads` with
the **offer-then-send** handshake the package report already uses. The offer is
identified by an explicit `full` flag, **not by an empty workloads list**: a
host genuinely running nothing sends an empty list as its full report, and
inferring the offer from emptiness leaves that host answering `need_full` every
60 seconds forever and never storing anything.

**The on-demand refresh returns no data.** `RefreshWorkloadsCmd` carries nothing
back; it makes the agent report through the normal RPC and the UI refetches. A
refresh that returned workloads inline would be a second writer for
`server_workloads`, arriving by a different route with its own serialisation and
its own opportunity to disagree with the periodic one. One writer, one shape.
Opening the panel dispatches a refresh because the panel has a Restart button on
it, and a stale row is a wrong action aimed at a container that already died.

Two operations do answer back, both over the bus, both with `Await` called
**before** dispatch: control actions reuse the existing `CommandResult`, and log
reads get `WorkloadLogsResult`. `CommandStream` republishes **every**
`CommandResult` onto `bus.ResultChannel` — publishing with no subscriber is a
no-op, so this costs nothing and avoids a second result path.

**The protected set is computed agent-side and enforced agent-side.**
`vantage-agent.service` on Linux, `VantageAgent` on Windows, plus the container
ID read from `/proc/self/cgroup` should the agent ever run in a container. As
with the console relay hardcoding
`127.0.0.1`, the control plane may name a target but the agent decides what it
will do to itself; a server-side denylist alone would be bypassed by the next
dispatch path someone adds, and the failure is unrecoverable from the UI. The
reported `Protected` flag is the courtesy that greys the button; the agent's own
check is the boundary. The API answers **409** when it fires — nothing failed.

Collection avoids parsing English: `docker ps -aq` then
`docker inspect --format '{{json .}}'`, because `docker ps` reports health and
uptime inside a human `Status` string that is localised and reworded between
releases. Compose stacks come from the `com.docker.compose.project` label, never
from YAML on disk — a compose file there may not be what is running. systemd
uses **column** output, not `--output=json`, which needs systemd 246+.

`DockerOK`/`DockerError` are two fields because there are three states: not
installed (common on this fleet, and not a fault), installed but not responding,
and running nothing. The UI must render the first as "not in use here" rather
than an empty list.

Logs are capped at **500 lines and 256KB, whichever binds first** — a line count
alone does not bound size, and 500 lines of 4KB JSON is 2MB across the bus. The
cap is mirrored in `services.MaxWorkloadLogLines` because the agent is a
separate module — a separate repository now — with an `internal/` tree, and the
constant cannot be shared; change one, change the other. There is **no follow mode**: the browser console already gives
a real terminal where `docker logs -f` works properly. Log reads and control
actions are **owner|admin and audited**, unlike the read-only snapshot — a
container's stdout is arbitrary and cannot be masked the way a workflow's can.

`server_workloads` is one document per server, mirroring `server_packages`, and
is in `ScopedCollections` (which `scopedCollectionsForPurge` derives from). There
is no history: a workload list is state, not a record.

**The wire contract is not in this repository at all.** `shared/grpc/pb` is
hand-written JSON-tagged structs over the custom codec in `shared/grpc/codec`,
and `proto/vantage/v1/vantage.proto` is documentation of them rather than a
generator input — nothing compiles it. Both live in `vantage-shared`, together,
because that co-location is the only thing making "add the message to both in
the same commit" possible.

It is **one** `pb` package serving both sides. There used to be two
(one in the agent, one in the server) and they had already
drifted: the agent's `UnimplementedVantageServer` was three methods stale and
carried no `ReportWorkloads` at all. The agent links the server half as dead
code, which the linker drops.

A message added to `vantage-shared` is not a message either side has until its
pin is bumped. What that buys is a mismatch that is a compile error rather than
two copies that both compiled and disagreed on the wire. What it costs is
ordering — **a wire change is three steps**: release `vantage-shared`, bump the
pin in `server/` (live at the next push to main), bump the pin in `agent/` (live
only at the next `agent/v*` tag). The control plane runs ahead of the fleet in
between, which was true before too; it is now explicit in two `go.mod` files
rather than implicit in a shared directory.

### Status pages

Two collections: `status_pages` is the page itself — title, banner, published
flag, and an ordered list of sections each holding entries that pair a
`monitor_id` with a per-page display name. `status_incidents` holds both
operator-authored incidents and maintenance windows, sharing one document
shape because they share a timeline, an impact and a set of affected
components; each carries an explicit `page_ids` rather than deriving it from
`affected_monitors`, because adding a monitor to a page later must not
retroactively republish that monitor's old incidents to a new audience.

**`services.assembleSnapshot` is the redaction boundary, and it is the only
one.** It takes a `snapshotInput` built from already-fetched
`models.Monitor`/`models.Rollup`/`models.Incident` documents and returns a
`StatusSnapshot` built entirely from a parallel, deliberately smaller
vocabulary (`PublicComponent`, `PublicIncident`, …) that has no field for a
target URL, host, port, expected status, keyword, failure message,
certificate expiry, latency, runner or notification channel — `models.Monitor`
itself never reaches an anonymous caller, only the handful of fields
`assembleSnapshot` chooses to copy out of it. Being a pure function of already-
fetched data (no DB calls inside it) is what makes the boundary testable
without a database, which is the only thing standing between an editor adding
a field to `PublicComponent` and that field being a hostname.

**An incident may only name components the page already carries.**
`services.checkAffectedOnPages` refuses an `affected_monitors` entry that no
page in the incident's `page_ids` lists, and the editor offers only the saved
page's components — labelled by their per-page display name, since that is the
name the reader sees. Naming an arbitrary monitor would publish a machine the
page deliberately does not, which is the same leak `assembleSnapshot`'s
redaction boundary exists to prevent, reached from the authoring side instead
of the read side. It is a separate pass rather than part of `validateIncident`
because it reads the database and `validateIncident` is a pure function of the
document. A component dropped from the page **after** an incident named it
makes the next edit of that incident fail, deliberately: the editor renders the
stale entry flagged and checked so it is one click from being dropped, and the
alternative is a page quietly publishing a component it no longer has.

Monitor-detected outages are **derived at read time, never copied**: each
snapshot assembly reads recent `incidents` for the page's monitors and folds
them into the timeline alongside the authored ones. There is no second
incidents table for automatic ones and no reconciliation between two records
of the same outage. A maintenance window in progress **repaints how a day is
drawn, never the uptime number** — `buildDays` computes each day's up/down
state and the 90-day percentage from rollups first, and
`applyMaintenanceRepaint` only overwrites today's display state afterward, so
a component that stayed up throughout a maintenance window still shows as up
in its history.

The public route, `GET /public/status/:pageId`, is mounted on the gin **root**,
outside `/api`, on purpose: `/api` carries `auth.Middleware`, `RequireScopes`,
`RateLimitTokens` and `RequireActiveLicense` by virtue of where it is mounted,
and a public route living there would need four exemptions — each one a hole a
later change to any of those four could widen back open. A missing page, an
unpublished page, and a page on the wrong host all answer the same 404;
inventing a distinct code for "exists but unpublished" would itself leak that
the page exists. A lapsed licence or a tier lacking `status_pages` answers 200
with `available:false` and a `reason`, never a 403 or a blank page — the
reader is a member of the public who can do nothing about either condition and
deserves an explanation, not a browser error.

**The instance is resolved from `X-Forwarded-Host`, not `Host`.** The public
page is server-rendered by `web/`, and the SSR fetch cannot set `Host` at all:
it is a forbidden header name and undici drops it silently, so the Go server
saw `server:8080` and every status page 404'd on every deployment. `web/`
forwards the visitor's host in `X-Forwarded-Host` (and their address in
`X-Forwarded-For`, or the whole deployment shares one rate-limit bucket), and
`publicStatusInstance` honours that header **only when `c.RemoteIP()` is in
`TRUSTED_PROXIES`** — it selects a tenant, so an untrusted peer must not be
able to name one. It uses `RemoteIP()` and not `ClientIP()` deliberately: the
latter is reconstructed from the very headers being judged.

**A host naming no slug falls back to the sole instance on a non-cloud
deployment.** `hostSlug` requires `<slug>.vantage.<tld>`; a self-hosted install
at `vantage.acme.com` or an IP has no slug and would otherwise 404 forever. It
has exactly one instance, resolved with the same count-then-read bootstrap
uses, cached alongside the slug lookups. More than one instance is a 404, not a
guess. A host that *does* name a slug which does not exist stays a 404 —
falling back there would serve one tenant's page on another's address.

Assembled snapshots are cached in Redis for **30 seconds**, keyed per
instance and page, and every authoring write (`UpdateStatusPage`,
`DeleteStatusPage`, and every incident mutation) invalidates its page's entry
immediately rather than waiting out the TTL — an operator posting an update
mid-incident should not wonder for half a minute whether it saved. A cache
miss, on Redis being down or on any read error, degrades to reassembly rather
than an error: the status page has to survive the outage it exists to report.
The public endpoint itself is rate limited to **120 requests per minute per
client address**, answering 429 with `Retry-After`, on the same fixed-window
pattern as `RateLimitTokens`.

**`TRUSTED_PROXIES` is load-bearing for that limiter, not cosmetic.** `main.go`
always calls `gin.SetTrustedProxies` with it; left unset, gin trusts no proxy
and `c.ClientIP()` falls back to the direct peer address — which, sat behind a
real reverse proxy, is the proxy's own address for every visitor. The rate
limiter then keys on one address for the whole fleet of readers, and the first
burst of legitimate traffic during an incident is what trips it. Set it to the
proxy's real address or CIDR, not merely a private range guess; the shipped
compose file and Helm chart default it to the RFC1918 ranges, which is right
for their own bundled reverse proxy but wrong the moment another one is
inserted in front. The same setting also decides the address recorded in
`audit_logs` and `console_sessions`.

### Agent self-update

`UpdateAgentCmd` carries a target version and Gitea base URL; the agent
downloads and replaces itself, from
`<gitea>/vantage/vantage-agent/releases/download/<tag>/…`.

**That repository path is compiled into the agent, not sent to it**, and it
changed when the agent moved out of this repository. Agents built before that
move look for `mrhid6/vantage`, where releases are no longer published, so the
push-button update in the UI fails for them with a 404. They are not stranded:
`/install`, `/install.ps1`, `/update` and `/update.ps1` are generated **here**,
at request time, so re-running the update one-liner on a host moves it onto a
build that knows the new address, after which self-update works again.

The ordering matters. This server must be deployed with the new paths *before*
the one-liner is any use, because it is this server that hands out the URL.

The six generators — two install scripts, two update scripts,
`GET /api/agent/latest-version` in `services/dispatch.go`, and the tag lookup
inside each — all name that repository. They must agree with wherever
`agent-release.yml` actually publishes, and nothing checks that they do.

### Backup and restore

`vantagectl` is `vantage-ctl` now — its own repository, with the command still
named `vantagectl`. It is not a subcommand of `server` and never was: `server`
imports the whole control-plane dependency graph, `spf13/cobra` has no business
in a process that also terminates gRPC streams, and above all **it has to run
when the control plane does not**. A backup or restore against a database with
no server container alive is the normal case, so it cannot be a mode of the
binary whose failure is the reason you reached for it.

Almost none of its logic is in that repository either: dump, restore, verify,
manifest and fingerprint are `shared/backup` in `vantage-shared`, and
`internal/cmd` holds only argument parsing and operator-facing output. That
split is what would let `server` import `shared/backup` later — a scheduled
in-process backup, say — without a second implementation to keep in sync.
`shared/cryptobox` is the same move one layer down: it is the **single**
AES-256-GCM implementation, and `server/internal/services/crypto.go` delegates
to it rather than keeping its own copy that `shared/backup` would otherwise have
had to duplicate to decrypt a probe value during `verify`.

**The one thing this repository owes it is `backup.ciphertextFields`**, which
lives in `vantage-shared` and mirrors `server/internal/models` **by hand** —
`shared/` is a separate module and cannot import it. The map naming each
collection's `*_enc` fields (`keys`, `secrets`, `auth_providers`,
`console_sessions`) must change in the same commit as any of those bson tags,
and that commit is now in a different repository from the tags it tracks. Wrong
field names are **silent**: `verify`'s live probe finds no ciphertext and
reports "this database stores no ciphertext yet", so the one gate that catches
what a key fingerprint cannot becomes a no-op. `settings` is deliberately in
neither that map nor `CiphertextCollections()` — its ESO read token is a
SHA-256 hash, not ciphertext.

The chart's optional backup CronJob runs that image, and **`backup.image` has
no default** — the template `fail`s without one rather than guessing, so a
cluster set up before the move keeps working until someone changes the value.
It is `gitea.hostxtra.co.uk/vantage/vantage-ctl:latest` now, was
`mrhid6/vantage/vantagectl:latest`; `chart-release.yml`'s render checks name the
new path.

The rest — the refusals around `--force` and `--confirm-db`, the key
fingerprint, live collection enumeration, verbatim index replay, the `.partial`
rename — is documented in `vantage-ctl`.

### API tokens and OpenAPI

A token is `vt_` plus 32 random bytes hex, shown once at creation and stored
only as sha256 — the same shape as `servers.agent_token_hash` and the ESO read
token, and for the same reason: nothing downstream ever needs the plaintext
back. It belongs to the user who created it, and its role can never exceed
theirs; see the `api_tokens` note under MongoDB Collections for how that stays
true across a demotion rather than only at issuance. Scopes are nine
resources — `servers`, `keys`, `secrets`, `workflows`, `monitors`, `vulns`,
`workloads`, `status`, `settings` — each split into `:read` and `:write`, with `:write`
satisfying a `:read` requirement on the same resource so a caller does not have
to hold both. Any signed-in member may mint and revoke their **own** tokens —
there is no `RequireRole` on `POST /tokens` or `DELETE /tokens/:id` — because
`roleRank` already bounds what a token can do to no more than its creator's
own role, so a member cannot use a token to reach past themselves. Owner and
admin additionally see and revoke every token in the instance — `all=true` on
`GET /tokens` is gated by `elevated()` in `api/tokens.go`, and
`RevokeAPIToken` in `services/tokens.go` checks the same owner/admin condition
before letting a revoke target somebody else's token — neither is a
`RequireRole` middleware.
The `settings:read`/`settings:write` entries in `routeScopes` govern a
**token-authenticated** caller reaching the token endpoints — `RequireScopes`
no-ops entirely for a cookie session — so they say nothing about which human
role may call these routes with a session; that is `roleRank` and `elevated()`,
not the scope map. Expiry is optional per token; `settings.api_token_max_days` caps how
far out a new one may be set, and when that cap is set a token requested with
no expiry is refused rather than silently capped — the policy governs
issuance only and never reaches back to invalidate a token already issued.
`RateLimitTokens` holds every token to 600 requests/minute in a Redis fixed
window, answering 429 with `Retry-After`; cookie sessions are untouched; it
exists so a runaway script cannot take an instance down, not as the general
API rate-limiting project some future ticket might build.

**The UI calls them API keys and lives at `/tokens`, not on `/settings`.**
The page is reachable at **every** role, which is the whole reason it is a page:
`/settings` is owner|admin throughout, so a card there hid a capability every
member has. `settings.api_token_max_days` stays on `/settings` because it is
instance policy rather than one person's credentials, and that split is exactly
what lets the page be ungated. The label differs from the identifiers on
purpose — the collection is `api_tokens`, the prefix is `vt_`, the routes are
`/api/tokens`, and renaming a published endpoint to match a nav label would
break every script already written against it.

`server/internal/api/docs/openapi.json` is a **generated, committed** OpenAPI
3.1 document — `swag v2` reading `@…` annotations off the handlers — served at
`GET /api/openapi.json` and rendered as a reference page by a vendored Scalar
bundle at `GET /api/docs`. `server-deploy.yml` regenerates it on every server
build and runs `git diff --exit-code` against the committed copy: a handler
whose annotation drifted from its code fails CI rather than shipping a
reference that lies. Scalar is vendored (`scalar.standalone.js`, served from
`GET /api/docs/scalar.js`) rather than pulled from a CDN, because the
reference page has to work on an air-gapped install with no outbound access at
all — the same requirement licence verification already meets.

### The public host

**vantage.hostxtra.co.uk is not served by this repository.** The marketing site
and its contact-form service are `vantage-site`; the documentation at `/docs` is
`vantage-docs`; the HQ console at `vantage-hq.hostxtra.co.uk` is
`vantage-admin`. Each carries its own compose fragment, and the host composes
them on top of this one:

```bash
# self-hosted install — the control plane and nothing else
docker compose -f deploy/docker/docker-compose.yml up -d

# vantage.hostxtra.co.uk — every repository's fragment together
docker compose \
  -f vantage/deploy/docker/docker-compose.yml \
  -f vantage-site/deploy/docker-compose.yml \
  -f vantage-docs/deploy/docker-compose.yml \
  -f vantage-admin/deploy/docker-compose.yml \
  up -d
```

`docker-compose.site.yml` is gone from this repository: every service it held
now lives with the repository that builds it. The self-hosted exclusion used to
be a rule about which file a service went in; it is the repository boundary now.

**The reverse proxy in front is shared and belongs to none of them.** On
vantage.hostxtra.co.uk that is an Nginx Proxy Manager, and its routing spans
repositories: `/docs` to `vantage-docs` — a location that must sort **above**
the catch-all — and everything else on that host to `vantage-site`. A
self-hosted install needs its own; see the compose note below for what it must
route.

One coupling survives the split and is easy to miss: the marketing site's
`/start` form posts account signups **straight to `vantage-admin`**, not to
anything here. The control plane is not touched until the customer later creates
a cloud instance from the portal.

### Signup and verification

Signup is **account-first**: it creates an HQ account and an unverified `customer_user` in admin's own database, nothing in the control plane. Only after a customer later creates a cloud instance from the portal (`POST /api/instances`, see Admin REST API) does an org, or rather an `instance`, come to exist — provisioned by `cloudprov`, with the owner's password hash copied from the HQ user rather than shared. `site_pending_signups` is gone; the contact-form service has no signup flow at
all, and lives in another repository besides.

- The token is 32 random bytes; only its **SHA-256 hash** is stored, so a leaked database yields no working links.
- Links expire after 24 hours (`VerifyWindow`).
- An unverified sign-in gets a distinct "check your email" error rather than the generic auth failure, because the address is already known to be theirs.
- If sending the verification email fails, the freshly inserted `customer_user` (and account, on first signup) is rolled back rather than left stranded holding the unique index on email.
- Rate limited per client IP, plus a honeypot field.

An account is a team, not a person. `customer_users.account_role` is `owner`,
`admin` or `member` — the same three words as the control plane's roles, on
purpose. Owners and admins invite people, create instances and grant instance
access; billing is owner-only.

An invitation creates a `customer_user` with an **empty password hash**, which
cannot authenticate, and the invitee sets their own at `/accept-invite`. An
inviter-chosen password would be a shared credential to every instance that
person is later granted. `GET /auth/verify` therefore peeks before it consumes:
a token belonging to a passwordless row answers `{"needs_password":true}` and is
left unspent.

### Email

`shared/mail` is the only email system. It owns the SMTP conversation, the RFC
5322 envelope and the look of every message; the control plane, `vantage-admin`
and `vantage-site` each import it and none of them builds a subject line, a MIME
part or a colour.
Before this existed the transport was copied three times, and the copies had
already diverged once — the 465-implicit-TLS fix landed in one of them while
the others silently delivered nothing.

`Sender` is a value, not a singleton: `server/internal/notify` builds one per
notification channel from the channel document in Mongo, while `vantage-site`'s
service builds one at boot and `vantage-admin` holds one in its own
`internal/mail.Default`, alongside its other boot-time singletons. Callers only ever see typed methods —
`SendVerification`, `SendExpiring`, `SendMonitorAlert`, `SendEnquiry` and the
rest, grouped by owner into `account.go`, `licence.go`, `billing.go`,
`monitor.go` and `contact.go`.

Every message is `multipart/alternative`, so each one is two templates:
`templates/<name>.html.tmpl` and `.txt.tmpl`, embedded with `go:embed`. They
define `subject`, `title`, `pill` and `body`; `layout.html.tmpl` and
`layout.txt.tmpl` provide the chrome and the helper templates (`p`, `lead`,
`button`, `well`, `note`, `rows`, `chip`) that the bodies compose. One template
set is parsed per message rather than one big set, because every message
defines those same four names. **`subject` is defined in the txt file only** —
`html/template` would escape an ampersand in an instance name and mail clients
show subjects verbatim.

`shared/mail/render_test.go` renders all of them and fails if a template exists
that no case covers, which is the only thing standing between a mistyped field
and a boot-time panic — the templates are parsed in `init()`.

### Shared provisioning

`shared/provision` (`instance.go`, `slug.go`, `user.go`) holds the slug rules, reserved names and instance/user creation logic that both `server` and `admin/internal/cloudprov` need, so there is no longer a second copy to drift: `cloudprov.CreateInstance` calls straight into it to create a control-plane instance and its owner from a customer request.

### Grants project, they do not federate

Granting someone access to a cloud instance writes a real control-plane `users`
row through `cloudprov`, with `auth_source: "hq"` and `hq_user_id` set. The
instance authenticates it exactly as it authenticates anyone else, with **no
runtime dependency on admin**. Revoking deletes that row — the control plane has
no disabled state, and a row that exists is a row that can sign in.

`instance_members` in admin's database is only admin's _index_ of those
projections; the control-plane row is the access. That is why a failed
`instance_members` insert unwinds the projection, and why the boot backfill can
rebuild the index from the control plane but never the other way round.

**Self-hosted instances are never projected into.** All three mutating member
endpoints refuse when `deployment != cloud`.

The HQ password is the single source of truth for every `hq`-sourced row.
`PUT /api/account/password` rehashes and has `cloudprov` copy the hash to every
projected row; propagation is best-effort, and `admin/internal/hqsync` compares
and repairs every 15 minutes. It is **its own package rather than a pass inside
`inject`** — `inject` writes three licence fields and nothing else, and that
narrowness is what makes admin's reach into the control plane reviewable.

The control plane refuses to change an `hq`-sourced user's role or delete it
(`services.ErrHQManaged`, 409). `web/` shows those rows read-only with a link to
the portal, but the API is the boundary; the UI is a courtesy. There is no local
password-change endpoint at all, so there is no competing writer for the hash.

**A rename moves the host, and the licence does not care.** `PUT
/api/instances/:id/name` re-derives the slug from the new name through
`provision.RenameSlug` — the same rules that named the instance at creation —
and writes the control plane first, because `instances.slug`'s unique index is
what settles a race between two accounts reaching for one name. A taken slug is
a refusal, not an `acme-2`: creation appends a counter because any free slug
will do, and a rename is a request for one specific host. A licence binds the
instance UUID, so nothing is reissued and Paddle is not called. The old host
keeps resolving for up to 60s (`instancehost.go`'s cache, which admin cannot
reach into), and `km_session` is host-only, so the customer signs in again on
the new address — the portal says so rather than redirecting them into a login
screen with no explanation. The 24h cooldown lives on `admin_instances.renamed_at`
because it is admin's policy; staff bypass it and must not write the field.

---

## Auth and Orgs

- **Bootstrap** — first run has no users. `GET /auth/bootstrap-status` drives `/setup`, `POST /auth/bootstrap` creates the first org plus its owner.
- **Local auth** — email + password (bcrypt), `POST /auth/login`.
- **Auth providers** — configured _per instance_ in `auth_providers`, any number of them, each named and independently enabled. Issuer, client ID and an encrypted client secret per provider. `/auth/oidc/:providerId/start` → `/auth/oidc/:providerId/callback`. Presets (Entra, Google, Okta, GitHub) are a Go table in `server/internal/auth/presets.go` and expand to a real issuer on save, so nothing downstream knows a preset existed. GitHub is OAuth2 rather than OIDC and takes its own branch, requiring an address that is both primary **and** verified — an unverified address is not proof of control.
- **Local login** — `settings.local_login_enabled`, a `*bool` because absent must mean enabled; a plain bool would disable password sign-in fleet-wide at upgrade. `services.CheckLockout` refuses any change leaving neither local login nor an enabled provider, and is enforced in the service layer so the settings path and the provider path cannot disagree.
- **Sessions** — opaque 32-byte hex ID in the `km_session` cookie, session body stored in Redis with a 24h TTL.
- **Roles** — `owner`, `admin`, `member`. `/api/settings` and `/api/org/*` require owner or admin.
- **Host/org guard** — `APP_ROOT_LABEL` (default `vantage`) defines the app root label. A request to `<slug>.vantage.<tld>` resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.

Unique indexes are a **security property**, not an optimisation. `users` is
unique on `(instance_id, email)` — one address is one user _within_ an instance,
and the same address may hold a user in several instances, because an account's
people are projected into each instance they are granted. This is sufficient only
because **every lookup by email is scoped by instance**; there is deliberately no
unscoped lookup anywhere, and adding one would let the login path return an
arbitrary one of several matching users. Instance slug, settings instance and ESO
token hash remain globally unique.

---

## gRPC API

```protobuf
service Vantage {
  rpc Register(RegisterRequest)              returns (RegisterResponse);
  rpc SyncKeys(SyncRequest)                  returns (SyncResponse);
  rpc UploadGeneratedKey(UploadKeyRequest)   returns (UploadKeyResponse);
  rpc ReportUpdates(ReportUpdatesRequest)    returns (ReportUpdatesResponse);
  rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse);
  rpc ReportInventory(InventoryReport)       returns (InventoryReportResponse);
  rpc SyncMonitors(SyncMonitorsRequest)      returns (SyncMonitorsResponse);
  rpc ReportChecks(ReportChecksRequest)      returns (ReportChecksResponse);
  rpc CommandStream(stream AgentMessage)     returns (stream ServerCommand);
}
```

`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.

`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`, `OpenProxyCmd`, `PingCmd`, `RefreshWorkloadsCmd`, `ControlWorkloadCmd`,
`WorkloadLogsCmd`.

**`PingCmd` is a liveness beat, and it is not redundant with gRPC keepalive.**
The server sends one every 20s on an otherwise idle command stream; the agent
treats 70s of silence as a dead stream and reconnects. Keepalive cannot do this
job behind an L7 proxy: the agent's HTTP/2 connection terminates at the proxy,
which answers pings on its own behalf, so a control-plane pod that dies leaves
the agent blocked in `Recv` on a stream that never delivers another message and
never errors — commands dispatched into it are silently lost while `SyncKeys`
keeps succeeding and the fleet list still shows the server `active`. The agent's
watchdog arms only **after** it has seen a first ping, so an older server that
sends none is treated as working rather than put into a reconnect loop.

Key-state polling stays on the 30s `SyncKeys` interval. Full message definitions live in `vantage-shared`, in `proto/vantage/v1/vantage.proto` beside the `grpc/pb` types it describes.

---

## REST API

Unauthenticated:

```
GET  /healthz            /readyz             # liveness / readiness probes
GET  /install            /install.ps1        # dynamic agent install scripts
GET  /update             /update.ps1
GET  /auth/bootstrap-status
POST /auth/bootstrap     /auth/login  /auth/logout
GET  /auth/me
GET  /auth/providers                          # {local_enabled, providers:[{id,name,preset}]} — no issuer, client ID or secret
GET  /api/secrets/:group/values               # bearer token (ESO)
```

Session-authed under `/api`:

```
servers      GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
             POST /servers/:id/{generate-key,update-agent,apply-updates}
keys         GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
             POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
             POST /steps/{import,seed-defaults,parse} · GET /steps/usage
             GET,POST /workflows · GET,PUT,DELETE /workflows/:id
             POST /workflows/:id/run · GET /workflows/:id/runs
             GET /runs/:runId · POST /runs/:runId/cancel
             GET /runs/:runId/servers/:serverId/logs[/stream]
monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
             GET /monitors/:id/{incidents,uptime}
channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
             POST /secrets/:group/reveal · DELETE /secrets/:group/:key
console      POST /console/connect · GET /console/tunnel (websocket)
vulns        GET /vulnerabilities · GET /vulnerabilities/summary
             POST /vulnerabilities/rescan                          (owner|admin)
             POST,DELETE /vulnerabilities/:id/accept               (owner|admin)
             GET /servers/:id/vulnerabilities · GET /servers/:id/packages
             GET /packages/search?name=
             GET,POST /vuln-rules · PUT,DELETE /vuln-rules/:id     (owner|admin)
workloads    GET /workloads · GET /servers/:id/workloads
             POST /servers/:id/workloads/refresh
             POST /servers/:id/workloads/:wid/action        (owner|admin)
             GET /servers/:id/workloads/:wid/logs           (owner|admin)
status-pages GET,POST /status-pages · GET,PUT,DELETE /status-pages/:pageId  (owner|admin)
             GET,POST /status-pages/:pageId/incidents
             PUT,DELETE /status-pages/:pageId/incidents/:incidentId
             POST /status-pages/:pageId/incidents/:incidentId/updates
audit        GET /audit
agent        GET /agent/latest-version
settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
licence      GET /license · POST /license                (POST: self-hosted only)
org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
providers    GET,POST /auth/providers · PUT,DELETE /auth/providers/:id
             POST /auth/providers/:id/{test,ack-notice} · GET /auth/presets  (owner|admin)
tokens       GET /tokens · GET /tokens/scopes · POST /tokens · DELETE /tokens/:id
             GET /openapi.json · GET /docs
```

`GET /license` reports `deployment`, and **`POST /license` answers 409 `cloud_managed` when it is `cloud`**. A cloud instance's licence is written by `admin/internal/inject` straight into the database and never through this endpoint, so the refusal cannot break injection — it only stops a customer pasting over a licence they do not own. `web/` hides the paste form and points at the HQ portal instead, but as with `hq`-managed users, the API is the boundary and the UI is the courtesy.

`POST /license` is also in `licenceExemptPaths`: pasting a valid licence has to work while the current one is expired, because it is the way out of degraded mode.

Free exists in both deployments, so it is no longer cloud-only by construction. The one-Free-per-account rule is enforced per account **and deployment**, in `licensing.checkFreeLimit` and in `createInstance`'s friendly pre-check — the two must stay scoped identically, because a pre-check stricter than the issuer refuses what would have worked. A self-hosted Free licence is claimed with `POST /api/instances/:id/claim-free` after the install is linked; the metered server count and per-instance feature toggles come from the instance's `entitlement`, which a licence snapshots at issue time (see the `catalogue`/`entitlements` note under MongoDB Collections).

---

## Vantage HQ (`vantage-admin`)

Lives in its own repository now, with its own session cookie (`admin_session`),
its own database and its own console at `vantage-hq.hostxtra.co.uk`. Its REST
surface, its Paddle integration and its `plans`/`catalogue`/`entitlements`
model are documented there, not here.

What matters on this side is the small set of things it does to the control
plane, each of which this codebase enforces:

- **Licences are injected, not pasted.** `inject` writes three licence fields
  straight into `instances`. `GET /license` reports `deployment`, and **`POST
  /license` answers 409 `cloud_managed` when it is `cloud`** — the refusal
  cannot break injection, it only stops a customer pasting over a licence they
  do not own.
- **Instances and owners are provisioned through `shared/provision`**, the same
  code path bootstrap uses, so there is one implementation of the slug rules
  and reserved names rather than two — see "Shared provisioning".
- **Members are projected, not federated** — `users.auth_source: "hq"` with
  `hq_user_id` set, refused for role changes and deletion by
  `services.ErrHQManaged`. See "Grants project, they do not federate", which is
  the contract in full.
- **`FREE_INSTANCE_REAP_AFTER` must match admin's value.** Admin names the date
  in its warning emails; this side performs the delete, because it is the only
  service that knows which collections carry `instance_id`.

## MongoDB Collections

`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `server_workloads` · `api_tokens` · `status_pages` · `status_incidents` · `migrations`

Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`.

Notes that are not obvious from the structs:

- `servers.agent_token_hash` stores SHA-256 of the token, never plaintext. `pre_reg_token` is cleared after `Register()`. `status` is `pending` → `active` on register, `offline` when `last_seen` passes the threshold (swept every 2 min).
- `servers.inventory` holds the latest metrics snapshot with separate `metrics_at` / `static_at` timestamps.
- `keys.private_key_enc` and `passphrase_enc` are AES-256-GCM; the JSON form exposes only `has_private_key` / `has_passphrase`.
- `assignments.revoked_at: null` means active. Revocation is soft, preserving audit history.
- `workflow_runs.steps_snapshot` freezes the resolved steps so editing the library never rewrites history.
- `console_sessions.token_consumed_at` is set atomically to enforce one-time use.
- `auth_providers.provider_id` is a short random identifier, not the Mongo `_id`: it appears in the callback URL a customer pastes into their IdP, and an `_id` there would publish a database key. `callback_notice` marks a provider migrated from the old single-provider shape, whose redirect URI therefore changed.
- `workflow_log_lines` is keyed `(run_id, server_id, seq)` — the index is not an optimisation, every read is a range scan over it. `workflow_log_seq` holds one counter document per `run_id/server_id`, which is what lets two pods interleave into one ordered log. Neither carries `instance_id`: they are reached only through a run, and a run is already scoped.
- `users.auth_source` is `local`, `oidc` or `hq`. An `hq` user was projected from a Vantage HQ account and carries `hq_user_id`; HQ owns its role, password and existence.
- `server_packages` holds a server's whole package set in **one** document, not one per package. The hash already established that something changed, so a report is a single atomic upsert with no delta logic to get wrong; ~2000 packages is ~150KB, well inside the 16MB limit. `scan_pending` lives on the document rather than in memory so a leader handover cannot lose it.
- `vuln_findings` is unique on `(instance_id, server_id, cve_id, package_name)`. That key is what makes a rescan an idempotent upsert rather than a duplicate factory, and what lets `first_seen` survive one. An empty `fixed_in` means no vendor fix exists — a real state, never "not vulnerable".
- `vulndb_meta` is a singleton and deliberately carries **no** `instance_id`: the vulnerability database is a property of the deployment, not a tenant. Same reasoning as `migrations`, and the reason it is absent from `services.ScopedCollections`.
- **`services.ScopedCollections` is the canonical registry of tenant-scoped collections**, and `scopedCollectionsForPurge` derives instance deletion from it rather than keeping a second list. A new collection carrying `instance_id` must be added there or its rows outlive the instance.
- `api_tokens` stores only `sha256` of the token, like `servers.agent_token_hash`. A token's effective role is `min(user.role, token.role)` **recomputed per request**, so demoting somebody demotes their tokens; deleting the user deletes them. Scopes are enforced from a map keyed on the registered gin route pattern, and `AssertScopeMapComplete` **fails boot** when an `/api` route is missing from it — a route added without an entry would otherwise be silently unreachable by every token.

Admin's database is its own and lives with `vantage-admin` — `accounts`,
`admin_instances`, `licenses`, `subscriptions`, `plans`, `catalogue`,
`entitlements`, `paddle_events`, `staff_users`, `customer_users`,
`instance_members`, `admin_audit`. Nothing here reads or writes it. Note in
particular that `instance_members` is admin's _index_ of the control-plane
`users` rows it projected, not the authority for them: the row in this
database is the access.

### Migrations

`services.RunMigrations()` runs at boot, recording markers in `migrations`:

- `0001_default_org_backfill`
- `0002_settings_org_backfill` (must run before 0003 — 0003 can create a `default` org, which pushes 0002 into its ambiguous multi-org branch)
- `0003_missed_org_scopes`
- `0005_auth_providers` — copies each `instance_oidc` document into `auth_providers`, ciphertext verbatim rather than decrypted and re-encrypted, so it does not need `KEY_ENCRYPTION_KEY` and cannot strand an instance's SSO configuration that has none set.

Index builders (`EnsureAuthIndexes`, `EnsureSettingsIndexes`) are fatal on failure; `EnsureSecretIndexes` and `EnsureWorkflowIndexes` only warn.

---

## Agent Lifecycle

The agent is `vantage-agent` now; its internals are documented there. What the
control plane depends on:

### Config file

Linux `/etc/vantage/config.yaml`, Windows `%ProgramData%\vantage\config.yaml`.
Directory `0700`, file `0600`.

```yaml
server_url: "vantage.yourdomain.com:9090"
server_id: "<uuid>"
pre_reg_token: "<token>" # removed after first successful Register()
agent_token: "" # written by agent after Register()
poll_interval: 30s
tls: true
```

### Startup

```
1. Load config
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
4. Enter SyncKeys poll loop (default 30s)
```

### Poll loop

```
1. SyncKeys(server_id, agent_token, agent_version)
2. Non-Linux hosts stop here — the key-management steps below are Linux-only; a Windows agent's other work (workflow steps, inventory, OS updates, workloads) runs from the goroutines started above, not from this loop
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600
```

### Install

Linux: systemd unit at `/etc/systemd/system/vantage-agent.service`,
`Restart=always`, runs as root — written by the install script this server
generates, not shipped as a file. Windows: MSI built by `vantage-agent`'s CI
(WiX), or its `installer/setup.ps1` registering the agent as a service via NSSM.

---

## Server Registration Flow

1. **Add Server** in the UI calls `POST /api/servers/new`, which generates a `server_id` and a pre-registration token (TTL 1 hour, single-use).
2. The UI shows a one-liner:
    ```bash
    curl -fsSL https://vantage.yourdomain.com/install | \
      bash -s -- --server-id=<id> --token=<token>
    ```
    Windows gets the `/install.ps1` equivalent.
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
4. The server flips to `active` on first sync.

`/install` is served dynamically, injecting the latest agent version from the Gitea API.

---

## Environment Variables (server)

| Name                       | Required        | Notes                                                                                                                                                                                                                                                                                                              |
| -------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GRPC_HOST`                | **yes**         | `host:port` agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC.                                                                                                                                                         |
| `MONGO_URI`                | no              | default `mongodb://localhost:27017`                                                                                                                                                                                                                                                                                |
| `MONGO_DB`                 | no              | default `vantage`                                                                                                                                                                                                                                                                                                  |
| `REDIS_USERNAME`           | no              | Redis 6+ ACL user. Leave empty for a legacy `requirepass` instance — go-redis then sends AUTH with one argument instead of two                                                                                                                                                                                     |
| `REDIS_PASSWORD`           | no              | empty for an unauthenticated Redis                                                                                                                                                                                                                                                                                 |
| `REDIS_ADDR`               | no              | default `localhost:6379`                                                                                                                                                                                                                                                                                           |
| `KEY_ENCRYPTION_KEY`       | yes in practice | 64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials.                                                                                                                                                                                                         |
| `GUACD_ADDR`               | no              | default `guacd:4822`                                                                                                                                                                                                                                                                                               |
| `PROXY_ADVERTISE_HOST`     | no              | default `server`; the hostname guacd resolves the control plane by, handed to guacd as the relay's address. Wrong here and every console session fails at connect                                                                                                                                                |
| `PROXY_LISTEN_HOST`        | no              | default `0.0.0.0`; the interface the ephemeral relay listener binds                                                                                                                                                                                                                                                |
| `APP_ROOT_LABEL`           | no              | default `vantage`; wrong value disables the host/session org guard                                                                                                                                                                                                                                                 |
| `TRUSTED_PROXIES`          | no              | comma-separated CIDRs or addresses gin trusts for `X-Forwarded-For`. Empty means trust none: `c.ClientIP()` falls back to the direct peer address, which behind a real reverse proxy is that proxy's own address for every visitor — the public status page's per-address rate limit then keys on one address for the whole fleet of readers. Also the address recorded in `audit_logs` and `console_sessions`. Compose and the Helm chart default it to the RFC1918 ranges, right for their own bundled proxy and wrong the moment another one is inserted in front |
| `POD_IP`                   | no              | this pod's own address, set by the Helm chart from the downward API. **Takes precedence over `PROXY_ADVERTISE_HOST`** — a console relay listener belongs to one replica, and a Service address names all of them                                                                                                     |
| `VANTAGE_MIGRATE_ONLY`     | no              | run schema setup (migrations, index builders, default-step seeding) and exit without serving. `GRPC_HOST` is not required in this mode. Set by the Helm chart's pre-upgrade Job                                                                                                                                                          |
| `VANTAGE_SKIP_MIGRATIONS`  | no              | serve without running schema setup, on the assumption a Job already did. Set by the chart's Deployment whenever `server.migrationJob.enabled`. Unset under Compose, where one process still migrates and then serves                                                                                                                    |
| `VANTAGE_TRIVY_DB_REF`     | no              | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for an air-gapped install, or to avoid the anonymous ghcr rate limit                                                                                                                                                                                    |
| `VANTAGE_VULNDB_DISABLED`  | no              | `true` disables the vulnerability database puller and scan loop entirely. Findings already written are still served, and still shown as stale                                                                                                                                                                       |
| `FREE_INSTANCE_REAP_AFTER` | no              | duration past a Free licence's expiry before the instance and all its data are deleted. **Empty disables the reaper, and empty is the default.** Set to `336h` on vantage.hostxtra.co.uk only — a self-hosted deployment must never reap. Must match admin's value, which only names the date in warning emails |

### Ingress (Helm, Traefik)

`ingress.enabled` publishes **two** hostnames, because the two audiences arrive over different protocols:

| Values                                       | Route                                                                                       |
| -------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `ingress.web.host` (+ `web.extraHosts`)      | browsers → `web:3000`                                                                        |
| `ingress.api.paths` (when `api.enabled`)     | `/api`, `/auth` → `<release>-server:8080`, bypassing the Next proxy                          |
| `ingress.grpc.host`                          | agents → a dedicated `<release>-server-grpc` Service on 9090, annotated `serversscheme: h2c` |

**`ingress.web.host` is normally a wildcard.** `*.vantage.example.com` is the per-tenant instance namespace — `APP_ROOT_LABEL` resolves the instance from the label. A Kubernetes wildcard host matches **exactly one** label, so it does not match the apex, and here that is correct rather than a gap: `vantage.hostxtra.co.uk` is the marketing site, which lives in `vantage-site` and which this chart does not deploy. `extraHosts` is for a genuine second name; adding the apex to it would put the control plane on the marketing host. Every host in the list gets identical paths.

**`ingress.api.enabled` routes `/api`, `/auth`, `/public`, `/install*` and `/update*` straight to the server, and it is not optional.** It defaults to **true** and the chart refuses to render with it off, because `web` proxies nothing: with those prefixes unrouted the UI loads and every request it makes 404s against Next. The value survives only for an installation whose own terminator sits in front of this ingress and routes them there instead. Traefik derives router priority from rule length, so `PathPrefix(/api)` outranks the catch-all `/` with no priority annotation needed.

**The gRPC route needs its own Service.** The server terminates no TLS; it speaks plain h2c and always has, with TLS terminated by whatever sits in front. Traefik will not use h2c to a backend unless the *Service* says so, and that annotation applies to every port on the Service — so annotating the shared two-port `<release>-server` would force h2c on its HTTP port too.

**`server.env.grpcHost` is not derived from `ingress.grpc.host`, and the chart refuses to render if they disagree.** Agents dial whatever `grpcHost` says, and it is baked into every install one-liner; left pointing at the in-cluster Service while agents arrive through the ingress, every install succeeds and every agent then fails to connect, with nothing in the control plane explaining why. Guessing at the port (443? 9090?) would be worse than stopping.

TLS is `ingress.tls.secretName` / `grpcSecretName` (pre-existing certificates) **or** `certResolver` (Traefik ACME). Setting neither while `tls.enabled` produces a TLS router with no certificate, so Traefik serves its self-signed default — which looks valid and is trusted by nothing. NOTES.txt warns on install rather than the chart failing, since it is a real if unusual choice behind another terminator.

---

**Neither compose file ships a reverse proxy, and both now need one.** `web:3000` serves the UI only; a request to `/api` there is a Next 404. Route `/api`, `/auth`, `/public`, `/install`, `/install.ps1`, `/update`, `/update.ps1` to `server:8080` and everything else to `web:3000` — on vantage.hostxtra.co.uk that is the Nginx Proxy Manager already in front, and it is what a self-hosted install has to configure before the UI works at all.

`deploy/docker/docker-compose.yml` runs four services: `redis`, `guacd`, `server` (8080 + 9090), `web` (3000). MongoDB is external. **That is the whole of a self-hosted install**, and it is now the only compose file here. vantage.hostxtra.co.uk adds three fragments from three other repositories — `vantage-site` (`site` 3003, `sitesvc` 8082), `vantage-docs` (`docsite` 3005) and `vantage-admin` (`admin` 8083, `adminsite` 3004) — composed together as shown under "The public host".

`LICENSE_SIGNING_KEY` appears in **no compose file in this repository**, and must never be added to one: admin is the only signer, and it now lives in `vantage-admin` along with its own compose fragment. `docker-compose.yml` should never mention `admin` or `adminsite` again — the separation used to be a rule someone had to remember, and is the repository boundary now. `server` reads `REDIS_ADDR`/`REDIS_USERNAME`/`REDIS_PASSWORD` so a Kubernetes install can point at a managed Redis; the base compose still hardcodes an unauthenticated `redis:6379` for it.

---

## Security

- gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
- Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's `0600` config.
- Pre-registration tokens are short-lived (1 hour) and single-use.
- AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
- Console session tokens are one-time; RDP credentials are consumed on tunnel open.
- ESO read token stored as a SHA-256 hash and rotatable.
- Unique indexes on `(instance_id, email)`, instance slug, settings instance and the ESO token hash are load-bearing for tenant isolation. So is the absence of any unscoped lookup by email.
- `authorized_keys` written `0600`, owned by root. The agent runs as root because it must.
- Every mutating API path writes an audit event.

---

## Frontend

Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in `web/lib/guacamole-common.js`.

All four apps are **one visual system**, anchored on the logo navy. What differs between them is which end of it they stand on:

| App             | Ground                     | Accent                           | Themes                      |
| --------------- | -------------------------- | -------------------------------- | --------------------------- |
| `web/` (here)   | `--ground` dark, `#071628` | `#5b9be8`                        | dark only, locked           |
| `vantage-site`   | token-based            | `#0b2a58` light / `#5b9be8` dark | light + dark                |
| `vantage-admin`  | token-based            | `#0b2a58` light / `#5b9be8` dark | light + dark, light default |
| `vantage-docs`   | token-based            | `#0b2a58` light / `#5b9be8` dark | light + dark, light default |

Only the first row is in this repository. The other three are listed because
the palette is one system across all four front ends regardless of which
repository they sit in.

`vantage-site`'s `web/app/globals.css` is the **origin**: it is the only one
carrying both light and dark values in full, and the other three copy its token
blocks **verbatim** — same names, same values. `web/` here holds the same tokens
but only the **dark** values, since it does not switch. **Nothing enforces the
match, and the four now sit in four repositories, so "change them in the same
commit" is not merely unenforced but impossible.** The drift window is however
long it takes to push four times. Treat a token change as an announcement
rather than a refactor.

`vantage-docs` is the one place the tokens are not consumed through Tailwind:
everything below its token block maps Docusaurus's `--ifm-*` variables onto
them. Docusaurus already stamps `data-theme` on `<html>`, which is the selector
the dark block keys on, so the built-in toggle needed no wiring. The rule holds
all the same — no rule in that file outside the token blocks carries a hex. Its
one concession is a favicon, which must, for the same reason the email layout
must: a browser tab cannot read a token.

Tailwind in all of them maps `var(--…)` references only, so **no component in
any of them may carry a hex value**. The names differ per app on purpose,
because each has its own subject: `vantage-site` calls the semantic three
`--up`/`--pend`/`--down` for monitor state, `vantage-admin` aliases them to
`valid`/`warn`/`expired` for licence state, and `web/` here to
`success`/`warning`/`danger`. Same colours, honest names on each side.

`web/` is locked to dark and the HQ console defaults to **light**, and that
pairing is the point: an operator with both open should never mistake one for
the other before clicking Reissue. Now that both are drawn from the same palette
the distinction rests **entirely** on the ground, so do not make dark the HQ
console's default and do not give `web/` a light theme. State never reads by
colour alone in either: every pill carries a distinct shape and a text label.

There is a **fifth** copy, and it is the one people forget:
`shared/mail/templates/layout.html.tmpl` in `vantage-shared` carries `web/`'s
dark values as literal hex. Email clients support neither `var()` nor a reliable
`prefers-color-scheme`, so the token indirection is simply not available there —
an email is read before the recipient clicks through to the control plane, and
the two should not look like different products.

`web/` stores its tokens as **RGB channel triplets** with the hex in a trailing comment, and derives `--token: rgb(var(--token-rgb))` from them. That is not a style preference: the console leans on Tailwind's opacity modifiers (`bg-danger/10`, `border-accent/50`, `ring-accent/30`) in a way the other two do not, and `<alpha-value>` only compiles against channels. Keep the hex comments — they are what lets the four token blocks still be diffed by eye, which matters more now that they cannot be diffed by `git`. `web/` also adds three tokens the marketing site has no use for: `--accent-hover` and `--down-hover` (it brightens with a CSS `filter`, which a Tailwind colour token cannot do) and `--well`, the floor beneath the ground for install one-liners, key blobs and run logs — surfaces showing machine output rather than interface.

`web/` is locked to dark and the HQ console defaults to **light**, and that pairing is the point: an operator with both open should never mistake one for the other before clicking Reissue. Now that both are drawn from the same palette the distinction rests **entirely** on the ground, so do not make dark the HQ console's default and do not give web/ a light theme. State never reads by colour alone in either: every pill carries a distinct shape and a text label. The same argument applies one level in: the **staff** masthead sits on `--panel-2` with a `STAFF` chip, so staff and customer screens are not identical either.

`web/` collapses Tailwind's radius scale — `md`, `lg` and `xl` all resolve to the shared 4px — rather than rewriting the ~140 `rounded-lg` classes across its pages. Every one of them meant "a panel corner", and `tailwind.config.ts` is now where that decision lives. `rounded-full` is untouched: status dots and pills still need it.

The HQ console's own shell, its `/staff/pricing` page and the catalogue coverage
ledger are documented in `vantage-admin`. They are still built from these
tokens, which is the only reason they are mentioned here at all.

| Route                                                           | Purpose                                                                 |
| --------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `/setup`                                                        | First-run bootstrap: create the first org and owner                     |
| `/login`                                                        | Local or OIDC sign-in                                                   |
| `/`                                                             | Fleet dashboard                                                         |
| `/servers`, `/servers/new`, `/servers/[id]`                     | Fleet list, install one-liner, server detail (keys, inventory, updates) |
| `/servers/[id]/console`                                         | Browser SSH/RDP/VNC session                                             |
| `/keys`, `/keys/[id]`                                           | Key library; assign and revoke per server                               |
| `/workflows`, `/workflows/[id]`, `/workflows/[id]/runs[/runId]` | Compose, run, and follow live logs                                      |
| `/steps`                                                        | Reusable step library                                                   |
| `/monitors`, `/monitors/new`, `/monitors/[id][/edit]`           | Checks, uptime, incidents                                               |
| `/secrets`, `/secrets/[group]`                                  | Vault                                                                   |
| `/tokens`                                                       | Personal API keys — reachable at **every** role, unlike `/settings`     |
| `/audit`                                                        | Audit log                                                               |
| `/settings`, `/settings/notifications`, `/settings/license`     | Members, OIDC, alerts, retention, ESO token · channels · licence        |

**The sidebar is grouped, and the groups are the nav's structure rather than
decoration.** `web/components/Sidebar.tsx` holds `navGroups` — Fleet, Access,
Automation, Instance — each rendered with a mono small-caps heading and a
hairline rule above it, the first group excepted. Grouping is by what the
operator is doing, not by which service answers: SSH keys, vault secrets and
API keys sit together under Access because all three are credentials. A group
whose every item is `adminOnly` disappears **whole**, heading and rule
included, for a member — a labelled section with nothing under it reads as
something that failed to load rather than something withheld.

**`/settings` is one page, not a section.** Members and single sign-on used to
live at `/settings/instance` with their own sidebar entry; they are now the
**Access** group at the top of `/settings`, above **Monitoring** and
**Integrations**. Splitting "who can sign in" away from "how this instance
behaves" made two half-pages and a nav entry called Instance that no one could
distinguish from Settings. `next.config.ts` keeps a permanent redirect from the
old path. The cards live in `web/components/settings/` rather than in the page,
which is also where the `Field`/`inputClass` pair the three of them share now
lives — one copy instead of the three that existed while they were apart.

---

## CI/CD — Gitea Actions

### `server-deploy.yml` — triggered on every push to `main`

Builds and pushes **two** images to the Gitea container registry: `server` and `web`. That is now the whole of this workflow. Everything else that was once built here belongs to the repository that owns it — `vantage-site`, `vantage-docs`, `vantage-admin` and `vantage-ctl` each publish their own, and `vantage-agent` cuts releases rather than images.

Note that despite the name, **this workflow does not deploy** — it only builds and pushes. There is no SSH step. Rolling images out is a separate manual step on the host:

```bash
# self-hosted
cd /opt/vantage && docker compose -f deploy/docker/docker-compose.yml pull && \
  docker compose -f deploy/docker/docker-compose.yml up -d --remove-orphans

# vantage.hostxtra.co.uk — all four repositories' fragments, see "The public host"

```

**Each image only rebuilds when its own inputs changed.** A `git diff` against `github.event.before` decides, which is why the checkout uses `fetch-depth: 0` — the default shallow clone has one commit and nothing to diff — and why `git` is installed in the `docker:dind` container. The mapping follows the build contexts exactly:

| Image                        | Rebuilds when                    |
| ---------------------------- | -------------------------------- |
| `server`                     | `server/`, `go.work`             |
| `web`                        | `web/` only                      |

**No path in this table names `shared/` any more**, and no fan-out rule replaces
it: `vantage-shared` is an external module pinned per service, so a service
rebuilds when its own `go.mod` moves, which its own directory pattern already
matches. What that removes is the failure where a `shared/` edit rebuilt three
images and one of them was not ready; what it adds is that nothing here reminds
you a pin is stale.

Every Go build in these workflows writes a netrc from `REGISTRY_USER` +
`RELEASE_TOKEN` before it runs, and sets `GOPRIVATE=gitea.hostxtra.co.uk/*`.
There is exactly **one** such place left here: `server-deploy.yml`'s single job.
The other repositories each carry their own, one per job, because jobs do not
share a filesystem — `vantage-agent`'s `msi` job is the one to remember, because
it is Windows, where Go reads `%USERPROFILE%\_netrc` and not `.netrc`. The docker builds pass
it on as `--secret id=netrc`, never a build arg. **`RELEASE_TOKEN` needs read
access to the `vantage` org** on top of its existing scopes; without it every Go
build fails at `go mod download` with a 404 on the module, which reads like a
missing tag rather than a missing permission. A change to the workflow file rebuilds everything, since
a build arg is baked into the image. So does anything that leaves no
trustworthy base commit: a manual `workflow_dispatch`, a new branch, or a
force-push whose old head is gone.

The gap this leaves: **changing a repo variable pushes no commit, so nothing rebuilds.** After editing `HQ_URL`, run the workflow manually — that is what `workflow_dispatch` is there for. Base images also stop being refreshed on a service nobody touches; a periodic manual run covers that.

### `chart-release.yml` — validates on every chart change, publishes on `chart/v*` tags

Two jobs' worth of work in one, split by trigger. Any push or PR touching `deploy/chart/` lints the chart and renders it four ways: defaults, a multi-replica install, external Redis and MongoDB, and a set of values that **must be refused**. That last one is the point — every safety rail in this chart is a template `fail`, and `helm lint` happily accepts a chart whose templates never execute, so only rendering proves they still fire.

Publishing runs only on a `chart/v*` tag, to the Gitea Helm registry at `/api/packages/<owner>/helm/api/charts`. **`Chart.yaml` is the source of truth for the version**; the tag only selects which one to publish, and a tag that disagrees with `Chart.yaml` fails rather than stamping over it — the alternative leaves the repository disagreeing with what shipped. A version already in the registry is rejected by Gitea, which is intended: published chart versions are immutable.

The registry host comes from `github.server_url`, so it cannot drift from the instance the workflow is running on. It authenticates with `REGISTRY_USER` + **`RELEASE_TOKEN`** — the pair `server-deploy.yml` actually uses for `docker login`. `REGISTRY_PASSWORD` is listed in the secrets table below but set by no workflow; passing an unset secret yields an empty password and Gitea answers `401 Failed to authenticate user`, which reads like a scope problem on a token that was never sent. The publish step therefore checks both are non-empty before it calls curl. `RELEASE_TOKEN` needs `write:package` in addition to `write:release`.

```bash
helm repo add vantage https://gitea.hostxtra.co.uk/api/packages/mrhid6/helm
helm install vantage vantage/vantage --version 0.1.0
```

### Tagging

```bash
git tag chart/v0.1.0 && git push origin chart/v0.1.0   # helm chart package
git push origin main                                   # server + web deploy
```

Only two things are tagged here now. Elsewhere: `vantage-agent` keeps the
`agent/v*` prefix, because the control plane greps release tag names for exactly
that string; `vantage-ctl` dropped its prefix for a bare `v*`, because nothing
reads it programmatically.

### Secrets / variables

| Name                    | Type     | Value                                                                                                                                                                                                                                                      |
| ----------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `RELEASE_TOKEN`         | Secret   | Gitea API token. Needs `write:release` (agent releases), `write:package` (container images and the Helm chart) and **read access to the `vantage` org**, which is where the private `vantage-shared` module lives — without that last one every Go build fails at `go mod download` with what looks like a missing tag. **This is the only token any workflow authenticates with** — `docker login`, the chart publish and the module netrc all pair it with `REGISTRY_USER` |
| `REGISTRY_USER`         | Secret   | Gitea username. Must own `RELEASE_TOKEN`, or basic auth is rejected                                                                                                                                                                                        |
| ~~`REGISTRY_PASSWORD`~~ | —        | **Not used.** Named here historically; no workflow reads it. Referencing an unset secret yields an empty password and a `401 Failed to authenticate user` that looks like a token scope problem. Use `RELEASE_TOKEN`                                        |
| `DOCKER_HOST`           | Variable | registry host used for image tags                                                                                                                                                                                                                          |
| ~~`API_URL`~~           | —        | **Gone.** `web` proxies nothing and holds no address for the control plane. `/api`, `/auth`, `/public`, `/install*` and `/update*` must be routed to `server:8080` by the reverse proxy in front of both; everything else goes to `web:3000`. One variable that could name the wrong host was one request path too many — pointed at the marketing site, `/public/status/…` answered a Next 404 indistinguishable from a status page that does not exist. |
| `HQ_URL`                | Variable | optional; browser URL of the HQ portal, baked into `web` so an `hq`-sourced member links to where they are managed. Empty on self-hosted, which renders a plain label instead.                                                                             |

`SITE_URL`, `SITE_API_URL`, `SITE_CONTACT_EMAIL`, `ADMIN_API_URL`, `ADMIN_ENV`,
`DOCS_URL`, `DOCS_BASE_URL`, `APP_URL` and every `PADDLE_*` name are set on the
repository that bakes them in — `vantage-site`, `vantage-docs` or
`vantage-admin` — and none of them is read by anything here. Two are set in
**two** repositories and must agree: `ADMIN_API_URL` (`vantage-site` bakes it
into the marketing site's signup form, `vantage-admin` into its own console) and
`SITE_URL`.


---

## Design Decisions

- **gRPC for agent traffic** — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
- **Outbound-only agents** — no inbound ports on managed servers, works behind NAT.
- **Poll for keys, push for commands** — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
- **Atomic `authorized_keys` rewrite** — temp file plus `os.Rename()`; a machine that dies mid-write keeps the old file.
- **Fingerprint diffing before write** — no disk churn on unchanged state.
- **Soft revocation** — `revoked_at` rather than deletes; preserves audit history.
- **Run snapshots** — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
- **Monitors run in two places** — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
- **Redis for sessions only** — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
- **guacd for console** — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
- **`org_id` on every document** — isolation enforced at the query layer, not by separate databases.
- **root only** — manages `/root/.ssh/authorized_keys`; no per-user key management.
- **Windows agents cover the fleet-management path** — register, heartbeat, run
  steps, report inventory, OS updates through the Windows Update COM API, and
  workloads (services plus containers, with control and logs). They still do no
  `authorized_keys` management, and no package inventory or CVE matching: a
  Windows agent never calls `ReportPackages`, so no `server_packages` document
  exists for it and it reports no package inventory at all — a different,
  earlier state than the `unsupported` a Linux distribution reaches when its
  family has no security feed.
- **Both `server` and `web` scale horizontally** — see "Running more than one server replica" below. `web` holds nothing; `server` holds per-agent state that is routed between replicas over Redis rather than duplicated.
- **Deletion lives in the control plane** — admin sends the warnings because it knows the billing address; the control plane performs the delete because it is the only service that knows which collections carry `instance_id`. Mirroring that list into admin would drift, and a drift there deletes the wrong rows.

## graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.

Rules:
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).