# Vantage

A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.

A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.

---

## Architecture Overview

```
┌──────────────────────────────────────────────┐
│  Next.js 16 Frontend (web, :3000)            │
│  servers · keys · workflows · monitors       │
│  secrets · audit · console · settings        │
└───────────────┬──────────────────────────────┘
                │ REST + cookie session
┌───────────────▼──────────────────────────────┐
│  Go Backend (server)                         │
│  :8080 REST (gin)   :9090 gRPC (agents)      │
│  MongoDB (state) · Redis (sessions)          │
│  monitor scheduler · workflow runner         │
│  guacd tunnel proxy for browser console      │
└───────────────┬──────────────────────────────┘
                │ gRPC (TLS) — outbound from agent only
┌───────────────▼──────────────────────────────┐
│  Go Agent (per server, Linux + Windows)      │
│  polls SyncKeys · CommandStream              │
│  rewrites authorized_keys (Linux only)       │
│  runs workflow steps · monitors · inventory  │
└──────────────────────────────────────────────┘
```

Multi-tenancy: every domain document carries `org_id`, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (`<slug>.vantage.<tld>`).

---

## Repository Structure

```
vantage/
├── agent/
│   ├── cmd/main.go                  # flags: -generate-key
│   └── internal/
│       ├── checker/                 # monitor check execution
│       ├── config/                  # config.yaml load/save
│       ├── exec/                    # workflow step execution
│       ├── grpc/                    # client + generated pb
│       ├── inventory/               # CPU/mem/disk collection (linux/other)
│       ├── keys/                    # authorized_keys read/diff/write
│       ├── monitors/                # agent-run monitor loop
│       ├── sync/                    # poll loop + command stream
│       └── updates/                 # OS package update check/apply
├── server/
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # REST handlers
│       ├── auth/                    # local, OIDC, session, middleware, orghost
│       ├── checker/                 # server-run monitor checks
│       ├── db/                      # mongo connect + Col()
│       ├── grpc/                    # gRPC server + generated pb
│       ├── models/                  # MongoDB documents
│       ├── monitorsched/            # server-side monitor scheduler
│       ├── notify/                  # channel dispatch: http, discord, slack, telegram, smtp
│       └── services/                # business logic + migrations
├── web/                             # the application UI (authenticated)
│   ├── app/(app)/                   # authed routes
│   ├── app/login, app/setup         # unauthed routes
│   ├── components/                  # ui/, workflows/, monitors/, Sidebar
│   └── lib/                         # api client, guac console, query client
├── site/                            # public marketing site
│   ├── app/                         # one directory per route
│   ├── components/                  # Nav, Footer, Logo, InstrumentPanel, forms
│   ├── assets/                      # image sources, not served
│   └── Dockerfile                   # same shape as web/: standalone, node, 3000
├── sitesvc/                         # public form: contact mail only
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # contact
│       └── store/                   # Mongo connect helper
├── admin/                           # licensing authority: the only signer
│   ├── cmd/main.go                  # boot: two Mongo connections, reconciler, HTTP
│   ├── cmd/adminctl/                # staff-add; deliberately has no HTTP surface
│   └── internal/
│       ├── api/                     # customer + staff handlers, route table
│       ├── auth/                    # staff, HQ customer and cloud-owner sessions
│       ├── inject/                  # licence write path into the control plane
│       ├── cloudprov/               # instance write path: creates instances + owners
│       ├── licensing/               # Issue, LinkInstance, Relink
│       ├── mail/                    # admin's boot-time shared/mail Sender
│       └── models/                  # accounts, instances, licences, plans
├── adminsite/                       # staff + customer console (vantage-hq)
│   ├── app/(customer)/              # overview, instance, link, billing
│   ├── app/(staff)/staff/           # operations, accounts, licences, plans, audit
│   ├── components/                  # AppBar, PageHeader, PageFrame, InstanceRecord
│   └── lib/                         # api client, session guards, formatters
├── docsite/                         # user documentation (Docusaurus, static)
│   ├── docs/                        # getting-started, vantage, hq, reference, operations
│   ├── src/css/custom.css           # site/'s tokens, copied, mapped onto --ifm-*
│   ├── sidebars.ts                  # authored by hand, not autogenerated
│   └── nginx.conf                   # serves the build under /docs
├── shared/                          # imported by server, sitesvc and admin
│   ├── mail/                        # the one email system: transport + tmpl templates
│   ├── license/                     # payload, sign, verify, trusted keys, plans
│   ├── models/                      # Instance, User, Settings
│   └── cmd/lkctl/                   # issue and inspect licences by hand
├── proto/vantage/v1/vantage.proto
├── installer/                       # Windows: setup.ps1, nssm.exe, WiX .wxs
├── deploy/                          # docker-compose.yml, agent.service
└── .gitea/workflows/                # agent-release.yml, server-deploy.yml
```

---

## Subsystems

### SSH keys

Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites `/root/.ssh/authorized_keys` atomically. Keys can also be generated _on_ a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.

### Workflows

A library of reusable **steps** (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into **workflows** targeting a set of servers. Running one snapshots the resolved steps into a `WorkflowRun`, then dispatches `RunStepCmd` over the agent command stream. Step stdout/stderr streams back as `StepOutputChunk` and is written to MongoDB (`workflow_log_lines`, one document per line); the UI streams it live. Steps support `on_failure: stop|continue|retry`, per-run env passed between steps via `output_env`, and a per-run workspace directory the agent cleans up at the end.

Default steps are seeded per org at boot (`SeedDefaultSteps`) from `VANTAGE_DEFAULT_STEPS_DIR`, which `server/Dockerfile` bakes to `/opt/default-steps` from the repo's `default_steps/`. Deliberately **not** under `/data` — that is a bind mount, so the library would be editable from the host. Adding a step there means committing a file and rebuilding, which is why `default_steps/` is in the `server` rebuild trigger. **Steps with `source: "default"` are read-only**: `UpdateStep`/`DeleteStep` refuse with `ErrDefaultStep` (409), because seeding rewrites them on every boot, so an edit would silently revert and a delete would come back. `web/` mirrors this — the step modal opens read-only, Delete is hidden, and the designer's per-step script override is `readOnly` for a default library step — but as elsewhere, the API is the boundary and the UI is the courtesy. Seeding writes straight to the collection rather than through `UpdateStep`, so the guard does not lock out the seeder. Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever).

### Scheduled workflows

A workflow may carry `schedule{enabled, cron, tz}` — standard **5-field** cron
and an IANA zone name, both validated at save time. `next_run_at` is
**persisted on the document, not held in memory**: a leader handover between
computing an occurrence and firing it would otherwise lose it or fire it twice,
the same argument that put `workflow_log_seq` in MongoDB.

`server/internal/workflowsched` ticks every 30s inside the **existing**
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched` and the sweepers —
one role, one lock. **The atomic claim, not the lock, is what prevents a double
fire**: the `UpdateOne` matches on the document *and* its current `next_run_at`
while setting the recomputed one, so a second process reaching the same workflow
matches nothing and does nothing. The lock only makes it cheap.

`workflowsched` **must not import `services`** — `services` already imports it
for `SetSchedule`'s call to `NextOccurrence`, and Go has no cycles.
`TriggerWorkflow` and `LogEvent` are therefore injected as `workflowsched.Deps`
from `main.go`. Firing goes through the same `TriggerWorkflow` a person uses,
with `"schedule"` as the actor, so there is no second dispatch path and the run
detail page needed no changes.

`main.go` imports `_ "time/tzdata"`, and it is load-bearing: `server/Dockerfile`
runs on `scratch`, which ships no zone database, so without it
`time.LoadLocation("Europe/London")` fails and every schedule silently falls
back to UTC — an hour wrong for half the year, in the direction nobody notices
until a maintenance window lands in business hours. It works on a developer
machine either way, which is exactly why it gets forgotten.

Skips are recorded and surfaced, not just logged: past the 1h grace window is
`missed`, an active run is `already_running`, and a schedule that no longer
parses is disabled rather than left spinning the loop every 30 seconds forever.

### Server tags and workflow targeting

A server carries `tags map[string]string` — lowercase `[a-z0-9_-]`, key ≤32,
value ≤64, 20 per server, `sys:` reserved. **There is no `tags` collection**: a
tag is a property of a server, not an entity, so `KnownTags` aggregates over
`servers` rather than reading a registry that would need reference counting to
know when a tag stopped existing. `PUT /api/servers/:id/tags` replaces the whole
map — last-write-wins over a small map beats merge semantics between two people
editing one server. The index is `{instance_id: 1, "tags.$**": 1}`, wildcard
because the queried key is chosen by the user at request time and cannot be named
in advance; `EnsureServerIndexes` warns rather than being fatal, since a missing
index degrades tag filtering to a scan of a small collection and is no reason to
refuse to serve the fleet list.

`services.ResolveTargets` is the **single** answer to which servers a workflow
touches — the run path and validation both go through it, so the readout and the
dispatch cannot disagree. It is the distinct union of `target_server_ids` and
`target_tags` (AND across keys), ordered by the fleet rather than by the
arguments, so two runs naming the same servers differently are still comparable
line by line. **An empty selector matches nothing** on purpose: "matches
everything" turns a cleared field in the designer into a fleet-wide run. Both
empty is `ErrNoTargets` (400), not a success over zero servers. Offline servers
are **not** filtered out — the dispatcher already answers 503 per server, and a
patch run that silently omits an unreachable machine is worse than one that
visibly fails on it.

**Both halves of the selector are edited in `EditWorkflowModal`** — the named
servers in a `DualListBox`, the tag rows directly beneath it — and saved
together by one `updateWorkflow`. The designer's Targets panel is **read-only**:
it reports the count and the tags and links to Edit. Splitting the two halves
across two screens meant a workflow's reach was decided in two places with no
one view showing both.

`web/lib/targets.ts` **duplicates the match logic in TypeScript** to draw the
resolved count without a round trip, since the browser already holds the fleet.
It is a second implementation of `UnionTargets` / `MatchesTags` and must change
in the same commit as the Go one — the same shape of hazard as the mirrored
token blocks. It is a shared module rather than inline in a component because
the logic had already been written twice, and the second copy — the workflows
list — counted `target_server_ids` alone, so a **tag-only workflow reported zero
targets** while running fine.

The server picker is a hand-built two-pane list, not `<select multiple>`: a
native multi-select paints its selected rows with the platform highlight colour,
which cannot be restyled across browsers and lands outside the token palette on
a dark ground.

### Monitors

HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.

### Notification channels

Per-org outbound destinations: `webhook`, `smtp`, `discord`, `slack`, `telegram`. Monitors reference channels by ID. Channels are testable from the UI.

### Secrets vault

Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via `secret_refs` (injected as env at execution), and read by Kubernetes External Secrets Operator via `GET /api/secrets/:group/values` using a bearer token whose SHA-256 hash is stored in settings.

### Browser console

`POST /api/console/connect` mints a one-time session token; `GET /api/console/tunnel`
upgrades to a WebSocket and proxies to **guacd** using `github.com/wwt/guac`.

guacd never dials the managed server. The server binds a single-use ephemeral
listener, pushes `OpenProxyCmd` down the agent's command stream, and the agent
opens a `ProxyStream` and relays the connection from its own **`127.0.0.1`** —
the host is hardcoded agent-side, so the control plane can name only a port.
This is what makes the console work on Vantage Cloud, where the customer's
server is behind NAT on a private address. It also means the console now
**requires a live agent** on every deployment: `consoleConnect` answers 409
`agent_offline` rather than hanging.

**guacd's Service is headless on purpose.** The server resolves `GUACD_ADDR` to
build the allow-list of sources permitted to claim a relay listener; a ClusterIP
resolves to the Service's virtual address while guacd connects from its *pod*
IP, so every relay connection is rejected and every session dies with
`waiting for guacd: i/o timeout`. Compose is immune — there the name resolves to
the address that connects.

SSH connections authenticate with a stored private key; RDP/VNC credentials are
encrypted, single-use, and consumed when the tunnel opens. None of them reach
the agent — the session is negotiated end-to-end between guacd and the target
daemon, so the agent relays bytes it cannot read.

### Running more than one server replica

An agent's `CommandStream` terminates on exactly **one** server process. Every
piece of coordination below exists because of that single fact: with several
replicas, the process asked to do something to an agent is almost never the
process holding that agent's stream.

`server/internal/bus` is the Redis message bus that closes the gap. It adds no
infrastructure — Redis was already required for sessions — and it is **not
optional on a single-replica deployment**: dispatch takes the bus path always,
so the code running in production is the code running everywhere, rather than a
rare cross-pod branch that only fails under load.

| Concern                | How it crosses replicas                                                                                                                                                                       |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Which pod owns an agent | `vantage:agent:<server_id>` holds the owner's node ID with a 30s TTL, renewed every 10s. `Dispatcher.IsConnected` is an `EXISTS` on it                                                        |
| Sending a command       | published to `vantage:cmd:<server_id>`; the owner pod acks on `vantage:ack:<command_id>`. **Request/ack, not a queue** — a command whose owner died must fail loudly (503) rather than queue. The envelope carries `node`, the presence holder resolved at publish time, and a pod ignores envelopes addressed elsewhere: the channel is a fan-out, and during a reconnect a half-open stream's pod is still subscribed. Unaddressed, it could ack first and queue the command onto a dead stream — the operator told it worked, the agent never seeing it. Presence renewal is owner-only (`RenewPresence`) for the same reason: a blind `SET` let the stale pod steal the key back every 10s |
| Step results            | the owner pod publishes to `vantage:res:<command_id>`; the pod driving the run subscribes **before** dispatching, or a fast agent answers into a channel nobody has joined                    |
| Step output             | never crosses. The dispatch envelope carries the secret mask list, so the owner pod masks and writes lines itself — unmasked bytes stay off the bus                                          |
| Console relay           | **not routed to the owner pod at all.** A `ProxyStream` is its own HTTP/2 request and an L7 proxy balances requests, not connections, so it does not follow the command stream — the listener therefore cannot be bound in advance. Whichever pod receives the stream binds it and announces **its own** address on `vantage:proxyaddr:<proxy_id>`; `vantage:proxypending:<proxy_id>` (30s, consumed atomically) is what authorises the claim, and the failure reason comes back on `vantage:proxyend:<proxy_id>` |
| Background jobs         | `bus.RunAsLeader` — one Redis lock named `housekeeping`                                                                                                                                       |

**Workflow logs are in MongoDB** (`workflow_log_lines`, one document per line,
with a `workflow_log_seq` counter document per run/server). Two pods write the
same log concurrently — the run's pod emits markers, the agent's pod emits
output — so ordering only means anything if both draw sequence numbers from the
same counter. `StepRun.log_offset` is that sequence number now, not a byte
offset. Writes are batched (128 lines or 250ms) and capped: 8 KB per line,
200k lines per server-run, after which one final `[vantage] log truncated`
marker is written and the rest is dropped. Without that cap a `yes` in a step
is a database incident. **Nothing writes to `/data` any more**, which is why
`server.persistence` now defaults to off and `VANTAGE_WORKFLOW_LOG_DIR` is gone.

**Shutdown order is load-bearing.** `main` traps SIGTERM, stops gRPC
(`GracefulStop`, 10s cap) and only then drains HTTP. Each `CommandStream`
handler releases its agent's presence claim on return, so a killed process
leaves `vantage:agent:<server_id>` behind for the rest of its 30s TTL — during
which other replicas dispatch to a pod that has exited and the caller sees
`agent offline` for a perfectly healthy agent. Draining HTTP first would hold
those claims for the length of the drain, which is why gRPC goes first. The
chart's `server.terminationGracePeriodSeconds` (30s) must stay above the
10s + 10s the stop sequence needs, or the kubelet SIGKILLs mid-shutdown and the
handling buys nothing.

The agent side of the same failure: `runCommandStream` resets its backoff only
after a stream that survived `streamHealthyAfter`. `connectAndHandleStream`
returns an error on *every* stream end, healthy ones included, so without that
reset the backoff only ever climbed — an agent pinned itself at the ceiling
after a handful of ordinary deploys and stayed there. The ceiling is 30s, not
minutes, because while the stream is down the agent still polls `SyncKeys` and
still reads as `active` in the fleet list while answering no commands at all.

**The leader lock is not an optimisation.** N replicas each running the monitor
scheduler means each check fires N times, each incident notification reaches the
customer N times, and each hourly rollup is written N times; N reapers race to
purge the same Free instance. `monitorsched`, `StartReaper`, `StartLogSweeper`,
`StartAuditSweeper` and the offline sweep therefore all run inside one
`RunAsLeader("housekeeping", …)` — one role, one lock. Each takes a context
cancelled the instant leadership is lost, and must return when it is.

Redis rather than a Kubernetes `Lease` so Compose takes the identical path: one
implementation to reason about, not two.

Two deployment requirements come with `replicaCount > 1`: every replica must
share **one** Redis (a per-pod Redis partitions the bus and every agent looks
offline to two thirds of the fleet), and `POD_IP` must be set — the chart does
it from the downward API — because `PROXY_ADVERTISE_HOST` names the Service, and
a Service cannot address the one pod holding a console listener.

### Inventory and OS updates

Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`).

### Package inventory and CVE findings

Agents report their installed packages hourly; the control plane matches them
against distribution security feeds and raises findings that link to the
existing `ApplyUpdatesCmd` patching path. Gated by the `vuln_scanning` licence
feature, **checked at collection rather than display** — an ungated instance
stores no inventory, and storage is the expensive half.

**Matching uses distribution feeds, never NVD version ranges.** Distributions
backport security fixes without changing the upstream version: Ubuntu's
`openssl 3.0.2-0ubuntu1.15` is patched against CVE-2023-0286 while NVD still
calls 3.0.2 vulnerable. Matching on NVD would report a fully patched fleet as
critical, and once the first report is mostly wrong nobody reads the second.
`trivy-db` is those feeds pre-merged; `server/internal/vulndb` pulls it as an
OCI artifact to an ephemeral directory. Version comparison is bought from
`go-deb-version`/`go-rpm-version`/`go-apk-version` because dpkg epochs, `~`
sorting before the empty string, and `rpmvercmp` are each a silent false
negative waiting to happen.

**Only the leader matches.** `ReportPackages` upserts the list and sets
`scan_pending`; it does not scan. `vulnsched` runs inside the existing
`bus.RunAsLeader("housekeeping", …)` and does the matching, because otherwise
every replica needs the ~50MB database resident and a database refresh has N
replicas rescanning the same fleet and sending N digests. The tick is also the
digest's batch boundary, which is what makes "one message, not five hundred"
structural rather than a debounce someone maintains.

Findings are **never deleted when a package is patched** — the state moves to
`fixed`, so "what did we remediate last quarter" stays answerable. Acceptance
requires a reason and an expiry, and reopens automatically: permanent dismissal
is where risk goes to be forgotten. An unsupported distribution reports
`status: unsupported`, never "0 findings"; claiming clean when the truth is
unknown is the same lie as a silently stale database, which is why
`vulndb_meta.pulled_at` is on screen rather than only in a log.

**`server/Dockerfile`'s runtime stage is `scratch`, so it carries an explicitly
copied `/tmp`.** The scheduler unpacks the database to a temporary directory,
and a scratch image has none — the failure is `vulnsched: temp dir: stat /tmp:
no such file or directory`, logged once at boot while every other subsystem
runs normally, so the only symptom is a fleet that never reports a finding.

Two environment variables: `VANTAGE_TRIVY_DB_REF` mirrors the artifact for
air-gapped installs, and `VANTAGE_VULNDB_DISABLED` switches the puller and
scheduler off entirely.

### Agent self-update

`UpdateAgentCmd` carries a target version and Gitea base URL; the agent downloads and replaces itself.

### Marketing site and sitesvc

`site/` is a separate Next.js app built exactly like `web/` — `output: "standalone"`, run by Node in a `node:26-alpine` image, listening on `3000` and published as `3003`. The contact form posts to `sitesvc`; account signup posts to `admin` (`NEXT_PUBLIC_ADMIN_API_URL`), which creates an HQ account, not an org — the control plane is not touched until the customer later creates a cloud instance from the portal.

`adminsite/` is built the same way and published as `3004`, served at **`vantage-hq.hostxtra.co.uk`** — deliberately _outside_ `*.vantage.hostxtra.co.uk`, because that namespace is per-tenant instance subdomains and `APP_ROOT_LABEL` resolves an org from the label before `vantage`. It shares `site/`'s design tokens verbatim (see Frontend below) and, unlike `web/`, does **not** proxy through a Next rewrite: the browser calls `admin` directly, so `ADMIN_API_URL` must be browser-reachable. Authenticated requests work cross-origin only because both hosts share the registrable domain `hostxtra.co.uk`, which keeps `admin_session`'s `SameSite=Lax` cookie in play.

**`ADMIN_ORIGIN` must list every browser origin that calls admin — currently two**: `https://vantage-hq.hostxtra.co.uk` for the console, and `https://vantage.hostxtra.co.uk` because the marketing site's `/start` form posts account signups to admin directly. It is comma-separated. A missing origin does not produce a 403: `cors()` simply omits the `Access-Control-Allow-Origin` header and still answers the preflight `204`, so the browser blocks the request and **admin logs nothing at all**. Symptom is a CORS preflight failure on an endpoint that works fine under curl.

`sitesvc/` (port `8082`) now owns only the contact flow:

| Form    | Endpoint            | Effect                                                                  |
| ------- | ------------------- | ----------------------------------------------------------------------- |
| Contact | `POST /api/contact` | Emails `support@hostxtra.co.uk`, `Reply-To` the sender. Nothing stored. |

Account signup lives in `admin` instead (`POST /auth/signup`, `GET /auth/verify?token=…`) — see Signup and verification below.

`site`, `sitesvc`, `admin` and `docsite` are deliberately **excluded from the self-hosted deployment**: `deploy/docker-compose.yml` mentions none of them, and they live in `deploy/docker-compose.site.yml` instead.

### Documentation site

`docsite/` is the user-facing documentation — Docusaurus 3 in docs-only mode (`routeBasePath: "/"`, no blog), one version tracking `main`, search indexed at build time by `@easyops-cn/docusaurus-search-local` so nothing external is keyed or called. It documents the **product**, not the codebase: this file remains the contributor's map, and the two are allowed to differ in altitude but not in fact. Five sections — Getting started, Vantage, Vantage HQ, Reference, Operations — with `sidebars.ts` authored by hand so ordering is a decision rather than a filename accident.

Unlike the three Next apps it builds to static files, so its runtime stage is `nginx:alpine-slim` rather than Node, and it listens on `80`. See the compose note below for the `/docs` prefix, which is the one thing about it that is easy to get wrong.

```bash
# self-hosted install — no marketing site, no sitesvc
docker compose up -d

# vantage.hostxtra.co.uk — control plane plus the public site
docker compose -f docker-compose.yml -f docker-compose.site.yml up -d
```

### Signup and verification

Signup is **account-first**: it creates an HQ account and an unverified `customer_user` in admin's own database, nothing in the control plane. Only after a customer later creates a cloud instance from the portal (`POST /api/instances`, see Admin REST API) does an org, or rather an `instance`, come to exist — provisioned by `cloudprov`, with the owner's password hash copied from the HQ user rather than shared. `site_pending_signups` is gone; sitesvc no longer has a signup flow at all.

- The token is 32 random bytes; only its **SHA-256 hash** is stored, so a leaked database yields no working links.
- Links expire after 24 hours (`VerifyWindow`).
- An unverified sign-in gets a distinct "check your email" error rather than the generic auth failure, because the address is already known to be theirs.
- If sending the verification email fails, the freshly inserted `customer_user` (and account, on first signup) is rolled back rather than left stranded holding the unique index on email.
- Rate limited per client IP, plus a honeypot field.

An account is a team, not a person. `customer_users.account_role` is `owner`,
`admin` or `member` — the same three words as the control plane's roles, on
purpose. Owners and admins invite people, create instances and grant instance
access; billing is owner-only.

An invitation creates a `customer_user` with an **empty password hash**, which
cannot authenticate, and the invitee sets their own at `/accept-invite`. An
inviter-chosen password would be a shared credential to every instance that
person is later granted. `GET /auth/verify` therefore peeks before it consumes:
a token belonging to a passwordless row answers `{"needs_password":true}` and is
left unspent.

### Email

`shared/mail` is the only email system. It owns the SMTP conversation, the RFC
5322 envelope and the look of every message; `server`, `admin` and `sitesvc`
each import it and none of them builds a subject line, a MIME part or a colour.
Before this existed the transport was copied three times, and the copies had
already diverged once — the 465-implicit-TLS fix landed in one of them while
the others silently delivered nothing.

`Sender` is a value, not a singleton: `server/internal/notify` builds one per
notification channel from the channel document in Mongo, while `sitesvc` builds
one at boot and `admin` holds one in `admin/internal/mail.Default`, alongside
its other boot-time singletons. Callers only ever see typed methods —
`SendVerification`, `SendExpiring`, `SendMonitorAlert`, `SendEnquiry` and the
rest, grouped by owner into `account.go`, `licence.go`, `billing.go`,
`monitor.go` and `contact.go`.

Every message is `multipart/alternative`, so each one is two templates:
`templates/<name>.html.tmpl` and `.txt.tmpl`, embedded with `go:embed`. They
define `subject`, `title`, `pill` and `body`; `layout.html.tmpl` and
`layout.txt.tmpl` provide the chrome and the helper templates (`p`, `lead`,
`button`, `well`, `note`, `rows`, `chip`) that the bodies compose. One template
set is parsed per message rather than one big set, because every message
defines those same four names. **`subject` is defined in the txt file only** —
`html/template` would escape an ampersand in an instance name and mail clients
show subjects verbatim.

`shared/mail/render_test.go` renders all of them and fails if a template exists
that no case covers, which is the only thing standing between a mistyped field
and a boot-time panic — the templates are parsed in `init()`.

### Shared provisioning

`shared/provision` (`instance.go`, `slug.go`, `user.go`) holds the slug rules, reserved names and instance/user creation logic that both `server` and `admin/internal/cloudprov` need, so there is no longer a second copy to drift: `cloudprov.CreateInstance` calls straight into it to create a control-plane instance and its owner from a customer request.

### Grants project, they do not federate

Granting someone access to a cloud instance writes a real control-plane `users`
row through `cloudprov`, with `auth_source: "hq"` and `hq_user_id` set. The
instance authenticates it exactly as it authenticates anyone else, with **no
runtime dependency on admin**. Revoking deletes that row — the control plane has
no disabled state, and a row that exists is a row that can sign in.

`instance_members` in admin's database is only admin's _index_ of those
projections; the control-plane row is the access. That is why a failed
`instance_members` insert unwinds the projection, and why the boot backfill can
rebuild the index from the control plane but never the other way round.

**Self-hosted instances are never projected into.** All three mutating member
endpoints refuse when `deployment != cloud`.

The HQ password is the single source of truth for every `hq`-sourced row.
`PUT /api/account/password` rehashes and has `cloudprov` copy the hash to every
projected row; propagation is best-effort, and `admin/internal/hqsync` compares
and repairs every 15 minutes. It is **its own package rather than a pass inside
`inject`** — `inject` writes three licence fields and nothing else, and that
narrowness is what makes admin's reach into the control plane reviewable.

The control plane refuses to change an `hq`-sourced user's role or delete it
(`services.ErrHQManaged`, 409). `web/` shows those rows read-only with a link to
the portal, but the API is the boundary; the UI is a courtesy. There is no local
password-change endpoint at all, so there is no competing writer for the hash.

---

## Auth and Orgs

- **Bootstrap** — first run has no users. `GET /auth/bootstrap-status` drives `/setup`, `POST /auth/bootstrap` creates the first org plus its owner.
- **Local auth** — email + password (bcrypt), `POST /auth/login`.
- **Auth providers** — configured _per instance_ in `auth_providers`, any number of them, each named and independently enabled. Issuer, client ID and an encrypted client secret per provider. `/auth/oidc/:providerId/start` → `/auth/oidc/:providerId/callback`. Presets (Entra, Google, Okta, GitHub) are a Go table in `server/internal/auth/presets.go` and expand to a real issuer on save, so nothing downstream knows a preset existed. GitHub is OAuth2 rather than OIDC and takes its own branch, requiring an address that is both primary **and** verified — an unverified address is not proof of control.
- **Local login** — `settings.local_login_enabled`, a `*bool` because absent must mean enabled; a plain bool would disable password sign-in fleet-wide at upgrade. `services.CheckLockout` refuses any change leaving neither local login nor an enabled provider, and is enforced in the service layer so the settings path and the provider path cannot disagree.
- **Sessions** — opaque 32-byte hex ID in the `km_session` cookie, session body stored in Redis with a 24h TTL.
- **Roles** — `owner`, `admin`, `member`. `/api/settings` and `/api/org/*` require owner or admin.
- **Host/org guard** — `APP_ROOT_LABEL` (default `vantage`) defines the app root label. A request to `<slug>.vantage.<tld>` resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.

Unique indexes are a **security property**, not an optimisation. `users` is
unique on `(instance_id, email)` — one address is one user _within_ an instance,
and the same address may hold a user in several instances, because an account's
people are projected into each instance they are granted. This is sufficient only
because **every lookup by email is scoped by instance**; there is deliberately no
unscoped lookup anywhere, and adding one would let the login path return an
arbitrary one of several matching users. Instance slug, settings instance and ESO
token hash remain globally unique.

---

## gRPC API

```protobuf
service Vantage {
  rpc Register(RegisterRequest)              returns (RegisterResponse);
  rpc SyncKeys(SyncRequest)                  returns (SyncResponse);
  rpc UploadGeneratedKey(UploadKeyRequest)   returns (UploadKeyResponse);
  rpc ReportUpdates(ReportUpdatesRequest)    returns (ReportUpdatesResponse);
  rpc ReportInventory(InventoryReport)       returns (InventoryReportResponse);
  rpc SyncMonitors(SyncMonitorsRequest)      returns (SyncMonitorsResponse);
  rpc ReportChecks(ReportChecksRequest)      returns (ReportChecksResponse);
  rpc CommandStream(stream AgentMessage)     returns (stream ServerCommand);
}
```

`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.

`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`, `OpenProxyCmd`, `PingCmd`.

**`PingCmd` is a liveness beat, and it is not redundant with gRPC keepalive.**
The server sends one every 20s on an otherwise idle command stream; the agent
treats 70s of silence as a dead stream and reconnects. Keepalive cannot do this
job behind an L7 proxy: the agent's HTTP/2 connection terminates at the proxy,
which answers pings on its own behalf, so a control-plane pod that dies leaves
the agent blocked in `Recv` on a stream that never delivers another message and
never errors — commands dispatched into it are silently lost while `SyncKeys`
keeps succeeding and the fleet list still shows the server `active`. The agent's
watchdog arms only **after** it has seen a first ping, so an older server that
sends none is treated as working rather than put into a reconnect loop.

Key-state polling stays on the 30s `SyncKeys` interval. Full message definitions live in `proto/vantage/v1/vantage.proto`.

---

## REST API

Unauthenticated:

```
GET  /healthz            /readyz             # liveness / readiness probes
GET  /install            /install.ps1        # dynamic agent install scripts
GET  /update             /update.ps1
GET  /auth/bootstrap-status
POST /auth/bootstrap     /auth/login  /auth/logout
GET  /auth/me
GET  /auth/providers                          # {local_enabled, providers:[{id,name,preset}]} — no issuer, client ID or secret
GET  /api/secrets/:group/values               # bearer token (ESO)
```

Session-authed under `/api`:

```
servers      GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
             POST /servers/:id/{generate-key,update-agent,apply-updates}
keys         GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
             POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
             POST /steps/{import,seed-defaults,parse} · GET /steps/usage
             GET,POST /workflows · GET,PUT,DELETE /workflows/:id
             POST /workflows/:id/run · GET /workflows/:id/runs
             GET /runs/:runId · POST /runs/:runId/cancel
             GET /runs/:runId/servers/:serverId/logs[/stream]
monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
             GET /monitors/:id/{incidents,uptime}
channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
             POST /secrets/:group/reveal · DELETE /secrets/:group/:key
console      POST /console/connect · GET /console/tunnel (websocket)
vulns        GET /vulnerabilities · GET /vulnerabilities/summary
             POST /vulnerabilities/rescan                          (owner|admin)
             POST,DELETE /vulnerabilities/:id/accept               (owner|admin)
             GET /servers/:id/vulnerabilities · GET /servers/:id/packages
             GET /packages/search?name=
             GET,POST /vuln-rules · PUT,DELETE /vuln-rules/:id     (owner|admin)
audit        GET /audit
agent        GET /agent/latest-version
settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
licence      GET /license · POST /license                (POST: self-hosted only)
org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
providers    GET,POST /auth/providers · PUT,DELETE /auth/providers/:id
             POST /auth/providers/:id/{test,ack-notice} · GET /auth/presets  (owner|admin)
```

`GET /license` reports `deployment`, and **`POST /license` answers 409 `cloud_managed` when it is `cloud`**. A cloud instance's licence is written by `admin/internal/inject` straight into the database and never through this endpoint, so the refusal cannot break injection — it only stops a customer pasting over a licence they do not own. `web/` hides the paste form and points at the HQ portal instead, but as with `hq`-managed users, the API is the boundary and the UI is the courtesy.

`POST /license` is also in `licenceExemptPaths`: pasting a valid licence has to work while the current one is expired, because it is the way out of degraded mode.

Free exists in both deployments, so it is no longer cloud-only by construction. The one-Free-per-account rule is enforced per account **and deployment**, in `licensing.checkFreeLimit` and in `createInstance`'s friendly pre-check — the two must stay scoped identically, because a pre-check stricter than the issuer refuses what would have worked. A self-hosted Free licence is claimed with `POST /api/instances/:id/claim-free` after the install is linked; the metered server count and per-instance feature toggles come from the instance's `entitlement`, which a licence snapshots at issue time (see the `catalogue`/`entitlements` note under MongoDB Collections).

---

## Admin REST API (`admin`, :8083)

A separate service with its own session cookie (`admin_session`) and its own database. Unauthenticated:

```
GET  /healthz
GET  /auth/me                                  # who am I; 401 drives the UI's redirects
POST /auth/staff/login   /auth/login  /auth/logout
POST /auth/signup                              # self-hosted only; honeypot + rate limited
GET  /auth/verify?token=…
POST /auth/accept-invite                       # an invitee sets their own password
POST /api/paddle/webhook                       # Paddle events; signature-verified, idempotent, no session
```

Customer-session (`/api`), every instance resolved through `ownedInstance`:

```
GET  /account                                  # account, instances, max_relinks
POST /instances                                # create a cloud instance (Free tier, one Free per account per deployment)
POST /instances/:id/renew                      # Free renewal; refuses outside the renewal window
POST /instances/:id/claim-free                 # issue Free on a linked self-hosted instance
POST /instances/link · /instances/:id/relink
GET  /instances/:id/entitlement
GET  /checkout/options                         # active plans + catalogue prices for the running PADDLE_ENV
POST /instances/self-hosted                    # create a paid-checkout placeholder (awaiting_link, no licence)
POST /instances/:id/claim-link                 # bind a paid placeholder to the real UUID and issue
PUT  /instances/:id/entitlement                # set desired config; pushes line items to Paddle (owner|admin)
POST /billing/portal                           # mint a Paddle customer-portal URL
GET  /instances/:id/license · /instances/:id/license/download
GET  /subscriptions
GET,POST /account/users · PUT /account/users/:id/role · DELETE /account/users/:id
PUT  /account/password                         # propagates to every projected user
GET,POST /instances/:id/members                # cloud only
PUT  /instances/:id/members/:uid/role · DELETE /instances/:id/members/:uid
```

Reading is open to any signed-in member; every mutation above except
`/account/password` (which is your own) sits behind `RequireAccountRole(owner,
admin)`. `:uid` is the **`customer_users.user_id`**, not the projected
control-plane user_id — the portal never has to know that one.

Staff-session (`/api/staff`):

```
GET,POST /accounts · GET /accounts/:id         # search by name, email, Paddle ID or instance UUID
GET,POST /instances · GET /instances/:id       # instance + account + licence history + injection state
POST /instances/:id/issue · /instances/:id/relink
GET  /licenses · /subscriptions · /audit · /plans · PUT /plans/:deployment/:tier
GET,PUT /catalogue
GET,PUT /instances/:id/entitlement
GET  /health/injection · /health/billing
```

**Customer endpoints answer 404, never 403, for another account's resource** — a 403 confirms the resource exists. Route-group guards in `adminsite/` mirror this, but the backend is the layer that matters.

### Billing (Paddle)

Paddle is merchant of record; `admin/internal/paddle` is a thin REST client (no vendor SDK) and the only place that talks to it. **Free is entirely outside Paddle** — the shipped self-serve Free flow owns its own renewal, so no £0 subscription exists; an account learns its `paddle_customer_id` from its first paid webhook. Checkout happens in the browser (`@paddle/paddle-js`, token baked into the adminsite build); the server only updates a live subscription (`PUT /instances/:id/entitlement`) and mints a portal session.

`POST /api/paddle/webhook` is the **only** issuing path for paid plans: signature-verified with `PADDLE_WEBHOOK_SECRET` (boot-required), idempotent via `paddle_events`, and a function of the subscription's _current_ line items — resolved back to a plan and configuration by `catalogue.ResolveItems`, so out-of-order delivery is correct by construction. A confirmed webhook promotes the entitlement `desired`→`granted` and signs from `granted` **only**; a checkout is built from `desired`. `subscription.canceled` and `past_due` take **no licence action** — the licence runs to its (grace-padded) expiry, then the existing lifecycle sweep lapses the instance. A renewal (`transaction.completed`, origin `subscription_recurring`) is the only moment a scheduled reduction collapses `desired` into `granted`. Self-hosted purchase creates a placeholder instance before payment (`POST /instances/self-hosted`); the licence is issued only once the customer pastes the install's real UUID (`POST /instances/:id/claim-link`), because a licence binds to that UUID.

## MongoDB Collections

`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `migrations`

Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`.

Notes that are not obvious from the structs:

- `servers.agent_token_hash` stores SHA-256 of the token, never plaintext. `pre_reg_token` is cleared after `Register()`. `status` is `pending` → `active` on register, `offline` when `last_seen` passes the threshold (swept every 2 min).
- `servers.inventory` holds the latest metrics snapshot with separate `metrics_at` / `static_at` timestamps.
- `keys.private_key_enc` and `passphrase_enc` are AES-256-GCM; the JSON form exposes only `has_private_key` / `has_passphrase`.
- `assignments.revoked_at: null` means active. Revocation is soft, preserving audit history.
- `workflow_runs.steps_snapshot` freezes the resolved steps so editing the library never rewrites history.
- `console_sessions.token_consumed_at` is set atomically to enforce one-time use.
- `auth_providers.provider_id` is a short random identifier, not the Mongo `_id`: it appears in the callback URL a customer pastes into their IdP, and an `_id` there would publish a database key. `callback_notice` marks a provider migrated from the old single-provider shape, whose redirect URI therefore changed.
- `workflow_log_lines` is keyed `(run_id, server_id, seq)` — the index is not an optimisation, every read is a range scan over it. `workflow_log_seq` holds one counter document per `run_id/server_id`, which is what lets two pods interleave into one ordered log. Neither carries `instance_id`: they are reached only through a run, and a run is already scoped.
- `users.auth_source` is `local`, `oidc` or `hq`. An `hq` user was projected from a Vantage HQ account and carries `hq_user_id`; HQ owns its role, password and existence.
- `server_packages` holds a server's whole package set in **one** document, not one per package. The hash already established that something changed, so a report is a single atomic upsert with no delta logic to get wrong; ~2000 packages is ~150KB, well inside the 16MB limit. `scan_pending` lives on the document rather than in memory so a leader handover cannot lose it.
- `vuln_findings` is unique on `(instance_id, server_id, cve_id, package_name)`. That key is what makes a rescan an idempotent upsert rather than a duplicate factory, and what lets `first_seen` survive one. An empty `fixed_in` means no vendor fix exists — a real state, never "not vulnerable".
- `vulndb_meta` is a singleton and deliberately carries **no** `instance_id`: the vulnerability database is a property of the deployment, not a tenant. Same reasoning as `migrations`, and the reason it is absent from `services.ScopedCollections`.
- **`services.ScopedCollections` is the canonical registry of tenant-scoped collections**, and `scopedCollectionsForPurge` derives instance deletion from it rather than keeping a second list. A new collection carrying `instance_id` must be added there or its rows outlive the instance.

Admin's own database is separate and holds `accounts` · `admin_instances` · `licenses` · `subscriptions` · `plans` · `catalogue` · `entitlements` · `paddle_events` · `staff_users` · `customer_users` · `instance_members` · `admin_audit`. `paddle_events` is the webhook idempotency log, unique on `event_id`: an event is claimed there before processing, and a duplicate of a handled event is a 200 no-op. `instance_members` is unique on `(instance_id, customer_user_id)` — one person holds at most one user in one instance, which makes a grant idempotent-by-refusal rather than silently doubling a projection. It is an _index_ of the control-plane rows, not the authority (see "Grants project, they do not federate"). Admin has no migrations collection; `models.Backfill` runs on every boot and is idempotent by filtering on the absence of what it writes.

`plans` is keyed on `(deployment, tier)` — six rows, two deployments times three tiers — and holds base allowances only. **Every Paddle price ID lives in `catalogue`**, one row per priceable component (`base`, `limit`, `feature`), because a metered plan is priced by several prices and one map on a plan row cannot express that. `entitlements` holds one row per instance with `desired` beside `granted`: the checkout is built from `desired`, a licence is only ever signed from `granted`, and an abandoned checkout therefore leaves a `desired` that reached nothing. The two Free plans have **no catalogue rows at all**, which is what keeps Free outside Paddle.

### Migrations

`services.RunMigrations()` runs at boot, recording markers in `migrations`:

- `0001_default_org_backfill`
- `0002_settings_org_backfill` (must run before 0003 — 0003 can create a `default` org, which pushes 0002 into its ambiguous multi-org branch)
- `0003_missed_org_scopes`
- `0005_auth_providers` — copies each `instance_oidc` document into `auth_providers`, ciphertext verbatim rather than decrypted and re-encrypted, so it does not need `KEY_ENCRYPTION_KEY` and cannot strand an instance's SSO configuration that has none set.

Index builders (`EnsureAuthIndexes`, `EnsureSettingsIndexes`) are fatal on failure; `EnsureSecretIndexes` and `EnsureWorkflowIndexes` only warn.

---

## Agent Lifecycle

### Config file

Linux `/etc/vantage/config.yaml`, Windows `%ProgramData%\vantage\config.yaml`. Directory `0700`, file `0600`.

```yaml
server_url: "vantage.yourdomain.com:9090"
server_id: "<uuid>"
pre_reg_token: "<token>" # removed after first successful Register()
agent_token: "" # written by agent after Register()
poll_interval: 30s
tls: true
```

### Startup

```
1. Load config
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
4. Enter SyncKeys poll loop (default 30s)
```

### Poll loop

```
1. SyncKeys(server_id, agent_token, agent_version)
2. Non-Linux hosts stop here — Windows agents register and heartbeat only
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600
```

### Install

Linux: systemd unit at `/etc/systemd/system/vantage-agent.service`, `Restart=always`, runs as root.
Windows: MSI built by CI (WiX), or `installer/setup.ps1` registering the agent as a service via NSSM.

---

## Server Registration Flow

1. **Add Server** in the UI calls `POST /api/servers/new`, which generates a `server_id` and a pre-registration token (TTL 1 hour, single-use).
2. The UI shows a one-liner:
    ```bash
    curl -fsSL https://vantage.yourdomain.com/install | \
      bash -s -- --server-id=<id> --token=<token>
    ```
    Windows gets the `/install.ps1` equivalent.
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
4. The server flips to `active` on first sync.

`/install` is served dynamically, injecting the latest agent version from the Gitea API.

---

## Environment Variables (server)

| Name                       | Required        | Notes                                                                                                                                                                                                                                                                                                              |
| -------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GRPC_HOST`                | **yes**         | `host:port` agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC.                                                                                                                                                         |
| `MONGO_URI`                | no              | default `mongodb://localhost:27017`                                                                                                                                                                                                                                                                                |
| `MONGO_DB`                 | no              | default `vantage`                                                                                                                                                                                                                                                                                                  |
| `REDIS_USERNAME`           | no              | Redis 6+ ACL user. Leave empty for a legacy `requirepass` instance — go-redis then sends AUTH with one argument instead of two                                                                                                                                                                                     |
| `REDIS_PASSWORD`           | no              | empty for an unauthenticated Redis                                                                                                                                                                                                                                                                                 |
| `REDIS_ADDR`               | no              | default `localhost:6379`                                                                                                                                                                                                                                                                                           |
| `KEY_ENCRYPTION_KEY`       | yes in practice | 64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials.                                                                                                                                                                                                         |
| `GUACD_ADDR`               | no              | default `guacd:4822`                                                                                                                                                                                                                                                                                               |
| `PROXY_ADVERTISE_HOST`     | no              | default `server`; the hostname guacd resolves the control plane by, handed to guacd as the relay's address. Wrong here and every console session fails at connect                                                                                                                                                |
| `PROXY_LISTEN_HOST`        | no              | default `0.0.0.0`; the interface the ephemeral relay listener binds                                                                                                                                                                                                                                                |
| `APP_ROOT_LABEL`           | no              | default `vantage`; wrong value disables the host/session org guard                                                                                                                                                                                                                                                 |
| `POD_IP`                   | no              | this pod's own address, set by the Helm chart from the downward API. **Takes precedence over `PROXY_ADVERTISE_HOST`** — a console relay listener belongs to one replica, and a Service address names all of them                                                                                                     |
| `VANTAGE_MIGRATE_ONLY`     | no              | run schema setup (migrations, index builders, default-step seeding) and exit without serving. `GRPC_HOST` is not required in this mode. Set by the Helm chart's pre-upgrade Job                                                                                                                                                          |
| `VANTAGE_SKIP_MIGRATIONS`  | no              | serve without running schema setup, on the assumption a Job already did. Set by the chart's Deployment whenever `server.migrationJob.enabled`. Unset under Compose, where one process still migrates and then serves                                                                                                                    |
| `VANTAGE_TRIVY_DB_REF`     | no              | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for an air-gapped install, or to avoid the anonymous ghcr rate limit                                                                                                                                                                                    |
| `VANTAGE_VULNDB_DISABLED`  | no              | `true` disables the vulnerability database puller and scan loop entirely. Findings already written are still served, and still shown as stale                                                                                                                                                                       |
| `FREE_INSTANCE_REAP_AFTER` | no              | duration past a Free licence's expiry before the instance and all its data are deleted. **Empty disables the reaper, and empty is the default.** Set to `336h` in `docker-compose.site.yml` only — a self-hosted deployment must never reap. Must match admin's value, which only names the date in warning emails |

**sitesvc** (`deploy/docker-compose.site.yml` only):

| Name                              | Required        | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| --------------------------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MONGO_URI`                       | yes             | **must point at the control plane's database.** sitesvc no longer provisions orgs itself, but it still refuses to start (`RequireMigratedDatabase`) against a database that has not run migration `0004` (the `orgs` → `instances` rename), and it (re)declares the shared `users.email` / `instances.slug` indexes at boot. The database name is read from the URI path; a URI without one is refused rather than defaulted. Note this differs from the server, which takes `MONGO_DB` separately. |
| `SMTP_HOST` / `SMTP_FROM`         | yes             | without them the contact form refuses (503) rather than silently dropping                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `SMTP_TO`                         | no              | default `support@hostxtra.co.uk`; contact enquiries only                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `SMTP_PORT`                       | no              | default `587`; `465` uses implicit TLS                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `SMTP_USERNAME` / `SMTP_PASSWORD` | no              | auth skipped when username is empty                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `SITE_ORIGIN`                     | yes in practice | comma-separated allowed origins; unset refuses every cross-origin browser request                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `TRUST_PROXY`                     | no              | only `true` behind a proxy that overwrites `X-Forwarded-For`, or clients spoof past the rate limiter                                                                                                                                                                                                                                                                                                                                                                                                |

### Ingress (Helm, Traefik)

`ingress.enabled` publishes **two** hostnames, because the two audiences arrive over different protocols:

| Values                                       | Route                                                                                       |
| -------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `ingress.web.host` (+ `web.extraHosts`)      | browsers → `web:3000`                                                                        |
| `ingress.api.paths` (when `api.enabled`)     | `/api`, `/auth` → `<release>-server:8080`, bypassing the Next proxy                          |
| `ingress.grpc.host`                          | agents → a dedicated `<release>-server-grpc` Service on 9090, annotated `serversscheme: h2c` |

**`ingress.web.host` is normally a wildcard.** `*.vantage.example.com` is the per-tenant instance namespace — `APP_ROOT_LABEL` resolves the instance from the label. A Kubernetes wildcard host matches **exactly one** label, so it does not match the apex, and here that is correct rather than a gap: `vantage.hostxtra.co.uk` is the marketing site (`site/`, in `docker-compose.site.yml`), which this chart does not deploy. `extraHosts` is for a genuine second name; adding the apex to it would put the control plane on the marketing host. Every host in the list gets identical paths.

**`ingress.api.enabled` routes `/api` and `/auth` straight to the server.** Both arrangements work — without it `web` proxies those prefixes onward itself (`web/next.config.ts`) — but edge routing is one hop shorter and matches what the Nginx Proxy Manager in front of the Docker deployment already does, so leaving it off makes the request path a different shape on Kubernetes than in production. It stays **off by default** because it only helps where the server is reachable on the same host and certificate as `web`; turning it on blindly moves the whole API onto a route that may not be provisioned. Traefik derives router priority from rule length, so `PathPrefix(/api)` outranks the catch-all `/` with no priority annotation needed.

**The gRPC route needs its own Service.** The server terminates no TLS; it speaks plain h2c and always has, with TLS terminated by whatever sits in front. Traefik will not use h2c to a backend unless the *Service* says so, and that annotation applies to every port on the Service — so annotating the shared two-port `<release>-server` would force h2c on its HTTP port too.

**`server.env.grpcHost` is not derived from `ingress.grpc.host`, and the chart refuses to render if they disagree.** Agents dial whatever `grpcHost` says, and it is baked into every install one-liner; left pointing at the in-cluster Service while agents arrive through the ingress, every install succeeds and every agent then fails to connect, with nothing in the control plane explaining why. Guessing at the port (443? 9090?) would be worse than stopping.

TLS is `ingress.tls.secretName` / `grpcSecretName` (pre-existing certificates) **or** `certResolver` (Traefik ACME). Setting neither while `tls.enabled` produces a TLS router with no certificate, so Traefik serves its self-signed default — which looks valid and is trusted by nothing. NOTES.txt warns on install rather than the chart failing, since it is a real if unusual choice behind another terminator.

---

`deploy/docker-compose.yml` runs four services: `redis`, `guacd`, `server` (8080 + 9090), `web` (3000). MongoDB is external. `deploy/docker-compose.site.yml` adds five more — `site` (3003), `sitesvc` (8082), `admin` (8083), `adminsite` (3004) and `docsite` (3005) — and is only used on vantage.hostxtra.co.uk.

`docsite` is the odd one: a **static** build served by `nginx:alpine-slim`, not a Node runtime, and it listens on `80` rather than `3000`. It is reached at **`vantage.hostxtra.co.uk/docs`** — a path on the marketing host, routed by its own Nginx Proxy Manager location, which must sort **above** the catch-all forwarding to `site:3003` or Next answers the 404. A path and not a subdomain because `*.vantage.hostxtra.co.uk` is the per-tenant instance namespace and `APP_ROOT_LABEL` would read a `docs.` label as a tenant slug. NPM forwards the **full** path upstream — it does not strip `/docs` — so `DOCS_BASE_URL`, the proxy location and the directory the image copies the build into (`/usr/share/nginx/html/docs`) must all agree. When they do not, the HTML loads and every asset 404s.

`LICENSE_SIGNING_KEY` appears in **exactly one service in exactly one compose file**: `admin` in `docker-compose.site.yml`. It must never be added to `server`, and the self-hosted `docker-compose.yml` must never mention `admin` or `adminsite` at all. Admin uses an external Redis via `REDIS_ADDR`/`REDIS_USERNAME`/`REDIS_PASSWORD`. `server` now reads the same three, so a Kubernetes install can point at a managed Redis; the base compose still hardcodes an unauthenticated `redis:6379` for it, so in Docker those credentials remain admin's alone.

---

## Security

- gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
- Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's `0600` config.
- Pre-registration tokens are short-lived (1 hour) and single-use.
- AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
- Console session tokens are one-time; RDP credentials are consumed on tunnel open.
- ESO read token stored as a SHA-256 hash and rotatable.
- Unique indexes on `(instance_id, email)`, instance slug, settings instance and the ESO token hash are load-bearing for tenant isolation. So is the absence of any unscoped lookup by email.
- `authorized_keys` written `0600`, owned by root. The agent runs as root because it must.
- Every mutating API path writes an audit event.

---

## Frontend

Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in `web/lib/guacamole-common.js`.

All four apps are **one visual system**, anchored on the logo navy. What differs between them is which end of it they stand on:

| App          | Ground                     | Accent                           | Themes                      |
| ------------ | -------------------------- | -------------------------------- | --------------------------- |
| `web/`       | `--ground` dark, `#071628` | `#5b9be8`                        | dark only, locked           |
| `site/`      | token-based                | `#0b2a58` light / `#5b9be8` dark | light + dark                |
| `adminsite/` | token-based                | `#0b2a58` light / `#5b9be8` dark | light + dark, light default |
| `docsite/`   | token-based                | `#0b2a58` light / `#5b9be8` dark | light + dark, light default |

`adminsite/app/globals.css` and `docsite/src/css/custom.css` hold `site/app/globals.css`'s token blocks **copied verbatim** — same names, same values. `web/app/globals.css` holds the same tokens too, but only the **dark** values, since it does not switch. **Change a token in all four files in the same commit; nothing enforces the match automatically**, the same shape of hazard as sitesvc's mirrored slug rules.

`docsite/` is the one place the tokens are not consumed through Tailwind: everything below its token block maps Docusaurus's `--ifm-*` variables onto them. Docusaurus already stamps `data-theme` on `<html>`, which is the selector `site/`'s dark block keys on, so the built-in toggle needed no wiring. The rule holds all the same — no rule in that file outside the token blocks carries a hex. The one concession is `docsite/static/img/favicon.svg`, which must, for the same reason the email layout must: a browser tab cannot read a token.

There is a **fifth** copy, and it is the one people forget: `shared/mail/templates/layout.html.tmpl` carries web/'s dark values as literal hex. Email clients support neither `var()` nor a reliable `prefers-color-scheme`, so the token indirection is simply not available there — an email is read before the recipient clicks through to the control plane, and the two should not look like different products. Every colour in the email system is in that one file, in the same way no component in the four web apps carries a hex.

Tailwind in all three maps `var(--…)` references only, so **no component in any of them may carry a hex value**. The names differ per app on purpose, because each app has its own subject: `site/` calls the semantic three `--up`/`--pend`/`--down` for monitor state, `adminsite/` aliases them to `valid`/`warn`/`expired` for licence state, and `web/` to `success`/`warning`/`danger`. Same colours, honest names on each side.

`web/` stores its tokens as **RGB channel triplets** with the hex in a trailing comment, and derives `--token: rgb(var(--token-rgb))` from them. That is not a style preference: the console leans on Tailwind's opacity modifiers (`bg-danger/10`, `border-accent/50`, `ring-accent/30`) in a way the other two do not, and `<alpha-value>` only compiles against channels. Keep the hex comments — they are what lets the three token blocks still be diffed by eye. `web/` also adds three tokens site/ has no use for: `--accent-hover` and `--down-hover` (site/ brightens with a CSS `filter`, which a Tailwind colour token cannot do) and `--well`, the floor beneath the ground for install one-liners, key blobs and run logs — surfaces showing machine output rather than interface.

`web/` is locked to dark and `adminsite/` defaults to **light**, and that pairing is the point: an operator with both open should never mistake one for the other before clicking Reissue. Now that both are drawn from the same palette the distinction rests **entirely** on the ground, so do not make dark the adminsite default and do not give web/ a light theme. State never reads by colour alone in either: every pill carries a distinct shape and a text label. The same argument applies one level in: the **staff** masthead sits on `--panel-2` with a `STAFF` chip, so staff and customer screens are not identical either.

`web/` collapses Tailwind's radius scale — `md`, `lg` and `xl` all resolve to site/'s 4px — rather than rewriting the ~140 `rounded-lg` classes across its pages. Every one of them meant "a panel corner", and `tailwind.config.ts` is now where that decision lives. `rounded-full` is untouched: status dots and pills still need it.

**The `adminsite/` shell.** `AppBar` is the single masthead — identity, nav, environment, account menu — and it belongs to the two authenticated layouts, never to `app/layout.tsx`, so `/login` and `/accept-invite` do not render navigation they cannot use. Nav active state is derived from `usePathname`; do not hardcode it. `PageHeader` gives every screen the same back link, title, actions and **record line** (the reference number in mono, click-to-copy) — the reference is what people paste into support tickets, so it has a fixed slot rather than a per-page treatment. `PageFrame` is the main-plus-320px-rail split; the rail carries only what is true account-wide, which is why there is no plan card in it — **tier, limits and expiry belong to a licence, and a licence belongs to one instance**, so an account holding a Free cloud instance and a Professional self-hosted one has no single plan.

Customer nav is three destinations — Overview, People, Billing. Settings is in the account menu because it is your password, not a place, and appearance lives there too: `AccountMenu` is the only thing that sets `data-theme`, which the token blocks have always supported in both directions.

`InstanceRecord` is one component open or closed, and it **replaced** `InstanceCard`. Closed it is a row; open it adds licence contents, members and actions. It defaults open when the instance is the only one or needs attention, and a manual toggle is remembered per instance in `localStorage`. Do not reintroduce a second summary component — the split is what left a one-instance account showing a third of a row and nothing else.

| Route                                                           | Purpose                                                                 |
| --------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `/setup`                                                        | First-run bootstrap: create the first org and owner                     |
| `/login`                                                        | Local or OIDC sign-in                                                   |
| `/`                                                             | Fleet dashboard                                                         |
| `/servers`, `/servers/new`, `/servers/[id]`                     | Fleet list, install one-liner, server detail (keys, inventory, updates) |
| `/servers/[id]/console`                                         | Browser SSH/RDP/VNC session                                             |
| `/keys`, `/keys/[id]`                                           | Key library; assign and revoke per server                               |
| `/workflows`, `/workflows/[id]`, `/workflows/[id]/runs[/runId]` | Compose, run, and follow live logs                                      |
| `/steps`                                                        | Reusable step library                                                   |
| `/monitors`, `/monitors/new`, `/monitors/[id][/edit]`           | Checks, uptime, incidents                                               |
| `/secrets`, `/secrets/[group]`                                  | Vault                                                                   |
| `/audit`                                                        | Audit log                                                               |
| `/settings`, `/settings/notifications`, `/settings/license`     | Members, OIDC, alerts, retention, ESO token · channels · licence        |

**`/settings` is one page, not a section.** Members and single sign-on used to
live at `/settings/instance` with their own sidebar entry; they are now the
**Access** group at the top of `/settings`, above **Monitoring** and
**Integrations**. Splitting "who can sign in" away from "how this instance
behaves" made two half-pages and a nav entry called Instance that no one could
distinguish from Settings. `next.config.ts` keeps a permanent redirect from the
old path. The cards live in `web/components/settings/` rather than in the page,
which is also where the `Field`/`inputClass` pair the three of them share now
lives — one copy instead of the three that existed while they were apart.

---

## CI/CD — Gitea Actions

### `agent-release.yml` — triggered by `agent/v*` tags

Builds `linux/amd64`, `linux/arm64`, `windows/amd64`, writes `checksums.txt`, creates a Gitea release. A second `msi` job on `windows-2022` packages the WiX installer.

```bash
GOOS=linux GOARCH=amd64 go build \
  -ldflags="-s -w -X main.Version=${VERSION}" \
  -o dist/vantage-agent-linux-amd64 ./cmd
```

### `server-deploy.yml` — triggered on every push to `main`

Builds and pushes seven images to the Gitea container registry: `server`, `web`, `site`, `sitesvc`, `admin`, `adminsite` and `docsite`.

Note that despite the name, **this workflow does not deploy** — it only builds and pushes. There is no SSH step. Rolling images out is a separate manual step on the host:

```bash
cd /opt/vantage && docker compose -f docker-compose.yml -f docker-compose.site.yml pull && \
  docker compose -f docker-compose.yml -f docker-compose.site.yml up -d --remove-orphans
```

**Each image only rebuilds when its own inputs changed.** A `git diff` against `github.event.before` decides, which is why the checkout uses `fetch-depth: 0` — the default shallow clone has one commit and nothing to diff — and why `git` is installed in the `docker:dind` container. The mapping follows the build contexts exactly:

| Image                        | Rebuilds when                             |
| ---------------------------- | ----------------------------------------- |
| `server`                     | `server/`, `shared/`, `proto/`, `go.work` |
| `admin`                      | `admin/`, `shared/`, `go.work`            |
| `sitesvc`                    | `sitesvc/`, `shared/`, `go.work`          |
| `web` · `site` · `adminsite` · `docsite` | their own directory only      |

`shared/` fans out to all three Go images because each of their Dockerfiles copies `shared/` from a root context — **if a fourth service ever imports `shared/`, add it to that list or it will ship stale**. A change to the workflow file rebuilds everything, since a build arg is baked into the image. So does anything that leaves no trustworthy base commit: a manual `workflow_dispatch`, a new branch, or a force-push whose old head is gone.

The gap this leaves: **changing a repo variable pushes no commit, so nothing rebuilds.** After editing `ADMIN_API_URL`, `HQ_URL` or `ADMIN_ENV`, run the workflow manually — that is what `workflow_dispatch` is there for. Base images also stop being refreshed on a service nobody touches; a periodic manual run covers that.

### `chart-release.yml` — validates on every chart change, publishes on `chart/v*` tags

Two jobs' worth of work in one, split by trigger. Any push or PR touching `deploy/chart/` lints the chart and renders it four ways: defaults, a multi-replica install, external Redis and MongoDB, and a set of values that **must be refused**. That last one is the point — every safety rail in this chart is a template `fail`, and `helm lint` happily accepts a chart whose templates never execute, so only rendering proves they still fire.

Publishing runs only on a `chart/v*` tag, to the Gitea Helm registry at `/api/packages/<owner>/helm/api/charts`. **`Chart.yaml` is the source of truth for the version**; the tag only selects which one to publish, and a tag that disagrees with `Chart.yaml` fails rather than stamping over it — the alternative leaves the repository disagreeing with what shipped. A version already in the registry is rejected by Gitea, which is intended: published chart versions are immutable.

The registry host comes from `github.server_url`, so it cannot drift from the instance the workflow is running on. It authenticates with `REGISTRY_USER` + **`RELEASE_TOKEN`** — the pair `server-deploy.yml` actually uses for `docker login`. `REGISTRY_PASSWORD` is listed in the secrets table below but set by no workflow; passing an unset secret yields an empty password and Gitea answers `401 Failed to authenticate user`, which reads like a scope problem on a token that was never sent. The publish step therefore checks both are non-empty before it calls curl. `RELEASE_TOKEN` needs `write:package` in addition to `write:release`.

```bash
helm repo add vantage https://gitea.hostxtra.co.uk/api/packages/mrhid6/helm
helm install vantage vantage/vantage --version 0.1.0
```

### Tagging

```bash
git tag agent/v1.0.0 && git push origin agent/v1.0.0   # agent release
git tag chart/v0.1.0 && git push origin chart/v0.1.0   # helm chart package
git push origin main                                    # server + web deploy
```

### Secrets / variables

| Name                    | Type     | Value                                                                                                                                                                                                                                                      |
| ----------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `RELEASE_TOKEN`         | Secret   | Gitea API token. Needs `write:release` (agent releases), `write:package` (container images and the Helm chart). **This is the only token any workflow authenticates with** — `docker login` and the chart publish both pair it with `REGISTRY_USER`         |
| `REGISTRY_USER`         | Secret   | Gitea username. Must own `RELEASE_TOKEN`, or basic auth is rejected                                                                                                                                                                                        |
| ~~`REGISTRY_PASSWORD`~~ | —        | **Not used.** Named here historically; no workflow reads it. Referencing an unset secret yields an empty password and a `401 Failed to authenticate user` that looks like a token scope problem. Use `RELEASE_TOKEN`                                        |
| `DOCKER_HOST`           | Variable | registry host used for image tags                                                                                                                                                                                                                          |
| `API_URL`               | **not** a CI variable | `web` reads it at **runtime**, from the container environment — `next.config.ts` is evaluated when `server.js` boots in standalone mode, and the rewrites it feeds are server-side, never browser-side. Default `http://localhost:8080`; compose sets `http://server:8080`. `NEXT_PUBLIC_API_URL` is still honoured as a fallback for existing deployments. |
| `SITE_API_URL`          | Variable | **browser-reachable** sitesvc URL, baked into the `site` image. Required — if empty, both forms report "not connected" and submit nowhere. Must also be in sitesvc's `SITE_ORIGIN`.                                                                        |
| `SITE_CONTACT_EMAIL`    | Variable | optional; address shown when a form is misconfigured                                                                                                                                                                                                       |
| `SITE_URL`              | Variable | browser URL of the marketing site, baked into `adminsite` so `/login` can point at `/start`. **Signup has no page in `adminsite` at all** — one signup form, on `site/`. Empty renders no link rather than one that 404s.                                  |
| `ADMIN_API_URL`         | Variable | **browser-reachable** admin URL, baked into **both** the `adminsite` and `site` images — `site/start` posts account signups straight to admin. Same footgun as `SITE_API_URL`: wrong here and every request fails at runtime with the not-connected panel. |
| `ADMIN_ENV`             | Variable | `production` or `sandbox`; drives the persistent environment badge. Anything but `sandbox` reads as production.                                                                                                                                            |
| `HQ_URL`                | Variable | optional; browser URL of the HQ portal, baked into `web` so an `hq`-sourced member links to where they are managed. Empty on self-hosted, which renders a plain label instead.                                                                             |
| `PADDLE_CLIENT_TOKEN`   | Variable | **browser** Paddle token, baked into the `adminsite` image for checkout. A repo-variable change pushes no commit, so rebuild `adminsite` manually via `workflow_dispatch` after editing it.                                                                |
| `PADDLE_ENV`            | Variable | `sandbox` or `production`; baked into `adminsite` AND read by `admin` at runtime. Selects which `catalogue` price IDs are served, and must match on both sides.                                                                                            |
| `PADDLE_API_KEY`        | Secret   | server-side Paddle key, read by `admin` at runtime. Boot-required.                                                                                                                                                                                         |
| `PADDLE_WEBHOOK_SECRET` | Secret   | webhook signature verification, read by `admin`. Boot-required — an unverified endpoint is one anyone can issue licences through.                                                                                                                          |
| `DOCS_URL`              | Variable | site `url` baked into `docsite`; `https://vantage.hostxtra.co.uk`. Empty falls back to that default rather than breaking the build.                                                                                                                        |
| `DOCS_BASE_URL`         | Variable | `/docs/`. Must match the NPM location and the directory the image serves from — all three, or the HTML loads and every asset 404s.                                                                                                                         |
| `APP_URL`               | Variable | control-plane link in `docsite`'s navbar.                                                                                                                                                                                                                  |

---

## Design Decisions

- **gRPC for agent traffic** — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
- **Outbound-only agents** — no inbound ports on managed servers, works behind NAT.
- **Poll for keys, push for commands** — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
- **Atomic `authorized_keys` rewrite** — temp file plus `os.Rename()`; a machine that dies mid-write keeps the old file.
- **Fingerprint diffing before write** — no disk churn on unchanged state.
- **Soft revocation** — `revoked_at` rather than deletes; preserves audit history.
- **Run snapshots** — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
- **Monitors run in two places** — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
- **Redis for sessions only** — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
- **guacd for console** — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
- **`org_id` on every document** — isolation enforced at the query layer, not by separate databases.
- **root only** — manages `/root/.ssh/authorized_keys`; no per-user key management.
- **Windows agents are second-class by design** — register, heartbeat, run steps, report inventory; no `authorized_keys` management.
- **Both `server` and `web` scale horizontally** — see "Running more than one server replica" below. `web` holds nothing; `server` holds per-agent state that is routed between replicas over Redis rather than duplicated.
- **Deletion lives in the control plane** — admin sends the warnings because it knows the billing address; the control plane performs the delete because it is the only service that knows which collections carry `instance_id`. Mirroring that list into admin would drift, and a drift there deletes the wrong rows.

## graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.

Rules:
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).