vantage/CLAUDE.md

Vantage

A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.

A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.

---

Architecture Overview

┌──────────────────────────────────────────────┐
│  Next.js 16 Frontend (web, :3000)            │
│  servers · keys · workflows · monitors       │
│  secrets · audit · console · settings        │
└───────────────┬──────────────────────────────┘
                │ REST + cookie session
┌───────────────▼──────────────────────────────┐
│  Go Backend (server)                         │
│  :8080 REST (gin)   :9090 gRPC (agents)      │
│  MongoDB (state) · Redis (sessions)          │
│  monitor scheduler · workflow runner         │
│  guacd tunnel proxy for browser console      │
└───────────────┬──────────────────────────────┘
                │ gRPC (TLS) — outbound from agent only
┌───────────────▼──────────────────────────────┐
│  Go Agent (per server, Linux + Windows)      │
│  polls SyncKeys · CommandStream              │
│  rewrites authorized_keys (Linux only)       │
│  runs workflow steps · monitors · inventory  │
└──────────────────────────────────────────────┘

Multi-tenancy: every domain document carries org_id, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (<slug>.vantage.<tld>).

---

Repository Structure

vantage/
├── agent/
│   ├── cmd/main.go                  # flags: -generate-key
│   └── internal/
│       ├── checker/                 # monitor check execution
│       ├── config/                  # config.yaml load/save
│       ├── exec/                    # workflow step execution
│       ├── grpc/                    # client + generated pb
│       ├── inventory/               # CPU/mem/disk collection (linux/other)
│       ├── keys/                    # authorized_keys read/diff/write
│       ├── monitors/                # agent-run monitor loop
│       ├── sync/                    # poll loop + command stream
│       └── updates/                 # OS package update check/apply
├── server/
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # REST handlers
│       ├── auth/                    # local, OIDC, session, middleware, orghost
│       ├── checker/                 # server-run monitor checks
│       ├── db/                      # mongo connect + Col()
│       ├── grpc/                    # gRPC server + generated pb
│       ├── models/                  # MongoDB documents
│       ├── monitorsched/            # server-side monitor scheduler
│       ├── notify/                  # channel dispatch: http, discord, slack, telegram, smtp
│       └── services/                # business logic + migrations
├── web/                             # the application UI (authenticated)
│   ├── app/(app)/                   # authed routes
│   ├── app/login, app/setup         # unauthed routes
│   ├── components/                  # ui/, workflows/, monitors/, Sidebar
│   └── lib/                         # api client, guac console, query client
├── site/                            # public marketing site
│   ├── app/                         # one directory per route
│   ├── components/                  # Nav, Footer, Logo, InstrumentPanel, forms
│   ├── assets/                      # image sources, not served
│   └── Dockerfile                   # same shape as web/: standalone, node, 3000
├── sitesvc/                         # public form: contact mail only
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # contact
│       └── store/                   # Mongo connect helper
├── admin/                           # licensing authority: the only signer
│   ├── cmd/main.go                  # boot: two Mongo connections, reconciler, HTTP
│   ├── cmd/adminctl/                # staff-add; deliberately has no HTTP surface
│   └── internal/
│       ├── api/                     # customer + staff handlers, route table
│       ├── auth/                    # staff, HQ customer and cloud-owner sessions
│       ├── inject/                  # licence write path into the control plane
│       ├── cloudprov/               # instance write path: creates instances + owners
│       ├── licensing/               # Issue, LinkInstance, Relink
│       ├── mail/                    # admin's boot-time shared/mail Sender
│       └── models/                  # accounts, instances, licences, plans
├── adminsite/                       # staff + customer console (vantage-hq)
│   ├── app/(customer)/              # overview, instance, link, billing
│   ├── app/(staff)/staff/           # operations, accounts, licences, plans, audit
│   ├── components/                  # AppBar, PageHeader, PageFrame, InstanceRecord
│   └── lib/                         # api client, session guards, formatters
├── docsite/                         # user documentation (Docusaurus, static)
│   ├── docs/                        # getting-started, vantage, hq, reference, operations
│   ├── src/css/custom.css           # site/'s tokens, copied, mapped onto --ifm-*
│   ├── sidebars.ts                  # authored by hand, not autogenerated
│   └── nginx.conf                   # serves the build under /docs
├── shared/                          # imported by server, sitesvc and admin
│   ├── mail/                        # the one email system: transport + tmpl templates
│   ├── license/                     # payload, sign, verify, trusted keys, plans
│   ├── models/                      # Instance, User, Settings
│   └── cmd/lkctl/                   # issue and inspect licences by hand
├── proto/vantage/v1/vantage.proto
├── installer/                       # Windows: setup.ps1, nssm.exe, WiX .wxs
├── deploy/                          # docker-compose.yml, agent.service
└── .gitea/workflows/                # agent-release.yml, server-deploy.yml

---

Subsystems

SSH keys

Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites /root/.ssh/authorized_keys atomically. Keys can also be generated on a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.

Workflows

A library of reusable steps (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into workflows targeting a set of servers. Running one snapshots the resolved steps into a WorkflowRun, then dispatches RunStepCmd over the agent command stream. Step stdout/stderr streams back as StepOutputChunk and is written to MongoDB (workflow_log_lines, one document per line); the UI streams it live. Steps support on_failure: stop|continue|retry, per-run env passed between steps via output_env, and a per-run workspace directory the agent cleans up at the end.

Default steps are seeded per org at boot (SeedDefaultSteps) from VANTAGE_DEFAULT_STEPS_DIR, which server/Dockerfile bakes to /opt/default-steps from the repo's default_steps/. Deliberately not under /data — that is a bind mount, so the library would be editable from the host. Adding a step there means committing a file and rebuilding, which is why default_steps/ is in the server rebuild trigger. Steps with source: "default" are read-only: UpdateStep/DeleteStep refuse with ErrDefaultStep (409), because seeding rewrites them on every boot, so an edit would silently revert and a delete would come back. web/ mirrors this — the step modal opens read-only, Delete is hidden, and the designer's per-step script override is readOnly for a default library step — but as elsewhere, the API is the boundary and the UI is the courtesy. Seeding writes straight to the collection rather than through UpdateStep, so the guard does not lock out the seeder. Logs are swept by retention (workflow_log_retention_days; nil = 30 days, 0 = forever).

Scheduled workflows

A workflow may carry schedule{enabled, cron, tz} — standard 5-field cron and an IANA zone name, both validated at save time. next_run_at is persisted on the document, not held in memory: a leader handover between computing an occurrence and firing it would otherwise lose it or fire it twice, the same argument that put workflow_log_seq in MongoDB.

server/internal/workflowsched ticks every 30s inside the existing bus.RunAsLeader("housekeeping", …) alongside monitorsched and the sweepers — one role, one lock. The atomic claim, not the lock, is what prevents a double fire: the UpdateOne matches on the document and its current next_run_at while setting the recomputed one, so a second process reaching the same workflow matches nothing and does nothing. The lock only makes it cheap.

workflowsched must not import services — services already imports it for SetSchedule's call to NextOccurrence, and Go has no cycles. TriggerWorkflow and LogEvent are therefore injected as workflowsched.Deps from main.go. Firing goes through the same TriggerWorkflow a person uses, with "schedule" as the actor, so there is no second dispatch path and the run detail page needed no changes.

main.go imports _ "time/tzdata", and it is load-bearing: server/Dockerfile runs on scratch, which ships no zone database, so without it time.LoadLocation("Europe/London") fails and every schedule silently falls back to UTC — an hour wrong for half the year, in the direction nobody notices until a maintenance window lands in business hours. It works on a developer machine either way, which is exactly why it gets forgotten.

Skips are recorded and surfaced, not just logged: past the 1h grace window is missed, an active run is already_running, and a schedule that no longer parses is disabled rather than left spinning the loop every 30 seconds forever.

Server tags and workflow targeting

A server carries tags map[string]string — lowercase [a-z0-9_-], key ≤32, value ≤64, 20 per server, sys: reserved. There is no tags collection: a tag is a property of a server, not an entity, so KnownTags aggregates over servers rather than reading a registry that would need reference counting to know when a tag stopped existing. PUT /api/servers/:id/tags replaces the whole map — last-write-wins over a small map beats merge semantics between two people editing one server. The index is {instance_id: 1, "tags.$**": 1}, wildcard because the queried key is chosen by the user at request time and cannot be named in advance; EnsureServerIndexes warns rather than being fatal, since a missing index degrades tag filtering to a scan of a small collection and is no reason to refuse to serve the fleet list.

services.ResolveTargets is the single answer to which servers a workflow touches — the run path and validation both go through it, so the readout and the dispatch cannot disagree. It is the distinct union of target_server_ids and target_tags (AND across keys), ordered by the fleet rather than by the arguments, so two runs naming the same servers differently are still comparable line by line. An empty selector matches nothing on purpose: "matches everything" turns a cleared field in the designer into a fleet-wide run. Both empty is ErrNoTargets (400), not a success over zero servers. Offline servers are not filtered out — the dispatcher already answers 503 per server, and a patch run that silently omits an unreachable machine is worse than one that visibly fails on it.

Both halves of the selector are edited in EditWorkflowModal — the named servers in a DualListBox, the tag rows directly beneath it — and saved together by one updateWorkflow. The designer's Targets panel is read-only: it reports the count and the tags and links to Edit. Splitting the two halves across two screens meant a workflow's reach was decided in two places with no one view showing both.

web/lib/targets.ts duplicates the match logic in TypeScript to draw the resolved count without a round trip, since the browser already holds the fleet. It is a second implementation of UnionTargets / MatchesTags and must change in the same commit as the Go one — the same shape of hazard as the mirrored token blocks. It is a shared module rather than inline in a component because the logic had already been written twice, and the second copy — the workflows list — counted target_server_ids alone, so a tag-only workflow reported zero targets while running fine.

The server picker is a hand-built two-pane list, not <select multiple>: a native multi-select paints its selected rows with the platform highlight colour, which cannot be restyled across browsers and lands outside the token palette on a dark ground.

Monitors

HTTP, TCP, ICMP and TLS checks. Each monitor has a runner: "server" (executed by the server-side scheduler) or a server_id (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond retries flip state to down, open an Incident, and notify. Hourly Rollup documents back the uptime graphs.

Notification channels

Per-org outbound destinations: webhook, smtp, discord, slack, telegram. Monitors reference channels by ID. Channels are testable from the UI.

Secrets vault

Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via secret_refs (injected as env at execution), and read by Kubernetes External Secrets Operator via GET /api/secrets/:group/values using a bearer token whose SHA-256 hash is stored in settings.

Browser console

POST /api/console/connect mints a one-time session token; GET /api/console/tunnel upgrades to a WebSocket and proxies to guacd using github.com/wwt/guac.

guacd never dials the managed server. The server binds a single-use ephemeral listener, pushes OpenProxyCmd down the agent's command stream, and the agent opens a ProxyStream and relays the connection from its own 127.0.0.1 — the host is hardcoded agent-side, so the control plane can name only a port. This is what makes the console work on Vantage Cloud, where the customer's server is behind NAT on a private address. It also means the console now requires a live agent on every deployment: consoleConnect answers 409 agent_offline rather than hanging.

guacd's Service is headless on purpose. The server resolves GUACD_ADDR to build the allow-list of sources permitted to claim a relay listener; a ClusterIP resolves to the Service's virtual address while guacd connects from its pod IP, so every relay connection is rejected and every session dies with waiting for guacd: i/o timeout. Compose is immune — there the name resolves to the address that connects.

SSH connections authenticate with a stored private key; RDP/VNC credentials are encrypted, single-use, and consumed when the tunnel opens. None of them reach the agent — the session is negotiated end-to-end between guacd and the target daemon, so the agent relays bytes it cannot read.

Running more than one server replica

An agent's CommandStream terminates on exactly one server process. Every piece of coordination below exists because of that single fact: with several replicas, the process asked to do something to an agent is almost never the process holding that agent's stream.

server/internal/bus is the Redis message bus that closes the gap. It adds no infrastructure — Redis was already required for sessions — and it is not optional on a single-replica deployment: dispatch takes the bus path always, so the code running in production is the code running everywhere, rather than a rare cross-pod branch that only fails under load.

Concern	How it crosses replicas
Which pod owns an agent	vantage:agent:<server_id> holds the owner's node ID with a 30s TTL, renewed every 10s. Dispatcher.IsConnected is an EXISTS on it
Sending a command	published to vantage:cmd:<server_id>; the owner pod acks on vantage:ack:<command_id>. Request/ack, not a queue — a command whose owner died must fail loudly (503) rather than queue. The envelope carries node, the presence holder resolved at publish time, and a pod ignores envelopes addressed elsewhere: the channel is a fan-out, and during a reconnect a half-open stream's pod is still subscribed. Unaddressed, it could ack first and queue the command onto a dead stream — the operator told it worked, the agent never seeing it. Presence renewal is owner-only (RenewPresence) for the same reason: a blind SET let the stale pod steal the key back every 10s
Step results	the owner pod publishes to vantage:res:<command_id>; the pod driving the run subscribes before dispatching, or a fast agent answers into a channel nobody has joined
Step output	never crosses. The dispatch envelope carries the secret mask list, so the owner pod masks and writes lines itself — unmasked bytes stay off the bus
Console relay	not routed to the owner pod at all. A ProxyStream is its own HTTP/2 request and an L7 proxy balances requests, not connections, so it does not follow the command stream — the listener therefore cannot be bound in advance. Whichever pod receives the stream binds it and announces its own address on vantage:proxyaddr:<proxy_id>; vantage:proxypending:<proxy_id> (30s, consumed atomically) is what authorises the claim, and the failure reason comes back on vantage:proxyend:<proxy_id>
Background jobs	bus.RunAsLeader — one Redis lock named housekeeping

Workflow logs are in MongoDB (workflow_log_lines, one document per line, with a workflow_log_seq counter document per run/server). Two pods write the same log concurrently — the run's pod emits markers, the agent's pod emits output — so ordering only means anything if both draw sequence numbers from the same counter. StepRun.log_offset is that sequence number now, not a byte offset. Writes are batched (128 lines or 250ms) and capped: 8 KB per line, 200k lines per server-run, after which one final [vantage] log truncated marker is written and the rest is dropped. Without that cap a yes in a step is a database incident. Nothing writes to /data any more, which is why server.persistence now defaults to off and VANTAGE_WORKFLOW_LOG_DIR is gone.

Shutdown order is load-bearing. main traps SIGTERM, stops gRPC (GracefulStop, 10s cap) and only then drains HTTP. Each CommandStream handler releases its agent's presence claim on return, so a killed process leaves vantage:agent:<server_id> behind for the rest of its 30s TTL — during which other replicas dispatch to a pod that has exited and the caller sees agent offline for a perfectly healthy agent. Draining HTTP first would hold those claims for the length of the drain, which is why gRPC goes first. The chart's server.terminationGracePeriodSeconds (30s) must stay above the 10s + 10s the stop sequence needs, or the kubelet SIGKILLs mid-shutdown and the handling buys nothing.

The agent side of the same failure: runCommandStream resets its backoff only after a stream that survived streamHealthyAfter. connectAndHandleStream returns an error on every stream end, healthy ones included, so without that reset the backoff only ever climbed — an agent pinned itself at the ceiling after a handful of ordinary deploys and stayed there. The ceiling is 30s, not minutes, because while the stream is down the agent still polls SyncKeys and still reads as active in the fleet list while answering no commands at all.

The leader lock is not an optimisation. N replicas each running the monitor scheduler means each check fires N times, each incident notification reaches the customer N times, and each hourly rollup is written N times; N reapers race to purge the same Free instance. monitorsched, StartReaper, StartLogSweeper, StartAuditSweeper and the offline sweep therefore all run inside one RunAsLeader("housekeeping", …) — one role, one lock. Each takes a context cancelled the instant leadership is lost, and must return when it is.

Redis rather than a Kubernetes Lease so Compose takes the identical path: one implementation to reason about, not two.

Two deployment requirements come with replicaCount > 1: every replica must share one Redis (a per-pod Redis partitions the bus and every agent looks offline to two thirds of the fleet), and POD_IP must be set — the chart does it from the downward API — because PROXY_ADVERTISE_HOST names the Service, and a Service cannot address the one pod holding a console listener.

Inventory and OS updates

Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (ApplyUpdatesCmd).

Package inventory and CVE findings

Agents report their installed packages hourly; the control plane matches them against distribution security feeds and raises findings that link to the existing ApplyUpdatesCmd patching path. Gated by the vuln_scanning licence feature, checked at collection rather than display — an ungated instance stores no inventory, and storage is the expensive half.

Matching uses distribution feeds, never NVD version ranges. Distributions backport security fixes without changing the upstream version: Ubuntu's openssl 3.0.2-0ubuntu1.15 is patched against CVE-2023-0286 while NVD still calls 3.0.2 vulnerable. Matching on NVD would report a fully patched fleet as critical, and once the first report is mostly wrong nobody reads the second. trivy-db is those feeds pre-merged; server/internal/vulndb pulls it as an OCI artifact to an ephemeral directory. Version comparison is bought from go-deb-version/go-rpm-version/go-apk-version because dpkg epochs, ~ sorting before the empty string, and rpmvercmp are each a silent false negative waiting to happen.

Only the leader matches. ReportPackages upserts the list and sets scan_pending; it does not scan. vulnsched runs inside the existing bus.RunAsLeader("housekeeping", …) and does the matching, because otherwise every replica needs the ~50MB database resident and a database refresh has N replicas rescanning the same fleet and sending N digests. The tick is also the digest's batch boundary, which is what makes "one message, not five hundred" structural rather than a debounce someone maintains.

Findings are never deleted when a package is patched — the state moves to fixed, so "what did we remediate last quarter" stays answerable. Acceptance requires a reason and an expiry, and reopens automatically: permanent dismissal is where risk goes to be forgotten. An unsupported distribution reports status: unsupported, never "0 findings"; claiming clean when the truth is unknown is the same lie as a silently stale database, which is why vulndb_meta.pulled_at is on screen rather than only in a log.

server/Dockerfile's runtime stage is scratch, so it carries an explicitly copied /tmp. The scheduler unpacks the database to a temporary directory, and a scratch image has none — the failure is vulnsched: temp dir: stat /tmp: no such file or directory, logged once at boot while every other subsystem runs normally, so the only symptom is a fleet that never reports a finding.

Two environment variables: VANTAGE_TRIVY_DB_REF mirrors the artifact for air-gapped installs, and VANTAGE_VULNDB_DISABLED switches the puller and scheduler off entirely.

Agent self-update

UpdateAgentCmd carries a target version and Gitea base URL; the agent downloads and replaces itself.

Marketing site and sitesvc

site/ is a separate Next.js app built exactly like web/ — output: "standalone", run by Node in a node:26-alpine image, listening on 3000 and published as 3003. The contact form posts to sitesvc; account signup posts to admin (NEXT_PUBLIC_ADMIN_API_URL), which creates an HQ account, not an org — the control plane is not touched until the customer later creates a cloud instance from the portal.

adminsite/ is built the same way and published as 3004, served at vantage-hq.hostxtra.co.uk — deliberately outside *.vantage.hostxtra.co.uk, because that namespace is per-tenant instance subdomains and APP_ROOT_LABEL resolves an org from the label before vantage. It shares site/'s design tokens verbatim (see Frontend below) and, unlike web/, does not proxy through a Next rewrite: the browser calls admin directly, so ADMIN_API_URL must be browser-reachable. Authenticated requests work cross-origin only because both hosts share the registrable domain hostxtra.co.uk, which keeps admin_session's SameSite=Lax cookie in play.

ADMIN_ORIGIN must list every browser origin that calls admin — currently two: https://vantage-hq.hostxtra.co.uk for the console, and https://vantage.hostxtra.co.uk because the marketing site's /start form posts account signups to admin directly. It is comma-separated. A missing origin does not produce a 403: cors() simply omits the Access-Control-Allow-Origin header and still answers the preflight 204, so the browser blocks the request and admin logs nothing at all. Symptom is a CORS preflight failure on an endpoint that works fine under curl.

sitesvc/ (port 8082) now owns only the contact flow:

Form	Endpoint	Effect
Contact	POST /api/contact	Emails support@hostxtra.co.uk, Reply-To the sender. Nothing stored.

Account signup lives in admin instead (POST /auth/signup, GET /auth/verify?token=…) — see Signup and verification below.

site, sitesvc, admin and docsite are deliberately excluded from the self-hosted deployment: deploy/docker-compose.yml mentions none of them, and they live in deploy/docker-compose.site.yml instead.

Documentation site

docsite/ is the user-facing documentation — Docusaurus 3 in docs-only mode (routeBasePath: "/", no blog), one version tracking main, search indexed at build time by @easyops-cn/docusaurus-search-local so nothing external is keyed or called. It documents the product, not the codebase: this file remains the contributor's map, and the two are allowed to differ in altitude but not in fact. Five sections — Getting started, Vantage, Vantage HQ, Reference, Operations — with sidebars.ts authored by hand so ordering is a decision rather than a filename accident.

Unlike the three Next apps it builds to static files, so its runtime stage is nginx:alpine-slim rather than Node, and it listens on 80. See the compose note below for the /docs prefix, which is the one thing about it that is easy to get wrong.

# self-hosted install — no marketing site, no sitesvc
docker compose up -d

# vantage.hostxtra.co.uk — control plane plus the public site
docker compose -f docker-compose.yml -f docker-compose.site.yml up -d

Signup and verification

Signup is account-first: it creates an HQ account and an unverified customer_user in admin's own database, nothing in the control plane. Only after a customer later creates a cloud instance from the portal (POST /api/instances, see Admin REST API) does an org, or rather an instance, come to exist — provisioned by cloudprov, with the owner's password hash copied from the HQ user rather than shared. site_pending_signups is gone; sitesvc no longer has a signup flow at all.

• The token is 32 random bytes; only its SHA-256 hash is stored, so a leaked database yields no working links.
• Links expire after 24 hours (VerifyWindow).
• An unverified sign-in gets a distinct "check your email" error rather than the generic auth failure, because the address is already known to be theirs.
• If sending the verification email fails, the freshly inserted customer_user (and account, on first signup) is rolled back rather than left stranded holding the unique index on email.
• Rate limited per client IP, plus a honeypot field.

An account is a team, not a person. customer_users.account_role is owner, admin or member — the same three words as the control plane's roles, on purpose. Owners and admins invite people, create instances and grant instance access; billing is owner-only.

An invitation creates a customer_user with an empty password hash, which cannot authenticate, and the invitee sets their own at /accept-invite. An inviter-chosen password would be a shared credential to every instance that person is later granted. GET /auth/verify therefore peeks before it consumes: a token belonging to a passwordless row answers {"needs_password":true} and is left unspent.

Email

shared/mail is the only email system. It owns the SMTP conversation, the RFC 5322 envelope and the look of every message; server, admin and sitesvc each import it and none of them builds a subject line, a MIME part or a colour. Before this existed the transport was copied three times, and the copies had already diverged once — the 465-implicit-TLS fix landed in one of them while the others silently delivered nothing.

Sender is a value, not a singleton: server/internal/notify builds one per notification channel from the channel document in Mongo, while sitesvc builds one at boot and admin holds one in admin/internal/mail.Default, alongside its other boot-time singletons. Callers only ever see typed methods — SendVerification, SendExpiring, SendMonitorAlert, SendEnquiry and the rest, grouped by owner into account.go, licence.go, billing.go, monitor.go and contact.go.

Every message is multipart/alternative, so each one is two templates: templates/<name>.html.tmpl and .txt.tmpl, embedded with go:embed. They define subject, title, pill and body; layout.html.tmpl and layout.txt.tmpl provide the chrome and the helper templates (p, lead, button, well, note, rows, chip) that the bodies compose. One template set is parsed per message rather than one big set, because every message defines those same four names. subject is defined in the txt file only — html/template would escape an ampersand in an instance name and mail clients show subjects verbatim.

shared/mail/render_test.go renders all of them and fails if a template exists that no case covers, which is the only thing standing between a mistyped field and a boot-time panic — the templates are parsed in init().

Shared provisioning

shared/provision (instance.go, slug.go, user.go) holds the slug rules, reserved names and instance/user creation logic that both server and admin/internal/cloudprov need, so there is no longer a second copy to drift: cloudprov.CreateInstance calls straight into it to create a control-plane instance and its owner from a customer request.

Grants project, they do not federate

Granting someone access to a cloud instance writes a real control-plane users row through cloudprov, with auth_source: "hq" and hq_user_id set. The instance authenticates it exactly as it authenticates anyone else, with no runtime dependency on admin. Revoking deletes that row — the control plane has no disabled state, and a row that exists is a row that can sign in.

instance_members in admin's database is only admin's index of those projections; the control-plane row is the access. That is why a failed instance_members insert unwinds the projection, and why the boot backfill can rebuild the index from the control plane but never the other way round.

Self-hosted instances are never projected into. All three mutating member endpoints refuse when deployment != cloud.

The HQ password is the single source of truth for every hq-sourced row. PUT /api/account/password rehashes and has cloudprov copy the hash to every projected row; propagation is best-effort, and admin/internal/hqsync compares and repairs every 15 minutes. It is its own package rather than a pass inside inject — inject writes three licence fields and nothing else, and that narrowness is what makes admin's reach into the control plane reviewable.

The control plane refuses to change an hq-sourced user's role or delete it (services.ErrHQManaged, 409). web/ shows those rows read-only with a link to the portal, but the API is the boundary; the UI is a courtesy. There is no local password-change endpoint at all, so there is no competing writer for the hash.

---

Auth and Orgs

• Bootstrap — first run has no users. GET /auth/bootstrap-status drives /setup, POST /auth/bootstrap creates the first org plus its owner.
• Local auth — email + password (bcrypt), POST /auth/login.
• Auth providers — configured per instance in auth_providers, any number of them, each named and independently enabled. Issuer, client ID and an encrypted client secret per provider. /auth/oidc/:providerId/start → /auth/oidc/:providerId/callback. Presets (Entra, Google, Okta, GitHub) are a Go table in server/internal/auth/presets.go and expand to a real issuer on save, so nothing downstream knows a preset existed. GitHub is OAuth2 rather than OIDC and takes its own branch, requiring an address that is both primary and verified — an unverified address is not proof of control.
• Local login — settings.local_login_enabled, a *bool because absent must mean enabled; a plain bool would disable password sign-in fleet-wide at upgrade. services.CheckLockout refuses any change leaving neither local login nor an enabled provider, and is enforced in the service layer so the settings path and the provider path cannot disagree.
• Sessions — opaque 32-byte hex ID in the km_session cookie, session body stored in Redis with a 24h TTL.
• Roles — owner, admin, member. /api/settings and /api/org/* require owner or admin.
• Host/org guard — APP_ROOT_LABEL (default vantage) defines the app root label. A request to <slug>.vantage.<tld> resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.

Unique indexes are a security property, not an optimisation. users is unique on (instance_id, email) — one address is one user within an instance, and the same address may hold a user in several instances, because an account's people are projected into each instance they are granted. This is sufficient only because every lookup by email is scoped by instance; there is deliberately no unscoped lookup anywhere, and adding one would let the login path return an arbitrary one of several matching users. Instance slug, settings instance and ESO token hash remain globally unique.

---

gRPC API

service Vantage {
  rpc Register(RegisterRequest)              returns (RegisterResponse);
  rpc SyncKeys(SyncRequest)                  returns (SyncResponse);
  rpc UploadGeneratedKey(UploadKeyRequest)   returns (UploadKeyResponse);
  rpc ReportUpdates(ReportUpdatesRequest)    returns (ReportUpdatesResponse);
  rpc ReportInventory(InventoryReport)       returns (InventoryReportResponse);
  rpc SyncMonitors(SyncMonitorsRequest)      returns (SyncMonitorsResponse);
  rpc ReportChecks(ReportChecksRequest)      returns (ReportChecksResponse);
  rpc CommandStream(stream AgentMessage)     returns (stream ServerCommand);
}

CommandStream is the only streaming RPC: the agent authenticates once with AgentReady, then the server pushes ServerCommands and the agent replies with CommandResult, StepResult, or StepOutputChunk.

ServerCommand variants: GenerateKeyCmd, DeleteKeyCmd, UpdateAgentCmd, ApplyUpdatesCmd, RunStepCmd, CleanupWorkspaceCmd, OpenProxyCmd, PingCmd.

PingCmd is a liveness beat, and it is not redundant with gRPC keepalive. The server sends one every 20s on an otherwise idle command stream; the agent treats 70s of silence as a dead stream and reconnects. Keepalive cannot do this job behind an L7 proxy: the agent's HTTP/2 connection terminates at the proxy, which answers pings on its own behalf, so a control-plane pod that dies leaves the agent blocked in Recv on a stream that never delivers another message and never errors — commands dispatched into it are silently lost while SyncKeys keeps succeeding and the fleet list still shows the server active. The agent's watchdog arms only after it has seen a first ping, so an older server that sends none is treated as working rather than put into a reconnect loop.

Key-state polling stays on the 30s SyncKeys interval. Full message definitions live in proto/vantage/v1/vantage.proto.

---

REST API

Unauthenticated:

GET  /healthz            /readyz             # liveness / readiness probes
GET  /install            /install.ps1        # dynamic agent install scripts
GET  /update             /update.ps1
GET  /auth/bootstrap-status
POST /auth/bootstrap     /auth/login  /auth/logout
GET  /auth/me
GET  /auth/providers                          # {local_enabled, providers:[{id,name,preset}]} — no issuer, client ID or secret
GET  /api/secrets/:group/values               # bearer token (ESO)

Session-authed under /api:

servers      GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
             POST /servers/:id/{generate-key,update-agent,apply-updates}
keys         GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
             POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
             POST /steps/{import,seed-defaults,parse} · GET /steps/usage
             GET,POST /workflows · GET,PUT,DELETE /workflows/:id
             POST /workflows/:id/run · GET /workflows/:id/runs
             GET /runs/:runId · POST /runs/:runId/cancel
             GET /runs/:runId/servers/:serverId/logs[/stream]
monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
             GET /monitors/:id/{incidents,uptime}
channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
             POST /secrets/:group/reveal · DELETE /secrets/:group/:key
console      POST /console/connect · GET /console/tunnel (websocket)
vulns        GET /vulnerabilities · GET /vulnerabilities/summary
             POST /vulnerabilities/rescan                          (owner|admin)
             POST,DELETE /vulnerabilities/:id/accept               (owner|admin)
             GET /servers/:id/vulnerabilities · GET /servers/:id/packages
             GET /packages/search?name=
             GET,POST /vuln-rules · PUT,DELETE /vuln-rules/:id     (owner|admin)
audit        GET /audit
agent        GET /agent/latest-version
settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
licence      GET /license · POST /license                (POST: self-hosted only)
org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
providers    GET,POST /auth/providers · PUT,DELETE /auth/providers/:id
             POST /auth/providers/:id/{test,ack-notice} · GET /auth/presets  (owner|admin)

GET /license reports deployment, and POST /license answers 409 cloud_managed when it is cloud. A cloud instance's licence is written by admin/internal/inject straight into the database and never through this endpoint, so the refusal cannot break injection — it only stops a customer pasting over a licence they do not own. web/ hides the paste form and points at the HQ portal instead, but as with hq-managed users, the API is the boundary and the UI is the courtesy.

POST /license is also in licenceExemptPaths: pasting a valid licence has to work while the current one is expired, because it is the way out of degraded mode.

Free exists in both deployments, so it is no longer cloud-only by construction. The one-Free-per-account rule is enforced per account and deployment, in licensing.checkFreeLimit and in createInstance's friendly pre-check — the two must stay scoped identically, because a pre-check stricter than the issuer refuses what would have worked. A self-hosted Free licence is claimed with POST /api/instances/:id/claim-free after the install is linked; the metered server count and per-instance feature toggles come from the instance's entitlement, which a licence snapshots at issue time (see the catalogue/entitlements note under MongoDB Collections).

---

Admin REST API (admin, :8083)

A separate service with its own session cookie (admin_session) and its own database. Unauthenticated:

GET  /healthz
GET  /auth/me                                  # who am I; 401 drives the UI's redirects
POST /auth/staff/login   /auth/login  /auth/logout
POST /auth/signup                              # self-hosted only; honeypot + rate limited
GET  /auth/verify?token=…
POST /auth/accept-invite                       # an invitee sets their own password
POST /api/paddle/webhook                       # Paddle events; signature-verified, idempotent, no session

Customer-session (/api), every instance resolved through ownedInstance:

GET  /account                                  # account, instances, max_relinks
POST /instances                                # create a cloud instance (Free tier, one Free per account per deployment)
POST /instances/:id/renew                      # Free renewal; refuses outside the renewal window
POST /instances/:id/claim-free                 # issue Free on a linked self-hosted instance
POST /instances/link · /instances/:id/relink
GET  /instances/:id/entitlement
GET  /checkout/options                         # active plans + catalogue prices for the running PADDLE_ENV
POST /instances/self-hosted                    # create a paid-checkout placeholder (awaiting_link, no licence)
POST /instances/:id/claim-link                 # bind a paid placeholder to the real UUID and issue
PUT  /instances/:id/entitlement                # set desired config; pushes line items to Paddle (owner|admin)
POST /billing/portal                           # mint a Paddle customer-portal URL
GET  /instances/:id/license · /instances/:id/license/download
GET  /subscriptions
GET,POST /account/users · PUT /account/users/:id/role · DELETE /account/users/:id
PUT  /account/password                         # propagates to every projected user
GET,POST /instances/:id/members                # cloud only
PUT  /instances/:id/members/:uid/role · DELETE /instances/:id/members/:uid

Reading is open to any signed-in member; every mutation above except /account/password (which is your own) sits behind RequireAccountRole(owner, admin). :uid is the customer_users.user_id, not the projected control-plane user_id — the portal never has to know that one.

Staff-session (/api/staff):

GET,POST /accounts · GET /accounts/:id         # search by name, email, Paddle ID or instance UUID
GET,POST /instances · GET /instances/:id       # instance + account + licence history + injection state
POST /instances/:id/issue · /instances/:id/relink
GET  /licenses · /subscriptions · /audit · /plans · PUT /plans/:deployment/:tier
GET,PUT /catalogue
GET,PUT /instances/:id/entitlement
GET  /health/injection · /health/billing

Customer endpoints answer 404, never 403, for another account's resource — a 403 confirms the resource exists. Route-group guards in adminsite/ mirror this, but the backend is the layer that matters.

Billing (Paddle)

Paddle is merchant of record; admin/internal/paddle is a thin REST client (no vendor SDK) and the only place that talks to it. Free is entirely outside Paddle — the shipped self-serve Free flow owns its own renewal, so no £0 subscription exists; an account learns its paddle_customer_id from its first paid webhook. Checkout happens in the browser (@paddle/paddle-js, token baked into the adminsite build); the server only updates a live subscription (PUT /instances/:id/entitlement) and mints a portal session.

POST /api/paddle/webhook is the only issuing path for paid plans: signature-verified with PADDLE_WEBHOOK_SECRET (boot-required), idempotent via paddle_events, and a function of the subscription's current line items — resolved back to a plan and configuration by catalogue.ResolveItems, so out-of-order delivery is correct by construction. A confirmed webhook promotes the entitlement desired→granted and signs from granted only; a checkout is built from desired. subscription.canceled and past_due take no licence action — the licence runs to its (grace-padded) expiry, then the existing lifecycle sweep lapses the instance. A renewal (transaction.completed, origin subscription_recurring) is the only moment a scheduled reduction collapses desired into granted. Self-hosted purchase creates a placeholder instance before payment (POST /instances/self-hosted); the licence is issued only once the customer pastes the install's real UUID (POST /instances/:id/claim-link), because a licence binds to that UUID.

MongoDB Collections

servers · keys · assignments · orgs · users · auth_providers · settings · secrets · workflows · workflow_steps · workflow_runs · workflow_log_lines · workflow_log_seq · monitors · incidents · monitor_rollups · notification_channels · console_sessions · audit_logs · server_packages · vuln_findings · vuln_alert_rules · vulndb_meta · migrations

Every document except migrations carries org_id. Struct definitions are the source of truth — see server/internal/models/.

Notes that are not obvious from the structs:

• servers.agent_token_hash stores SHA-256 of the token, never plaintext. pre_reg_token is cleared after Register(). status is pending → active on register, offline when last_seen passes the threshold (swept every 2 min).
• servers.inventory holds the latest metrics snapshot with separate metrics_at / static_at timestamps.
• keys.private_key_enc and passphrase_enc are AES-256-GCM; the JSON form exposes only has_private_key / has_passphrase.
• assignments.revoked_at: null means active. Revocation is soft, preserving audit history.
• workflow_runs.steps_snapshot freezes the resolved steps so editing the library never rewrites history.
• console_sessions.token_consumed_at is set atomically to enforce one-time use.
• auth_providers.provider_id is a short random identifier, not the Mongo _id: it appears in the callback URL a customer pastes into their IdP, and an _id there would publish a database key. callback_notice marks a provider migrated from the old single-provider shape, whose redirect URI therefore changed.
• workflow_log_lines is keyed (run_id, server_id, seq) — the index is not an optimisation, every read is a range scan over it. workflow_log_seq holds one counter document per run_id/server_id, which is what lets two pods interleave into one ordered log. Neither carries instance_id: they are reached only through a run, and a run is already scoped.
• users.auth_source is local, oidc or hq. An hq user was projected from a Vantage HQ account and carries hq_user_id; HQ owns its role, password and existence.
• server_packages holds a server's whole package set in one document, not one per package. The hash already established that something changed, so a report is a single atomic upsert with no delta logic to get wrong; ~2000 packages is ~150KB, well inside the 16MB limit. scan_pending lives on the document rather than in memory so a leader handover cannot lose it.
• vuln_findings is unique on (instance_id, server_id, cve_id, package_name). That key is what makes a rescan an idempotent upsert rather than a duplicate factory, and what lets first_seen survive one. An empty fixed_in means no vendor fix exists — a real state, never "not vulnerable".
• vulndb_meta is a singleton and deliberately carries no instance_id: the vulnerability database is a property of the deployment, not a tenant. Same reasoning as migrations, and the reason it is absent from services.ScopedCollections.
• services.ScopedCollections is the canonical registry of tenant-scoped collections, and scopedCollectionsForPurge derives instance deletion from it rather than keeping a second list. A new collection carrying instance_id must be added there or its rows outlive the instance.

Admin's own database is separate and holds accounts · admin_instances · licenses · subscriptions · plans · catalogue · entitlements · paddle_events · staff_users · customer_users · instance_members · admin_audit. paddle_events is the webhook idempotency log, unique on event_id: an event is claimed there before processing, and a duplicate of a handled event is a 200 no-op. instance_members is unique on (instance_id, customer_user_id) — one person holds at most one user in one instance, which makes a grant idempotent-by-refusal rather than silently doubling a projection. It is an index of the control-plane rows, not the authority (see "Grants project, they do not federate"). Admin has no migrations collection; models.Backfill runs on every boot and is idempotent by filtering on the absence of what it writes.

plans is keyed on (deployment, tier) — six rows, two deployments times three tiers — and holds base allowances only. Every Paddle price ID lives in catalogue, one row per priceable component (base, limit, feature), because a metered plan is priced by several prices and one map on a plan row cannot express that. entitlements holds one row per instance with desired beside granted: the checkout is built from desired, a licence is only ever signed from granted, and an abandoned checkout therefore leaves a desired that reached nothing. The two Free plans have no catalogue rows at all, which is what keeps Free outside Paddle.

Migrations

services.RunMigrations() runs at boot, recording markers in migrations:

• 0001_default_org_backfill
• 0002_settings_org_backfill (must run before 0003 — 0003 can create a default org, which pushes 0002 into its ambiguous multi-org branch)
• 0003_missed_org_scopes
• 0005_auth_providers — copies each instance_oidc document into auth_providers, ciphertext verbatim rather than decrypted and re-encrypted, so it does not need KEY_ENCRYPTION_KEY and cannot strand an instance's SSO configuration that has none set.

Index builders (EnsureAuthIndexes, EnsureSettingsIndexes) are fatal on failure; EnsureSecretIndexes and EnsureWorkflowIndexes only warn.

---

Agent Lifecycle

Config file

Linux /etc/vantage/config.yaml, Windows %ProgramData%\vantage\config.yaml. Directory 0700, file 0600.

server_url: "vantage.yourdomain.com:9090"
server_id: "<uuid>"
pre_reg_token: "<token>" # removed after first successful Register()
agent_token: "" # written by agent after Register()
poll_interval: 30s
tls: true

Startup

1. Load config
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
4. Enter SyncKeys poll loop (default 30s)

Poll loop

1. SyncKeys(server_id, agent_token, agent_version)
2. Non-Linux hosts stop here — Windows agents register and heartbeat only
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600

Install

Linux: systemd unit at /etc/systemd/system/vantage-agent.service, Restart=always, runs as root. Windows: MSI built by CI (WiX), or installer/setup.ps1 registering the agent as a service via NSSM.

---

Server Registration Flow

1. Add Server in the UI calls POST /api/servers/new, which generates a server_id and a pre-registration token (TTL 1 hour, single-use).
2. The UI shows a one-liner:

   curl -fsSL https://vantage.yourdomain.com/install | \
     bash -s -- --server-id=<id> --token=<token>
   

   Windows gets the /install.ps1 equivalent.
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
4. The server flips to active on first sync.

/install is served dynamically, injecting the latest agent version from the Gitea API.

---

Environment Variables (server)

Name	Required	Notes
GRPC_HOST	yes	host:port agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC.
MONGO_URI	no	default mongodb://localhost:27017
MONGO_DB	no	default vantage
REDIS_USERNAME	no	Redis 6+ ACL user. Leave empty for a legacy requirepass instance — go-redis then sends AUTH with one argument instead of two
REDIS_PASSWORD	no	empty for an unauthenticated Redis
REDIS_ADDR	no	default localhost:6379
KEY_ENCRYPTION_KEY	yes in practice	64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials.
GUACD_ADDR	no	default guacd:4822
PROXY_ADVERTISE_HOST	no	default server; the hostname guacd resolves the control plane by, handed to guacd as the relay's address. Wrong here and every console session fails at connect
PROXY_LISTEN_HOST	no	default 0.0.0.0; the interface the ephemeral relay listener binds
APP_ROOT_LABEL	no	default vantage; wrong value disables the host/session org guard
POD_IP	no	this pod's own address, set by the Helm chart from the downward API. Takes precedence over PROXY_ADVERTISE_HOST — a console relay listener belongs to one replica, and a Service address names all of them
VANTAGE_MIGRATE_ONLY	no	run schema setup (migrations, index builders, default-step seeding) and exit without serving. GRPC_HOST is not required in this mode. Set by the Helm chart's pre-upgrade Job
VANTAGE_SKIP_MIGRATIONS	no	serve without running schema setup, on the assumption a Job already did. Set by the chart's Deployment whenever server.migrationJob.enabled. Unset under Compose, where one process still migrates and then serves
VANTAGE_TRIVY_DB_REF	no	default ghcr.io/aquasecurity/trivy-db:2. Point at a mirror for an air-gapped install, or to avoid the anonymous ghcr rate limit
VANTAGE_VULNDB_DISABLED	no	true disables the vulnerability database puller and scan loop entirely. Findings already written are still served, and still shown as stale
FREE_INSTANCE_REAP_AFTER	no	duration past a Free licence's expiry before the instance and all its data are deleted. Empty disables the reaper, and empty is the default. Set to 336h in docker-compose.site.yml only — a self-hosted deployment must never reap. Must match admin's value, which only names the date in warning emails

sitesvc (deploy/docker-compose.site.yml only):

Name	Required	Notes
MONGO_URI	yes	must point at the control plane's database. sitesvc no longer provisions orgs itself, but it still refuses to start (RequireMigratedDatabase) against a database that has not run migration 0004 (the orgs → instances rename), and it (re)declares the shared users.email / instances.slug indexes at boot. The database name is read from the URI path; a URI without one is refused rather than defaulted. Note this differs from the server, which takes MONGO_DB separately.
SMTP_HOST / SMTP_FROM	yes	without them the contact form refuses (503) rather than silently dropping
SMTP_TO	no	default support@hostxtra.co.uk; contact enquiries only
SMTP_PORT	no	default 587; 465 uses implicit TLS
SMTP_USERNAME / SMTP_PASSWORD	no	auth skipped when username is empty
SITE_ORIGIN	yes in practice	comma-separated allowed origins; unset refuses every cross-origin browser request
TRUST_PROXY	no	only true behind a proxy that overwrites X-Forwarded-For, or clients spoof past the rate limiter

Ingress (Helm, Traefik)

ingress.enabled publishes two hostnames, because the two audiences arrive over different protocols:

Values	Route
ingress.web.host (+ web.extraHosts)	browsers → web:3000
ingress.api.paths (when api.enabled)	/api, /auth → <release>-server:8080, bypassing the Next proxy
ingress.grpc.host	agents → a dedicated <release>-server-grpc Service on 9090, annotated serversscheme: h2c

ingress.web.host is normally a wildcard. *.vantage.example.com is the per-tenant instance namespace — APP_ROOT_LABEL resolves the instance from the label. A Kubernetes wildcard host matches exactly one label, so it does not match the apex, and here that is correct rather than a gap: vantage.hostxtra.co.uk is the marketing site (site/, in docker-compose.site.yml), which this chart does not deploy. extraHosts is for a genuine second name; adding the apex to it would put the control plane on the marketing host. Every host in the list gets identical paths.

ingress.api.enabled routes /api and /auth straight to the server. Both arrangements work — without it web proxies those prefixes onward itself (web/next.config.ts) — but edge routing is one hop shorter and matches what the Nginx Proxy Manager in front of the Docker deployment already does, so leaving it off makes the request path a different shape on Kubernetes than in production. It stays off by default because it only helps where the server is reachable on the same host and certificate as web; turning it on blindly moves the whole API onto a route that may not be provisioned. Traefik derives router priority from rule length, so PathPrefix(/api) outranks the catch-all / with no priority annotation needed.

The gRPC route needs its own Service. The server terminates no TLS; it speaks plain h2c and always has, with TLS terminated by whatever sits in front. Traefik will not use h2c to a backend unless the Service says so, and that annotation applies to every port on the Service — so annotating the shared two-port <release>-server would force h2c on its HTTP port too.

server.env.grpcHost is not derived from ingress.grpc.host, and the chart refuses to render if they disagree. Agents dial whatever grpcHost says, and it is baked into every install one-liner; left pointing at the in-cluster Service while agents arrive through the ingress, every install succeeds and every agent then fails to connect, with nothing in the control plane explaining why. Guessing at the port (443? 9090?) would be worse than stopping.

TLS is ingress.tls.secretName / grpcSecretName (pre-existing certificates) or certResolver (Traefik ACME). Setting neither while tls.enabled produces a TLS router with no certificate, so Traefik serves its self-signed default — which looks valid and is trusted by nothing. NOTES.txt warns on install rather than the chart failing, since it is a real if unusual choice behind another terminator.

---

deploy/docker-compose.yml runs four services: redis, guacd, server (8080 + 9090), web (3000). MongoDB is external. deploy/docker-compose.site.yml adds five more — site (3003), sitesvc (8082), admin (8083), adminsite (3004) and docsite (3005) — and is only used on vantage.hostxtra.co.uk.

docsite is the odd one: a static build served by nginx:alpine-slim, not a Node runtime, and it listens on 80 rather than 3000. It is reached at vantage.hostxtra.co.uk/docs — a path on the marketing host, routed by its own Nginx Proxy Manager location, which must sort above the catch-all forwarding to site:3003 or Next answers the 404. A path and not a subdomain because *.vantage.hostxtra.co.uk is the per-tenant instance namespace and APP_ROOT_LABEL would read a docs. label as a tenant slug. NPM forwards the full path upstream — it does not strip /docs — so DOCS_BASE_URL, the proxy location and the directory the image copies the build into (/usr/share/nginx/html/docs) must all agree. When they do not, the HTML loads and every asset 404s.

LICENSE_SIGNING_KEY appears in exactly one service in exactly one compose file: admin in docker-compose.site.yml. It must never be added to server, and the self-hosted docker-compose.yml must never mention admin or adminsite at all. Admin uses an external Redis via REDIS_ADDR/REDIS_USERNAME/REDIS_PASSWORD. server now reads the same three, so a Kubernetes install can point at a managed Redis; the base compose still hardcodes an unauthenticated redis:6379 for it, so in Docker those credentials remain admin's alone.

---

Security

• gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
• Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's 0600 config.
• Pre-registration tokens are short-lived (1 hour) and single-use.
• AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
• Console session tokens are one-time; RDP credentials are consumed on tunnel open.
• ESO read token stored as a SHA-256 hash and rotatable.
• Unique indexes on (instance_id, email), instance slug, settings instance and the ESO token hash are load-bearing for tenant isolation. So is the absence of any unscoped lookup by email.
• authorized_keys written 0600, owned by root. The agent runs as root because it must.
• Every mutating API path writes an audit event.

---

Frontend

Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in web/lib/guacamole-common.js.

All four apps are one visual system, anchored on the logo navy. What differs between them is which end of it they stand on:

App	Ground	Accent	Themes
web/	--ground dark, #071628	#5b9be8	dark only, locked
site/	token-based	#0b2a58 light / #5b9be8 dark	light + dark
adminsite/	token-based	#0b2a58 light / #5b9be8 dark	light + dark, light default
docsite/	token-based	#0b2a58 light / #5b9be8 dark	light + dark, light default

adminsite/app/globals.css and docsite/src/css/custom.css hold site/app/globals.css's token blocks copied verbatim — same names, same values. web/app/globals.css holds the same tokens too, but only the dark values, since it does not switch. Change a token in all four files in the same commit; nothing enforces the match automatically, the same shape of hazard as sitesvc's mirrored slug rules.

docsite/ is the one place the tokens are not consumed through Tailwind: everything below its token block maps Docusaurus's --ifm-* variables onto them. Docusaurus already stamps data-theme on <html>, which is the selector site/'s dark block keys on, so the built-in toggle needed no wiring. The rule holds all the same — no rule in that file outside the token blocks carries a hex. The one concession is docsite/static/img/favicon.svg, which must, for the same reason the email layout must: a browser tab cannot read a token.

There is a fifth copy, and it is the one people forget: shared/mail/templates/layout.html.tmpl carries web/'s dark values as literal hex. Email clients support neither var() nor a reliable prefers-color-scheme, so the token indirection is simply not available there — an email is read before the recipient clicks through to the control plane, and the two should not look like different products. Every colour in the email system is in that one file, in the same way no component in the four web apps carries a hex.

Tailwind in all three maps var(--…) references only, so no component in any of them may carry a hex value. The names differ per app on purpose, because each app has its own subject: site/ calls the semantic three --up/--pend/--down for monitor state, adminsite/ aliases them to valid/warn/expired for licence state, and web/ to success/warning/danger. Same colours, honest names on each side.

web/ stores its tokens as RGB channel triplets with the hex in a trailing comment, and derives --token: rgb(var(--token-rgb)) from them. That is not a style preference: the console leans on Tailwind's opacity modifiers (bg-danger/10, border-accent/50, ring-accent/30) in a way the other two do not, and <alpha-value> only compiles against channels. Keep the hex comments — they are what lets the three token blocks still be diffed by eye. web/ also adds three tokens site/ has no use for: --accent-hover and --down-hover (site/ brightens with a CSS filter, which a Tailwind colour token cannot do) and --well, the floor beneath the ground for install one-liners, key blobs and run logs — surfaces showing machine output rather than interface.

web/ is locked to dark and adminsite/ defaults to light, and that pairing is the point: an operator with both open should never mistake one for the other before clicking Reissue. Now that both are drawn from the same palette the distinction rests entirely on the ground, so do not make dark the adminsite default and do not give web/ a light theme. State never reads by colour alone in either: every pill carries a distinct shape and a text label. The same argument applies one level in: the staff masthead sits on --panel-2 with a STAFF chip, so staff and customer screens are not identical either.

web/ collapses Tailwind's radius scale — md, lg and xl all resolve to site/'s 4px — rather than rewriting the ~140 rounded-lg classes across its pages. Every one of them meant "a panel corner", and tailwind.config.ts is now where that decision lives. rounded-full is untouched: status dots and pills still need it.

The adminsite/ shell. AppBar is the single masthead — identity, nav, environment, account menu — and it belongs to the two authenticated layouts, never to app/layout.tsx, so /login and /accept-invite do not render navigation they cannot use. Nav active state is derived from usePathname; do not hardcode it. PageHeader gives every screen the same back link, title, actions and record line (the reference number in mono, click-to-copy) — the reference is what people paste into support tickets, so it has a fixed slot rather than a per-page treatment. PageFrame is the main-plus-320px-rail split; the rail carries only what is true account-wide, which is why there is no plan card in it — tier, limits and expiry belong to a licence, and a licence belongs to one instance, so an account holding a Free cloud instance and a Professional self-hosted one has no single plan.

Customer nav is three destinations — Overview, People, Billing. Settings is in the account menu because it is your password, not a place, and appearance lives there too: AccountMenu is the only thing that sets data-theme, which the token blocks have always supported in both directions.

InstanceRecord is one component open or closed, and it replaced InstanceCard. Closed it is a row; open it adds licence contents, members and actions. It defaults open when the instance is the only one or needs attention, and a manual toggle is remembered per instance in localStorage. Do not reintroduce a second summary component — the split is what left a one-instance account showing a third of a row and nothing else.

Route	Purpose
/setup	First-run bootstrap: create the first org and owner
/login	Local or OIDC sign-in
/	Fleet dashboard
/servers, /servers/new, /servers/[id]	Fleet list, install one-liner, server detail (keys, inventory, updates)
/servers/[id]/console	Browser SSH/RDP/VNC session
/keys, /keys/[id]	Key library; assign and revoke per server
/workflows, /workflows/[id], /workflows/[id]/runs[/runId]	Compose, run, and follow live logs
/steps	Reusable step library
/monitors, /monitors/new, /monitors/[id][/edit]	Checks, uptime, incidents
/secrets, /secrets/[group]	Vault
/audit	Audit log
/settings, /settings/notifications, /settings/license	Members, OIDC, alerts, retention, ESO token · channels · licence

/settings is one page, not a section. Members and single sign-on used to live at /settings/instance with their own sidebar entry; they are now the Access group at the top of /settings, above Monitoring and Integrations. Splitting "who can sign in" away from "how this instance behaves" made two half-pages and a nav entry called Instance that no one could distinguish from Settings. next.config.ts keeps a permanent redirect from the old path. The cards live in web/components/settings/ rather than in the page, which is also where the Field/inputClass pair the three of them share now lives — one copy instead of the three that existed while they were apart.

---

CI/CD — Gitea Actions

agent-release.yml — triggered by agent/v* tags

Builds linux/amd64, linux/arm64, windows/amd64, writes checksums.txt, creates a Gitea release. A second msi job on windows-2022 packages the WiX installer.

GOOS=linux GOARCH=amd64 go build \
  -ldflags="-s -w -X main.Version=${VERSION}" \
  -o dist/vantage-agent-linux-amd64 ./cmd

server-deploy.yml — triggered on every push to main

Builds and pushes seven images to the Gitea container registry: server, web, site, sitesvc, admin, adminsite and docsite.

Note that despite the name, this workflow does not deploy — it only builds and pushes. There is no SSH step. Rolling images out is a separate manual step on the host:

cd /opt/vantage && docker compose -f docker-compose.yml -f docker-compose.site.yml pull && \
  docker compose -f docker-compose.yml -f docker-compose.site.yml up -d --remove-orphans

Each image only rebuilds when its own inputs changed. A git diff against github.event.before decides, which is why the checkout uses fetch-depth: 0 — the default shallow clone has one commit and nothing to diff — and why git is installed in the docker:dind container. The mapping follows the build contexts exactly:

Image	Rebuilds when
server	server/, shared/, proto/, go.work
admin	admin/, shared/, go.work
sitesvc	sitesvc/, shared/, go.work
web · site · adminsite · docsite	their own directory only

shared/ fans out to all three Go images because each of their Dockerfiles copies shared/ from a root context — if a fourth service ever imports shared/, add it to that list or it will ship stale. A change to the workflow file rebuilds everything, since a build arg is baked into the image. So does anything that leaves no trustworthy base commit: a manual workflow_dispatch, a new branch, or a force-push whose old head is gone.

The gap this leaves: changing a repo variable pushes no commit, so nothing rebuilds. After editing ADMIN_API_URL, HQ_URL or ADMIN_ENV, run the workflow manually — that is what workflow_dispatch is there for. Base images also stop being refreshed on a service nobody touches; a periodic manual run covers that.

chart-release.yml — validates on every chart change, publishes on chart/v* tags

Two jobs' worth of work in one, split by trigger. Any push or PR touching deploy/chart/ lints the chart and renders it four ways: defaults, a multi-replica install, external Redis and MongoDB, and a set of values that must be refused. That last one is the point — every safety rail in this chart is a template fail, and helm lint happily accepts a chart whose templates never execute, so only rendering proves they still fire.

Publishing runs only on a chart/v* tag, to the Gitea Helm registry at /api/packages/<owner>/helm/api/charts. Chart.yaml is the source of truth for the version; the tag only selects which one to publish, and a tag that disagrees with Chart.yaml fails rather than stamping over it — the alternative leaves the repository disagreeing with what shipped. A version already in the registry is rejected by Gitea, which is intended: published chart versions are immutable.

The registry host comes from github.server_url, so it cannot drift from the instance the workflow is running on. It authenticates with REGISTRY_USER + RELEASE_TOKEN — the pair server-deploy.yml actually uses for docker login. REGISTRY_PASSWORD is listed in the secrets table below but set by no workflow; passing an unset secret yields an empty password and Gitea answers 401 Failed to authenticate user, which reads like a scope problem on a token that was never sent. The publish step therefore checks both are non-empty before it calls curl. RELEASE_TOKEN needs write:package in addition to write:release.

helm repo add vantage https://gitea.hostxtra.co.uk/api/packages/mrhid6/helm
helm install vantage vantage/vantage --version 0.1.0

Tagging

git tag agent/v1.0.0 && git push origin agent/v1.0.0   # agent release
git tag chart/v0.1.0 && git push origin chart/v0.1.0   # helm chart package
git push origin main                                    # server + web deploy

Secrets / variables

Name	Type	Value
RELEASE_TOKEN	Secret	Gitea API token. Needs write:release (agent releases), write:package (container images and the Helm chart). This is the only token any workflow authenticates with — docker login and the chart publish both pair it with REGISTRY_USER
REGISTRY_USER	Secret	Gitea username. Must own RELEASE_TOKEN, or basic auth is rejected
REGISTRY_PASSWORD	—	Not used. Named here historically; no workflow reads it. Referencing an unset secret yields an empty password and a 401 Failed to authenticate user that looks like a token scope problem. Use RELEASE_TOKEN
DOCKER_HOST	Variable	registry host used for image tags
API_URL	not a CI variable	web reads it at runtime, from the container environment — next.config.ts is evaluated when server.js boots in standalone mode, and the rewrites it feeds are server-side, never browser-side. Default http://localhost:8080; compose sets http://server:8080. NEXT_PUBLIC_API_URL is still honoured as a fallback for existing deployments.
SITE_API_URL	Variable	browser-reachable sitesvc URL, baked into the site image. Required — if empty, both forms report "not connected" and submit nowhere. Must also be in sitesvc's SITE_ORIGIN.
SITE_CONTACT_EMAIL	Variable	optional; address shown when a form is misconfigured
SITE_URL	Variable	browser URL of the marketing site, baked into adminsite so /login can point at /start. Signup has no page in adminsite at all — one signup form, on site/. Empty renders no link rather than one that 404s.
ADMIN_API_URL	Variable	browser-reachable admin URL, baked into both the adminsite and site images — site/start posts account signups straight to admin. Same footgun as SITE_API_URL: wrong here and every request fails at runtime with the not-connected panel.
ADMIN_ENV	Variable	production or sandbox; drives the persistent environment badge. Anything but sandbox reads as production.
HQ_URL	Variable	optional; browser URL of the HQ portal, baked into web so an hq-sourced member links to where they are managed. Empty on self-hosted, which renders a plain label instead.
PADDLE_CLIENT_TOKEN	Variable	browser Paddle token, baked into the adminsite image for checkout. A repo-variable change pushes no commit, so rebuild adminsite manually via workflow_dispatch after editing it.
PADDLE_ENV	Variable	sandbox or production; baked into adminsite AND read by admin at runtime. Selects which catalogue price IDs are served, and must match on both sides.
PADDLE_API_KEY	Secret	server-side Paddle key, read by admin at runtime. Boot-required.
PADDLE_WEBHOOK_SECRET	Secret	webhook signature verification, read by admin. Boot-required — an unverified endpoint is one anyone can issue licences through.
DOCS_URL	Variable	site url baked into docsite; https://vantage.hostxtra.co.uk. Empty falls back to that default rather than breaking the build.
DOCS_BASE_URL	Variable	/docs/. Must match the NPM location and the directory the image serves from — all three, or the HTML loads and every asset 404s.
APP_URL	Variable	control-plane link in docsite's navbar.

---

Design Decisions

• gRPC for agent traffic — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
• Outbound-only agents — no inbound ports on managed servers, works behind NAT.
• Poll for keys, push for commands — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
• Atomic authorized_keys rewrite — temp file plus os.Rename(); a machine that dies mid-write keeps the old file.
• Fingerprint diffing before write — no disk churn on unchanged state.
• Soft revocation — revoked_at rather than deletes; preserves audit history.
• Run snapshots — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
• Monitors run in two places — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
• Redis for sessions only — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
• guacd for console — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
• org_id on every document — isolation enforced at the query layer, not by separate databases.
• root only — manages /root/.ssh/authorized_keys; no per-user key management.
• Windows agents are second-class by design — register, heartbeat, run steps, report inventory; no authorized_keys management.
• Both server and web scale horizontally — see "Running more than one server replica" below. web holds nothing; server holds per-agent state that is routed between replicas over Redis rather than duplicated.
• Deletion lives in the control plane — admin sends the warnings because it knows the billing address; the control plane performs the delete because it is the only service that knows which collections carry instance_id. Mirroring that list into admin would drift, and a drift there deletes the wrong rows.

graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.

Rules:

• For codebase questions, first run graphify query "<question>" when graphify-out/graph.json exists. Use graphify path "<A>" "<B>" for relationships and graphify explain "<concept>" for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
• If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
• Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
• After modifying code, run graphify update . to keep the graph current (AST-only, no API cost).