# Vantage A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management. A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands. --- ## Architecture Overview ``` ┌──────────────────────────────────────────────┐ │ Next.js 16 Frontend (web, :3000) │ │ servers · keys · workflows · monitors │ │ secrets · audit · console · settings │ └───────────────┬──────────────────────────────┘ │ REST + cookie session ┌───────────────▼──────────────────────────────┐ │ Go Backend (server) │ │ :8080 REST (gin) :9090 gRPC (agents) │ │ MongoDB (state) · Redis (sessions) │ │ monitor scheduler · workflow runner │ │ guacd tunnel proxy for browser console │ └───────────────┬──────────────────────────────┘ │ gRPC (TLS) — outbound from agent only ┌───────────────▼──────────────────────────────┐ │ Go Agent (per server, Linux + Windows) │ │ polls SyncKeys · CommandStream │ │ rewrites authorized_keys (Linux only) │ │ runs workflow steps · monitors · inventory │ └──────────────────────────────────────────────┘ ``` Multi-tenancy: every domain document carries `org_id`, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (`.vantage.`). --- ## Repository Structure ``` vantage/ ├── agent/ │ ├── cmd/main.go # flags: -generate-key │ └── internal/ │ ├── checker/ # monitor check execution │ ├── config/ # config.yaml load/save │ ├── exec/ # workflow step execution │ ├── grpc/ # client + generated pb │ ├── inventory/ # CPU/mem/disk collection (linux/other) │ ├── keys/ # authorized_keys read/diff/write │ ├── monitors/ # agent-run monitor loop │ ├── sync/ # poll loop + command stream │ └── updates/ # OS package update check/apply ├── server/ │ ├── cmd/main.go │ └── internal/ │ ├── api/ # REST handlers │ ├── auth/ # local, OIDC, session, middleware, orghost │ ├── checker/ # server-run monitor checks │ ├── db/ # mongo connect + Col() │ ├── grpc/ # gRPC server + generated pb │ ├── models/ # MongoDB documents │ ├── monitorsched/ # server-side monitor scheduler │ ├── notify/ # channel dispatch: http, discord, slack, telegram, smtp │ └── services/ # business logic + migrations ├── web/ # the application UI (authenticated) │ ├── app/(app)/ # authed routes │ ├── app/login, app/setup # unauthed routes │ ├── components/ # ui/, workflows/, monitors/, Sidebar │ └── lib/ # api client, guac console, query client ├── site/ # public marketing site │ ├── app/ # one directory per route │ ├── components/ # Nav, Footer, Logo, InstrumentPanel, forms │ ├── assets/ # image sources, not served │ └── Dockerfile # same shape as web/: standalone, node, 3000 ├── sitesvc/ # public form: contact mail only │ ├── cmd/main.go │ └── internal/ │ ├── api/ # contact │ └── store/ # Mongo connect helper ├── admin/ # licensing authority: the only signer │ ├── cmd/main.go # boot: two Mongo connections, reconciler, HTTP │ ├── cmd/adminctl/ # staff-add; deliberately has no HTTP surface │ └── internal/ │ ├── api/ # customer + staff handlers, route table │ ├── auth/ # staff, HQ customer and cloud-owner sessions │ ├── inject/ # licence write path into the control plane │ ├── cloudprov/ # instance write path: creates instances + owners │ ├── licensing/ # Issue, LinkInstance, Relink │ ├── mail/ # admin's boot-time shared/mail Sender │ └── models/ # accounts, instances, licences, plans ├── adminsite/ # staff + customer console (vantage-hq) │ ├── app/(customer)/ # overview, instance, link, billing │ ├── app/(staff)/staff/ # operations, accounts, licences, plans, audit │ ├── components/ # AppBar, PageHeader, PageFrame, InstanceRecord │ └── lib/ # api client, session guards, formatters ├── shared/ # imported by server, sitesvc and admin │ ├── mail/ # the one email system: transport + tmpl templates │ ├── license/ # payload, sign, verify, trusted keys, plans │ ├── models/ # Instance, User, Settings │ └── cmd/lkctl/ # issue and inspect licences by hand ├── proto/vantage/v1/vantage.proto ├── installer/ # Windows: setup.ps1, nssm.exe, WiX .wxs ├── deploy/ # docker-compose.yml, agent.service └── .gitea/workflows/ # agent-release.yml, server-deploy.yml ``` --- ## Subsystems ### SSH keys Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites `/root/.ssh/authorized_keys` atomically. Keys can also be generated _on_ a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted. ### Workflows A library of reusable **steps** (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into **workflows** targeting a set of servers. Running one snapshots the resolved steps into a `WorkflowRun`, then dispatches `RunStepCmd` over the agent command stream. Step stdout/stderr streams back as `StepOutputChunk` and is written to a log file on disk; the UI streams it live. Steps support `on_failure: stop|continue|retry`, per-run env passed between steps via `output_env`, and a per-run workspace directory the agent cleans up at the end. Default steps are seeded per org at boot (`SeedDefaultSteps`) from `VANTAGE_DEFAULT_STEPS_DIR`, which `server/Dockerfile` bakes to `/opt/default-steps` from the repo's `default_steps/`. Deliberately **not** under `/data` — that is a bind mount, so the library would be editable from the host. Adding a step there means committing a file and rebuilding, which is why `default_steps/` is in the `server` rebuild trigger. Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever). ### Monitors HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs. ### Notification channels Per-org outbound destinations: `webhook`, `smtp`, `discord`, `slack`, `telegram`. Monitors reference channels by ID. Channels are testable from the UI. ### Secrets vault Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via `secret_refs` (injected as env at execution), and read by Kubernetes External Secrets Operator via `GET /api/secrets/:group/values` using a bearer token whose SHA-256 hash is stored in settings. ### Browser console `POST /api/console/connect` mints a one-time session token; `GET /api/console/tunnel` upgrades to a WebSocket and proxies to **guacd** (Apache Guacamole daemon) using `github.com/wwt/guac`. SSH connections authenticate with a stored private key; RDP/VNC credentials are encrypted, single-use, and consumed when the tunnel opens. ### Inventory and OS updates Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`). ### Agent self-update `UpdateAgentCmd` carries a target version and Gitea base URL; the agent downloads and replaces itself. ### Marketing site and sitesvc `site/` is a separate Next.js app built exactly like `web/` — `output: "standalone"`, run by Node in a `node:26-alpine` image, listening on `3000` and published as `3003`. The contact form posts to `sitesvc`; account signup posts to `admin` (`NEXT_PUBLIC_ADMIN_API_URL`), which creates an HQ account, not an org — the control plane is not touched until the customer later creates a cloud instance from the portal. `adminsite/` is built the same way and published as `3004`, served at **`vantage-hq.hostxtra.co.uk`** — deliberately *outside* `*.vantage.hostxtra.co.uk`, because that namespace is per-tenant instance subdomains and `APP_ROOT_LABEL` resolves an org from the label before `vantage`. It shares `site/`'s design tokens verbatim (see Frontend below) and, unlike `web/`, does **not** proxy through a Next rewrite: the browser calls `admin` directly, so `ADMIN_API_URL` must be browser-reachable. Authenticated requests work cross-origin only because both hosts share the registrable domain `hostxtra.co.uk`, which keeps `admin_session`'s `SameSite=Lax` cookie in play. **`ADMIN_ORIGIN` must list every browser origin that calls admin — currently two**: `https://vantage-hq.hostxtra.co.uk` for the console, and `https://vantage.hostxtra.co.uk` because the marketing site's `/start` form posts account signups to admin directly. It is comma-separated. A missing origin does not produce a 403: `cors()` simply omits the `Access-Control-Allow-Origin` header and still answers the preflight `204`, so the browser blocks the request and **admin logs nothing at all**. Symptom is a CORS preflight failure on an endpoint that works fine under curl. `sitesvc/` (port `8082`) now owns only the contact flow: | Form | Endpoint | Effect | | ------- | --------------------- | ----------------------------------------------------------------------- | | Contact | `POST /api/contact` | Emails `support@hostxtra.co.uk`, `Reply-To` the sender. Nothing stored. | Account signup lives in `admin` instead (`POST /auth/signup`, `GET /auth/verify?token=…`) — see Signup and verification below. `site`, `sitesvc` and `admin` are deliberately **excluded from the self-hosted deployment**: `deploy/docker-compose.yml` mentions none of them, and they live in `deploy/docker-compose.site.yml` instead. ```bash # self-hosted install — no marketing site, no sitesvc docker compose up -d # vantage.hostxtra.co.uk — control plane plus the public site docker compose -f docker-compose.yml -f docker-compose.site.yml up -d ``` ### Signup and verification Signup is **account-first**: it creates an HQ account and an unverified `customer_user` in admin's own database, nothing in the control plane. Only after a customer later creates a cloud instance from the portal (`POST /api/instances`, see Admin REST API) does an org, or rather an `instance`, come to exist — provisioned by `cloudprov`, with the owner's password hash copied from the HQ user rather than shared. `site_pending_signups` is gone; sitesvc no longer has a signup flow at all. - The token is 32 random bytes; only its **SHA-256 hash** is stored, so a leaked database yields no working links. - Links expire after 24 hours (`VerifyWindow`). - An unverified sign-in gets a distinct "check your email" error rather than the generic auth failure, because the address is already known to be theirs. - If sending the verification email fails, the freshly inserted `customer_user` (and account, on first signup) is rolled back rather than left stranded holding the unique index on email. - Rate limited per client IP, plus a honeypot field. An account is a team, not a person. `customer_users.account_role` is `owner`, `admin` or `member` — the same three words as the control plane's roles, on purpose. Owners and admins invite people, create instances and grant instance access; billing is owner-only. An invitation creates a `customer_user` with an **empty password hash**, which cannot authenticate, and the invitee sets their own at `/accept-invite`. An inviter-chosen password would be a shared credential to every instance that person is later granted. `GET /auth/verify` therefore peeks before it consumes: a token belonging to a passwordless row answers `{"needs_password":true}` and is left unspent. ### Email `shared/mail` is the only email system. It owns the SMTP conversation, the RFC 5322 envelope and the look of every message; `server`, `admin` and `sitesvc` each import it and none of them builds a subject line, a MIME part or a colour. Before this existed the transport was copied three times, and the copies had already diverged once — the 465-implicit-TLS fix landed in one of them while the others silently delivered nothing. `Sender` is a value, not a singleton: `server/internal/notify` builds one per notification channel from the channel document in Mongo, while `sitesvc` builds one at boot and `admin` holds one in `admin/internal/mail.Default`, alongside its other boot-time singletons. Callers only ever see typed methods — `SendVerification`, `SendExpiring`, `SendMonitorAlert`, `SendEnquiry` and the rest, grouped by owner into `account.go`, `licence.go`, `billing.go`, `monitor.go` and `contact.go`. Every message is `multipart/alternative`, so each one is two templates: `templates/.html.tmpl` and `.txt.tmpl`, embedded with `go:embed`. They define `subject`, `title`, `pill` and `body`; `layout.html.tmpl` and `layout.txt.tmpl` provide the chrome and the helper templates (`p`, `lead`, `button`, `well`, `note`, `rows`, `chip`) that the bodies compose. One template set is parsed per message rather than one big set, because every message defines those same four names. **`subject` is defined in the txt file only** — `html/template` would escape an ampersand in an instance name and mail clients show subjects verbatim. `shared/mail/render_test.go` renders all of them and fails if a template exists that no case covers, which is the only thing standing between a mistyped field and a boot-time panic — the templates are parsed in `init()`. ### Shared provisioning `shared/provision` (`instance.go`, `slug.go`, `user.go`) holds the slug rules, reserved names and instance/user creation logic that both `server` and `admin/internal/cloudprov` need, so there is no longer a second copy to drift: `cloudprov.CreateInstance` calls straight into it to create a control-plane instance and its owner from a customer request. ### Grants project, they do not federate Granting someone access to a cloud instance writes a real control-plane `users` row through `cloudprov`, with `auth_source: "hq"` and `hq_user_id` set. The instance authenticates it exactly as it authenticates anyone else, with **no runtime dependency on admin**. Revoking deletes that row — the control plane has no disabled state, and a row that exists is a row that can sign in. `instance_members` in admin's database is only admin's *index* of those projections; the control-plane row is the access. That is why a failed `instance_members` insert unwinds the projection, and why the boot backfill can rebuild the index from the control plane but never the other way round. **Self-hosted instances are never projected into.** All three mutating member endpoints refuse when `deployment != cloud`. The HQ password is the single source of truth for every `hq`-sourced row. `PUT /api/account/password` rehashes and has `cloudprov` copy the hash to every projected row; propagation is best-effort, and `admin/internal/hqsync` compares and repairs every 15 minutes. It is **its own package rather than a pass inside `inject`** — `inject` writes three licence fields and nothing else, and that narrowness is what makes admin's reach into the control plane reviewable. The control plane refuses to change an `hq`-sourced user's role or delete it (`services.ErrHQManaged`, 409). `web/` shows those rows read-only with a link to the portal, but the API is the boundary; the UI is a courtesy. There is no local password-change endpoint at all, so there is no competing writer for the hash. --- ## Auth and Orgs - **Bootstrap** — first run has no users. `GET /auth/bootstrap-status` drives `/setup`, `POST /auth/bootstrap` creates the first org plus its owner. - **Local auth** — email + password (bcrypt), `POST /auth/login`. - **OIDC** — configured _per org_ (`org_oidc`), issuer + client ID + encrypted client secret. `/auth/oidc/start` → `/auth/oidc/callback`. - **Sessions** — opaque 32-byte hex ID in the `km_session` cookie, session body stored in Redis with a 24h TTL. - **Roles** — `owner`, `admin`, `member`. `/api/settings` and `/api/org/*` require owner or admin. - **Host/org guard** — `APP_ROOT_LABEL` (default `vantage`) defines the app root label. A request to `.vantage.` resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s. Unique indexes are a **security property**, not an optimisation. `users` is unique on `(instance_id, email)` — one address is one user *within* an instance, and the same address may hold a user in several instances, because an account's people are projected into each instance they are granted. This is sufficient only because **every lookup by email is scoped by instance**; there is deliberately no unscoped lookup anywhere, and adding one would let the login path return an arbitrary one of several matching users. Instance slug, settings instance and ESO token hash remain globally unique. --- ## gRPC API ```protobuf service Vantage { rpc Register(RegisterRequest) returns (RegisterResponse); rpc SyncKeys(SyncRequest) returns (SyncResponse); rpc UploadGeneratedKey(UploadKeyRequest) returns (UploadKeyResponse); rpc ReportUpdates(ReportUpdatesRequest) returns (ReportUpdatesResponse); rpc ReportInventory(InventoryReport) returns (InventoryReportResponse); rpc SyncMonitors(SyncMonitorsRequest) returns (SyncMonitorsResponse); rpc ReportChecks(ReportChecksRequest) returns (ReportChecksResponse); rpc CommandStream(stream AgentMessage) returns (stream ServerCommand); } ``` `CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`. `ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`. Key-state polling stays on the 30s `SyncKeys` interval. Full message definitions live in `proto/vantage/v1/vantage.proto`. --- ## REST API Unauthenticated: ``` GET /install /install.ps1 # dynamic agent install scripts GET /update /update.ps1 GET /auth/bootstrap-status POST /auth/bootstrap /auth/login /auth/logout GET /auth/me /auth/oidc/start /auth/oidc/callback GET /api/secrets/:group/values # bearer token (ESO) ``` Session-authed under `/api`: ``` servers GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id POST /servers/:id/{generate-key,update-agent,apply-updates} keys GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId workflows GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export POST /steps/{import,seed-defaults,parse} · GET /steps/usage GET,POST /workflows · GET,PUT,DELETE /workflows/:id POST /workflows/:id/run · GET /workflows/:id/runs GET /runs/:runId · POST /runs/:runId/cancel GET /runs/:runId/servers/:serverId/logs[/stream] monitors GET,POST /monitors · GET,PUT,DELETE /monitors/:id GET /monitors/:id/{incidents,uptime} channels GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test secrets GET,POST /secrets · GET,PUT,DELETE /secrets/:group POST /secrets/:group/reveal · DELETE /secrets/:group/:key console POST /console/connect · GET /console/tunnel (websocket) audit GET /audit agent GET /agent/latest-version settings GET,PUT /settings · POST /settings/secrets-token (owner|admin) licence GET /license · POST /license (POST: self-hosted only) org GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id GET,PUT /org/oidc (owner|admin) ``` `GET /license` reports `deployment`, and **`POST /license` answers 409 `cloud_managed` when it is `cloud`**. A cloud instance's licence is written by `admin/internal/inject` straight into the database and never through this endpoint, so the refusal cannot break injection — it only stops a customer pasting over a licence they do not own. `web/` hides the paste form and points at the HQ portal instead, but as with `hq`-managed users, the API is the boundary and the UI is the courtesy. `POST /license` is also in `licenceExemptPaths`: pasting a valid licence has to work while the current one is expired, because it is the way out of degraded mode. Free exists in both deployments, so it is no longer cloud-only by construction. The one-Free-per-account rule is enforced per account **and deployment**, in `licensing.checkFreeLimit` and in `createInstance`'s friendly pre-check — the two must stay scoped identically, because a pre-check stricter than the issuer refuses what would have worked. A self-hosted Free licence is claimed with `POST /api/instances/:id/claim-free` after the install is linked; the metered server count and per-instance feature toggles come from the instance's `entitlement`, which a licence snapshots at issue time (see the `catalogue`/`entitlements` note under MongoDB Collections). --- ## Admin REST API (`admin`, :8083) A separate service with its own session cookie (`admin_session`) and its own database. Unauthenticated: ``` GET /healthz GET /auth/me # who am I; 401 drives the UI's redirects POST /auth/staff/login /auth/login /auth/logout POST /auth/signup # self-hosted only; honeypot + rate limited GET /auth/verify?token=… POST /auth/accept-invite # an invitee sets their own password POST /api/paddle/webhook # Paddle events; signature-verified, idempotent, no session ``` Customer-session (`/api`), every instance resolved through `ownedInstance`: ``` GET /account # account, instances, max_relinks POST /instances # create a cloud instance (Free tier, one Free per account per deployment) POST /instances/:id/renew # Free renewal; refuses outside the renewal window POST /instances/:id/claim-free # issue Free on a linked self-hosted instance POST /instances/link · /instances/:id/relink GET /instances/:id/entitlement GET /checkout/options # active plans + catalogue prices for the running PADDLE_ENV POST /instances/self-hosted # create a paid-checkout placeholder (awaiting_link, no licence) POST /instances/:id/claim-link # bind a paid placeholder to the real UUID and issue PUT /instances/:id/entitlement # set desired config; pushes line items to Paddle (owner|admin) POST /billing/portal # mint a Paddle customer-portal URL GET /instances/:id/license · /instances/:id/license/download GET /subscriptions GET,POST /account/users · PUT /account/users/:id/role · DELETE /account/users/:id PUT /account/password # propagates to every projected user GET,POST /instances/:id/members # cloud only PUT /instances/:id/members/:uid/role · DELETE /instances/:id/members/:uid ``` Reading is open to any signed-in member; every mutation above except `/account/password` (which is your own) sits behind `RequireAccountRole(owner, admin)`. `:uid` is the **`customer_users.user_id`**, not the projected control-plane user_id — the portal never has to know that one. Staff-session (`/api/staff`): ``` GET,POST /accounts · GET /accounts/:id # search by name, email, Paddle ID or instance UUID GET,POST /instances · GET /instances/:id # instance + account + licence history + injection state POST /instances/:id/issue · /instances/:id/relink GET /licenses · /subscriptions · /audit · /plans · PUT /plans/:deployment/:tier GET,PUT /catalogue GET,PUT /instances/:id/entitlement GET /health/injection · /health/billing ``` **Customer endpoints answer 404, never 403, for another account's resource** — a 403 confirms the resource exists. Route-group guards in `adminsite/` mirror this, but the backend is the layer that matters. ### Billing (Paddle) Paddle is merchant of record; `admin/internal/paddle` is a thin REST client (no vendor SDK) and the only place that talks to it. **Free is entirely outside Paddle** — the shipped self-serve Free flow owns its own renewal, so no £0 subscription exists; an account learns its `paddle_customer_id` from its first paid webhook. Checkout happens in the browser (`@paddle/paddle-js`, token baked into the adminsite build); the server only updates a live subscription (`PUT /instances/:id/entitlement`) and mints a portal session. `POST /api/paddle/webhook` is the **only** issuing path for paid plans: signature-verified with `PADDLE_WEBHOOK_SECRET` (boot-required), idempotent via `paddle_events`, and a function of the subscription's *current* line items — resolved back to a plan and configuration by `catalogue.ResolveItems`, so out-of-order delivery is correct by construction. A confirmed webhook promotes the entitlement `desired`→`granted` and signs from `granted` **only**; a checkout is built from `desired`. `subscription.canceled` and `past_due` take **no licence action** — the licence runs to its (grace-padded) expiry, then the existing lifecycle sweep lapses the instance. A renewal (`transaction.completed`, origin `subscription_recurring`) is the only moment a scheduled reduction collapses `desired` into `granted`. Self-hosted purchase creates a placeholder instance before payment (`POST /instances/self-hosted`); the licence is issued only once the customer pastes the install's real UUID (`POST /instances/:id/claim-link`), because a licence binds to that UUID. ## MongoDB Collections `servers` · `keys` · `assignments` · `orgs` · `users` · `org_oidc` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `migrations` Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`. Notes that are not obvious from the structs: - `servers.agent_token_hash` stores SHA-256 of the token, never plaintext. `pre_reg_token` is cleared after `Register()`. `status` is `pending` → `active` on register, `offline` when `last_seen` passes the threshold (swept every 2 min). - `servers.inventory` holds the latest metrics snapshot with separate `metrics_at` / `static_at` timestamps. - `keys.private_key_enc` and `passphrase_enc` are AES-256-GCM; the JSON form exposes only `has_private_key` / `has_passphrase`. - `assignments.revoked_at: null` means active. Revocation is soft, preserving audit history. - `workflow_runs.steps_snapshot` freezes the resolved steps so editing the library never rewrites history. - `console_sessions.token_consumed_at` is set atomically to enforce one-time use. - `users.auth_source` is `local`, `oidc` or `hq`. An `hq` user was projected from a Vantage HQ account and carries `hq_user_id`; HQ owns its role, password and existence. Admin's own database is separate and holds `accounts` · `admin_instances` · `licenses` · `subscriptions` · `plans` · `catalogue` · `entitlements` · `paddle_events` · `staff_users` · `customer_users` · `instance_members` · `admin_audit`. `paddle_events` is the webhook idempotency log, unique on `event_id`: an event is claimed there before processing, and a duplicate of a handled event is a 200 no-op. `instance_members` is unique on `(instance_id, customer_user_id)` — one person holds at most one user in one instance, which makes a grant idempotent-by-refusal rather than silently doubling a projection. It is an *index* of the control-plane rows, not the authority (see "Grants project, they do not federate"). Admin has no migrations collection; `models.Backfill` runs on every boot and is idempotent by filtering on the absence of what it writes. `plans` is keyed on `(deployment, tier)` — six rows, two deployments times three tiers — and holds base allowances only. **Every Paddle price ID lives in `catalogue`**, one row per priceable component (`base`, `limit`, `feature`), because a metered plan is priced by several prices and one map on a plan row cannot express that. `entitlements` holds one row per instance with `desired` beside `granted`: the checkout is built from `desired`, a licence is only ever signed from `granted`, and an abandoned checkout therefore leaves a `desired` that reached nothing. The two Free plans have **no catalogue rows at all**, which is what keeps Free outside Paddle. ### Migrations `services.RunMigrations()` runs at boot, recording markers in `migrations`: - `0001_default_org_backfill` - `0002_settings_org_backfill` (must run before 0003 — 0003 can create a `default` org, which pushes 0002 into its ambiguous multi-org branch) - `0003_missed_org_scopes` Index builders (`EnsureAuthIndexes`, `EnsureSettingsIndexes`) are fatal on failure; `EnsureSecretIndexes` and `EnsureWorkflowIndexes` only warn. --- ## Agent Lifecycle ### Config file Linux `/etc/vantage/config.yaml`, Windows `%ProgramData%\vantage\config.yaml`. Directory `0700`, file `0600`. ```yaml server_url: "vantage.yourdomain.com:9090" server_id: "" pre_reg_token: "" # removed after first successful Register() agent_token: "" # written by agent after Register() poll_interval: 30s tls: true ``` ### Startup ``` 1. Load config 2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect 3. Start goroutines: command stream · update check (hourly) · inventory · monitors 4. Enter SyncKeys poll loop (default 30s) ``` ### Poll loop ``` 1. SyncKeys(server_id, agent_token, agent_version) 2. Non-Linux hosts stop here — Windows agents register and heartbeat only 3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write 4. Changed → write .tmp, os.Rename() over the real file, chmod 0600 ``` ### Install Linux: systemd unit at `/etc/systemd/system/vantage-agent.service`, `Restart=always`, runs as root. Windows: MSI built by CI (WiX), or `installer/setup.ps1` registering the agent as a service via NSSM. --- ## Server Registration Flow 1. **Add Server** in the UI calls `POST /api/servers/new`, which generates a `server_id` and a pre-registration token (TTL 1 hour, single-use). 2. The UI shows a one-liner: ```bash curl -fsSL https://vantage.yourdomain.com/install | \ bash -s -- --server-id= --token= ``` Windows gets the `/install.ps1` equivalent. 3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service. 4. The server flips to `active` on first sync. `/install` is served dynamically, injecting the latest agent version from the Gitea API. --- ## Environment Variables (server) | Name | Required | Notes | | -------------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GRPC_HOST` | **yes** | `host:port` agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC. | | `MONGO_URI` | no | default `mongodb://localhost:27017` | | `MONGO_DB` | no | default `vantage` | | `REDIS_ADDR` | no | default `localhost:6379` | | `KEY_ENCRYPTION_KEY` | yes in practice | 64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials. | | `GITEA_HOST` | yes | used to build install scripts and agent download URLs | | `GUACD_ADDR` | no | default `guacd:4822` | | `APP_ROOT_LABEL` | no | default `vantage`; wrong value disables the host/session org guard | | `VANTAGE_WORKFLOW_LOG_DIR` | no | where run logs are written | | `FREE_INSTANCE_REAP_AFTER` | no | duration past a Free licence's expiry before the instance and all its data are deleted. **Empty disables the reaper, and empty is the default.** Set to `336h` in `docker-compose.site.yml` only — a self-hosted deployment must never reap. Must match admin's value, which only names the date in warning emails | **sitesvc** (`deploy/docker-compose.site.yml` only): | Name | Required | Notes | | --------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `MONGO_URI` | yes | **must point at the control plane's database.** sitesvc no longer provisions orgs itself, but it still refuses to start (`RequireMigratedDatabase`) against a database that has not run migration `0004` (the `orgs` → `instances` rename), and it (re)declares the shared `users.email` / `instances.slug` indexes at boot. The database name is read from the URI path; a URI without one is refused rather than defaulted. Note this differs from the server, which takes `MONGO_DB` separately. | | `SMTP_HOST` / `SMTP_FROM` | yes | without them the contact form refuses (503) rather than silently dropping | | `SMTP_TO` | no | default `support@hostxtra.co.uk`; contact enquiries only | | `SMTP_PORT` | no | default `587`; `465` uses implicit TLS | | `SMTP_USERNAME` / `SMTP_PASSWORD` | no | auth skipped when username is empty | | `SITE_ORIGIN` | yes in practice | comma-separated allowed origins; unset refuses every cross-origin browser request | | `TRUST_PROXY` | no | only `true` behind a proxy that overwrites `X-Forwarded-For`, or clients spoof past the rate limiter | `deploy/docker-compose.yml` runs four services: `redis`, `guacd`, `server` (8080 + 9090), `web` (3000). MongoDB is external. `deploy/docker-compose.site.yml` adds four more — `site` (3003), `sitesvc` (8082), `admin` (8083) and `adminsite` (3004) — and is only used on vantage.hostxtra.co.uk. `LICENSE_SIGNING_KEY` appears in **exactly one service in exactly one compose file**: `admin` in `docker-compose.site.yml`. It must never be added to `server`, and the self-hosted `docker-compose.yml` must never mention `admin` or `adminsite` at all. Admin uses an external Redis via `REDIS_ADDR`/`REDIS_USERNAME`/`REDIS_PASSWORD`; the base compose hardcodes `redis:6379` for `server`, so those variables reach admin only. --- ## Security - gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers. - Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's `0600` config. - Pre-registration tokens are short-lived (1 hour) and single-use. - AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials. - Console session tokens are one-time; RDP credentials are consumed on tunnel open. - ESO read token stored as a SHA-256 hash and rotatable. - Unique indexes on `(instance_id, email)`, instance slug, settings instance and the ESO token hash are load-bearing for tenant isolation. So is the absence of any unscoped lookup by email. - `authorized_keys` written `0600`, owned by root. The agent runs as root because it must. - Every mutating API path writes an audit event. --- ## Frontend Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in `web/lib/guacamole-common.js`. All three apps are **one visual system**, anchored on the logo navy. What differs between them is which end of it they stand on: | App | Ground | Accent | Themes | | --- | --- | --- | --- | | `web/` | `--ground` dark, `#071628` | `#5b9be8` | dark only, locked | | `site/` | token-based | `#0b2a58` light / `#5b9be8` dark | light + dark | | `adminsite/` | token-based | `#0b2a58` light / `#5b9be8` dark | light + dark, light default | `adminsite/app/globals.css` holds `site/app/globals.css`'s token blocks **copied verbatim** — same names, same values. `web/app/globals.css` holds the same tokens too, but only the **dark** values, since it does not switch. **Change a token in all three files in the same commit; nothing enforces the match automatically**, the same shape of hazard as sitesvc's mirrored slug rules. There is a **fourth** copy, and it is the one people forget: `shared/mail/templates/layout.html.tmpl` carries web/'s dark values as literal hex. Email clients support neither `var()` nor a reliable `prefers-color-scheme`, so the token indirection is simply not available there — an email is read before the recipient clicks through to the control plane, and the two should not look like different products. Every colour in the email system is in that one file, in the same way no component in the three web apps carries a hex. Tailwind in all three maps `var(--…)` references only, so **no component in any of them may carry a hex value**. The names differ per app on purpose, because each app has its own subject: `site/` calls the semantic three `--up`/`--pend`/`--down` for monitor state, `adminsite/` aliases them to `valid`/`warn`/`expired` for licence state, and `web/` to `success`/`warning`/`danger`. Same colours, honest names on each side. `web/` stores its tokens as **RGB channel triplets** with the hex in a trailing comment, and derives `--token: rgb(var(--token-rgb))` from them. That is not a style preference: the console leans on Tailwind's opacity modifiers (`bg-danger/10`, `border-accent/50`, `ring-accent/30`) in a way the other two do not, and `` only compiles against channels. Keep the hex comments — they are what lets the three token blocks still be diffed by eye. `web/` also adds three tokens site/ has no use for: `--accent-hover` and `--down-hover` (site/ brightens with a CSS `filter`, which a Tailwind colour token cannot do) and `--well`, the floor beneath the ground for install one-liners, key blobs and run logs — surfaces showing machine output rather than interface. `web/` is locked to dark and `adminsite/` defaults to **light**, and that pairing is the point: an operator with both open should never mistake one for the other before clicking Reissue. Now that both are drawn from the same palette the distinction rests **entirely** on the ground, so do not make dark the adminsite default and do not give web/ a light theme. State never reads by colour alone in either: every pill carries a distinct shape and a text label. The same argument applies one level in: the **staff** masthead sits on `--panel-2` with a `STAFF` chip, so staff and customer screens are not identical either. `web/` collapses Tailwind's radius scale — `md`, `lg` and `xl` all resolve to site/'s 4px — rather than rewriting the ~140 `rounded-lg` classes across its pages. Every one of them meant "a panel corner", and `tailwind.config.ts` is now where that decision lives. `rounded-full` is untouched: status dots and pills still need it. **The `adminsite/` shell.** `AppBar` is the single masthead — identity, nav, environment, account menu — and it belongs to the two authenticated layouts, never to `app/layout.tsx`, so `/login` and `/accept-invite` do not render navigation they cannot use. Nav active state is derived from `usePathname`; do not hardcode it. `PageHeader` gives every screen the same back link, title, actions and **record line** (the reference number in mono, click-to-copy) — the reference is what people paste into support tickets, so it has a fixed slot rather than a per-page treatment. `PageFrame` is the main-plus-320px-rail split; the rail carries only what is true account-wide, which is why there is no plan card in it — **tier, limits and expiry belong to a licence, and a licence belongs to one instance**, so an account holding a Free cloud instance and a Professional self-hosted one has no single plan. Customer nav is three destinations — Overview, People, Billing. Settings is in the account menu because it is your password, not a place, and appearance lives there too: `AccountMenu` is the only thing that sets `data-theme`, which the token blocks have always supported in both directions. `InstanceRecord` is one component open or closed, and it **replaced** `InstanceCard`. Closed it is a row; open it adds licence contents, members and actions. It defaults open when the instance is the only one or needs attention, and a manual toggle is remembered per instance in `localStorage`. Do not reintroduce a second summary component — the split is what left a one-instance account showing a third of a row and nothing else. | Route | Purpose | | --------------------------------------------------------------- | ----------------------------------------------------------------------- | | `/setup` | First-run bootstrap: create the first org and owner | | `/login` | Local or OIDC sign-in | | `/` | Fleet dashboard | | `/servers`, `/servers/new`, `/servers/[id]` | Fleet list, install one-liner, server detail (keys, inventory, updates) | | `/servers/[id]/console` | Browser SSH/RDP/VNC session | | `/keys`, `/keys/[id]` | Key library; assign and revoke per server | | `/workflows`, `/workflows/[id]`, `/workflows/[id]/runs[/runId]` | Compose, run, and follow live logs | | `/steps` | Reusable step library | | `/monitors`, `/monitors/new`, `/monitors/[id][/edit]` | Checks, uptime, incidents | | `/secrets`, `/secrets/[group]` | Vault | | `/audit` | Audit log | | `/settings`, `/settings/notifications`, `/settings/license` | Members, OIDC, alerts, retention, ESO token · channels · licence | **`/settings` is one page, not a section.** Members and single sign-on used to live at `/settings/instance` with their own sidebar entry; they are now the **Access** group at the top of `/settings`, above **Monitoring** and **Integrations**. Splitting "who can sign in" away from "how this instance behaves" made two half-pages and a nav entry called Instance that no one could distinguish from Settings. `next.config.ts` keeps a permanent redirect from the old path. The cards live in `web/components/settings/` rather than in the page, which is also where the `Field`/`inputClass` pair the three of them share now lives — one copy instead of the three that existed while they were apart. --- ## CI/CD — Gitea Actions ### `agent-release.yml` — triggered by `agent/v*` tags Builds `linux/amd64`, `linux/arm64`, `windows/amd64`, writes `checksums.txt`, creates a Gitea release. A second `msi` job on `windows-2022` packages the WiX installer. ```bash GOOS=linux GOARCH=amd64 go build \ -ldflags="-s -w -X main.Version=${VERSION}" \ -o dist/vantage-agent-linux-amd64 ./cmd ``` ### `server-deploy.yml` — triggered on every push to `main` Builds and pushes six images to the Gitea container registry: `server`, `web`, `site`, `sitesvc`, `admin` and `adminsite`. Note that despite the name, **this workflow does not deploy** — it only builds and pushes. There is no SSH step. Rolling images out is a separate manual step on the host: ```bash cd /opt/vantage && docker compose -f docker-compose.yml -f docker-compose.site.yml pull && \ docker compose -f docker-compose.yml -f docker-compose.site.yml up -d --remove-orphans ``` **Each image only rebuilds when its own inputs changed.** A `git diff` against `github.event.before` decides, which is why the checkout uses `fetch-depth: 0` — the default shallow clone has one commit and nothing to diff — and why `git` is installed in the `docker:dind` container. The mapping follows the build contexts exactly: | Image | Rebuilds when | | --- | --- | | `server` | `server/`, `shared/`, `proto/`, `go.work` | | `admin` | `admin/`, `shared/`, `go.work` | | `sitesvc` | `sitesvc/`, `shared/`, `go.work` | | `web` · `site` · `adminsite` | their own directory only | `shared/` fans out to all three Go images because each of their Dockerfiles copies `shared/` from a root context — **if a fourth service ever imports `shared/`, add it to that list or it will ship stale**. A change to the workflow file rebuilds everything, since a build arg is baked into the image. So does anything that leaves no trustworthy base commit: a manual `workflow_dispatch`, a new branch, or a force-push whose old head is gone. The gap this leaves: **changing a repo variable pushes no commit, so nothing rebuilds.** After editing `API_URL`, `ADMIN_API_URL`, `HQ_URL` or `ADMIN_ENV`, run the workflow manually — that is what `workflow_dispatch` is there for. Base images also stop being refreshed on a service nobody touches; a periodic manual run covers that. ### Tagging ```bash git tag agent/v1.0.0 && git push origin agent/v1.0.0 # agent release git push origin main # server + web deploy ``` ### Secrets / variables | Name | Type | Value | | -------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RELEASE_TOKEN` | Secret | Gitea API token, `write:release` | | `REGISTRY_USER` | Secret | Gitea username | | `REGISTRY_PASSWORD` | Secret | Gitea token, `write:packages` | | `GITEA_HOST` | Variable | `gitea.hostxtra.co.uk` | | `DOCKER_HOST` | Variable | registry host used for image tags | | `API_URL` | Variable | baked into the `web` image at build time | | `SITE_API_URL` | Variable | **browser-reachable** sitesvc URL, baked into the `site` image. Required — if empty, both forms report "not connected" and submit nowhere. Must also be in sitesvc's `SITE_ORIGIN`. | | `SITE_CONTACT_EMAIL` | Variable | optional; address shown when a form is misconfigured | | `ADMIN_API_URL` | Variable | **browser-reachable** admin URL, baked into **both** the `adminsite` and `site` images — `site/start` posts account signups straight to admin. Same footgun as `SITE_API_URL`: wrong here and every request fails at runtime with the not-connected panel. | | `ADMIN_ENV` | Variable | `production` or `sandbox`; drives the persistent environment badge. Anything but `sandbox` reads as production. | | `HQ_URL` | Variable | optional; browser URL of the HQ portal, baked into `web` so an `hq`-sourced member links to where they are managed. Empty on self-hosted, which renders a plain label instead. | | `PADDLE_CLIENT_TOKEN`| Variable | **browser** Paddle token, baked into the `adminsite` image for checkout. A repo-variable change pushes no commit, so rebuild `adminsite` manually via `workflow_dispatch` after editing it. | | `PADDLE_ENV` | Variable | `sandbox` or `production`; baked into `adminsite` AND read by `admin` at runtime. Selects which `catalogue` price IDs are served, and must match on both sides. | | `PADDLE_API_KEY` | Secret | server-side Paddle key, read by `admin` at runtime. Boot-required. | | `PADDLE_WEBHOOK_SECRET` | Secret | webhook signature verification, read by `admin`. Boot-required — an unverified endpoint is one anyone can issue licences through. | --- ## Design Decisions - **gRPC for agent traffic** — strong typing and cheap versioning; polling for state, one bidirectional stream for commands. - **Outbound-only agents** — no inbound ports on managed servers, works behind NAT. - **Poll for keys, push for commands** — a 30s key poll is fine, but running a workflow step should not wait up to 30s. - **Atomic `authorized_keys` rewrite** — temp file plus `os.Rename()`; a machine that dies mid-write keeps the old file. - **Fingerprint diffing before write** — no disk churn on unchanged state. - **Soft revocation** — `revoked_at` rather than deletes; preserves audit history. - **Run snapshots** — workflow runs freeze their resolved steps so editing a step never rewrites past runs. - **Monitors run in two places** — server-side for external endpoints, agent-side for anything only reachable from inside the target network. - **Redis for sessions only** — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else. - **guacd for console** — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials. - **`org_id` on every document** — isolation enforced at the query layer, not by separate databases. - **root only** — manages `/root/.ssh/authorized_keys`; no per-user key management. - **Windows agents are second-class by design** — register, heartbeat, run steps, report inventory; no `authorized_keys` management. - **Deletion lives in the control plane** — admin sends the warnings because it knows the billing address; the control plane performs the delete because it is the only service that knows which collections carry `instance_id`. Mirroring that list into admin would drift, and a drift there deletes the wrong rows.