vantage/claude.md

Vantage

A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.

A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.

---

Architecture Overview

┌──────────────────────────────────────────────┐
│  Next.js 16 Frontend (web, :3000)            │
│  servers · keys · workflows · monitors       │
│  secrets · audit · console · settings        │
└───────────────┬──────────────────────────────┘
                │ REST + cookie session
┌───────────────▼──────────────────────────────┐
│  Go Backend (server)                         │
│  :8080 REST (gin)   :9090 gRPC (agents)      │
│  MongoDB (state) · Redis (sessions)          │
│  monitor scheduler · workflow runner         │
│  guacd tunnel proxy for browser console      │
└───────────────┬──────────────────────────────┘
                │ gRPC (TLS) — outbound from agent only
┌───────────────▼──────────────────────────────┐
│  Go Agent (per server, Linux + Windows)      │
│  polls SyncKeys · CommandStream              │
│  rewrites authorized_keys (Linux only)       │
│  runs workflow steps · monitors · inventory  │
└──────────────────────────────────────────────┘

Multi-tenancy: every domain document carries org_id, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (<slug>.vantage.<tld>).

---

Repository Structure

vantage/
├── agent/
│   ├── cmd/main.go                  # flags: -generate-key
│   └── internal/
│       ├── checker/                 # monitor check execution
│       ├── config/                  # config.yaml load/save
│       ├── exec/                    # workflow step execution
│       ├── grpc/                    # client + generated pb
│       ├── inventory/               # CPU/mem/disk collection (linux/other)
│       ├── keys/                    # authorized_keys read/diff/write
│       ├── monitors/                # agent-run monitor loop
│       ├── sync/                    # poll loop + command stream
│       └── updates/                 # OS package update check/apply
├── server/
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # REST handlers
│       ├── auth/                    # local, OIDC, session, middleware, orghost
│       ├── checker/                 # server-run monitor checks
│       ├── db/                      # mongo connect + Col()
│       ├── grpc/                    # gRPC server + generated pb
│       ├── models/                  # MongoDB documents
│       ├── monitorsched/            # server-side monitor scheduler
│       ├── notify/                  # smtp, http, templating, dispatch
│       └── services/                # business logic + migrations
├── web/                             # the application UI (authenticated)
│   ├── app/(app)/                   # authed routes
│   ├── app/login, app/setup         # unauthed routes
│   ├── components/                  # ui/, workflows/, monitors/, Sidebar
│   └── lib/                         # api client, guac console, query client
├── site/                            # public marketing site
│   ├── app/                         # one directory per route
│   ├── components/                  # Nav, Footer, Logo, InstrumentPanel, forms
│   ├── assets/                      # image sources, not served
│   └── Dockerfile                   # same shape as web/: standalone, node, 3000
├── sitesvc/                         # public forms: contact mail + signup
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # contact, signup, verify
│       ├── mail/                    # SMTP
│       ├── models/                  # mirrors server org/user + pending signup
│       ├── provision/               # slug rules mirrored from the control plane
│       └── store/                   # mongo: pending signups, org/user creation
├── proto/vantage/v1/vantage.proto
├── installer/                       # Windows: setup.ps1, nssm.exe, WiX .wxs
├── deploy/                          # docker-compose.yml, agent.service
└── .gitea/workflows/                # agent-release.yml, server-deploy.yml

---

Subsystems

SSH keys

Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites /root/.ssh/authorized_keys atomically. Keys can also be generated on a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.

Workflows

A library of reusable steps (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into workflows targeting a set of servers. Running one snapshots the resolved steps into a WorkflowRun, then dispatches RunStepCmd over the agent command stream. Step stdout/stderr streams back as StepOutputChunk and is written to a log file on disk; the UI streams it live. Steps support on_failure: stop|continue|retry, per-run env passed between steps via output_env, and a per-run workspace directory the agent cleans up at the end.

Default steps are seeded per org at boot (SeedDefaultSteps). Logs are swept by retention (workflow_log_retention_days; nil = 30 days, 0 = forever).

Monitors

HTTP, TCP, ICMP and TLS checks. Each monitor has a runner: "server" (executed by the server-side scheduler) or a server_id (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond retries flip state to down, open an Incident, and notify. Hourly Rollup documents back the uptime graphs.

Notification channels

Per-org outbound destinations: webhook, smtp, discord, slack, telegram. Monitors reference channels by ID. Channels are testable from the UI.

Secrets vault

Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via secret_refs (injected as env at execution), and read by Kubernetes External Secrets Operator via GET /api/secrets/:group/values using a bearer token whose SHA-256 hash is stored in settings.

Browser console

POST /api/console/connect mints a one-time session token; GET /api/console/tunnel upgrades to a WebSocket and proxies to guacd (Apache Guacamole daemon) using github.com/wwt/guac. SSH connections authenticate with a stored private key; RDP/VNC credentials are encrypted, single-use, and consumed when the tunnel opens.

Inventory and OS updates

Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (ApplyUpdatesCmd).

Agent self-update

UpdateAgentCmd carries a target version and Gitea base URL; the agent downloads and replaces itself.

Marketing site and sitesvc

site/ is a separate Next.js app built exactly like web/ — output: "standalone", run by Node in a node:26-alpine image, listening on 3000 and published as 3001. Both of its forms post to sitesvc; the control plane is not involved and has no public signup endpoint.

sitesvc/ (port 8082) owns both flows end to end:

Form	Endpoint	Effect
Contact	POST /api/contact	Emails support@hostxtra.co.uk, Reply-To the sender. Nothing stored.
Create organisation	POST /api/signup	Records a pending signup and emails a verification link.
Verification link	GET /api/verify?token=…	Creates the org and its owner, then redirects to the org's sign-in page (APP_LOGIN_URL with {slug} filled in).

All three are deliberately excluded from the self-hosted deployment: deploy/docker-compose.yml mentions none of them, and they live in deploy/docker-compose.site.yml instead.

# self-hosted install — no marketing site, no sitesvc
docker compose up -d

# vantage.hostxtra.co.uk — control plane plus the public site
docker compose -f docker-compose.yml -f docker-compose.site.yml up -d

Signup and verification

Nothing is written to orgs or users until the emailed link is opened. A signup lands in sitesvc's own site_pending_signups collection holding the org name, the address, and the password already bcrypt-hashed at cost 12. The consequence is worth stating: an address nobody controls can never occupy an email, hold an organisation slug, or produce an account that can sign in. It also means the control plane's login path needs no concept of "unverified".

• The token is 32 random bytes; only its SHA-256 hash is stored, so a leaked database yields no working links.
• Verify deletes the pending record atomically before provisioning (FindOneAndDelete), so a double-clicked link cannot create two organisations — the second delete matches nothing.
• Links expire after 24 hours, and a TTL index lets Mongo drop abandoned signups so password hashes do not linger.
• Re-submitting the form for the same address replaces the previous pending record, so only the newest link works.
• If the owner insert fails after the org is created, the org is rolled back rather than stranded holding a slug. The rollback refuses to touch an org that has users.
• Rate limited to 3 signups per client IP per hour, plus a honeypot field.

The one piece of duplicated logic

sitesvc/internal/provision and sitesvc/internal/models mirror the control plane's slug rules, reserved names, bcrypt cost and document shapes. They are duplicated rather than imported because sitesvc is a separate module that deliberately does not depend on the server.

Nothing enforces the match automatically. If the control plane's Slugify, reservedSlugs, CreateOrg or CreateUser change, update sitesvc/internal/provision in the same commit — a divergence would provision tenants under rules the app does not agree with.

sitesvc also (re)declares the unique indexes on users.email and orgs.slug at boot so it does not depend on the server having started first. Creating an existing index is a no-op.

---

Auth and Orgs

• Bootstrap — first run has no users. GET /auth/bootstrap-status drives /setup, POST /auth/bootstrap creates the first org plus its owner.
• Local auth — email + password (bcrypt), POST /auth/login.
• OIDC — configured per org (org_oidc), issuer + client ID + encrypted client secret. /auth/oidc/start → /auth/oidc/callback.
• Sessions — opaque 32-byte hex ID in the km_session cookie, session body stored in Redis with a 24h TTL.
• Roles — owner, admin, member. /api/settings and /api/org/* require owner or admin.
• Host/org guard — APP_ROOT_LABEL (default vantage) defines the app root label. A request to <slug>.vantage.<tld> resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.

Unique indexes on user email and org slug are a security property, not an optimisation: GetUserByEmail does an unscoped FindOne, so duplicates would let the OIDC cross-org guard compare against an arbitrary user. Same for duplicate settings docs and duplicate ESO token hashes.

---

gRPC API

service Vantage {
  rpc Register(RegisterRequest)              returns (RegisterResponse);
  rpc SyncKeys(SyncRequest)                  returns (SyncResponse);
  rpc UploadGeneratedKey(UploadKeyRequest)   returns (UploadKeyResponse);
  rpc ReportUpdates(ReportUpdatesRequest)    returns (ReportUpdatesResponse);
  rpc ReportInventory(InventoryReport)       returns (InventoryReportResponse);
  rpc SyncMonitors(SyncMonitorsRequest)      returns (SyncMonitorsResponse);
  rpc ReportChecks(ReportChecksRequest)      returns (ReportChecksResponse);
  rpc CommandStream(stream AgentMessage)     returns (stream ServerCommand);
}

CommandStream is the only streaming RPC: the agent authenticates once with AgentReady, then the server pushes ServerCommands and the agent replies with CommandResult, StepResult, or StepOutputChunk.

ServerCommand variants: GenerateKeyCmd, DeleteKeyCmd, UpdateAgentCmd, ApplyUpdatesCmd, RunStepCmd, CleanupWorkspaceCmd.

Key-state polling stays on the 30s SyncKeys interval. Full message definitions live in proto/vantage/v1/vantage.proto.

---

REST API

Unauthenticated:

GET  /install            /install.ps1        # dynamic agent install scripts
GET  /update             /update.ps1
GET  /auth/bootstrap-status
POST /auth/bootstrap     /auth/login  /auth/logout
GET  /auth/me            /auth/oidc/start    /auth/oidc/callback
GET  /api/secrets/:group/values               # bearer token (ESO)

Session-authed under /api:

servers      GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
             POST /servers/:id/{generate-key,update-agent,apply-updates}
keys         GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
             POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
             POST /steps/{import,seed-defaults,parse} · GET /steps/usage
             GET,POST /workflows · GET,PUT,DELETE /workflows/:id
             POST /workflows/:id/run · GET /workflows/:id/runs
             GET /runs/:runId · POST /runs/:runId/cancel
             GET /runs/:runId/servers/:serverId/logs[/stream]
monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
             GET /monitors/:id/{incidents,uptime}
channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
             POST /secrets/:group/reveal · DELETE /secrets/:group/:key
console      POST /console/connect · GET /console/tunnel (websocket)
audit        GET /audit
agent        GET /agent/latest-version
settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
             GET,PUT /org/oidc                                       (owner|admin)

---

MongoDB Collections

servers · keys · assignments · orgs · users · org_oidc · settings · secrets · workflows · workflow_steps · workflow_runs · monitors · incidents · monitor_rollups · notification_channels · console_sessions · audit_logs · migrations

Every document except migrations carries org_id. Struct definitions are the source of truth — see server/internal/models/.

site_pending_signups is written only by sitesvc and holds unverified signups; the control plane neither reads nor knows about it.

Notes that are not obvious from the structs:

• servers.agent_token_hash stores SHA-256 of the token, never plaintext. pre_reg_token is cleared after Register(). status is pending → active on register, offline when last_seen passes the threshold (swept every 2 min).
• servers.inventory holds the latest metrics snapshot with separate metrics_at / static_at timestamps.
• keys.private_key_enc and passphrase_enc are AES-256-GCM; the JSON form exposes only has_private_key / has_passphrase.
• assignments.revoked_at: null means active. Revocation is soft, preserving audit history.
• workflow_runs.steps_snapshot freezes the resolved steps so editing the library never rewrites history.
• console_sessions.token_consumed_at is set atomically to enforce one-time use.

Migrations

services.RunMigrations() runs at boot, recording markers in migrations:

• 0001_default_org_backfill
• 0002_settings_org_backfill (must run before 0003 — 0003 can create a default org, which pushes 0002 into its ambiguous multi-org branch)
• 0003_missed_org_scopes

Index builders (EnsureAuthIndexes, EnsureSettingsIndexes) are fatal on failure; EnsureSecretIndexes and EnsureWorkflowIndexes only warn.

---

Agent Lifecycle

Config file

Linux /etc/vantage/config.yaml, Windows %ProgramData%\vantage\config.yaml. Directory 0700, file 0600.

server_url: "vantage.yourdomain.com:9090"
server_id: "<uuid>"
pre_reg_token: "<token>" # removed after first successful Register()
agent_token: "" # written by agent after Register()
poll_interval: 30s
tls: true

Startup

1. Load config
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
4. Enter SyncKeys poll loop (default 30s)

Poll loop

1. SyncKeys(server_id, agent_token, agent_version)
2. Non-Linux hosts stop here — Windows agents register and heartbeat only
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600

Install

Linux: systemd unit at /etc/systemd/system/vantage-agent.service, Restart=always, runs as root. Windows: MSI built by CI (WiX), or installer/setup.ps1 registering the agent as a service via NSSM.

---

Server Registration Flow

1. Add Server in the UI calls POST /api/servers/new, which generates a server_id and a pre-registration token (TTL 1 hour, single-use).
2. The UI shows a one-liner:

   curl -fsSL https://vantage.yourdomain.com/install | \
     bash -s -- --server-id=<id> --token=<token>
   

   Windows gets the /install.ps1 equivalent.
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
4. The server flips to active on first sync.

/install is served dynamically, injecting the latest agent version from the Gitea API.

---

Environment Variables (server)

Name	Required	Notes
GRPC_HOST	yes	host:port agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC.
MONGO_URI	no	default mongodb://localhost:27017
MONGO_DB	no	default vantage
REDIS_ADDR	no	default localhost:6379
KEY_ENCRYPTION_KEY	yes in practice	64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials.
GITEA_HOST	yes	used to build install scripts and agent download URLs
GUACD_ADDR	no	default guacd:4822
APP_ROOT_LABEL	no	default vantage; wrong value disables the host/session org guard
VANTAGE_WORKFLOW_LOG_DIR	no	where run logs are written

sitesvc (deploy/docker-compose.site.yml only):

Name	Required	Notes
MONGO_URI	yes	must point at the control plane's database, or the app will not see organisations created here. The database name is read from the URI path (mongodb://user:pass@host:27017/vantage?authSource=vantage); a URI without one is refused at boot rather than defaulted. Note this differs from the server, which takes MONGO_DB separately.
PUBLIC_URL	yes	sitesvc's own public base URL; verification links are built from it
APP_LOGIN_URL	no	template for the org sign-in URL a verified owner is redirected to. {slug} is replaced with the new org's slug (each org has its own subdomain), e.g. https://{slug}.vantage.hostxtra.co.uk/login. A value without {slug} is used verbatim; empty means a plain confirmation page.
SMTP_HOST / SMTP_FROM	yes	without them both forms refuse (503) rather than silently dropping
SMTP_TO	no	default support@hostxtra.co.uk; contact enquiries only
SMTP_PORT	no	default 587; 465 uses implicit TLS
SMTP_USERNAME / SMTP_PASSWORD	no	auth skipped when username is empty
SITE_ORIGIN	yes in practice	comma-separated allowed origins; unset refuses every cross-origin browser request
TRUST_PROXY	no	only true behind a proxy that overwrites X-Forwarded-For, or clients spoof past the rate limiter

deploy/docker-compose.yml runs four services: redis, guacd, server (8080 + 9090), web (3000). MongoDB is external. deploy/docker-compose.site.yml adds the public marketing site on 3001 and is only used on vantage.hostxtra.co.uk.

---

Security

• gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
• Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's 0600 config.
• Pre-registration tokens are short-lived (1 hour) and single-use.
• AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
• Console session tokens are one-time; RDP credentials are consumed on tunnel open.
• ESO read token stored as a SHA-256 hash and rotatable.
• Unique indexes on user email, org slug, settings org, and ESO token hash are load-bearing for tenant isolation.
• authorized_keys written 0600, owned by root. The agent runs as root because it must.
• Every mutating API path writes an audit event.

---

Frontend

Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in web/lib/guacamole-common.js.

Route	Purpose
/setup	First-run bootstrap: create the first org and owner
/login	Local or OIDC sign-in
/	Fleet dashboard
/servers, /servers/new, /servers/[id]	Fleet list, install one-liner, server detail (keys, inventory, updates)
/servers/[id]/console	Browser SSH/RDP/VNC session
/keys, /keys/[id]	Key library; assign and revoke per server
/workflows, /workflows/[id], /workflows/[id]/runs[/runId]	Compose, run, and follow live logs
/steps	Reusable step library
/monitors, /monitors/new, /monitors/[id][/edit]	Checks, uptime, incidents
/secrets, /secrets/[group]	Vault
/audit	Audit log
/settings, /settings/org, /settings/notifications	Alerts, members, OIDC, channels

---

CI/CD — Gitea Actions

agent-release.yml — triggered by agent/v* tags

Builds linux/amd64, linux/arm64, windows/amd64, writes checksums.txt, creates a Gitea release. A second msi job on windows-2022 packages the WiX installer.

GOOS=linux GOARCH=amd64 go build \
  -ldflags="-s -w -X main.Version=${VERSION}" \
  -o dist/vantage-agent-linux-amd64 ./cmd

server-deploy.yml — triggered on every push to main

Builds and pushes four images to the Gitea container registry: server, web, site and sitesvc.

Note that despite the name, this workflow does not deploy — it only builds and pushes. There is no SSH step and no path filter; every push to main rebuilds all three images. Rolling them out is a separate manual step on the host:

cd /opt/vantage && docker compose -f docker-compose.yml -f docker-compose.site.yml pull && \
  docker compose -f docker-compose.yml -f docker-compose.site.yml up -d --remove-orphans

Tagging

git tag agent/v1.0.0 && git push origin agent/v1.0.0   # agent release
git push origin main                                    # server + web deploy

Secrets / variables

Name	Type	Value
RELEASE_TOKEN	Secret	Gitea API token, write:release
REGISTRY_USER	Secret	Gitea username
REGISTRY_PASSWORD	Secret	Gitea token, write:packages
GITEA_HOST	Variable	gitea.hostxtra.co.uk
DOCKER_HOST	Variable	registry host used for image tags
API_URL	Variable	baked into the web image at build time
SITE_API_URL	Variable	browser-reachable sitesvc URL, baked into the site image. Required — if empty, both forms report "not connected" and submit nowhere. Must also be in sitesvc's SITE_ORIGIN.
SITE_CONTACT_EMAIL	Variable	optional; address shown when a form is misconfigured

---

Design Decisions

• gRPC for agent traffic — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
• Outbound-only agents — no inbound ports on managed servers, works behind NAT.
• Poll for keys, push for commands — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
• Atomic authorized_keys rewrite — temp file plus os.Rename(); a machine that dies mid-write keeps the old file.
• Fingerprint diffing before write — no disk churn on unchanged state.
• Soft revocation — revoked_at rather than deletes; preserves audit history.
• Run snapshots — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
• Monitors run in two places — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
• Redis for sessions only — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
• guacd for console — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
• org_id on every document — isolation enforced at the query layer, not by separate databases.
• root only — manages /root/.ssh/authorized_keys; no per-user key management.
• Windows agents are second-class by design — register, heartbeat, run steps, report inventory; no authorized_keys management.