vantage/claude.md

Vantage

A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.

A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.

---

Architecture Overview

┌──────────────────────────────────────────────┐
│  Next.js 16 Frontend (web, :3000)            │
│  servers · keys · workflows · monitors       │
│  secrets · audit · console · settings        │
└───────────────┬──────────────────────────────┘
                │ REST + cookie session
┌───────────────▼──────────────────────────────┐
│  Go Backend (server)                         │
│  :8080 REST (gin)   :9090 gRPC (agents)      │
│  MongoDB (state) · Redis (sessions)          │
│  monitor scheduler · workflow runner         │
│  guacd tunnel proxy for browser console      │
└───────────────┬──────────────────────────────┘
                │ gRPC (TLS) — outbound from agent only
┌───────────────▼──────────────────────────────┐
│  Go Agent (per server, Linux + Windows)      │
│  polls SyncKeys · CommandStream              │
│  rewrites authorized_keys (Linux only)       │
│  runs workflow steps · monitors · inventory  │
└──────────────────────────────────────────────┘

Multi-tenancy: every domain document carries org_id, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (<slug>.vantage.<tld>).

---

Repository Structure

vantage/
├── agent/
│   ├── cmd/main.go                  # flags: -generate-key
│   └── internal/
│       ├── checker/                 # monitor check execution
│       ├── config/                  # config.yaml load/save
│       ├── exec/                    # workflow step execution
│       ├── grpc/                    # client + generated pb
│       ├── inventory/               # CPU/mem/disk collection (linux/other)
│       ├── keys/                    # authorized_keys read/diff/write
│       ├── monitors/                # agent-run monitor loop
│       ├── sync/                    # poll loop + command stream
│       └── updates/                 # OS package update check/apply
├── server/
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # REST handlers
│       ├── auth/                    # local, OIDC, session, middleware, orghost
│       ├── checker/                 # server-run monitor checks
│       ├── db/                      # mongo connect + Col()
│       ├── grpc/                    # gRPC server + generated pb
│       ├── models/                  # MongoDB documents
│       ├── monitorsched/            # server-side monitor scheduler
│       ├── notify/                  # smtp, http, templating, dispatch
│       └── services/                # business logic + migrations
├── web/
│   ├── app/(app)/                   # authed routes
│   ├── app/login, app/setup         # unauthed routes
│   ├── components/                  # ui/, workflows/, monitors/, Sidebar
│   └── lib/                         # api client, guac console, query client
├── proto/vantage/v1/vantage.proto
├── installer/                       # Windows: setup.ps1, nssm.exe, WiX .wxs
├── deploy/                          # docker-compose.yml, agent.service
└── .gitea/workflows/                # agent-release.yml, server-deploy.yml

---

Subsystems

SSH keys

Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites /root/.ssh/authorized_keys atomically. Keys can also be generated on a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.

Workflows

A library of reusable steps (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into workflows targeting a set of servers. Running one snapshots the resolved steps into a WorkflowRun, then dispatches RunStepCmd over the agent command stream. Step stdout/stderr streams back as StepOutputChunk and is written to a log file on disk; the UI streams it live. Steps support on_failure: stop|continue|retry, per-run env passed between steps via output_env, and a per-run workspace directory the agent cleans up at the end.

Default steps are seeded per org at boot (SeedDefaultSteps). Logs are swept by retention (workflow_log_retention_days; nil = 30 days, 0 = forever).

Monitors

HTTP, TCP, ICMP and TLS checks. Each monitor has a runner: "server" (executed by the server-side scheduler) or a server_id (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond retries flip state to down, open an Incident, and notify. Hourly Rollup documents back the uptime graphs.

Notification channels

Per-org outbound destinations: webhook, smtp, discord, slack, telegram. Monitors reference channels by ID. Channels are testable from the UI.

Secrets vault

Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via secret_refs (injected as env at execution), and read by Kubernetes External Secrets Operator via GET /api/secrets/:group/values using a bearer token whose SHA-256 hash is stored in settings.

Browser console

POST /api/console/connect mints a one-time session token; GET /api/console/tunnel upgrades to a WebSocket and proxies to guacd (Apache Guacamole daemon) using github.com/wwt/guac. SSH connections authenticate with a stored private key; RDP/VNC credentials are encrypted, single-use, and consumed when the tunnel opens.

Inventory and OS updates

Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (ApplyUpdatesCmd).

Agent self-update

UpdateAgentCmd carries a target version and Gitea base URL; the agent downloads and replaces itself.

---

Auth and Orgs

• Bootstrap — first run has no users. GET /auth/bootstrap-status drives /setup, POST /auth/bootstrap creates the first org plus its owner.
• Local auth — email + password (bcrypt), POST /auth/login.
• OIDC — configured per org (org_oidc), issuer + client ID + encrypted client secret. /auth/oidc/start → /auth/oidc/callback.
• Sessions — opaque 32-byte hex ID in the km_session cookie, session body stored in Redis with a 24h TTL.
• Roles — owner, admin, member. /api/settings and /api/org/* require owner or admin.
• Host/org guard — APP_ROOT_LABEL (default vantage) defines the app root label. A request to <slug>.vantage.<tld> resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.

Unique indexes on user email and org slug are a security property, not an optimisation: GetUserByEmail does an unscoped FindOne, so duplicates would let the OIDC cross-org guard compare against an arbitrary user. Same for duplicate settings docs and duplicate ESO token hashes.

---

gRPC API

service Vantage {
  rpc Register(RegisterRequest)              returns (RegisterResponse);
  rpc SyncKeys(SyncRequest)                  returns (SyncResponse);
  rpc UploadGeneratedKey(UploadKeyRequest)   returns (UploadKeyResponse);
  rpc ReportUpdates(ReportUpdatesRequest)    returns (ReportUpdatesResponse);
  rpc ReportInventory(InventoryReport)       returns (InventoryReportResponse);
  rpc SyncMonitors(SyncMonitorsRequest)      returns (SyncMonitorsResponse);
  rpc ReportChecks(ReportChecksRequest)      returns (ReportChecksResponse);
  rpc CommandStream(stream AgentMessage)     returns (stream ServerCommand);
}

CommandStream is the only streaming RPC: the agent authenticates once with AgentReady, then the server pushes ServerCommands and the agent replies with CommandResult, StepResult, or StepOutputChunk.

ServerCommand variants: GenerateKeyCmd, DeleteKeyCmd, UpdateAgentCmd, ApplyUpdatesCmd, RunStepCmd, CleanupWorkspaceCmd.

Key-state polling stays on the 30s SyncKeys interval. Full message definitions live in proto/vantage/v1/vantage.proto.

---

REST API

Unauthenticated:

GET  /install            /install.ps1        # dynamic agent install scripts
GET  /update             /update.ps1
GET  /auth/bootstrap-status
POST /auth/bootstrap     /auth/login  /auth/logout
GET  /auth/me            /auth/oidc/start    /auth/oidc/callback
GET  /api/secrets/:group/values               # bearer token (ESO)

Session-authed under /api:

servers      GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
             POST /servers/:id/{generate-key,update-agent,apply-updates}
keys         GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
             POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
             POST /steps/{import,seed-defaults,parse} · GET /steps/usage
             GET,POST /workflows · GET,PUT,DELETE /workflows/:id
             POST /workflows/:id/run · GET /workflows/:id/runs
             GET /runs/:runId · POST /runs/:runId/cancel
             GET /runs/:runId/servers/:serverId/logs[/stream]
monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
             GET /monitors/:id/{incidents,uptime}
channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
             POST /secrets/:group/reveal · DELETE /secrets/:group/:key
console      POST /console/connect · GET /console/tunnel (websocket)
audit        GET /audit
agent        GET /agent/latest-version
settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
             GET,PUT /org/oidc                                       (owner|admin)

---

MongoDB Collections

servers · keys · assignments · orgs · users · org_oidc · settings · secrets · workflows · workflow_steps · workflow_runs · monitors · incidents · monitor_rollups · notification_channels · console_sessions · audit_logs · migrations

Every document except migrations carries org_id. Struct definitions are the source of truth — see server/internal/models/.

Notes that are not obvious from the structs:

• servers.agent_token_hash stores SHA-256 of the token, never plaintext. pre_reg_token is cleared after Register(). status is pending → active on register, offline when last_seen passes the threshold (swept every 2 min).
• servers.inventory holds the latest metrics snapshot with separate metrics_at / static_at timestamps.
• keys.private_key_enc and passphrase_enc are AES-256-GCM; the JSON form exposes only has_private_key / has_passphrase.
• assignments.revoked_at: null means active. Revocation is soft, preserving audit history.
• workflow_runs.steps_snapshot freezes the resolved steps so editing the library never rewrites history.
• console_sessions.token_consumed_at is set atomically to enforce one-time use.

Migrations

services.RunMigrations() runs at boot, recording markers in migrations:

• 0001_default_org_backfill
• 0002_settings_org_backfill (must run before 0003 — 0003 can create a default org, which pushes 0002 into its ambiguous multi-org branch)
• 0003_missed_org_scopes

Index builders (EnsureAuthIndexes, EnsureSettingsIndexes) are fatal on failure; EnsureSecretIndexes and EnsureWorkflowIndexes only warn.

---

Agent Lifecycle

Config file

Linux /etc/vantage/config.yaml, Windows %ProgramData%\vantage\config.yaml. Directory 0700, file 0600.

server_url: "vantage.yourdomain.com:9090"
server_id: "<uuid>"
pre_reg_token: "<token>"   # removed after first successful Register()
agent_token: ""            # written by agent after Register()
poll_interval: 30s
tls: true

Startup

1. Load config
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
4. Enter SyncKeys poll loop (default 30s)

Poll loop

1. SyncKeys(server_id, agent_token, agent_version)
2. Non-Linux hosts stop here — Windows agents register and heartbeat only
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600

Install

Linux: systemd unit at /etc/systemd/system/vantage-agent.service, Restart=always, runs as root. Windows: MSI built by CI (WiX), or installer/setup.ps1 registering the agent as a service via NSSM.

---

Server Registration Flow

1. Add Server in the UI calls POST /api/servers/new, which generates a server_id and a pre-registration token (TTL 1 hour, single-use).
2. The UI shows a one-liner:

   curl -fsSL https://vantage.yourdomain.com/install | \
     bash -s -- --server-id=<id> --token=<token>
   

   Windows gets the /install.ps1 equivalent.
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
4. The server flips to active on first sync.

/install is served dynamically, injecting the latest agent version from the Gitea API.

---

Environment Variables (server)

Name	Required	Notes
GRPC_HOST	yes	host:port agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC.
MONGO_URI	no	default mongodb://localhost:27017
MONGO_DB	no	default vantage
REDIS_ADDR	no	default localhost:6379
KEY_ENCRYPTION_KEY	yes in practice	64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials.
GITEA_HOST	yes	used to build install scripts and agent download URLs
GUACD_ADDR	no	default guacd:4822
APP_ROOT_LABEL	no	default vantage; wrong value disables the host/session org guard
VANTAGE_WORKFLOW_LOG_DIR	no	where run logs are written

deploy/docker-compose.yml runs four services: redis, guacd, server (8080 + 9090), web (3000). MongoDB is external.

---

Security

• gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
• Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's 0600 config.
• Pre-registration tokens are short-lived (1 hour) and single-use.
• AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
• Console session tokens are one-time; RDP credentials are consumed on tunnel open.
• ESO read token stored as a SHA-256 hash and rotatable.
• Unique indexes on user email, org slug, settings org, and ESO token hash are load-bearing for tenant isolation.
• authorized_keys written 0600, owned by root. The agent runs as root because it must.
• Every mutating API path writes an audit event.

---

Frontend

Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in web/lib/guacamole-common.js.

Route	Purpose
/setup	First-run bootstrap: create the first org and owner
/login	Local or OIDC sign-in
/	Fleet dashboard
/servers, /servers/new, /servers/[id]	Fleet list, install one-liner, server detail (keys, inventory, updates)
/servers/[id]/console	Browser SSH/RDP/VNC session
/keys, /keys/[id]	Key library; assign and revoke per server
/workflows, /workflows/[id], /workflows/[id]/runs[/runId]	Compose, run, and follow live logs
/steps	Reusable step library
/monitors, /monitors/new, /monitors/[id][/edit]	Checks, uptime, incidents
/secrets, /secrets/[group]	Vault
/audit	Audit log
/settings, /settings/org, /settings/notifications	Alerts, members, OIDC, channels

---

CI/CD — Gitea Actions

agent-release.yml — triggered by agent/v* tags

Builds linux/amd64, linux/arm64, windows/amd64, writes checksums.txt, creates a Gitea release. A second msi job on windows-2022 packages the WiX installer.

GOOS=linux GOARCH=amd64 go build \
  -ldflags="-s -w -X main.Version=${VERSION}" \
  -o dist/vantage-agent-linux-amd64 ./cmd

server-deploy.yml — triggered on push to main

Builds and pushes the server and web images to the Gitea container registry, then deploys over SSH:

cd /opt/vantage && docker compose pull && docker compose up -d --remove-orphans

Tagging

git tag agent/v1.0.0 && git push origin agent/v1.0.0   # agent release
git push origin main                                    # server + web deploy

Secrets / variables

Name	Type	Value
RELEASE_TOKEN	Secret	Gitea API token, write:release
REGISTRY_USER	Secret	Gitea username
REGISTRY_PASSWORD	Secret	Gitea token, write:packages
DEPLOY_HOST	Secret	server VM host
DEPLOY_USER	Secret	SSH user for deploy
DEPLOY_SSH_KEY	Secret	deploy private key
GITEA_HOST	Variable	gitea.hostxtra.co.uk

---

Design Decisions

• gRPC for agent traffic — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
• Outbound-only agents — no inbound ports on managed servers, works behind NAT.
• Poll for keys, push for commands — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
• Atomic authorized_keys rewrite — temp file plus os.Rename(); a machine that dies mid-write keeps the old file.
• Fingerprint diffing before write — no disk churn on unchanged state.
• Soft revocation — revoked_at rather than deletes; preserves audit history.
• Run snapshots — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
• Monitors run in two places — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
• Redis for sessions only — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
• guacd for console — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
• org_id on every document — isolation enforced at the query layer, not by separate databases.
• root only — manages /root/.ssh/authorized_keys; no per-user key management.
• Windows agents are second-class by design — register, heartbeat, run steps, report inventory; no authorized_keys management.