378 lines
20 KiB
Markdown
378 lines
20 KiB
Markdown
# Vantage
|
|
|
|
A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.
|
|
|
|
A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
┌──────────────────────────────────────────────┐
|
|
│ Next.js 16 Frontend (web, :3000) │
|
|
│ servers · keys · workflows · monitors │
|
|
│ secrets · audit · console · settings │
|
|
└───────────────┬──────────────────────────────┘
|
|
│ REST + cookie session
|
|
┌───────────────▼──────────────────────────────┐
|
|
│ Go Backend (server) │
|
|
│ :8080 REST (gin) :9090 gRPC (agents) │
|
|
│ MongoDB (state) · Redis (sessions) │
|
|
│ monitor scheduler · workflow runner │
|
|
│ guacd tunnel proxy for browser console │
|
|
└───────────────┬──────────────────────────────┘
|
|
│ gRPC (TLS) — outbound from agent only
|
|
┌───────────────▼──────────────────────────────┐
|
|
│ Go Agent (per server, Linux + Windows) │
|
|
│ polls SyncKeys · CommandStream │
|
|
│ rewrites authorized_keys (Linux only) │
|
|
│ runs workflow steps · monitors · inventory │
|
|
└──────────────────────────────────────────────┘
|
|
```
|
|
|
|
Multi-tenancy: every domain document carries `org_id`, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (`<slug>.vantage.<tld>`).
|
|
|
|
---
|
|
|
|
## Repository Structure
|
|
|
|
```
|
|
vantage/
|
|
├── agent/
|
|
│ ├── cmd/main.go # flags: -generate-key
|
|
│ └── internal/
|
|
│ ├── checker/ # monitor check execution
|
|
│ ├── config/ # config.yaml load/save
|
|
│ ├── exec/ # workflow step execution
|
|
│ ├── grpc/ # client + generated pb
|
|
│ ├── inventory/ # CPU/mem/disk collection (linux/other)
|
|
│ ├── keys/ # authorized_keys read/diff/write
|
|
│ ├── monitors/ # agent-run monitor loop
|
|
│ ├── sync/ # poll loop + command stream
|
|
│ └── updates/ # OS package update check/apply
|
|
├── server/
|
|
│ ├── cmd/main.go
|
|
│ └── internal/
|
|
│ ├── api/ # REST handlers
|
|
│ ├── auth/ # local, OIDC, session, middleware, orghost
|
|
│ ├── checker/ # server-run monitor checks
|
|
│ ├── db/ # mongo connect + Col()
|
|
│ ├── grpc/ # gRPC server + generated pb
|
|
│ ├── models/ # MongoDB documents
|
|
│ ├── monitorsched/ # server-side monitor scheduler
|
|
│ ├── notify/ # smtp, http, templating, dispatch
|
|
│ └── services/ # business logic + migrations
|
|
├── web/
|
|
│ ├── app/(app)/ # authed routes
|
|
│ ├── app/login, app/setup # unauthed routes
|
|
│ ├── components/ # ui/, workflows/, monitors/, Sidebar
|
|
│ └── lib/ # api client, guac console, query client
|
|
├── proto/vantage/v1/vantage.proto
|
|
├── installer/ # Windows: setup.ps1, nssm.exe, WiX .wxs
|
|
├── deploy/ # docker-compose.yml, agent.service
|
|
└── .gitea/workflows/ # agent-release.yml, server-deploy.yml
|
|
```
|
|
|
|
---
|
|
|
|
## Subsystems
|
|
|
|
### SSH keys
|
|
Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites `/root/.ssh/authorized_keys` atomically. Keys can also be generated *on* a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.
|
|
|
|
### Workflows
|
|
A library of reusable **steps** (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into **workflows** targeting a set of servers. Running one snapshots the resolved steps into a `WorkflowRun`, then dispatches `RunStepCmd` over the agent command stream. Step stdout/stderr streams back as `StepOutputChunk` and is written to a log file on disk; the UI streams it live. Steps support `on_failure: stop|continue|retry`, per-run env passed between steps via `output_env`, and a per-run workspace directory the agent cleans up at the end.
|
|
|
|
Default steps are seeded per org at boot (`SeedDefaultSteps`). Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever).
|
|
|
|
### Monitors
|
|
HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.
|
|
|
|
### Notification channels
|
|
Per-org outbound destinations: `webhook`, `smtp`, `discord`, `slack`, `telegram`. Monitors reference channels by ID. Channels are testable from the UI.
|
|
|
|
### Secrets vault
|
|
Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via `secret_refs` (injected as env at execution), and read by Kubernetes External Secrets Operator via `GET /api/secrets/:group/values` using a bearer token whose SHA-256 hash is stored in settings.
|
|
|
|
### Browser console
|
|
`POST /api/console/connect` mints a one-time session token; `GET /api/console/tunnel` upgrades to a WebSocket and proxies to **guacd** (Apache Guacamole daemon) using `github.com/wwt/guac`. SSH connections authenticate with a stored private key; RDP/VNC credentials are encrypted, single-use, and consumed when the tunnel opens.
|
|
|
|
### Inventory and OS updates
|
|
Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`).
|
|
|
|
### Agent self-update
|
|
`UpdateAgentCmd` carries a target version and Gitea base URL; the agent downloads and replaces itself.
|
|
|
|
---
|
|
|
|
## Auth and Orgs
|
|
|
|
- **Bootstrap** — first run has no users. `GET /auth/bootstrap-status` drives `/setup`, `POST /auth/bootstrap` creates the first org plus its owner.
|
|
- **Local auth** — email + password (bcrypt), `POST /auth/login`.
|
|
- **OIDC** — configured *per org* (`org_oidc`), issuer + client ID + encrypted client secret. `/auth/oidc/start` → `/auth/oidc/callback`.
|
|
- **Sessions** — opaque 32-byte hex ID in the `km_session` cookie, session body stored in Redis with a 24h TTL.
|
|
- **Roles** — `owner`, `admin`, `member`. `/api/settings` and `/api/org/*` require owner or admin.
|
|
- **Host/org guard** — `APP_ROOT_LABEL` (default `vantage`) defines the app root label. A request to `<slug>.vantage.<tld>` resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.
|
|
|
|
Unique indexes on user email and org slug are a **security property**, not an optimisation: `GetUserByEmail` does an unscoped `FindOne`, so duplicates would let the OIDC cross-org guard compare against an arbitrary user. Same for duplicate settings docs and duplicate ESO token hashes.
|
|
|
|
---
|
|
|
|
## gRPC API
|
|
|
|
```protobuf
|
|
service Vantage {
|
|
rpc Register(RegisterRequest) returns (RegisterResponse);
|
|
rpc SyncKeys(SyncRequest) returns (SyncResponse);
|
|
rpc UploadGeneratedKey(UploadKeyRequest) returns (UploadKeyResponse);
|
|
rpc ReportUpdates(ReportUpdatesRequest) returns (ReportUpdatesResponse);
|
|
rpc ReportInventory(InventoryReport) returns (InventoryReportResponse);
|
|
rpc SyncMonitors(SyncMonitorsRequest) returns (SyncMonitorsResponse);
|
|
rpc ReportChecks(ReportChecksRequest) returns (ReportChecksResponse);
|
|
rpc CommandStream(stream AgentMessage) returns (stream ServerCommand);
|
|
}
|
|
```
|
|
|
|
`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.
|
|
|
|
`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`.
|
|
|
|
Key-state polling stays on the 30s `SyncKeys` interval. Full message definitions live in `proto/vantage/v1/vantage.proto`.
|
|
|
|
---
|
|
|
|
## REST API
|
|
|
|
Unauthenticated:
|
|
|
|
```
|
|
GET /install /install.ps1 # dynamic agent install scripts
|
|
GET /update /update.ps1
|
|
GET /auth/bootstrap-status
|
|
POST /auth/bootstrap /auth/login /auth/logout
|
|
GET /auth/me /auth/oidc/start /auth/oidc/callback
|
|
GET /api/secrets/:group/values # bearer token (ESO)
|
|
```
|
|
|
|
Session-authed under `/api`:
|
|
|
|
```
|
|
servers GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
|
|
POST /servers/:id/{generate-key,update-agent,apply-updates}
|
|
keys GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
|
|
POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
|
|
workflows GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
|
|
POST /steps/{import,seed-defaults,parse} · GET /steps/usage
|
|
GET,POST /workflows · GET,PUT,DELETE /workflows/:id
|
|
POST /workflows/:id/run · GET /workflows/:id/runs
|
|
GET /runs/:runId · POST /runs/:runId/cancel
|
|
GET /runs/:runId/servers/:serverId/logs[/stream]
|
|
monitors GET,POST /monitors · GET,PUT,DELETE /monitors/:id
|
|
GET /monitors/:id/{incidents,uptime}
|
|
channels GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
|
|
secrets GET,POST /secrets · GET,PUT,DELETE /secrets/:group
|
|
POST /secrets/:group/reveal · DELETE /secrets/:group/:key
|
|
console POST /console/connect · GET /console/tunnel (websocket)
|
|
audit GET /audit
|
|
agent GET /agent/latest-version
|
|
settings GET,PUT /settings · POST /settings/secrets-token (owner|admin)
|
|
org GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
|
|
GET,PUT /org/oidc (owner|admin)
|
|
```
|
|
|
|
---
|
|
|
|
## MongoDB Collections
|
|
|
|
`servers` · `keys` · `assignments` · `orgs` · `users` · `org_oidc` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `migrations`
|
|
|
|
Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`.
|
|
|
|
Notes that are not obvious from the structs:
|
|
|
|
- `servers.agent_token_hash` stores SHA-256 of the token, never plaintext. `pre_reg_token` is cleared after `Register()`. `status` is `pending` → `active` on register, `offline` when `last_seen` passes the threshold (swept every 2 min).
|
|
- `servers.inventory` holds the latest metrics snapshot with separate `metrics_at` / `static_at` timestamps.
|
|
- `keys.private_key_enc` and `passphrase_enc` are AES-256-GCM; the JSON form exposes only `has_private_key` / `has_passphrase`.
|
|
- `assignments.revoked_at: null` means active. Revocation is soft, preserving audit history.
|
|
- `workflow_runs.steps_snapshot` freezes the resolved steps so editing the library never rewrites history.
|
|
- `console_sessions.token_consumed_at` is set atomically to enforce one-time use.
|
|
|
|
### Migrations
|
|
|
|
`services.RunMigrations()` runs at boot, recording markers in `migrations`:
|
|
|
|
- `0001_default_org_backfill`
|
|
- `0002_settings_org_backfill` (must run before 0003 — 0003 can create a `default` org, which pushes 0002 into its ambiguous multi-org branch)
|
|
- `0003_missed_org_scopes`
|
|
|
|
Index builders (`EnsureAuthIndexes`, `EnsureSettingsIndexes`) are fatal on failure; `EnsureSecretIndexes` and `EnsureWorkflowIndexes` only warn.
|
|
|
|
---
|
|
|
|
## Agent Lifecycle
|
|
|
|
### Config file
|
|
|
|
Linux `/etc/vantage/config.yaml`, Windows `%ProgramData%\vantage\config.yaml`. Directory `0700`, file `0600`.
|
|
|
|
```yaml
|
|
server_url: "vantage.yourdomain.com:9090"
|
|
server_id: "<uuid>"
|
|
pre_reg_token: "<token>" # removed after first successful Register()
|
|
agent_token: "" # written by agent after Register()
|
|
poll_interval: 30s
|
|
tls: true
|
|
```
|
|
|
|
### Startup
|
|
|
|
```
|
|
1. Load config
|
|
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
|
|
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
|
|
4. Enter SyncKeys poll loop (default 30s)
|
|
```
|
|
|
|
### Poll loop
|
|
|
|
```
|
|
1. SyncKeys(server_id, agent_token, agent_version)
|
|
2. Non-Linux hosts stop here — Windows agents register and heartbeat only
|
|
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
|
|
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600
|
|
```
|
|
|
|
### Install
|
|
|
|
Linux: systemd unit at `/etc/systemd/system/vantage-agent.service`, `Restart=always`, runs as root.
|
|
Windows: MSI built by CI (WiX), or `installer/setup.ps1` registering the agent as a service via NSSM.
|
|
|
|
---
|
|
|
|
## Server Registration Flow
|
|
|
|
1. **Add Server** in the UI calls `POST /api/servers/new`, which generates a `server_id` and a pre-registration token (TTL 1 hour, single-use).
|
|
2. The UI shows a one-liner:
|
|
```bash
|
|
curl -fsSL https://vantage.yourdomain.com/install | \
|
|
bash -s -- --server-id=<id> --token=<token>
|
|
```
|
|
Windows gets the `/install.ps1` equivalent.
|
|
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
|
|
4. The server flips to `active` on first sync.
|
|
|
|
`/install` is served dynamically, injecting the latest agent version from the Gitea API.
|
|
|
|
---
|
|
|
|
## Environment Variables (server)
|
|
|
|
| Name | Required | Notes |
|
|
| --- | --- | --- |
|
|
| `GRPC_HOST` | **yes** | `host:port` agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC. |
|
|
| `MONGO_URI` | no | default `mongodb://localhost:27017` |
|
|
| `MONGO_DB` | no | default `vantage` |
|
|
| `REDIS_ADDR` | no | default `localhost:6379` |
|
|
| `KEY_ENCRYPTION_KEY` | yes in practice | 64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials. |
|
|
| `GITEA_HOST` | yes | used to build install scripts and agent download URLs |
|
|
| `GUACD_ADDR` | no | default `guacd:4822` |
|
|
| `APP_ROOT_LABEL` | no | default `vantage`; wrong value disables the host/session org guard |
|
|
| `VANTAGE_WORKFLOW_LOG_DIR` | no | where run logs are written |
|
|
|
|
`deploy/docker-compose.yml` runs four services: `redis`, `guacd`, `server` (8080 + 9090), `web` (3000). MongoDB is external.
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
- gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
|
|
- Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's `0600` config.
|
|
- Pre-registration tokens are short-lived (1 hour) and single-use.
|
|
- AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
|
|
- Console session tokens are one-time; RDP credentials are consumed on tunnel open.
|
|
- ESO read token stored as a SHA-256 hash and rotatable.
|
|
- Unique indexes on user email, org slug, settings org, and ESO token hash are load-bearing for tenant isolation.
|
|
- `authorized_keys` written `0600`, owned by root. The agent runs as root because it must.
|
|
- Every mutating API path writes an audit event.
|
|
|
|
---
|
|
|
|
## Frontend
|
|
|
|
Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in `web/lib/guacamole-common.js`.
|
|
|
|
| Route | Purpose |
|
|
| --- | --- |
|
|
| `/setup` | First-run bootstrap: create the first org and owner |
|
|
| `/login` | Local or OIDC sign-in |
|
|
| `/` | Fleet dashboard |
|
|
| `/servers`, `/servers/new`, `/servers/[id]` | Fleet list, install one-liner, server detail (keys, inventory, updates) |
|
|
| `/servers/[id]/console` | Browser SSH/RDP/VNC session |
|
|
| `/keys`, `/keys/[id]` | Key library; assign and revoke per server |
|
|
| `/workflows`, `/workflows/[id]`, `/workflows/[id]/runs[/runId]` | Compose, run, and follow live logs |
|
|
| `/steps` | Reusable step library |
|
|
| `/monitors`, `/monitors/new`, `/monitors/[id][/edit]` | Checks, uptime, incidents |
|
|
| `/secrets`, `/secrets/[group]` | Vault |
|
|
| `/audit` | Audit log |
|
|
| `/settings`, `/settings/org`, `/settings/notifications` | Alerts, members, OIDC, channels |
|
|
|
|
---
|
|
|
|
## CI/CD — Gitea Actions
|
|
|
|
### `agent-release.yml` — triggered by `agent/v*` tags
|
|
|
|
Builds `linux/amd64`, `linux/arm64`, `windows/amd64`, writes `checksums.txt`, creates a Gitea release. A second `msi` job on `windows-2022` packages the WiX installer.
|
|
|
|
```bash
|
|
GOOS=linux GOARCH=amd64 go build \
|
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
|
-o dist/vantage-agent-linux-amd64 ./cmd
|
|
```
|
|
|
|
### `server-deploy.yml` — triggered on push to `main`
|
|
|
|
Builds and pushes the `server` and `web` images to the Gitea container registry, then deploys over SSH:
|
|
|
|
```bash
|
|
cd /opt/vantage && docker compose pull && docker compose up -d --remove-orphans
|
|
```
|
|
|
|
### Tagging
|
|
|
|
```bash
|
|
git tag agent/v1.0.0 && git push origin agent/v1.0.0 # agent release
|
|
git push origin main # server + web deploy
|
|
```
|
|
|
|
### Secrets / variables
|
|
|
|
| Name | Type | Value |
|
|
| --- | --- | --- |
|
|
| `RELEASE_TOKEN` | Secret | Gitea API token, `write:release` |
|
|
| `REGISTRY_USER` | Secret | Gitea username |
|
|
| `REGISTRY_PASSWORD` | Secret | Gitea token, `write:packages` |
|
|
| `DEPLOY_HOST` | Secret | server VM host |
|
|
| `DEPLOY_USER` | Secret | SSH user for deploy |
|
|
| `DEPLOY_SSH_KEY` | Secret | deploy private key |
|
|
| `GITEA_HOST` | Variable | `gitea.hostxtra.co.uk` |
|
|
|
|
---
|
|
|
|
## Design Decisions
|
|
|
|
- **gRPC for agent traffic** — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
|
|
- **Outbound-only agents** — no inbound ports on managed servers, works behind NAT.
|
|
- **Poll for keys, push for commands** — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
|
|
- **Atomic `authorized_keys` rewrite** — temp file plus `os.Rename()`; a machine that dies mid-write keeps the old file.
|
|
- **Fingerprint diffing before write** — no disk churn on unchanged state.
|
|
- **Soft revocation** — `revoked_at` rather than deletes; preserves audit history.
|
|
- **Run snapshots** — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
|
|
- **Monitors run in two places** — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
|
|
- **Redis for sessions only** — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
|
|
- **guacd for console** — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
|
|
- **`org_id` on every document** — isolation enforced at the query layer, not by separate databases.
|
|
- **root only** — manages `/root/.ssh/authorized_keys`; no per-user key management.
|
|
- **Windows agents are second-class by design** — register, heartbeat, run steps, report inventory; no `authorized_keys` management.
|