# Vantage

A self-hosted, multi-tenant infrastructure control plane. It started as SSH key management and has grown into fleet management: SSH key assignment, workflow/script execution, service monitoring, a secrets vault, a browser console (SSH/RDP/VNC), and OS update management.

A central server (Go + Next.js + MongoDB + Redis) drives a lightweight Go agent installed on each managed server. Agents poll over gRPC and also hold a bidirectional command stream for push-style commands.

---

## Architecture Overview

```
┌──────────────────────────────────────────────┐
│  Next.js 16 Frontend (web, :3000)            │
│  servers · keys · workflows · monitors       │
│  secrets · audit · console · settings        │
└───────────────┬──────────────────────────────┘
                │ REST + cookie session
┌───────────────▼──────────────────────────────┐
│  Go Backend (server)                         │
│  :8080 REST (gin)   :9090 gRPC (agents)      │
│  MongoDB (state) · Redis (sessions)          │
│  monitor scheduler · workflow runner         │
│  guacd tunnel proxy for browser console      │
└───────────────┬──────────────────────────────┘
                │ gRPC (TLS) — outbound from agent only
┌───────────────▼──────────────────────────────┐
│  Go Agent (per server, Linux + Windows)      │
│  polls SyncKeys · CommandStream              │
│  rewrites authorized_keys (Linux only)       │
│  runs workflow steps · monitors · inventory  │
└──────────────────────────────────────────────┘
```

Multi-tenancy: every domain document carries `org_id`, and every service query is scoped by it. Org is resolved from the session, and optionally cross-checked against the request host (`<slug>.vantage.<tld>`).

---

## Repository Structure

```
vantage/
├── agent/
│   ├── cmd/main.go                  # flags: -generate-key
│   └── internal/
│       ├── checker/                 # monitor check execution
│       ├── config/                  # config.yaml load/save
│       ├── exec/                    # workflow step execution
│       ├── grpc/                    # client + generated pb
│       ├── inventory/               # CPU/mem/disk collection (linux/other)
│       ├── keys/                    # authorized_keys read/diff/write
│       ├── monitors/                # agent-run monitor loop
│       ├── sync/                    # poll loop + command stream
│       └── updates/                 # OS package update check/apply
├── server/
│   ├── cmd/main.go
│   └── internal/
│       ├── api/                     # REST handlers
│       ├── auth/                    # local, OIDC, session, middleware, orghost
│       ├── checker/                 # server-run monitor checks
│       ├── db/                      # mongo connect + Col()
│       ├── grpc/                    # gRPC server + generated pb
│       ├── models/                  # MongoDB documents
│       ├── monitorsched/            # server-side monitor scheduler
│       ├── notify/                  # smtp, http, templating, dispatch
│       └── services/                # business logic + migrations
├── web/
│   ├── app/(app)/                   # authed routes
│   ├── app/login, app/setup         # unauthed routes
│   ├── components/                  # ui/, workflows/, monitors/, Sidebar
│   └── lib/                         # api client, guac console, query client
├── proto/vantage/v1/vantage.proto
├── installer/                       # Windows: setup.ps1, nssm.exe, WiX .wxs
├── deploy/                          # docker-compose.yml, agent.service
└── .gitea/workflows/                # agent-release.yml, server-deploy.yml
```

---

## Subsystems

### SSH keys
Upload a public key, assign it per server, revoke softly. The agent diffs desired vs on-disk state and rewrites `/root/.ssh/authorized_keys` atomically. Keys can also be generated *on* a server by the agent; the private half can optionally be uploaded and is stored AES-256-GCM encrypted.

### Workflows
A library of reusable **steps** (bash or PowerShell scripts with declared inputs, outputs, and secret refs) composed into **workflows** targeting a set of servers. Running one snapshots the resolved steps into a `WorkflowRun`, then dispatches `RunStepCmd` over the agent command stream. Step stdout/stderr streams back as `StepOutputChunk` and is written to a log file on disk; the UI streams it live. Steps support `on_failure: stop|continue|retry`, per-run env passed between steps via `output_env`, and a per-run workspace directory the agent cleans up at the end.

Default steps are seeded per org at boot (`SeedDefaultSteps`). Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever).

### Monitors
HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.

### Notification channels
Per-org outbound destinations: `webhook`, `smtp`, `discord`, `slack`, `telegram`. Monitors reference channels by ID. Channels are testable from the UI.

### Secrets vault
Key/value pairs grouped by name, encrypted at rest with AES-256-GCM. Consumed two ways: referenced by workflow steps via `secret_refs` (injected as env at execution), and read by Kubernetes External Secrets Operator via `GET /api/secrets/:group/values` using a bearer token whose SHA-256 hash is stored in settings.

### Browser console
`POST /api/console/connect` mints a one-time session token; `GET /api/console/tunnel` upgrades to a WebSocket and proxies to **guacd** (Apache Guacamole daemon) using `github.com/wwt/guac`. SSH connections authenticate with a stored private key; RDP/VNC credentials are encrypted, single-use, and consumed when the tunnel opens.

### Inventory and OS updates
Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`).

### Agent self-update
`UpdateAgentCmd` carries a target version and Gitea base URL; the agent downloads and replaces itself.

---

## Auth and Orgs

- **Bootstrap** — first run has no users. `GET /auth/bootstrap-status` drives `/setup`, `POST /auth/bootstrap` creates the first org plus its owner.
- **Local auth** — email + password (bcrypt), `POST /auth/login`.
- **OIDC** — configured *per org* (`org_oidc`), issuer + client ID + encrypted client secret. `/auth/oidc/start` → `/auth/oidc/callback`.
- **Sessions** — opaque 32-byte hex ID in the `km_session` cookie, session body stored in Redis with a 24h TTL.
- **Roles** — `owner`, `admin`, `member`. `/api/settings` and `/api/org/*` require owner or admin.
- **Host/org guard** — `APP_ROOT_LABEL` (default `vantage`) defines the app root label. A request to `<slug>.vantage.<tld>` resolves that org from the slug and rejects sessions belonging to a different one. Org lookups are cached for 60s.

Unique indexes on user email and org slug are a **security property**, not an optimisation: `GetUserByEmail` does an unscoped `FindOne`, so duplicates would let the OIDC cross-org guard compare against an arbitrary user. Same for duplicate settings docs and duplicate ESO token hashes.

---

## gRPC API

```protobuf
service Vantage {
  rpc Register(RegisterRequest)              returns (RegisterResponse);
  rpc SyncKeys(SyncRequest)                  returns (SyncResponse);
  rpc UploadGeneratedKey(UploadKeyRequest)   returns (UploadKeyResponse);
  rpc ReportUpdates(ReportUpdatesRequest)    returns (ReportUpdatesResponse);
  rpc ReportInventory(InventoryReport)       returns (InventoryReportResponse);
  rpc SyncMonitors(SyncMonitorsRequest)      returns (SyncMonitorsResponse);
  rpc ReportChecks(ReportChecksRequest)      returns (ReportChecksResponse);
  rpc CommandStream(stream AgentMessage)     returns (stream ServerCommand);
}
```

`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.

`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`.

Key-state polling stays on the 30s `SyncKeys` interval. Full message definitions live in `proto/vantage/v1/vantage.proto`.

---

## REST API

Unauthenticated:

```
GET  /install            /install.ps1        # dynamic agent install scripts
GET  /update             /update.ps1
GET  /auth/bootstrap-status
POST /auth/bootstrap     /auth/login  /auth/logout
GET  /auth/me            /auth/oidc/start    /auth/oidc/callback
GET  /api/secrets/:group/values               # bearer token (ESO)
```

Session-authed under `/api`:

```
servers      GET,POST /servers · GET,POST /servers/new · GET,DELETE /servers/:id
             POST /servers/:id/{generate-key,update-agent,apply-updates}
keys         GET,POST /keys · GET,DELETE /keys/:id · GET /keys/:id/private-key
             POST /keys/:id/assign · DELETE /keys/:id/assign/:serverId
workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
             POST /steps/{import,seed-defaults,parse} · GET /steps/usage
             GET,POST /workflows · GET,PUT,DELETE /workflows/:id
             POST /workflows/:id/run · GET /workflows/:id/runs
             GET /runs/:runId · POST /runs/:runId/cancel
             GET /runs/:runId/servers/:serverId/logs[/stream]
monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
             GET /monitors/:id/{incidents,uptime}
channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
             POST /secrets/:group/reveal · DELETE /secrets/:group/:key
console      POST /console/connect · GET /console/tunnel (websocket)
audit        GET /audit
agent        GET /agent/latest-version
settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
             GET,PUT /org/oidc                                       (owner|admin)
```

---

## MongoDB Collections

`servers` · `keys` · `assignments` · `orgs` · `users` · `org_oidc` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `migrations`

Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`.

Notes that are not obvious from the structs:

- `servers.agent_token_hash` stores SHA-256 of the token, never plaintext. `pre_reg_token` is cleared after `Register()`. `status` is `pending` → `active` on register, `offline` when `last_seen` passes the threshold (swept every 2 min).
- `servers.inventory` holds the latest metrics snapshot with separate `metrics_at` / `static_at` timestamps.
- `keys.private_key_enc` and `passphrase_enc` are AES-256-GCM; the JSON form exposes only `has_private_key` / `has_passphrase`.
- `assignments.revoked_at: null` means active. Revocation is soft, preserving audit history.
- `workflow_runs.steps_snapshot` freezes the resolved steps so editing the library never rewrites history.
- `console_sessions.token_consumed_at` is set atomically to enforce one-time use.

### Migrations

`services.RunMigrations()` runs at boot, recording markers in `migrations`:

- `0001_default_org_backfill`
- `0002_settings_org_backfill` (must run before 0003 — 0003 can create a `default` org, which pushes 0002 into its ambiguous multi-org branch)
- `0003_missed_org_scopes`

Index builders (`EnsureAuthIndexes`, `EnsureSettingsIndexes`) are fatal on failure; `EnsureSecretIndexes` and `EnsureWorkflowIndexes` only warn.

---

## Agent Lifecycle

### Config file

Linux `/etc/vantage/config.yaml`, Windows `%ProgramData%\vantage\config.yaml`. Directory `0700`, file `0600`.

```yaml
server_url: "vantage.yourdomain.com:9090"
server_id: "<uuid>"
pre_reg_token: "<token>"   # removed after first successful Register()
agent_token: ""            # written by agent after Register()
poll_interval: 30s
tls: true
```

### Startup

```
1. Load config
2. If pre_reg_token present → Register() → save agent_token, clear pre_reg_token, reconnect
3. Start goroutines: command stream · update check (hourly) · inventory · monitors
4. Enter SyncKeys poll loop (default 30s)
```

### Poll loop

```
1. SyncKeys(server_id, agent_token, agent_version)
2. Non-Linux hosts stop here — Windows agents register and heartbeat only
3. Diff desired keys against /root/.ssh/authorized_keys; unchanged → no write
4. Changed → write .tmp, os.Rename() over the real file, chmod 0600
```

### Install

Linux: systemd unit at `/etc/systemd/system/vantage-agent.service`, `Restart=always`, runs as root.
Windows: MSI built by CI (WiX), or `installer/setup.ps1` registering the agent as a service via NSSM.

---

## Server Registration Flow

1. **Add Server** in the UI calls `POST /api/servers/new`, which generates a `server_id` and a pre-registration token (TTL 1 hour, single-use).
2. The UI shows a one-liner:
   ```bash
   curl -fsSL https://vantage.yourdomain.com/install | \
     bash -s -- --server-id=<id> --token=<token>
   ```
   Windows gets the `/install.ps1` equivalent.
3. The script detects arch, downloads the agent from the Gitea release, verifies the SHA-256 checksum, writes the config, installs and starts the service.
4. The server flips to `active` on first sync.

`/install` is served dynamically, injecting the latest agent version from the Gitea API.

---

## Environment Variables (server)

| Name | Required | Notes |
| --- | --- | --- |
| `GRPC_HOST` | **yes** | `host:port` agents dial. Boot fails without it — there is no safe default; falling back to the web host would hand agents a port that does not speak gRPC. |
| `MONGO_URI` | no | default `mongodb://localhost:27017` |
| `MONGO_DB` | no | default `vantage` |
| `REDIS_ADDR` | no | default `localhost:6379` |
| `KEY_ENCRYPTION_KEY` | yes in practice | 64-char hex (32 bytes) for AES-256-GCM. Required for private keys, secrets, OIDC secrets, RDP credentials. |
| `GITEA_HOST` | yes | used to build install scripts and agent download URLs |
| `GUACD_ADDR` | no | default `guacd:4822` |
| `APP_ROOT_LABEL` | no | default `vantage`; wrong value disables the host/session org guard |
| `VANTAGE_WORKFLOW_LOG_DIR` | no | where run logs are written |

`deploy/docker-compose.yml` runs four services: `redis`, `guacd`, `server` (8080 + 9090), `web` (3000). MongoDB is external.

---

## Security

- gRPC over TLS; agents connect outbound only, no inbound firewall holes on managed servers.
- Per-server agent token stored as SHA-256 on the server, plaintext only in the agent's `0600` config.
- Pre-registration tokens are short-lived (1 hour) and single-use.
- AES-256-GCM at rest for private keys, key passphrases, vault secrets, OIDC client secrets, RDP/VNC credentials.
- Console session tokens are one-time; RDP credentials are consumed on tunnel open.
- ESO read token stored as a SHA-256 hash and rotatable.
- Unique indexes on user email, org slug, settings org, and ESO token hash are load-bearing for tenant isolation.
- `authorized_keys` written `0600`, owned by root. The agent runs as root because it must.
- Every mutating API path writes an audit event.

---

## Frontend

Next.js 16 (App Router) + React 18, Tailwind 3, TanStack Query. Guacamole client bundled locally in `web/lib/guacamole-common.js`.

| Route | Purpose |
| --- | --- |
| `/setup` | First-run bootstrap: create the first org and owner |
| `/login` | Local or OIDC sign-in |
| `/` | Fleet dashboard |
| `/servers`, `/servers/new`, `/servers/[id]` | Fleet list, install one-liner, server detail (keys, inventory, updates) |
| `/servers/[id]/console` | Browser SSH/RDP/VNC session |
| `/keys`, `/keys/[id]` | Key library; assign and revoke per server |
| `/workflows`, `/workflows/[id]`, `/workflows/[id]/runs[/runId]` | Compose, run, and follow live logs |
| `/steps` | Reusable step library |
| `/monitors`, `/monitors/new`, `/monitors/[id][/edit]` | Checks, uptime, incidents |
| `/secrets`, `/secrets/[group]` | Vault |
| `/audit` | Audit log |
| `/settings`, `/settings/org`, `/settings/notifications` | Alerts, members, OIDC, channels |

---

## CI/CD — Gitea Actions

### `agent-release.yml` — triggered by `agent/v*` tags

Builds `linux/amd64`, `linux/arm64`, `windows/amd64`, writes `checksums.txt`, creates a Gitea release. A second `msi` job on `windows-2022` packages the WiX installer.

```bash
GOOS=linux GOARCH=amd64 go build \
  -ldflags="-s -w -X main.Version=${VERSION}" \
  -o dist/vantage-agent-linux-amd64 ./cmd
```

### `server-deploy.yml` — triggered on push to `main`

Builds and pushes the `server` and `web` images to the Gitea container registry, then deploys over SSH:

```bash
cd /opt/vantage && docker compose pull && docker compose up -d --remove-orphans
```

### Tagging

```bash
git tag agent/v1.0.0 && git push origin agent/v1.0.0   # agent release
git push origin main                                    # server + web deploy
```

### Secrets / variables

| Name | Type | Value |
| --- | --- | --- |
| `RELEASE_TOKEN` | Secret | Gitea API token, `write:release` |
| `REGISTRY_USER` | Secret | Gitea username |
| `REGISTRY_PASSWORD` | Secret | Gitea token, `write:packages` |
| `DEPLOY_HOST` | Secret | server VM host |
| `DEPLOY_USER` | Secret | SSH user for deploy |
| `DEPLOY_SSH_KEY` | Secret | deploy private key |
| `GITEA_HOST` | Variable | `gitea.hostxtra.co.uk` |

---

## Design Decisions

- **gRPC for agent traffic** — strong typing and cheap versioning; polling for state, one bidirectional stream for commands.
- **Outbound-only agents** — no inbound ports on managed servers, works behind NAT.
- **Poll for keys, push for commands** — a 30s key poll is fine, but running a workflow step should not wait up to 30s.
- **Atomic `authorized_keys` rewrite** — temp file plus `os.Rename()`; a machine that dies mid-write keeps the old file.
- **Fingerprint diffing before write** — no disk churn on unchanged state.
- **Soft revocation** — `revoked_at` rather than deletes; preserves audit history.
- **Run snapshots** — workflow runs freeze their resolved steps so editing a step never rewrites past runs.
- **Monitors run in two places** — server-side for external endpoints, agent-side for anything only reachable from inside the target network.
- **Redis for sessions only** — all durable state stays in MongoDB; losing Redis logs everyone out and nothing else.
- **guacd for console** — protocol handling is Guacamole's problem, not ours; we proxy the WebSocket and manage credentials.
- **`org_id` on every document** — isolation enforced at the query layer, not by separate databases.
- **root only** — manages `/root/.ssh/authorized_keys`; no per-user key management.
- **Windows agents are second-class by design** — register, heartbeat, run steps, report inventory; no `authorized_keys` management.