diff --git a/docs/superpowers/specs/2026-09-07-control-plane-backup-restore-design.md b/docs/superpowers/specs/2026-09-07-control-plane-backup-restore-design.md new file mode 100644 index 0000000..f1ac675 --- /dev/null +++ b/docs/superpowers/specs/2026-09-07-control-plane-backup-restore-design.md @@ -0,0 +1,314 @@ +# Control plane backup and restore + +Date: 2026-09-07 +Status: approved, ready for implementation planning + +## Problem + +Vantage has no backup story. A self-hosted deployment holds its entire state in +MongoDB and encrypts the sensitive half of it — SSH private keys, key +passphrases, vault secrets, OIDC client secrets, RDP and VNC credentials — with +AES-256-GCM under a single 32-byte key supplied as the `KEY_ENCRYPTION_KEY` +environment variable. + +That key is a bare value. It carries no identifier, is not wrapped, and is not +recorded anywhere alongside the data it protects. Restoring a database without +it produces a control plane whose every secret is permanently unreadable, and +nothing in the product tells an operator this before it happens. + +`mongodump` exists and operators can use it, but it says nothing about the +encryption key, so the most common way to lose everything is to hold a perfectly +good database dump and no key. + +## Goal + +A standalone command-line tool that backs up and restores a whole Vantage +deployment, and that makes the key relationship impossible to get wrong by +accident. + +Explicitly not a goal: point-in-time recovery, incremental backups, built-in +storage backends, encryption of the archive itself, per-tenant export, and +backups scheduled from inside the server. Each is a separate decision and +several are better served by tools the operator already has. + +## Design + +### Scope of a backup + +One backup covers one MongoDB database: every collection in it, whether or not +that collection is tenant-scoped. A deployment-level disaster recovery tool that +skipped `migrations` or `vulndb_meta` would restore a database the server +refuses to boot against. + +Collections are enumerated live with `ListCollectionNames` rather than read from +a hardcoded list. This is the opposite choice to `services.ScopedCollections`, +and deliberately so: that list can afford to be hand-maintained because +`AssertNoScopedCollectionMissed` fails boot when it drifts. A backup tool has no +such assertion available, so a second hand-maintained registry would drift +silently and the first symptom would be a restore missing a collection nobody +noticed was added. + +`--exclude` accepts collection names for the volume-heavy ones — +`workflow_log_lines`, `monitor_samples`, `audit_logs`. Whatever is excluded is +recorded in the manifest, so an archive can never claim to be complete when it +is not. + +Redis is not backed up. It holds sessions only; losing it logs everyone out and +nothing else, which is already the documented behaviour. The restore output says +so explicitly rather than leaving an operator to wonder. + +### Where the code lives + +Two units. + +`shared/backup/` holds the logic: dump, restore, manifest construction, archive +reading and writing, and key fingerprinting. It depends on the MongoDB driver +and the standard library, and on no CLI framework. Keeping it in `shared/` and +free of cobra is what lets `server` import it later if backups scheduled from +inside the control plane are ever built, without pulling a command-line parser +into the server binary. + +`vantagectl/` is a new module in `go.work`, importing `shared`. It holds the +cobra command tree and nothing else. A separate module rather than a package +under `shared/` because adding cobra to `shared/go.mod` would put cobra and +pflag into the module graph of `server`, `admin` and `sitesvc`, none of which +use them. Binaries are unaffected — Go links only what is imported — but three +`go.sum` files would grow and three CI builds would fetch a dependency they do +not need. `agent/` is already a separate module for the same reason. + +The tool imports nothing from `server/`. No `db.Col()`, no `services`, no config +loader, and it never dials the REST or gRPC API. It needs only network reach to +MongoDB, a database name, and `KEY_ENCRYPTION_KEY` in its own environment. This +is what lets it run against a control plane that is down, half-migrated, or was +deleted an hour ago — which is the only condition under which anyone runs a +restore. + +### Dump implementation + +The dump is written against the MongoDB driver, not by shelling out to +`mongodump`. + +Two reasons. `server`'s runtime image is `scratch` and carries no shell and no +mongo tools, so a wrapper would depend on a matching `mongodump` version being +installed on whatever host runs the tool. And the manifest must be written by +the same process that read the documents, or the fingerprint and per-collection +checksums are claims about data the writer never saw. + +The cost is that BSON round-tripping is ours to get right. Documents are written +as raw BSON exactly as the driver returns them, without an intermediate map, so +`ObjectId`, `Decimal128`, `DateTime`, binary subtypes and nulls survive +unchanged. A round-trip test asserting byte-equal BSON is the guard. + +### Archive format + +A gzipped tar named `vantage-backup--.tar.gz`: + +``` +manifest.json +collections/.bson concatenated raw BSON documents +indexes/.json index specifications +``` + +`manifest.json` carries: + +| Field | Purpose | +| --- | --- | +| `format_version` | Currently `1`. Restore refuses an unknown version rather than guessing at it | +| `created_at` | RFC3339, UTC | +| `vantage_version` | Build stamp of the tool that wrote the archive | +| `hostname` | Provenance; which machine produced this | +| `mongo_db` | Source database name | +| `mongo_server_version` | Restore warns on a major version gap | +| `key_fingerprint` | `sha256` of the raw 32 key bytes, hex, or `null`. Never the key | +| `collections[]` | Per collection: name, document count, uncompressed bytes, `sha256` of the `.bson` member | +| `excluded[]` | Collection names passed to `--exclude` | + +Per-collection checksums mean a truncated or corrupted archive is detected +before a single document is written, rather than halfway through a restore. + +### Key custody + +The key never enters the archive. The archive is exactly as sensitive as a +`mongodump` of the same database, and no more. + +What the archive carries is `sha256` of the raw key bytes. A hash of the key +proves identity without being a hint at the value, which is what allows an +operator to answer "will this archive restore into this deployment" without +holding both in front of them. + +Backup refuses to run when `KEY_ENCRYPTION_KEY` is unset or malformed. An +archive full of ciphertext whose key was never recorded is worse than no archive +at all, because it looks like a backup. `--allow-no-key` exists for a deployment +that genuinely stores no encrypted material; it stamps `key_fingerprint: null`, +which restore then reports loudly rather than treating as a match. + +Restore compares the archive's fingerprint against the key in the current +environment: + +- Fingerprints match: proceed. +- Fingerprints differ: refuse, printing both. +- Archive has a fingerprint, environment has no key: refuse. +- `--ignore-key-mismatch`: proceed, having first printed exactly which + collections hold ciphertext that will be undecryptable — `keys`, `secrets`, + `auth_providers`, `console_sessions`, `settings`. + +### Restore semantics + +The order is fixed: + +1. Read `manifest.json` and check `format_version`. +2. Verify every archive member against its manifest checksum. Nothing is written + before this passes. +3. Apply the fingerprint rules above. +4. Inspect the target: `ListCollectionNames` and document counts. A non-empty + database is refused, printing what was found. `--force` proceeds. +5. Per collection: under `--force`, drop it first; then bulk-insert in batches + of 1000 with `ordered=false`. +6. Replay index specifications from `indexes/.json`, skipping `_id_`. +7. Print a summary: collection, documents restored, indexes created. + +Restore is not idempotent, and says so. A second run without `--force` is +refused because step 4 now finds data. A restore interrupted during step 5 +leaves a partial database that the next run refuses to touch — correct, because +the alternative is a silent merge. There are no merge or upsert semantics at +all: merging two control planes reconciles nothing and produces a fleet that +half works, and upserting by `_id` resurrects rows deleted since the backup, +which for revoked keys and deleted users is a security regression wearing the +costume of a convenience. + +Index replay is fatal per collection when a unique index fails to build, and a +warning when a non-unique one does. A unique index that cannot be created means +the restored data violates it, and the unique indexes here — `(instance_id, +email)`, instance slug, settings instance, the ESO token hash — are +tenant-isolation properties rather than optimisations. The failure names the +offending index. + +### Destructive confirmation + +Restore under `--force` requires a typed confirmation when stdin is a TTY. + +When stdin is not a TTY — a Kubernetes Job, a CI step, a cron entry — the +confirmation comes from `--confirm-db `, whose value must equal the +resolved target database name or restore refuses. Naming the database in the +argument means a copy-pasted restore command carries its intended target with +it and cannot destroy a different one. + +A dynamic flag name containing the database name was considered and rejected: +cobra registers flags before parsing, and the target database is not known at +registration time. + +### Command surface + +``` +vantagectl root; prints help +├── backup --out DIR|- --exclude a,b --allow-no-key +├── restore ARCHIVE --force --confirm-db NAME --ignore-key-mismatch +├── inspect ARCHIVE +└── verify ARCHIVE +``` + +Persistent flags on the root command, so every subcommand accepts them and they +are documented once: `--mongo-uri` (env `MONGO_URI`), `--db` (env `MONGO_DB`, +falling back to the URI path), `--log-level`. + +Environment fallback is wired with an explicit `Changed` check on each flag +rather than through viper. Viper is a configuration-file and remote-config +system; this tool reads no configuration file, and pulling it in to call +`os.Getenv` would make the largest dependency in the binary the one doing the +smallest job. + +`inspect` prints the manifest — when the archive was made, by what version, +which collections it holds, how many documents, what was excluded, and the key +fingerprint — and contacts no database. It is what an operator runs to find out +whether an archive they have found is worth anything. + +`verify` adds a live check: whether the archive's fingerprint matches the key in +the current environment, and whether it matches the database being pointed at. +This is the command that distinguishes "we have backups" from "we have backups +that will restore", and the documentation recommends running it on a schedule. + +`--out -` streams the tarball to stdout, so piping into `aws s3 cp -`, `restic` +or `age` covers storage and archive encryption without the tool growing backends +of its own. + +### Distribution + +Three ways to run it, because the deployments that need it run Docker Compose, +Kubernetes, or neither. + +**Loose binary.** A new `.gitea/workflows/vantagectl-release.yml`, triggered on +`vantagectl/v*` tags, shaped like `agent-release.yml`. Builds `linux/amd64`, +`linux/arm64`, `darwin/arm64` and `windows/amd64` with `CGO_ENABLED=0`, writes +`checksums.txt`, and creates a Gitea release. + +**Container image.** `deploy/docker/vantagectl.Dockerfile` produces a `scratch` +image holding the static binary and an explicitly copied `/tmp`, which the +archive is staged in before compression — the same omission that silently +disabled `vulnsched` on a scratch image. Pushed by `server-deploy.yml` as an +eighth image. + +```bash +docker run --rm --network vantage_default \ + -e MONGO_URI -e MONGO_DB -e KEY_ENCRYPTION_KEY \ + -v /backups:/out \ + gitea.hostxtra.co.uk/mrhid6/vantagectl backup --out /out +``` + +**Kubernetes.** The chart gains `backup.enabled`, defaulting to **false**, +rendering a `CronJob` that runs the same image and mounts the existing MongoDB +and `KEY_ENCRYPTION_KEY` secrets by reference rather than re-declaring them. +Output goes to a PVC named in values. The default is off because a backup with +nowhere durable to land is a false sense of safety and the chart cannot know +where that is; `NOTES.txt` says so on install. + +Restore in Kubernetes is the same image run as a one-shot `Job`. The chart ships +no restore manifest: a restore is an operator decision with a confirmation +attached to it, and must never be something a `helm upgrade` can trigger. + +`server-deploy.yml`'s rebuild trigger table gains a `vantagectl` row — +`vantagectl/`, `shared/`, `go.work` — which makes `shared/` fan out to four Go +images rather than three. That table is already called out in `CLAUDE.md` as a +place where a missed entry ships a stale image. + +## Testing + +`shared/backup` is tested against a real MongoDB, via `testcontainers-go` if the +module graph tolerates it and otherwise behind a `MONGO_TEST_URI` environment +variable that skips when unset. + +Required cases: + +- Round trip: seed one document of every awkward BSON type — `ObjectId`, + `Decimal128`, `DateTime`, binary, null, nested arrays — back up, restore into + a second database, assert byte-equal BSON. +- A single corrupted byte in a `.bson` member causes restore to refuse before + writing anything. +- Fingerprint mismatch is refused; `--ignore-key-mismatch` proceeds and names + the ciphertext-bearing collections. +- A non-empty target is refused; `--force` replaces it. +- An excluded collection is absent from the archive and named in the manifest. +- A unique index that cannot be built aborts the restore, naming the index. + +Fingerprint computation is a pure function and is tested without a database. + +## Documentation + +`docsite/docs/operations/backup-and-restore.md`, covering: + +- What `KEY_ENCRYPTION_KEY` is, that it is not in the backup, and that losing it + is unrecoverable. This comes first on the page, not as a note at the bottom. +- The three run modes above, each as a command that can be copied. +- A restore drill: restore into a scratch database and run `verify`, because an + untested backup is a hypothesis. +- What is not covered: Redis sessions, the vulnerability database (re-pulled + automatically), and agent state on managed servers — agents reconnect on their + own and `servers.agent_token_hash` is in the backup, so no re-enrolment is + needed. + +`CLAUDE.md` gains a section describing the tool, since a new module, a new +image, a new workflow and a new chart toggle are each something that drifts +quietly. + +## Open questions + +None. Every decision above was settled during design.