docs: Design for control plane backup and restore
Standalone vantagectl CLI (cobra, own module) that dumps and restores a whole Vantage MongoDB database, stamping a sha256 fingerprint of KEY_ENCRYPTION_KEY into the manifest so a restore cannot silently produce a database whose secrets are unreadable. The key itself never enters the archive.
This commit is contained in:
@@ -0,0 +1,314 @@
|
||||
# Control plane backup and restore
|
||||
|
||||
Date: 2026-09-07
|
||||
Status: approved, ready for implementation planning
|
||||
|
||||
## Problem
|
||||
|
||||
Vantage has no backup story. A self-hosted deployment holds its entire state in
|
||||
MongoDB and encrypts the sensitive half of it — SSH private keys, key
|
||||
passphrases, vault secrets, OIDC client secrets, RDP and VNC credentials — with
|
||||
AES-256-GCM under a single 32-byte key supplied as the `KEY_ENCRYPTION_KEY`
|
||||
environment variable.
|
||||
|
||||
That key is a bare value. It carries no identifier, is not wrapped, and is not
|
||||
recorded anywhere alongside the data it protects. Restoring a database without
|
||||
it produces a control plane whose every secret is permanently unreadable, and
|
||||
nothing in the product tells an operator this before it happens.
|
||||
|
||||
`mongodump` exists and operators can use it, but it says nothing about the
|
||||
encryption key, so the most common way to lose everything is to hold a perfectly
|
||||
good database dump and no key.
|
||||
|
||||
## Goal
|
||||
|
||||
A standalone command-line tool that backs up and restores a whole Vantage
|
||||
deployment, and that makes the key relationship impossible to get wrong by
|
||||
accident.
|
||||
|
||||
Explicitly not a goal: point-in-time recovery, incremental backups, built-in
|
||||
storage backends, encryption of the archive itself, per-tenant export, and
|
||||
backups scheduled from inside the server. Each is a separate decision and
|
||||
several are better served by tools the operator already has.
|
||||
|
||||
## Design
|
||||
|
||||
### Scope of a backup
|
||||
|
||||
One backup covers one MongoDB database: every collection in it, whether or not
|
||||
that collection is tenant-scoped. A deployment-level disaster recovery tool that
|
||||
skipped `migrations` or `vulndb_meta` would restore a database the server
|
||||
refuses to boot against.
|
||||
|
||||
Collections are enumerated live with `ListCollectionNames` rather than read from
|
||||
a hardcoded list. This is the opposite choice to `services.ScopedCollections`,
|
||||
and deliberately so: that list can afford to be hand-maintained because
|
||||
`AssertNoScopedCollectionMissed` fails boot when it drifts. A backup tool has no
|
||||
such assertion available, so a second hand-maintained registry would drift
|
||||
silently and the first symptom would be a restore missing a collection nobody
|
||||
noticed was added.
|
||||
|
||||
`--exclude` accepts collection names for the volume-heavy ones —
|
||||
`workflow_log_lines`, `monitor_samples`, `audit_logs`. Whatever is excluded is
|
||||
recorded in the manifest, so an archive can never claim to be complete when it
|
||||
is not.
|
||||
|
||||
Redis is not backed up. It holds sessions only; losing it logs everyone out and
|
||||
nothing else, which is already the documented behaviour. The restore output says
|
||||
so explicitly rather than leaving an operator to wonder.
|
||||
|
||||
### Where the code lives
|
||||
|
||||
Two units.
|
||||
|
||||
`shared/backup/` holds the logic: dump, restore, manifest construction, archive
|
||||
reading and writing, and key fingerprinting. It depends on the MongoDB driver
|
||||
and the standard library, and on no CLI framework. Keeping it in `shared/` and
|
||||
free of cobra is what lets `server` import it later if backups scheduled from
|
||||
inside the control plane are ever built, without pulling a command-line parser
|
||||
into the server binary.
|
||||
|
||||
`vantagectl/` is a new module in `go.work`, importing `shared`. It holds the
|
||||
cobra command tree and nothing else. A separate module rather than a package
|
||||
under `shared/` because adding cobra to `shared/go.mod` would put cobra and
|
||||
pflag into the module graph of `server`, `admin` and `sitesvc`, none of which
|
||||
use them. Binaries are unaffected — Go links only what is imported — but three
|
||||
`go.sum` files would grow and three CI builds would fetch a dependency they do
|
||||
not need. `agent/` is already a separate module for the same reason.
|
||||
|
||||
The tool imports nothing from `server/`. No `db.Col()`, no `services`, no config
|
||||
loader, and it never dials the REST or gRPC API. It needs only network reach to
|
||||
MongoDB, a database name, and `KEY_ENCRYPTION_KEY` in its own environment. This
|
||||
is what lets it run against a control plane that is down, half-migrated, or was
|
||||
deleted an hour ago — which is the only condition under which anyone runs a
|
||||
restore.
|
||||
|
||||
### Dump implementation
|
||||
|
||||
The dump is written against the MongoDB driver, not by shelling out to
|
||||
`mongodump`.
|
||||
|
||||
Two reasons. `server`'s runtime image is `scratch` and carries no shell and no
|
||||
mongo tools, so a wrapper would depend on a matching `mongodump` version being
|
||||
installed on whatever host runs the tool. And the manifest must be written by
|
||||
the same process that read the documents, or the fingerprint and per-collection
|
||||
checksums are claims about data the writer never saw.
|
||||
|
||||
The cost is that BSON round-tripping is ours to get right. Documents are written
|
||||
as raw BSON exactly as the driver returns them, without an intermediate map, so
|
||||
`ObjectId`, `Decimal128`, `DateTime`, binary subtypes and nulls survive
|
||||
unchanged. A round-trip test asserting byte-equal BSON is the guard.
|
||||
|
||||
### Archive format
|
||||
|
||||
A gzipped tar named `vantage-backup-<db>-<RFC3339>.tar.gz`:
|
||||
|
||||
```
|
||||
manifest.json
|
||||
collections/<name>.bson concatenated raw BSON documents
|
||||
indexes/<name>.json index specifications
|
||||
```
|
||||
|
||||
`manifest.json` carries:
|
||||
|
||||
| Field | Purpose |
|
||||
| --- | --- |
|
||||
| `format_version` | Currently `1`. Restore refuses an unknown version rather than guessing at it |
|
||||
| `created_at` | RFC3339, UTC |
|
||||
| `vantage_version` | Build stamp of the tool that wrote the archive |
|
||||
| `hostname` | Provenance; which machine produced this |
|
||||
| `mongo_db` | Source database name |
|
||||
| `mongo_server_version` | Restore warns on a major version gap |
|
||||
| `key_fingerprint` | `sha256` of the raw 32 key bytes, hex, or `null`. Never the key |
|
||||
| `collections[]` | Per collection: name, document count, uncompressed bytes, `sha256` of the `.bson` member |
|
||||
| `excluded[]` | Collection names passed to `--exclude` |
|
||||
|
||||
Per-collection checksums mean a truncated or corrupted archive is detected
|
||||
before a single document is written, rather than halfway through a restore.
|
||||
|
||||
### Key custody
|
||||
|
||||
The key never enters the archive. The archive is exactly as sensitive as a
|
||||
`mongodump` of the same database, and no more.
|
||||
|
||||
What the archive carries is `sha256` of the raw key bytes. A hash of the key
|
||||
proves identity without being a hint at the value, which is what allows an
|
||||
operator to answer "will this archive restore into this deployment" without
|
||||
holding both in front of them.
|
||||
|
||||
Backup refuses to run when `KEY_ENCRYPTION_KEY` is unset or malformed. An
|
||||
archive full of ciphertext whose key was never recorded is worse than no archive
|
||||
at all, because it looks like a backup. `--allow-no-key` exists for a deployment
|
||||
that genuinely stores no encrypted material; it stamps `key_fingerprint: null`,
|
||||
which restore then reports loudly rather than treating as a match.
|
||||
|
||||
Restore compares the archive's fingerprint against the key in the current
|
||||
environment:
|
||||
|
||||
- Fingerprints match: proceed.
|
||||
- Fingerprints differ: refuse, printing both.
|
||||
- Archive has a fingerprint, environment has no key: refuse.
|
||||
- `--ignore-key-mismatch`: proceed, having first printed exactly which
|
||||
collections hold ciphertext that will be undecryptable — `keys`, `secrets`,
|
||||
`auth_providers`, `console_sessions`, `settings`.
|
||||
|
||||
### Restore semantics
|
||||
|
||||
The order is fixed:
|
||||
|
||||
1. Read `manifest.json` and check `format_version`.
|
||||
2. Verify every archive member against its manifest checksum. Nothing is written
|
||||
before this passes.
|
||||
3. Apply the fingerprint rules above.
|
||||
4. Inspect the target: `ListCollectionNames` and document counts. A non-empty
|
||||
database is refused, printing what was found. `--force` proceeds.
|
||||
5. Per collection: under `--force`, drop it first; then bulk-insert in batches
|
||||
of 1000 with `ordered=false`.
|
||||
6. Replay index specifications from `indexes/<name>.json`, skipping `_id_`.
|
||||
7. Print a summary: collection, documents restored, indexes created.
|
||||
|
||||
Restore is not idempotent, and says so. A second run without `--force` is
|
||||
refused because step 4 now finds data. A restore interrupted during step 5
|
||||
leaves a partial database that the next run refuses to touch — correct, because
|
||||
the alternative is a silent merge. There are no merge or upsert semantics at
|
||||
all: merging two control planes reconciles nothing and produces a fleet that
|
||||
half works, and upserting by `_id` resurrects rows deleted since the backup,
|
||||
which for revoked keys and deleted users is a security regression wearing the
|
||||
costume of a convenience.
|
||||
|
||||
Index replay is fatal per collection when a unique index fails to build, and a
|
||||
warning when a non-unique one does. A unique index that cannot be created means
|
||||
the restored data violates it, and the unique indexes here — `(instance_id,
|
||||
email)`, instance slug, settings instance, the ESO token hash — are
|
||||
tenant-isolation properties rather than optimisations. The failure names the
|
||||
offending index.
|
||||
|
||||
### Destructive confirmation
|
||||
|
||||
Restore under `--force` requires a typed confirmation when stdin is a TTY.
|
||||
|
||||
When stdin is not a TTY — a Kubernetes Job, a CI step, a cron entry — the
|
||||
confirmation comes from `--confirm-db <name>`, whose value must equal the
|
||||
resolved target database name or restore refuses. Naming the database in the
|
||||
argument means a copy-pasted restore command carries its intended target with
|
||||
it and cannot destroy a different one.
|
||||
|
||||
A dynamic flag name containing the database name was considered and rejected:
|
||||
cobra registers flags before parsing, and the target database is not known at
|
||||
registration time.
|
||||
|
||||
### Command surface
|
||||
|
||||
```
|
||||
vantagectl root; prints help
|
||||
├── backup --out DIR|- --exclude a,b --allow-no-key
|
||||
├── restore ARCHIVE --force --confirm-db NAME --ignore-key-mismatch
|
||||
├── inspect ARCHIVE
|
||||
└── verify ARCHIVE
|
||||
```
|
||||
|
||||
Persistent flags on the root command, so every subcommand accepts them and they
|
||||
are documented once: `--mongo-uri` (env `MONGO_URI`), `--db` (env `MONGO_DB`,
|
||||
falling back to the URI path), `--log-level`.
|
||||
|
||||
Environment fallback is wired with an explicit `Changed` check on each flag
|
||||
rather than through viper. Viper is a configuration-file and remote-config
|
||||
system; this tool reads no configuration file, and pulling it in to call
|
||||
`os.Getenv` would make the largest dependency in the binary the one doing the
|
||||
smallest job.
|
||||
|
||||
`inspect` prints the manifest — when the archive was made, by what version,
|
||||
which collections it holds, how many documents, what was excluded, and the key
|
||||
fingerprint — and contacts no database. It is what an operator runs to find out
|
||||
whether an archive they have found is worth anything.
|
||||
|
||||
`verify` adds a live check: whether the archive's fingerprint matches the key in
|
||||
the current environment, and whether it matches the database being pointed at.
|
||||
This is the command that distinguishes "we have backups" from "we have backups
|
||||
that will restore", and the documentation recommends running it on a schedule.
|
||||
|
||||
`--out -` streams the tarball to stdout, so piping into `aws s3 cp -`, `restic`
|
||||
or `age` covers storage and archive encryption without the tool growing backends
|
||||
of its own.
|
||||
|
||||
### Distribution
|
||||
|
||||
Three ways to run it, because the deployments that need it run Docker Compose,
|
||||
Kubernetes, or neither.
|
||||
|
||||
**Loose binary.** A new `.gitea/workflows/vantagectl-release.yml`, triggered on
|
||||
`vantagectl/v*` tags, shaped like `agent-release.yml`. Builds `linux/amd64`,
|
||||
`linux/arm64`, `darwin/arm64` and `windows/amd64` with `CGO_ENABLED=0`, writes
|
||||
`checksums.txt`, and creates a Gitea release.
|
||||
|
||||
**Container image.** `deploy/docker/vantagectl.Dockerfile` produces a `scratch`
|
||||
image holding the static binary and an explicitly copied `/tmp`, which the
|
||||
archive is staged in before compression — the same omission that silently
|
||||
disabled `vulnsched` on a scratch image. Pushed by `server-deploy.yml` as an
|
||||
eighth image.
|
||||
|
||||
```bash
|
||||
docker run --rm --network vantage_default \
|
||||
-e MONGO_URI -e MONGO_DB -e KEY_ENCRYPTION_KEY \
|
||||
-v /backups:/out \
|
||||
gitea.hostxtra.co.uk/mrhid6/vantagectl backup --out /out
|
||||
```
|
||||
|
||||
**Kubernetes.** The chart gains `backup.enabled`, defaulting to **false**,
|
||||
rendering a `CronJob` that runs the same image and mounts the existing MongoDB
|
||||
and `KEY_ENCRYPTION_KEY` secrets by reference rather than re-declaring them.
|
||||
Output goes to a PVC named in values. The default is off because a backup with
|
||||
nowhere durable to land is a false sense of safety and the chart cannot know
|
||||
where that is; `NOTES.txt` says so on install.
|
||||
|
||||
Restore in Kubernetes is the same image run as a one-shot `Job`. The chart ships
|
||||
no restore manifest: a restore is an operator decision with a confirmation
|
||||
attached to it, and must never be something a `helm upgrade` can trigger.
|
||||
|
||||
`server-deploy.yml`'s rebuild trigger table gains a `vantagectl` row —
|
||||
`vantagectl/`, `shared/`, `go.work` — which makes `shared/` fan out to four Go
|
||||
images rather than three. That table is already called out in `CLAUDE.md` as a
|
||||
place where a missed entry ships a stale image.
|
||||
|
||||
## Testing
|
||||
|
||||
`shared/backup` is tested against a real MongoDB, via `testcontainers-go` if the
|
||||
module graph tolerates it and otherwise behind a `MONGO_TEST_URI` environment
|
||||
variable that skips when unset.
|
||||
|
||||
Required cases:
|
||||
|
||||
- Round trip: seed one document of every awkward BSON type — `ObjectId`,
|
||||
`Decimal128`, `DateTime`, binary, null, nested arrays — back up, restore into
|
||||
a second database, assert byte-equal BSON.
|
||||
- A single corrupted byte in a `.bson` member causes restore to refuse before
|
||||
writing anything.
|
||||
- Fingerprint mismatch is refused; `--ignore-key-mismatch` proceeds and names
|
||||
the ciphertext-bearing collections.
|
||||
- A non-empty target is refused; `--force` replaces it.
|
||||
- An excluded collection is absent from the archive and named in the manifest.
|
||||
- A unique index that cannot be built aborts the restore, naming the index.
|
||||
|
||||
Fingerprint computation is a pure function and is tested without a database.
|
||||
|
||||
## Documentation
|
||||
|
||||
`docsite/docs/operations/backup-and-restore.md`, covering:
|
||||
|
||||
- What `KEY_ENCRYPTION_KEY` is, that it is not in the backup, and that losing it
|
||||
is unrecoverable. This comes first on the page, not as a note at the bottom.
|
||||
- The three run modes above, each as a command that can be copied.
|
||||
- A restore drill: restore into a scratch database and run `verify`, because an
|
||||
untested backup is a hypothesis.
|
||||
- What is not covered: Redis sessions, the vulnerability database (re-pulled
|
||||
automatically), and agent state on managed servers — agents reconnect on their
|
||||
own and `servers.agent_token_hash` is in the backup, so no re-enrolment is
|
||||
needed.
|
||||
|
||||
`CLAUDE.md` gains a section describing the tool, since a new module, a new
|
||||
image, a new workflow and a new chart toggle are each something that drifts
|
||||
quietly.
|
||||
|
||||
## Open questions
|
||||
|
||||
None. Every decision above was settled during design.
|
||||
Reference in New Issue
Block a user