feat: Add vantagectl, the control plane backup and restore CLI
Vantage encrypts SSH private keys, vault secrets, SSO client secrets and console credentials with KEY_ENCRYPTION_KEY. That key is in no backup and is not recoverable, so restoring a database without it produces a control plane whose every secret is permanently unreadable. vantagectl dumps and restores a whole database and stamps a SHA-256 fingerprint of the key into every archive, so a restore refuses rather than producing that database. The key itself is never written. - shared/cryptobox: AES-GCM extracted so the server and the CLI share one implementation rather than a copy that can drift - shared/backup: manifest, tar+gzip archive with per-member checksums verified before the first write, dump, restore, verify - vantagectl: its own module, so cobra stays out of the server, admin and sitesvc module graphs - container image, tagged release workflow, CI rebuild trigger - optional Helm CronJob, off by default - operator documentation, reconciled with the existing backups page
This commit is contained in:
@@ -0,0 +1,226 @@
|
||||
---
|
||||
id: backup-and-restore
|
||||
title: Backup and restore
|
||||
sidebar_label: Backup and restore
|
||||
---
|
||||
|
||||
`vantagectl` is a separate command-line tool that backs up and restores the
|
||||
MongoDB database behind a Vantage control plane. It talks to MongoDB directly,
|
||||
never to the Vantage API, so it works against a control plane that is down,
|
||||
half-migrated, or gone — exactly the situation a backup tool has to survive.
|
||||
|
||||
For the store-level overview — what holds what, and why the database alone is
|
||||
not a backup — see [Backups](./backups.md). This page covers the tool.
|
||||
|
||||
:::danger The key comes first
|
||||
Vantage encrypts SSH private keys, key passphrases, vault secrets, SSO client
|
||||
secrets and console credentials with `KEY_ENCRYPTION_KEY`. **It is not in your
|
||||
backup, and it is not recoverable.** A database restored without it is
|
||||
permanently unreadable — not degraded, not partially readable, unreadable.
|
||||
|
||||
Store it wherever you store the credentials you could not rebuild: a password
|
||||
manager, a secrets vault outside this control plane, a piece of paper in a
|
||||
safe. Anywhere but next to the archive.
|
||||
:::
|
||||
|
||||
## What a backup holds
|
||||
|
||||
Every collection in the database, the index definitions each one needs to be
|
||||
useful again, and a SHA-256 **fingerprint** of `KEY_ENCRYPTION_KEY` — never the
|
||||
key itself. The fingerprint is what lets a later `restore` or `verify` tell you
|
||||
that the key you are holding is the wrong one, before it writes a database
|
||||
nobody can read.
|
||||
|
||||
## What it does not hold
|
||||
|
||||
- **Redis sessions.** Everyone signs in again after a restore, which is already
|
||||
true whenever Redis itself restarts.
|
||||
- **The vulnerability database.** It is re-pulled automatically on next boot.
|
||||
- **Agent state on managed servers.** Nothing needs re-enrolling: agents
|
||||
reconnect on their own, because `servers.agent_token_hash` — the thing an
|
||||
agent authenticates with — is itself in the backup.
|
||||
|
||||
## Taking a backup
|
||||
|
||||
The loose binary:
|
||||
|
||||
```bash
|
||||
export MONGO_URI=mongodb://localhost:27017
|
||||
export MONGO_DB=vantage
|
||||
export KEY_ENCRYPTION_KEY=<your 64-char hex key>
|
||||
vantagectl backup --out /backups
|
||||
```
|
||||
|
||||
The container:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-e MONGO_URI=mongodb://mongo:27017 \
|
||||
-e MONGO_DB=vantage \
|
||||
-e KEY_ENCRYPTION_KEY=<your 64-char hex key> \
|
||||
-v /backups:/backups \
|
||||
gitea.hostxtra.co.uk/mrhid6/vantage/vantagectl:latest backup --out /backups
|
||||
```
|
||||
|
||||
Kubernetes, as a scheduled `CronJob` the Helm chart can render for you:
|
||||
|
||||
```yaml
|
||||
backup:
|
||||
enabled: true
|
||||
schedule: "0 2 * * *"
|
||||
image: "gitea.hostxtra.co.uk/mrhid6/vantage/vantagectl:latest"
|
||||
pvcName: "vantage-backups"
|
||||
```
|
||||
|
||||
`backup.enabled` defaults to `false`, and the chart refuses to render if it is
|
||||
turned on without both `backup.image` and `backup.pvcName` — a backup needs a
|
||||
known image and somewhere durable to land, and guessing at either is worse than
|
||||
refusing to start. `backup.exclude` names collections to leave out (recorded in
|
||||
the archive's manifest, so an archive never claims to be complete when it is
|
||||
not), and `backup.successfulJobsHistoryLimit` / `backup.failedJobsHistoryLimit`
|
||||
/ `backup.resources` behave exactly as they do on any other `CronJob`.
|
||||
|
||||
`backup` refuses to run without `KEY_ENCRYPTION_KEY` set in the environment,
|
||||
unless you pass `--allow-no-key` — for a deployment that genuinely stores no
|
||||
encrypted data. Everywhere else, treat the refusal as the tool doing its job.
|
||||
|
||||
## Where to put the archive
|
||||
|
||||
`--out -` streams the tarball to stdout instead of writing a file, and every
|
||||
line of progress output goes to stderr — so piping the archive into something
|
||||
else is always safe, nothing progress-related lands in the stream.
|
||||
|
||||
Into `restic`:
|
||||
|
||||
```bash
|
||||
vantagectl backup --out - | restic backup --stdin --stdin-filename vantage.tar.gz
|
||||
```
|
||||
|
||||
Into S3:
|
||||
|
||||
```bash
|
||||
vantagectl backup --out - | aws s3 cp - s3://my-backups/vantage-$(date +%F).tar.gz
|
||||
```
|
||||
|
||||
An archive is as sensitive as a raw database dump — it carries every SSH key
|
||||
assignment, every secret group, every session-adjacent setting, in a form the
|
||||
right `KEY_ENCRYPTION_KEY` can decrypt. Whatever you pipe it into should
|
||||
encrypt it at rest; `vantagectl` itself does not.
|
||||
|
||||
## Checking a backup is real
|
||||
|
||||
```bash
|
||||
vantagectl verify /backups/vantage-backup-vantage-20260907T020000Z.tar.gz \
|
||||
--mongo-uri mongodb://localhost:27017 --db vantage
|
||||
```
|
||||
|
||||
Each line of output answers a different question:
|
||||
|
||||
- **`Archive`** — every member's checksum still matches; the tarball has not
|
||||
been truncated or corrupted.
|
||||
- **`Archive key`** / **`Your key`** — the fingerprint stored in the archive
|
||||
next to the fingerprint of the `KEY_ENCRYPTION_KEY` in your environment.
|
||||
- **`Key match`** — whether those two fingerprints agree.
|
||||
- **`Live probe`** — given `--mongo-uri`, `verify` goes one step further and
|
||||
decrypts a real ciphertext value from that database with the key you hold.
|
||||
A fingerprint match proves two archives agree about a key; only the probe
|
||||
proves the key in your hand actually reads the data.
|
||||
|
||||
`verify` exits non-zero the moment anything above is wrong, which is what makes
|
||||
it worth putting on a schedule — a backup job that "succeeded" last night is
|
||||
not the same claim as a backup that will actually restore.
|
||||
|
||||
## Looking inside an archive
|
||||
|
||||
`inspect` prints an archive's manifest and touches no database at all — no
|
||||
`--mongo-uri`, no key. It is what to run against an archive of unknown origin,
|
||||
before deciding whether it is the one you want:
|
||||
|
||||
```bash
|
||||
vantagectl inspect /backups/vantage-backup-vantage-20260907T020000Z.tar.gz
|
||||
```
|
||||
|
||||
It reports when the archive was taken and on which host, the Vantage and
|
||||
MongoDB versions behind it, the database it came from, the key fingerprint (or
|
||||
that it carries none), every collection with its document count and size, and
|
||||
anything `--exclude` left out. Opening the archive verifies every member's
|
||||
checksum on the way, so a corrupt archive fails here too.
|
||||
|
||||
Reach for `verify` instead when the question is whether the key you hold opens
|
||||
it; reach for `inspect` when the question is what it is.
|
||||
|
||||
## Restoring
|
||||
|
||||
`restore` expects the target database to be empty. Pointed at one that already
|
||||
holds data, it refuses outright: there are no merge semantics, because merging
|
||||
two control planes reconciles nothing and upserting old data over new would
|
||||
resurrect revoked keys and deleted users.
|
||||
|
||||
```bash
|
||||
vantagectl restore /backups/vantage-backup-vantage-20260907T020000Z.tar.gz \
|
||||
--mongo-uri mongodb://localhost:27017 --db vantage_restore
|
||||
```
|
||||
|
||||
To overwrite a database that is not empty, add `--force`, which drops each
|
||||
collection named in the archive before loading it. `--force` always needs a
|
||||
second assurance, in one of two forms:
|
||||
|
||||
- `--confirm-db NAME`, naming the target exactly. A mismatch is refused. This
|
||||
works everywhere — on a terminal and in a Kubernetes Job, a CI step or a cron
|
||||
entry alike — and is the form to script.
|
||||
- Nothing, on a terminal: `--force` alone prompts you to type the target
|
||||
database's name back, a deliberate pause before something destructive.
|
||||
|
||||
Without a terminal and without `--confirm-db`, `--force` is refused: there is
|
||||
nobody there to prompt. Naming the database in the command itself means a
|
||||
copy-pasted invocation carries its intended target with it and cannot destroy a
|
||||
different one by accident.
|
||||
|
||||
`--force` drops only the collections the archive carries. Anything else already
|
||||
in the target is left alone and named in a warning, so an archive taken with
|
||||
`--exclude workflow_log_lines` restored over a live database tells you the old
|
||||
log lines are still there, joined to freshly restored runs. Dropping them
|
||||
instead would delete data you never asked to delete.
|
||||
|
||||
`restore` also refuses when the archive's key fingerprint does not match the
|
||||
`KEY_ENCRYPTION_KEY` in your environment — see "When the key is wrong" below.
|
||||
|
||||
## The restore drill
|
||||
|
||||
An untested backup is a hypothesis, not a backup. Rehearse the whole path,
|
||||
monthly:
|
||||
|
||||
1. Restore last night's archive into a scratch database:
|
||||
```bash
|
||||
vantagectl restore /backups/vantage-backup-vantage-<date>.tar.gz \
|
||||
--mongo-uri mongodb://localhost:27017 --db vantage_drill
|
||||
```
|
||||
2. Run `verify` against the result to confirm the data that landed is actually
|
||||
readable with your current key:
|
||||
```bash
|
||||
vantagectl verify /backups/vantage-backup-vantage-<date>.tar.gz \
|
||||
--mongo-uri mongodb://localhost:27017 --db vantage_drill
|
||||
```
|
||||
3. Drop the scratch database. It served its purpose.
|
||||
|
||||
The failure this catches is not "the archive is corrupt" — `verify` alone
|
||||
catches that. It is "the archive is fine but nobody can actually stand a
|
||||
control plane back up from it," which only a real restore proves.
|
||||
|
||||
## When the key is wrong
|
||||
|
||||
If `restore` finds the archive's key fingerprint does not match the
|
||||
`KEY_ENCRYPTION_KEY` you are running with, it stops. Passing
|
||||
`--ignore-key-mismatch` proceeds anyway, but says plainly which collections
|
||||
will come back with ciphertext nobody can read:
|
||||
|
||||
- `keys` — SSH private keys and passphrases
|
||||
- `secrets` — the vault
|
||||
- `auth_providers` — OIDC/SSO client secrets
|
||||
- `console_sessions` — RDP/VNC credentials
|
||||
|
||||
There is no way to recover that ciphertext afterwards. If you have reached
|
||||
this point, the right key was lost along with the chance to read those rows —
|
||||
the fix is to re-enter each of them by hand (re-upload SSH keys, re-save vault
|
||||
secrets, reconfigure SSO), not to keep searching for a way to decrypt what is
|
||||
already in the database.
|
||||
@@ -24,7 +24,23 @@ values is permanently unreadable.
|
||||
Store the key somewhere other than the server it protects.
|
||||
:::
|
||||
|
||||
## Backing up MongoDB
|
||||
:::info Use `vantagectl`
|
||||
[**Backup and restore**](./backup-and-restore.md) is the supported way to take
|
||||
and restore a backup. It writes an archive that carries a fingerprint of
|
||||
`KEY_ENCRYPTION_KEY` — never the key — so a restore taken with the wrong key
|
||||
**refuses** rather than silently producing a database whose secrets nobody can
|
||||
read. It also checksums every archive member before writing anything, and
|
||||
refuses to restore into a database that already holds data. A plain
|
||||
`mongodump` does none of that: it records nothing about which key the data was
|
||||
encrypted under, so a restore from one succeeds even when the key is wrong and
|
||||
the failure only shows up later, as unreadable secrets.
|
||||
|
||||
The rest of this page, past the table above, describes the `mongodump` /
|
||||
`mongorestore` fallback for an operator who does not have `vantagectl`
|
||||
available. Prefer the linked page.
|
||||
:::
|
||||
|
||||
## Backing up MongoDB (fallback, without `vantagectl`)
|
||||
|
||||
With the bundled Mongo container:
|
||||
|
||||
@@ -33,6 +49,13 @@ docker compose exec -T mongo mongodump --archive --gzip --db vantage \
|
||||
> /backups/vantage-$(date +%F).archive.gz
|
||||
```
|
||||
|
||||
:::warning
|
||||
This archive records nothing about which `KEY_ENCRYPTION_KEY` it was taken
|
||||
under. Restoring it with the wrong key produces a database that looks intact
|
||||
and is not — every secret in it is silently unreadable until something tries
|
||||
to decrypt one.
|
||||
:::
|
||||
|
||||
Restoring:
|
||||
|
||||
```bash
|
||||
@@ -69,12 +92,13 @@ What it does **not** do is reconcile the world. After a restore:
|
||||
|
||||
| What | When |
|
||||
| ----------------- | ----------------------------------------------------- |
|
||||
| MongoDB dump | Nightly, retained per your policy |
|
||||
| Backup | Nightly, retained per your policy |
|
||||
| Environment file | On change, held in a password manager or secret store |
|
||||
| Restore rehearsal | Occasionally, into a throwaway host |
|
||||
|
||||
Rehearse a restore now and again. It is the step most often skipped, and the one
|
||||
that finds the problems.
|
||||
that finds the problems. See [Backup and restore](./backup-and-restore.md) for
|
||||
the drill, and for `verify`, which checks a backup is real without a restore.
|
||||
|
||||
## Cloud instances
|
||||
|
||||
|
||||
+1
-1
@@ -49,7 +49,7 @@ const sidebars: SidebarsConfig = {
|
||||
{
|
||||
type: "category",
|
||||
label: "Operations",
|
||||
items: ["operations/upgrading", "operations/backups", "operations/agent-updates"],
|
||||
items: ["operations/upgrading", "operations/backups", "operations/backup-and-restore", "operations/agent-updates"],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user