vantage-docs/docs/reference/troubleshooting.md

id, title, sidebar_label

id	title	sidebar_label
troubleshooting	Troubleshooting	Troubleshooting

Symptoms, in the order people hit them.

The server will not start

Exits immediately on boot. Almost always a missing GRPC_HOST — the server refuses to start rather than guess a value that would break every agent later.

Fails during index creation. The auth and settings index builders are fatal on failure by design: those unique indexes are what enforce tenant isolation, so starting without them is worse than not starting. Check the MongoDB user's permissions and whether a conflicting index already exists.

Starts, but every secret operation errors. KEY_ENCRYPTION_KEY is missing or is not 64 hex characters.

Nobody can sign in

/setup appears when users already exist. The server is pointed at a different database than you think. Check the database name in MONGO_URI — it comes from the URI path, not a separate variable.

Sessions do not stick. Redis is unreachable, or the cookie is being dropped because the site is served over plain HTTP.

"Wrong organisation" style rejections. The host and session guard is comparing the request host's label against the session's organisation. Check APP_ROOT_LABEL.

OIDC redirects and then fails. The callback URL registered with the provider must match exactly. Keep one local owner account so a broken provider is not a lockout.

A server never becomes active

Work through it in this order:

1. Is the agent running? systemctl status vantage-agent.
2. What does it say? journalctl -u vantage-agent -f.
3. Can that machine reach the endpoint? Test GRPC_HOST from the machine, not from the control plane host.
4. Was the token already used or expired? It is single-use and lives one hour — create a fresh enrolment rather than reusing the old command.

Symptom	Cause
Registers, then goes offline within minutes	Something permits the short Register call but drops the long-lived stream. Usually a proxy or idle-timeout middlebox
Stays pending forever	Registration never happened. Token spent, or the endpoint unreachable
Flaps between active and offline	Intermittent path, or a poll interval longer than the offline threshold

Remember the offline sweep runs every two minutes, so status is never instantaneous.

Keys are not appearing on a machine

• It is a Windows server. Key management is Linux-only, by design.
• The agent is not running. Nothing polls, nothing writes.
• The key is assigned but revoked. Revocation is soft; check the assignment state rather than the key.
• Someone edited authorized_keys by hand. The agent rewrites the file to match the desired set; hand-added keys disappear on the next change.

A workflow run fails or hangs

• Hangs at dispatch. The target's command stream is not connected — the server may be offline.
• Fails immediately with an interpreter error. A bash step on a Windows target, or PowerShell on Linux.
• A value does not reach the next step. Values pass through the file at $WORKFLOW_ENV, one KEY=value per line. Declaring an output does not export it.
• A secret is empty. The group is not in the step's secret_refs, or the key name differs from the environment variable you are reading.
• Logs stop mid-run. A reverse proxy read timeout cut the stream. The run itself continues; reload the page.

The console will not connect

Symptom	Cause
Connects, then closes at once	guacd unreachable. Check GUACD_ADDR and that the container is running
SSH rejects the key	The stored key has no private half, or is not on the target
RDP fails on retry	Credentials are single-use and consumed at tunnel open — enter them again
Hangs at "connecting"	The control plane cannot reach the target on the protocol port. The agent's reachability is irrelevant here
Fails only in production	The reverse proxy is not forwarding WebSocket upgrade headers

Monitors report down when the service is up

• The check is running from the control plane and the endpoint is only reachable internally. Switch the runner to an agent on a machine that can see it.
• The keyword no longer appears in the response body.
• Retries are 0, so a single dropped packet flips the state.

Notifications are not arriving

Use the channel Test button — it goes through the real delivery path, so a test that arrives proves credentials, network path and destination.

If the test fails: a webhook returning 300 or above counts as a failure, SMTP needs host, port, from and to, and Telegram needs both token and chat_id.

Licence problems

Symptom	Cause
409 cloud_managed when pasting	It is a cloud instance. Licences are written by HQ; there is nothing to paste
Licence rejected as not matching	It is bound to a different instance UUID. Relink in HQ
Instance degraded despite a valid-looking licence	It has expired past its grace period. Pasting still works — that endpoint stays available specifically so it can
Cannot enrol another server	The server allowance is reached. Raise it in HQ or remove one

HQ portal problems

A request fails in the browser but works under curl. The browser origin is missing from ADMIN_ORIGIN. This produces no log line at all in admin — the preflight is answered 204 without the allow-origin header, and the browser blocks the real request.

A price or plan looks wrong after an edit. Repository variables are baked into images at build time and editing one pushes no commit, so nothing rebuilds. Trigger the build manually. See CI/CD.

Gathering information before asking for help

docker compose ps
docker compose logs --tail=200 server
journalctl -u vantage-agent --no-pager -n 200   # on the affected machine

Include your instance UUID from Settings → Licence — it is the reference support works from.