vantage-docs/docs/reference/troubleshooting.md

id, title, sidebar_label

id	title	sidebar_label
troubleshooting	Troubleshooting	Troubleshooting

Symptoms, in the order people hit them.

The server will not start

Exits immediately on boot. Almost always a missing GRPC_HOST. The server refuses to start rather than guess a value that would break every agent later.

Fails while preparing the database. Vantage stops rather than run without the safeguards it sets up at startup. Check the MongoDB user's permissions and whether an old, conflicting index is already there.

Starts, but every secret operation errors. KEY_ENCRYPTION_KEY is missing or is not 64 hex characters.

Nobody can sign in

Every request 404s and the interface loads fine. Your reverse proxy sends the whole hostname to web:3000. /api, /auth, /public, /install* and /update* belong to server:8080 and the web app forwards nothing — see Ports and networking. The tell is curl -si https://<your-host>/auth/bootstrap-status returning HTML with x-powered-by: Next.js instead of JSON.

/setup appears when users already exist. The server is pointed at a different database than you think. Check the database name in MONGO_URI, which is taken from the end of the URI.

Sessions do not stick. Redis is unreachable, or the cookie is being dropped because the site is served over plain HTTP.

"Wrong organisation" style rejections. Vantage compares the address you browsed to against the instance your session belongs to. On a custom domain, check APP_ROOT_LABEL.

OIDC redirects and then fails. The callback URL registered with the provider must match exactly. Keep one local owner account so a broken provider is not a lockout.

A server never becomes active

Work through it in this order:

1. Is the agent running? systemctl status vantage-agent.
2. What does it say? journalctl -u vantage-agent -f.
3. Can that machine reach the endpoint? Test GRPC_HOST from the machine, not from the control plane host.
4. Was the token already used, or older than an hour? Generate a fresh install command rather than reusing the old one.

Symptom	Cause
Registers, then goes offline within minutes	The short registration call gets through but the long-lived connection is dropped, usually by a proxy or an idle timeout
Stays pending forever	Registration never happened. Token spent, or the endpoint unreachable
Flaps between active and offline	Intermittent path, or a poll interval longer than the offline threshold

Remember the offline sweep runs every two minutes, so status is never instantaneous.

Keys are not appearing on a machine

• It is a Windows server. Key management is Linux-only, by design.
• The agent is not running. Nothing polls, nothing writes.
• The key is assigned but revoked. Revocation is soft; check the assignment state rather than the key.
• Someone edited authorized_keys by hand. The agent rewrites the file to match the desired set; hand-added keys disappear on the next change.

A workflow run fails or hangs

• Hangs at dispatch. The target's command stream is not connected; the server may be offline.
• Fails immediately with an interpreter error. A bash step on a Windows target, or PowerShell on Linux.
• A value does not reach the next step. Values pass through the file at $WORKFLOW_ENV, one KEY=value per line. Listing an output does not pass it on by itself.
• A secret is empty. The group is not attached to that step, or the key name differs from the variable you are reading.
• Logs stop mid-run. A reverse proxy read timeout cut the stream. The run itself continues; reload the page.

The console will not connect

Symptom	Cause
Connects, then closes at once	guacd unreachable. Check GUACD_ADDR and that the container is running
SSH rejects the key	The stored key has no private half, or is not on the target
RDP fails on retry	Credentials are single-use and consumed at tunnel open. Enter them again
Hangs, then disconnects	The agent could not reach the service on that machine, or setting up the session timed out. The audit log records which
Fails only in production	The reverse proxy is not forwarding WebSocket upgrade headers

Monitors report down when the service is up

• The check is running from the control plane and the endpoint is only reachable internally. Switch the runner to an agent on a machine that can see it.
• The keyword no longer appears in the response body.
• Retries are 0, so a single dropped packet flips the state.

The check gets a 403, 429 or a CAPTCHA page

The endpoint is fine and answers a browser normally, but the monitor records a status it never sees by hand. Something between Vantage and the service is blocking automated traffic: a CDN, a WAF, a bot-protection product, a reverse proxy rule, or a rate limiter. The response usually comes from that layer and never reaches the origin at all, so nothing appears in the application's own logs.

Two things make it hard to spot. The check runs from the control plane's or the agent's address rather than yours, and those addresses are often datacenter ranges that bot protection scores badly. And a browser test proves nothing, because a browser is exactly what the blocking layer is willing to serve.

Every HTTP check Vantage makes identifies itself:

User-Agent: Vantage-Monitor/1.0 (+https://vantage.hostxtra.co.uk)

That string is the hook to allow the check through. In whichever product is doing the blocking, add a rule that skips bot protection, managed rules and rate limiting for requests carrying it — Cloudflare, AWS WAF, Azure Front Door, Akamai, Fastly, Imperva, Sucuri, ModSecurity, nginx and HAProxy all match on a request header. The shape of the rule is the same everywhere:

If the host is yours, the path is the one being monitored, and the User-Agent contains Vantage-Monitor, then skip the protection.

Three details are worth getting right:

• Match on contains, not equality. The version in the string moves. An exact match breaks silently on an upgrade, and the symptom is a monitor that goes down on deploy day.
• Keep the rule narrow. Scope it to the specific host and path being monitored. A User-Agent is not a secret — anyone can send it — so a rule that skips protection site-wide on that string alone is a bypass you have published.
• Allow the source address too, where you can. Combining the User-Agent with the checker's IP is stronger than either alone. Find the address in your blocking product's own event log; it is whichever client IP was blocked on the monitored path.

If the endpoint genuinely needs authentication rather than an exception, monitor a purpose-built health path that does not, and leave the protected paths protected.

Notifications are not arriving

Use the channel Test button. It goes through the real delivery path, so a test that arrives proves credentials, network path and destination.

If the test fails: a webhook returning 300 or above counts as a failure, SMTP needs host, port, from and to, and Telegram needs both token and chat_id.

Licence problems

Symptom	Cause
"Managed by Vantage HQ" when pasting	It is a cloud instance, which is licensed for you. There is nothing to paste
Licence rejected as not matching	It was issued to a different instance ID. Relink it in Vantage HQ
Instance degraded despite a valid-looking licence	It expired more than a few days ago. Pasting a new one still works, which is how you recover
Cannot enrol another server	The server allowance is reached. Raise it in HQ or remove one

A status page 404s or shows no data

404, and it should be published. Check the Published toggle on the page's editor — an unpublished page answers not found for everyone, including you, with no session exemption. Also check the host: the public URL is <your-instance>.vantage.<yourdomain>/status/<page-id>, the same per-instance subdomain everything else in Vantage uses. A wrong or missing subdomain resolves to no instance at all, which is also a 404.

Third possibility: /public is not routed to the server. Check with curl -si https://<your-instance>.vantage.<yourdomain>/public/status/<page-id> — JSON is correct, HTML carrying x-powered-by: Next.js means the proxy sent that prefix to the web app.

Loads, but shows an explanation instead of components. This is not a fault — it is the page working as designed. It means either the licence has lapsed (a self-hosted instance past its grace period, or a cloud instance between billing events) or the current tier does not include the Status pages feature. Fix the licence or the plan and the same link starts serving data again with no republish needed.

One component reads Unknown. The monitor behind it was deleted while still listed on the page. Nothing is checking it any more, so the page says so rather than showing a stale up or down. Remove the component from the page, or point it at a replacement monitor, in the page's editor.

HQ portal problems

The portal is a hosted service, so problems with it are ours to fix rather than yours to configure. If a page fails to load, an action reports an error, or a plan or price looks wrong after a change, contact support with your instance UUID and roughly when it happened.

Gathering information before asking for help

docker compose ps
docker compose logs --tail=200 server
journalctl -u vantage-agent --no-pager -n 200   # on the affected machine

Include your instance ID from the Licence page, which is the reference support works from.