--- id: troubleshooting title: Troubleshooting sidebar_label: Troubleshooting --- Symptoms, in the order people hit them. ## The server will not start **Exits immediately on boot.** Almost always a missing `GRPC_HOST`. The server refuses to start rather than guess a value that would break every agent later. **Fails while preparing the database.** Vantage stops rather than run without the safeguards it sets up at startup. Check the MongoDB user's permissions and whether an old, conflicting index is already there. **Starts, but every secret operation errors.** `KEY_ENCRYPTION_KEY` is missing or is not 64 hex characters. ## Nobody can sign in **Every request 404s and the interface loads fine.** Your reverse proxy sends the whole hostname to `web:3000`. `/api`, `/auth`, `/public`, `/install*` and `/update*` belong to `server:8080` and the web app forwards nothing - see [Ports and networking](./ports-and-networking.md#reverse-proxy-notes). The tell is `curl -si https:///auth/bootstrap-status` returning HTML with `x-powered-by: Next.js` instead of JSON. **`/setup` appears when users already exist.** The server is pointed at a different database than you think. Check the database name in `MONGO_URI`, which is taken from the end of the URI. **Sessions do not stick.** Redis is unreachable, or the cookie is being dropped because the site is served over plain HTTP. **"Wrong organisation" style rejections.** Vantage compares the address you browsed to against the instance your session belongs to. On a custom domain, check `APP_ROOT_LABEL`. **OIDC redirects and then fails.** The callback URL registered with the provider must match exactly. Keep one local owner account so a broken provider is not a lockout. ## A server never becomes active Work through it in this order: 1. Is the agent running? `systemctl status vantage-agent`. 2. What does it say? `journalctl -u vantage-agent -f`. 3. Can that machine reach the endpoint? Test `GRPC_HOST` **from the machine**, not from the control plane host. 4. Was the token already used, or older than an hour? Generate a fresh install command rather than reusing the old one. | Symptom | Cause | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Registers, then goes `offline` within minutes | The short registration call gets through but the long-lived connection is dropped, usually by a proxy or an idle timeout | | Stays `pending` forever | Registration never happened. Token spent, or the endpoint unreachable | | Flaps between `active` and `offline` | Intermittent path, or a poll interval longer than the offline threshold | Remember the offline sweep runs every two minutes, so status is never instantaneous. ## Keys are not appearing on a machine - **It is a Windows server.** Key management is Linux-only, by design. - **The agent is not running.** Nothing polls, nothing writes. - **The key is assigned but revoked.** Revocation is soft; check the assignment state rather than the key. - **Someone edited `authorized_keys` by hand.** The agent rewrites the file to match the desired set; hand-added keys disappear on the next change. ## A workflow run fails or hangs - **Hangs at dispatch.** The target's command stream is not connected; the server may be `offline`. - **Fails immediately with an interpreter error.** A bash step on a Windows target, or PowerShell on Linux. - **A value does not reach the next step.** Values pass through the file at `$WORKFLOW_ENV`, one `KEY=value` per line. Listing an output does not pass it on by itself. - **A secret is empty.** The group is not attached to that step, or the key name differs from the variable you are reading. - **Logs stop mid-run.** A reverse proxy read timeout cut the stream. The run itself continues; reload the page. ## The console will not connect | Symptom | Cause | | ----------------------------- | --------------------------------------------------------------------------------------------------------------- | | Connects, then closes at once | guacd unreachable. Check `GUACD_ADDR` and that the container is running | | SSH rejects the key | The stored key has no private half, or is not on the target | | RDP fails on retry | Credentials are single-use and consumed at tunnel open. Enter them again | | Hangs, then disconnects | The agent could not reach the service on that machine, or setting up the session timed out. The audit log records which | | Fails only in production | The reverse proxy is not forwarding WebSocket upgrade headers | ## Monitors report down when the service is up - The check is running from the control plane and the endpoint is only reachable internally. Switch the runner to an agent on a machine that can see it. - The keyword no longer appears in the response body. - Retries are `0`, so a single dropped packet flips the state. ### The check gets a 403, 429 or a CAPTCHA page The endpoint is fine and answers a browser normally, but the monitor records a status it never sees by hand. Something between Vantage and the service is blocking automated traffic: a CDN, a WAF, a bot-protection product, a reverse proxy rule, or a rate limiter. The response usually comes from that layer and never reaches the origin at all, so nothing appears in the application's own logs. Two things make it hard to spot. The check runs from the control plane's or the agent's address rather than yours, and those addresses are often datacenter ranges that bot protection scores badly. And a browser test proves nothing, because a browser is exactly what the blocking layer is willing to serve. Every HTTP check Vantage makes identifies itself: ``` User-Agent: Vantage-Monitor/1.0 (+https://vantage.hostxtra.co.uk) ``` That string is the hook to allow the check through. In whichever product is doing the blocking, add a rule that skips bot protection, managed rules and rate limiting for requests carrying it - Cloudflare, AWS WAF, Azure Front Door, Akamai, Fastly, Imperva, Sucuri, ModSecurity, nginx and HAProxy all match on a request header. The shape of the rule is the same everywhere: > If the host is *yours*, the path is *the one being monitored*, and the > User-Agent contains `Vantage-Monitor`, then skip the protection. Three details are worth getting right: - **Match on `contains`, not equality.** The version in the string moves. An exact match breaks silently on an upgrade, and the symptom is a monitor that goes down on deploy day. - **Keep the rule narrow.** Scope it to the specific host and path being monitored. A User-Agent is not a secret - anyone can send it - so a rule that skips protection site-wide on that string alone is a bypass you have published. - **Allow the source address too, where you can.** Combining the User-Agent with the checker's IP is stronger than either alone. Find the address in your blocking product's own event log; it is whichever client IP was blocked on the monitored path. If the endpoint genuinely needs authentication rather than an exception, monitor a purpose-built health path that does not, and leave the protected paths protected. ## Notifications are not arriving Use the channel **Test** button. It goes through the real delivery path, so a test that arrives proves credentials, network path and destination. If the test fails: a webhook returning 300 or above counts as a failure, SMTP needs `host`, `port`, `from` and `to`, and Telegram needs both `token` and `chat_id`. ## Licence problems | Symptom | Cause | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | "Managed by Vantage HQ" when pasting | It is a cloud instance, which is licensed for you. There is nothing to paste | | Licence rejected as not matching | It was issued to a different instance ID. Relink it in Vantage HQ | | Instance degraded despite a valid-looking licence | It expired more than a few days ago. Pasting a new one still works, which is how you recover | | Cannot enrol another server | The server allowance is reached. Raise it in HQ or remove one | ## A status page 404s or shows no data **404, and it should be published.** Check the **Published** toggle on the page's editor - an unpublished page answers *not found* for everyone, including you, with no session exemption. Also check the host: the public URL is `.vantage./status/`, the same per-instance subdomain everything else in Vantage uses. A wrong or missing subdomain resolves to no instance at all, which is also a 404. Third possibility: `/public` is not routed to the server. Check with `curl -si https://.vantage./public/status/` - JSON is correct, HTML carrying `x-powered-by: Next.js` means the proxy sent that prefix to the web app. **Loads, but shows an explanation instead of components.** This is not a fault - it is the page working as designed. It means either the licence has lapsed (a self-hosted instance past its grace period, or a cloud instance between billing events) or the current tier does not include the **Status pages** feature. Fix the licence or the plan and the same link starts serving data again with no republish needed. **One component reads `Unknown`.** The monitor behind it was deleted while still listed on the page. Nothing is checking it any more, so the page says so rather than showing a stale up or down. Remove the component from the page, or point it at a replacement monitor, in the page's editor. ## HQ portal problems The portal is a hosted service, so problems with it are ours to fix rather than yours to configure. If a page fails to load, an action reports an error, or a plan or price looks wrong after a change, contact support with your instance UUID and roughly when it happened. ## Gathering information before asking for help ```bash docker compose ps docker compose logs --tail=200 server journalctl -u vantage-agent --no-pager -n 200 # on the affected machine ``` Include your instance ID from the **Licence** page, which is the reference support works from.