diff --git a/docs/getting-started/self-hosted-install.md b/docs/getting-started/self-hosted-install.md index b339bcf..0d149a5 100644 --- a/docs/getting-started/self-hosted-install.md +++ b/docs/getting-started/self-hosted-install.md @@ -112,6 +112,17 @@ map $http_upgrade $connection_upgrade { '' close; } +# Heartbeat ping URLs carry a credential. Log them with the token replaced; +# the header form (X-Vantage-Token) is never logged by this format. +map $request_uri $vantage_log_uri { + "~^/public/hb/(?!start(?:[/?]|$)|fail(?:[/?]|$))[^/?]+(?.*)$" "/public/hb/***$hb_rest"; + default $request_uri; +} + +log_format vantage '$remote_addr - $remote_user [$time_local] ' + '"$request_method $vantage_log_uri $server_protocol" ' + '$status $body_bytes_sent "$http_referer" "$http_user_agent"'; + upstream vantage_server { server server:8080; keepalive 16; @@ -127,6 +138,8 @@ server { listen [::]:80; server_name _; + access_log /var/log/nginx/access.log vantage; + client_max_body_size 10m; proxy_http_version 1.1; @@ -170,6 +183,12 @@ The `Upgrade`/`Connection` headers and `proxy_buffering off` are not optional: without them the browser console cannot open its WebSocket and live workflow logs arrive in bursts or not at all. +The `map` and `log_format` at the top are optional but recommended. A +[heartbeat monitor](../vantage/heartbeat-monitors.md) ping URL contains a secret +token, and this format writes it to the access log as `***`. If you use your +own proxy instead, mask `/public/hb/` the same way, or have jobs send +the token in the `X-Vantage-Token` header. + ### Terminating TLS in nginx The shipped config speaks plain HTTP, which is right when another proxy or load diff --git a/docs/vantage/heartbeat-monitors.md b/docs/vantage/heartbeat-monitors.md new file mode 100644 index 0000000..7120fd9 --- /dev/null +++ b/docs/vantage/heartbeat-monitors.md @@ -0,0 +1,128 @@ +--- +id: heartbeat-monitors +title: Heartbeat monitors +sidebar_label: Heartbeat monitors +--- + +Backups and cron jobs fail silently. Nothing goes down, the job just doesn't +run. A heartbeat monitor turns that silence into an incident: your job calls a +URL each time it finishes, and Vantage alerts when the call stops arriving. + +It is the opposite of every other [monitor](./monitors.md). Vantage does not +check anything; it waits to be told. + +## Creating one + +1. Go to **Monitors** and choose **New monitor**. +2. Pick **Heartbeat**. +3. Set **Expected every (minutes)**: how often the job runs. +4. Set **Grace (minutes)**: how late a ping may be before it counts as missed. The + default is 5 minutes. +5. Attach [notification channels](./notification-channels.md) and save. + +The next page shows the **ping URL**, with ready-made `curl` commands. + +:::warning Copy the URL now +The URL contains a secret token, and it is shown only this once. Vantage +stores a hash of the token, not the token itself, so it cannot show it to you +again. If you lose it, use **Rotate token** on the monitor page to get a new +one. +::: + +A new heartbeat stays **pending** until its first ping. It will not raise an +incident before the job has ever run, so you can create the monitor before you +deploy the job. + +## Sending pings + +Add a call to the end of your job: + +```bash +# success +curl -fsS -m 10 --retry 3 https://vantage.example.com/public/hb/ +``` + +Two more calls are optional: + +| Call | Meaning | +| ------------------------------ | ------------------------------------------------------------------ | +| `/public/hb/` | The job succeeded. The monitor goes up. | +| `/public/hb//start` | The job started. Vantage measures the time until the next success. | +| `/public/hb//fail` | The job failed. An incident opens straight away. | + +Each accepts `GET` or `POST`, and answers `OK`. + +A typical cron job using all three: + +```bash +URL=https://vantage.example.com/public/hb/ +curl -fsS -m 10 "$URL/start" +if backup-job 2>/tmp/backup.err; then + curl -fsS -m 10 --retry 3 "$URL" +else + tail -c 1024 /tmp/backup.err | curl -fsS -m 10 --data-binary @- "$URL/fail" +fi +``` + +The body of a `/fail` request becomes the incident's cause and appears in the +alert, so sending the end of the job's error output tells whoever gets paged +what went wrong. Only the first 1 KB is kept. + +### Keeping the token out of URLs + +URLs end up in logs: your own proxy's, a load balancer's, an ingress +controller's. If that matters, send the token in a header instead and call the +path without it: + +```bash +curl -fsS -m 10 -X POST -H "X-Vantage-Token: " https://vantage.example.com/public/hb +curl -fsS -m 10 -X POST -H "X-Vantage-Token: " https://vantage.example.com/public/hb/start +curl -fsS -m 10 -X POST -H "X-Vantage-Token: " https://vantage.example.com/public/hb/fail +``` + +If a request carries a token in both places, the one in the URL is used. + +Vantage's own request log, and the access log of the nginx bundled with the +[self-hosted install](../getting-started/self-hosted-install.md#4-the-reverse-proxy), +replace the token in a ping URL with `***`. Nginx's error log and any proxy you +run in front of Vantage (on Kubernetes, the ingress controller) are not masked, +so the header is the safer choice there. + +## When an incident opens + +| Situation | Incident cause | +| ---------------------------------------------------------- | --------------------------------- | +| No ping within the expected period plus grace | `no ping since