docs(monitors): heartbeat monitors and metric alerts guides; mask ping tokens in bundled nginx log
Deploy / deploy (push) Successful in 4m3s
Deploy / deploy (push) Successful in 4m3s
This commit is contained in:
@@ -112,6 +112,17 @@ map $http_upgrade $connection_upgrade {
|
||||
'' close;
|
||||
}
|
||||
|
||||
# Heartbeat ping URLs carry a credential. Log them with the token replaced;
|
||||
# the header form (X-Vantage-Token) is never logged by this format.
|
||||
map $request_uri $vantage_log_uri {
|
||||
"~^/public/hb/(?!start(?:[/?]|$)|fail(?:[/?]|$))[^/?]+(?<hb_rest>.*)$" "/public/hb/***$hb_rest";
|
||||
default $request_uri;
|
||||
}
|
||||
|
||||
log_format vantage '$remote_addr - $remote_user [$time_local] '
|
||||
'"$request_method $vantage_log_uri $server_protocol" '
|
||||
'$status $body_bytes_sent "$http_referer" "$http_user_agent"';
|
||||
|
||||
upstream vantage_server {
|
||||
server server:8080;
|
||||
keepalive 16;
|
||||
@@ -127,6 +138,8 @@ server {
|
||||
listen [::]:80;
|
||||
server_name _;
|
||||
|
||||
access_log /var/log/nginx/access.log vantage;
|
||||
|
||||
client_max_body_size 10m;
|
||||
|
||||
proxy_http_version 1.1;
|
||||
@@ -170,6 +183,12 @@ The `Upgrade`/`Connection` headers and `proxy_buffering off` are not optional:
|
||||
without them the browser console cannot open its WebSocket and live workflow
|
||||
logs arrive in bursts or not at all.
|
||||
|
||||
The `map` and `log_format` at the top are optional but recommended. A
|
||||
[heartbeat monitor](../vantage/heartbeat-monitors.md) ping URL contains a secret
|
||||
token, and this format writes it to the access log as `***`. If you use your
|
||||
own proxy instead, mask `/public/hb/<token>` the same way, or have jobs send
|
||||
the token in the `X-Vantage-Token` header.
|
||||
|
||||
### Terminating TLS in nginx
|
||||
|
||||
The shipped config speaks plain HTTP, which is right when another proxy or load
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
id: heartbeat-monitors
|
||||
title: Heartbeat monitors
|
||||
sidebar_label: Heartbeat monitors
|
||||
---
|
||||
|
||||
Backups and cron jobs fail silently. Nothing goes down, the job just doesn't
|
||||
run. A heartbeat monitor turns that silence into an incident: your job calls a
|
||||
URL each time it finishes, and Vantage alerts when the call stops arriving.
|
||||
|
||||
It is the opposite of every other [monitor](./monitors.md). Vantage does not
|
||||
check anything; it waits to be told.
|
||||
|
||||
## Creating one
|
||||
|
||||
1. Go to **Monitors** and choose **New monitor**.
|
||||
2. Pick **Heartbeat**.
|
||||
3. Set **Expected every (minutes)**: how often the job runs.
|
||||
4. Set **Grace (minutes)**: how late a ping may be before it counts as missed. The
|
||||
default is 5 minutes.
|
||||
5. Attach [notification channels](./notification-channels.md) and save.
|
||||
|
||||
The next page shows the **ping URL**, with ready-made `curl` commands.
|
||||
|
||||
:::warning Copy the URL now
|
||||
The URL contains a secret token, and it is shown only this once. Vantage
|
||||
stores a hash of the token, not the token itself, so it cannot show it to you
|
||||
again. If you lose it, use **Rotate token** on the monitor page to get a new
|
||||
one.
|
||||
:::
|
||||
|
||||
A new heartbeat stays **pending** until its first ping. It will not raise an
|
||||
incident before the job has ever run, so you can create the monitor before you
|
||||
deploy the job.
|
||||
|
||||
## Sending pings
|
||||
|
||||
Add a call to the end of your job:
|
||||
|
||||
```bash
|
||||
# success
|
||||
curl -fsS -m 10 --retry 3 https://vantage.example.com/public/hb/<token>
|
||||
```
|
||||
|
||||
Two more calls are optional:
|
||||
|
||||
| Call | Meaning |
|
||||
| ------------------------------ | ------------------------------------------------------------------ |
|
||||
| `/public/hb/<token>` | The job succeeded. The monitor goes up. |
|
||||
| `/public/hb/<token>/start` | The job started. Vantage measures the time until the next success. |
|
||||
| `/public/hb/<token>/fail` | The job failed. An incident opens straight away. |
|
||||
|
||||
Each accepts `GET` or `POST`, and answers `OK`.
|
||||
|
||||
A typical cron job using all three:
|
||||
|
||||
```bash
|
||||
URL=https://vantage.example.com/public/hb/<token>
|
||||
curl -fsS -m 10 "$URL/start"
|
||||
if backup-job 2>/tmp/backup.err; then
|
||||
curl -fsS -m 10 --retry 3 "$URL"
|
||||
else
|
||||
tail -c 1024 /tmp/backup.err | curl -fsS -m 10 --data-binary @- "$URL/fail"
|
||||
fi
|
||||
```
|
||||
|
||||
The body of a `/fail` request becomes the incident's cause and appears in the
|
||||
alert, so sending the end of the job's error output tells whoever gets paged
|
||||
what went wrong. Only the first 1 KB is kept.
|
||||
|
||||
### Keeping the token out of URLs
|
||||
|
||||
URLs end up in logs: your own proxy's, a load balancer's, an ingress
|
||||
controller's. If that matters, send the token in a header instead and call the
|
||||
path without it:
|
||||
|
||||
```bash
|
||||
curl -fsS -m 10 -X POST -H "X-Vantage-Token: <token>" https://vantage.example.com/public/hb
|
||||
curl -fsS -m 10 -X POST -H "X-Vantage-Token: <token>" https://vantage.example.com/public/hb/start
|
||||
curl -fsS -m 10 -X POST -H "X-Vantage-Token: <token>" https://vantage.example.com/public/hb/fail
|
||||
```
|
||||
|
||||
If a request carries a token in both places, the one in the URL is used.
|
||||
|
||||
Vantage's own request log, and the access log of the nginx bundled with the
|
||||
[self-hosted install](../getting-started/self-hosted-install.md#4-the-reverse-proxy),
|
||||
replace the token in a ping URL with `***`. Nginx's error log and any proxy you
|
||||
run in front of Vantage (on Kubernetes, the ingress controller) are not masked,
|
||||
so the header is the safer choice there.
|
||||
|
||||
## When an incident opens
|
||||
|
||||
| Situation | Incident cause |
|
||||
| ---------------------------------------------------------- | --------------------------------- |
|
||||
| No ping within the expected period plus grace | `no ping since <time>` |
|
||||
| A `/start` was not followed by a success within the grace | `started <time>, never finished` |
|
||||
| The job called `/fail` | `reported failure: <body>` |
|
||||
|
||||
Overdue heartbeats are checked every 30 seconds, so an alert can arrive up to
|
||||
half a minute after the deadline. Only a success ping closes the incident.
|
||||
|
||||
The grace time does two jobs: it is how late a regular ping may be, and how
|
||||
long a run may take after `/start`. Set it longer than your slowest normal run.
|
||||
|
||||
## Duration
|
||||
|
||||
When a job calls `/start` and then succeeds, the time between the two is
|
||||
recorded and shown as **Duration** on the monitor page, in place of the
|
||||
response time other monitors show. A backup that used to take 4 minutes and now
|
||||
takes 40 is worth knowing about before it starts overrunning.
|
||||
|
||||
## Limits and behaviour worth knowing
|
||||
|
||||
- **One request per second per token and call type.** A `/start` and the
|
||||
success ping in the same second are both accepted; two success pings in the
|
||||
same second are not, and the second gets `429`.
|
||||
- **Unknown token, disabled monitor:** both answer `404`, with no hint which.
|
||||
- **Disabling and re-enabling** a heartbeat resets it to pending, so a monitor
|
||||
switched off for a month does not page the moment it is switched back on.
|
||||
- **Rotating the token** stops the old URL working immediately, so update the
|
||||
job straight after.
|
||||
- A heartbeat is never run by an agent and has no interval or retries.
|
||||
|
||||
## API tokens and agents
|
||||
|
||||
Creating a heartbeat through the API or the [MCP agent](./mcp.md) returns the
|
||||
token once in the create response, as `heartbeat_token`. A monitor created
|
||||
through MCP is saved disabled, like any other.
|
||||
+2
-2
@@ -100,7 +100,7 @@ surface is a presentation layer over existing authority, not a new one.
|
||||
| `get_server` | Get one server's OS, online state and tags. | `mcp:read`, `servers:read` |
|
||||
| `list_monitors` | List monitors and their current state. | `mcp:read`, `monitors:read` |
|
||||
| `get_monitor_status` | Get one monitor's up/down/pending state, last check and last error. | `mcp:read`, `monitors:read` |
|
||||
| `list_incidents` | List monitor incidents (outages), most recent first. | `mcp:read`, `monitors:read` |
|
||||
| `list_incidents` | List monitor incidents (outages), most recent first. A tag-restricted token only sees metric alert incidents for servers inside its tags. | `mcp:read`, `monitors:read` |
|
||||
| `get_monitor_samples` | Get one monitor's recent raw check results. | `mcp:read`, `monitors:read` |
|
||||
| `list_workflows` | List workflows with step count, target count and whether each is scheduled. | `mcp:read`, `workflows:read` |
|
||||
| `get_workflow` | Get one workflow's ordered steps, targets and schedule. | `mcp:read`, `workflows:read` |
|
||||
@@ -119,7 +119,7 @@ surface is a presentation layer over existing authority, not a new one.
|
||||
| `assign_key` | Assign an SSH key to real servers. | `mcp:write`, `keys:write` |
|
||||
| `create_step` | Create a reusable workflow step. | `mcp:write`, `workflows:write` |
|
||||
| `create_workflow` | Create a workflow from existing step IDs. | `mcp:write`, `workflows:write` |
|
||||
| `create_monitor` | Create a monitor, saved disabled. | `mcp:write`, `monitors:write` |
|
||||
| `create_monitor` | Create a monitor, saved disabled. Supports every type, including `heartbeat` (the ping token is returned once) and `metric`. | `mcp:write`, `monitors:write` |
|
||||
|
||||
A known limitation worth calling out on `search_fleet`: its `version_below`
|
||||
argument is not implemented. Passing it gets you an error explaining that
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
id: metric-alerts
|
||||
title: Metric alerts
|
||||
sidebar_label: Metric alerts
|
||||
---
|
||||
|
||||
Agents already report disk, memory, load, reboot status and the state of every
|
||||
container and systemd unit. A metric alert turns those reports into incidents:
|
||||
"disk above 90% on any production server", "a systemd unit failed", "a reboot
|
||||
has been pending for more than 7 days".
|
||||
|
||||
One rule covers as many servers as its tags match. There is nothing to install:
|
||||
alerts use what agents already send, on any agent version.
|
||||
|
||||
## Creating a rule
|
||||
|
||||
1. Go to **Monitors** and choose **New monitor**.
|
||||
2. Pick **Server metric**.
|
||||
3. Under **Servers**, choose which servers by tag. Leave it empty to watch every server.
|
||||
4. Choose the **Metric** (the condition below) and its **Threshold**.
|
||||
5. Set **For (minutes)**: how long the condition must hold before it alerts.
|
||||
6. Attach [notification channels](./notification-channels.md) and save.
|
||||
|
||||
Tags are resolved each time the rule is checked, so a server tagged `env=prod`
|
||||
tomorrow is covered from then on, and a server whose tag is removed stops being
|
||||
watched.
|
||||
|
||||
## Conditions
|
||||
|
||||
| Metric | Alerts when | Threshold |
|
||||
| --------------------- | ---------------------------------------------------------- | --------- |
|
||||
| Disk used | Used space is at or above the threshold | percent |
|
||||
| Disk free below | Free space is at or below the threshold | GB |
|
||||
| Memory used | Used memory is at or above the threshold | percent |
|
||||
| Load per core | 1-minute load divided by CPU cores is at or above it | ratio |
|
||||
| Systemd unit failed | Any unit on the server is `failed` | none |
|
||||
| Container unhealthy | Any container's health check reports `unhealthy` | none |
|
||||
| Reboot pending for | The server has needed a reboot for at least this long | days |
|
||||
| Agent offline for | The agent has not been seen for at least this long | minutes |
|
||||
|
||||
The two disk metrics take an optional **Mount**, such as `/var`. Without one,
|
||||
every mount is checked and the fullest one is reported.
|
||||
|
||||
## How a rule decides
|
||||
|
||||
Rules are checked every 30 seconds, against each matching server separately.
|
||||
|
||||
- When a server first meets the condition, it becomes **pending**.
|
||||
- If the condition is still met after **For (minutes)**, the server goes
|
||||
**down**, an incident opens for that server, and the channels fire.
|
||||
- As soon as the condition clears, the server is **up** and its incident
|
||||
closes. A pending server that clears never alerts.
|
||||
|
||||
"For" is continuous: a server that drops below the threshold for one check
|
||||
starts the count again. Use it to ignore a nightly backup filling a disk for
|
||||
ten minutes, or a load spike during a deploy. Set it to 0 to alert on the first
|
||||
check.
|
||||
|
||||
### Per-server incidents
|
||||
|
||||
Each server has its own state and its own incident. If a rule matches 40
|
||||
servers and 3 are breaching, you get 3 incidents, and each alert names its
|
||||
server:
|
||||
|
||||
```text
|
||||
[Vantage] Disk full (metric) on web-01 is DOWN: /var 94.2% used
|
||||
```
|
||||
|
||||
The monitors list shows the rule as a whole, for example
|
||||
`3 of 40 servers breaching`. The rule's own page lists every matching server
|
||||
with its state, the current value and how long it has been breaching.
|
||||
|
||||
Webhook payloads carry a `server_name` field for metric alerts; see
|
||||
[notification channels](./notification-channels.md#webhook).
|
||||
|
||||
## Stale data
|
||||
|
||||
Agents send metrics every 30 seconds. If a server's last metrics are more than
|
||||
5 minutes old, the rule skips that server and keeps its previous state. A
|
||||
powered-off server does not suddenly clear, or trip, a disk alert.
|
||||
|
||||
To be told about the server itself going quiet, add an **Agent offline for**
|
||||
rule. It is the one metric that does not need fresh reports.
|
||||
|
||||
Some reports arrive less often:
|
||||
|
||||
- **Reboot pending** is refreshed with the agent's full inventory, every 15
|
||||
minutes and at agent start.
|
||||
- **Units and containers** are reported every 60 seconds.
|
||||
|
||||
## When servers leave a rule
|
||||
|
||||
If a server stops matching (its tag changed, or it was deleted) while it has an
|
||||
open incident, the incident is closed quietly, with no recovery message. Nothing
|
||||
recovered; the server simply stopped being watched.
|
||||
|
||||
## Tag-restricted API keys
|
||||
|
||||
An [API key restricted to tags](../reference/api-tokens.md#tag-restrictions) can
|
||||
only create a metric rule whose tags include all of its own. A key restricted
|
||||
to `env=staging` can create a rule for `env=staging role=web`, but not one for
|
||||
every server, and not one for `env=prod`.
|
||||
|
||||
The same key cannot rename, disable, change or delete an existing rule that
|
||||
reaches further than its tags, and it only sees incidents and per-server states
|
||||
for servers inside them.
|
||||
|
||||
## Status pages
|
||||
|
||||
A metric rule can be added to a [status page](./status-pages.md) like any other
|
||||
monitor. Its uptime reflects whether any matching server was down.
|
||||
@@ -4,8 +4,11 @@ title: Monitors
|
||||
sidebar_label: Monitors
|
||||
---
|
||||
|
||||
Monitors check that something is answering. Four types, two places they can run
|
||||
from, and a notification path when they stop being satisfied.
|
||||
Monitors check that something is answering. Four check types, two places they
|
||||
can run from, and a notification path when they stop being satisfied. Two more
|
||||
types watch for things that don't answer a check:
|
||||
[heartbeat monitors](./heartbeat-monitors.md) and
|
||||
[metric alerts](./metric-alerts.md).
|
||||
|
||||
## Types
|
||||
|
||||
@@ -19,6 +22,15 @@ from, and a notification path when they stop being satisfied.
|
||||
An `http` monitor with a keyword is usually the one you want for an application:
|
||||
a 200 that returns an error page still fails the keyword.
|
||||
|
||||
| Type | Watches | Guide |
|
||||
| ----------- | --------------------------------------------------------------------------- | ------------------------------------------- |
|
||||
| `heartbeat` | A job that calls a Vantage URL when it runs; alerts when the call stops | [Heartbeat monitors](./heartbeat-monitors.md) |
|
||||
| `metric` | Disk, memory, load, units, containers, reboots and agents on tagged servers | [Metric alerts](./metric-alerts.md) |
|
||||
|
||||
These two have no runner, interval or retries. The rest of this page is about
|
||||
the four check types; state, incidents, notifications and uptime work the same
|
||||
for all six.
|
||||
|
||||
## Where a check runs
|
||||
|
||||
Every monitor has a **runner**:
|
||||
|
||||
@@ -30,6 +30,9 @@ Posts JSON to a URL you choose.
|
||||
}
|
||||
```
|
||||
|
||||
Alerts from a [metric alert](./metric-alerts.md) add `"server_name"` with the
|
||||
server that breached. Other monitor types leave it out.
|
||||
|
||||
Any response of 300 or above counts as a delivery failure. The request times out
|
||||
after 10 seconds.
|
||||
|
||||
|
||||
@@ -26,6 +26,8 @@ const sidebars: SidebarsConfig = {
|
||||
"vantage/ssh-keys",
|
||||
"vantage/workflows",
|
||||
"vantage/monitors",
|
||||
"vantage/heartbeat-monitors",
|
||||
"vantage/metric-alerts",
|
||||
"vantage/vulnerabilities",
|
||||
"vantage/patching",
|
||||
"vantage/workloads",
|
||||
|
||||
Reference in New Issue
Block a user