docs(spec): metric alerts and heartbeat monitors design
This commit is contained in:
@@ -0,0 +1,184 @@
|
||||
# Metric alerts and heartbeat monitors
|
||||
|
||||
Date: 2026-09-17
|
||||
Source: competitive gap review, "Alerts on agent metrics" (effort M) and "Heartbeat (push) monitors" (effort S).
|
||||
|
||||
## Goal
|
||||
|
||||
1. Alert on data agents already report (disk, memory, load, failed units, unhealthy containers, reboot pending, agent offline), targeted by tag so one rule covers the fleet.
|
||||
2. Alert when a push-based job (backup, cron) stops pinging, reports failure, or starts and never finishes.
|
||||
|
||||
Both reuse the existing monitor model, states, incidents, notification channels, groups and status pages.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Scheduled workflows pinging a heartbeat automatically.
|
||||
- Metric history graphs (separate gap review item).
|
||||
- Agent-side evaluation. No agent release is required.
|
||||
|
||||
## Data model
|
||||
|
||||
New constants in `models/monitor.go`: `MonitorMetric = "metric"`, `MonitorHeartbeat = "heartbeat"`.
|
||||
|
||||
`MonitorTarget` gains:
|
||||
|
||||
| Field | Type | Used by | Meaning |
|
||||
|---|---|---|---|
|
||||
| `Selector` | `map[string]string` | metric | Server tag selector. Empty means the whole org fleet. |
|
||||
| `Metric` | `string` | metric | One of the metric kinds below. |
|
||||
| `Threshold` | `float64` | metric | Breach threshold, unit depends on kind. |
|
||||
| `Mount` | `string` | metric (disk kinds) | Mountpoint. Empty means any mount breaches. |
|
||||
| `PeriodSec` | `int` | heartbeat | Expected time between pings. |
|
||||
| `GraceSec` | `int` | heartbeat | Extra time allowed before overdue, and the max run time after `/start`. |
|
||||
|
||||
`Monitor` gains `ForSec int` (metric): the condition must hold continuously this long before a server goes down. 0 means on first evaluation.
|
||||
|
||||
`Monitor` gains `HeartbeatTokenHash string` (`json:"-"`), SHA-256 of the ping token. The plaintext token is returned only on create and on rotate.
|
||||
|
||||
`MonitorState` gains, for heartbeats: `LastPingAt *time.Time`, `StartedAt *time.Time`.
|
||||
|
||||
`Incident` gains `ServerID string` (`omitempty`). Empty for every non-metric monitor.
|
||||
|
||||
`Inventory` gains `RebootRequiredSince *time.Time`, set by `StoreInventory` when `reboot_required` turns true and unset when it turns false.
|
||||
|
||||
New collection `monitor_server_states`, one document per (metric monitor, matching server):
|
||||
|
||||
```go
|
||||
type MonitorServerState struct {
|
||||
InstanceID string // org
|
||||
MonitorID string
|
||||
ServerID string
|
||||
Status string // up | down | pending
|
||||
BreachSince *time.Time // first evaluation where the condition held; nil when clear
|
||||
Value float64 // last evaluated value, for display
|
||||
Message string // e.g. "/var 94.2% used"
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
```
|
||||
|
||||
Unique index on `(monitor_id, server_id)`.
|
||||
|
||||
### Metric kinds
|
||||
|
||||
| Kind | Source | Breach when | Threshold unit |
|
||||
|---|---|---|---|
|
||||
| `disk_pct` | `inventory.partitions` | used/total*100 >= threshold on `Mount` (or any mount) | percent |
|
||||
| `disk_free_gb` | `inventory.partitions` | (total-used)/1e9 <= threshold | GB |
|
||||
| `mem_pct` | `inventory.memory` | used/total*100 >= threshold | percent |
|
||||
| `load_per_core` | `inventory.cpu.load1 / cores` | >= threshold | ratio |
|
||||
| `unit_failed` | workloads, kind `unit` | any unit state `failed` (threshold unused) | - |
|
||||
| `container_unhealthy` | workloads, kind `container` | any health `unhealthy` (threshold unused) | - |
|
||||
| `reboot_pending_days` | `inventory.reboot_required_since` | now - since >= threshold days | days |
|
||||
| `agent_offline_min` | `server.last_seen` | now - last_seen >= threshold minutes | minutes |
|
||||
|
||||
Validation on create/update: `Metric` is a known kind; `Threshold` > 0 for kinds that use it; `disk_pct` and `mem_pct` threshold <= 100; `Mount`, if set, is an absolute path; `Selector` keys and values follow existing tag rules. `IntervalSec`, `Runner` and `Retries` are ignored for metric and heartbeat monitors and stored as `runner = "server"`, so `ListServerScheduledMonitors` must exclude these two types from the pull scheduler.
|
||||
|
||||
Heartbeat validation: `PeriodSec` >= 60, `GraceSec` >= 0 (default 300 when unset).
|
||||
|
||||
## Shared transition logic
|
||||
|
||||
Extract from `ingestResult` in `services/monitors.go`:
|
||||
|
||||
```go
|
||||
func applyTransition(ctx context.Context, m *models.Monitor, serverID, prev, next, message string, now time.Time)
|
||||
```
|
||||
|
||||
It opens an incident (with `ServerID`) on a change to down, resolves the matching open incident (same `monitor_id` and `server_id`) on down to up, and calls `notifyTransition`. `ingestResult` calls it with `serverID = ""`. Behaviour of existing monitor types does not change.
|
||||
|
||||
`notify.Event` gains `ServerName string`. Channel formatters include it when set ("disk on web-01: /var 94.2% used").
|
||||
|
||||
## Metric evaluation
|
||||
|
||||
New package `server/internal/metricsched`, started from `main.go` beside `monitorsched`. Like monitors, it runs regardless of licence state.
|
||||
|
||||
Every 30 seconds, for each enabled metric monitor:
|
||||
|
||||
1. Resolve servers with `ListServersFiltered(instanceID, selector)`.
|
||||
2. Load existing `monitor_server_states` for the monitor.
|
||||
3. For each server:
|
||||
- For kinds other than `agent_offline_min`: skip (keep previous state) when `inventory.metrics_at` is older than 5 minutes or missing. A dead agent must not flap other alerts; `agent_offline_min` covers it.
|
||||
- Evaluate the kind to get `(breach bool, value float64, message string)`.
|
||||
- Breach: set `BreachSince` if nil. If `now - BreachSince >= ForSec`, next = down, otherwise next = pending when previously up or new.
|
||||
- No breach: `BreachSince = nil`, next = up.
|
||||
- Upsert the state doc. On a status change call `applyTransition` with the server ID. A new server's first state is never treated as a transition from down.
|
||||
4. State docs for servers no longer matched (tag removed, server deleted): resolve any open incident without notifying, and delete the doc.
|
||||
5. Roll up the parent `Monitor.State`: status = down if any child down, else pending if any pending, else up (no matching servers means up). Message is "N of M servers breaching". `last_check_at = now`. Write one `MonitorSample` (up = no child down, latency 0) and increment the hourly rollup, so uptime graphs and status pages work unchanged.
|
||||
|
||||
Deleting a metric monitor deletes its `monitor_server_states`.
|
||||
|
||||
Evaluators are pure functions `func(kind string, t models.MonitorTarget, srv models.Server, wls []models.Workload, now time.Time) (bool, float64, string)` so they can be table-tested without Mongo. Workloads are loaded only for monitors whose kind needs them.
|
||||
|
||||
## Heartbeats
|
||||
|
||||
### Public endpoints
|
||||
|
||||
Registered outside `/api`, unauthenticated, no scope declarations needed:
|
||||
|
||||
- `GET|POST /hb/:token`: success ping
|
||||
- `GET|POST /hb/:token/start`: run started
|
||||
- `GET|POST /hb/:token/fail`: run failed
|
||||
|
||||
Lookup is by SHA-256 of the token. Unknown token or disabled monitor gives 404. Rate limit is one accepted request per second per token (in-process, Redis-backed if a limiter helper already exists); excess gives 429. The response body is `OK`. Request bodies over 1 KB are truncated; only `/fail` uses the body.
|
||||
|
||||
Behaviour:
|
||||
|
||||
- **ping**: `LastPingAt = now`. If `StartedAt` is set, duration = now - StartedAt, write a sample with `LatencyMs = duration`, then clear `StartedAt`. Otherwise write a sample with latency 0. Next status is up; call `applyTransition` if changed.
|
||||
- **start**: `StartedAt = now`. No status change.
|
||||
- **fail**: clear `StartedAt`, write a down sample, next status is down immediately with message `reported failure: <body>` (body defaults to empty). The failure is also recorded as `LastPingAt` so the overdue timer restarts from it.
|
||||
|
||||
All state writes go through one conditional update per request, so concurrent pings don't lose updates.
|
||||
|
||||
### Overdue sweep
|
||||
|
||||
The same 30-second `metricsched` loop checks enabled heartbeat monitors:
|
||||
|
||||
- New monitor with no ping yet: pending, never down. The overdue clock starts at the first ping.
|
||||
- `now > LastPingAt + PeriodSec + GraceSec`: down, message `no ping since <time>`.
|
||||
- `StartedAt != nil && now > StartedAt + GraceSec`: down, message `started <time>, never finished`.
|
||||
|
||||
Recovery happens only through a ping.
|
||||
|
||||
### Token management
|
||||
|
||||
- Create returns `heartbeat_token` and `heartbeat_url` once.
|
||||
- `POST /api/monitors/:id/rotate-token` issues a new token and invalidates the old one. It needs `routeScopes` (`monitors:write`) and `serverScopedRoutes` entries.
|
||||
- The UI shows the URL only right after create or rotate, with a copy button and curl examples for ping, start and fail.
|
||||
|
||||
## API
|
||||
|
||||
- Existing create/update monitor endpoints accept the new types and fields. OpenAPI annotations are updated.
|
||||
- `GET /api/monitors/:id/servers` returns `monitor_server_states` for a metric monitor, with hostnames. It needs `routeScopes` (`monitors:read`) and `serverScopedRoutes` entries, and filters rows to servers visible to the acting token.
|
||||
- A tag-scoped token may create or update a metric monitor only when its selector includes every key/value of the token's scope. Otherwise the response is 403.
|
||||
- The MCP `create_monitor` tool accepts the new types.
|
||||
|
||||
## Web UI
|
||||
|
||||
- Monitor form: a type picker adds Metric and Heartbeat. Metric shows a selector editor (reuse the tag selector component), a kind dropdown, a threshold with a unit label, mount (disk kinds only) and "for N minutes". Heartbeat shows period and grace.
|
||||
- Monitors list: metric rows show "N of M servers breaching". Heartbeat rows show last ping as relative time.
|
||||
- Metric detail page: a per-server table with server, status, value, since and message, linking to the server page. The incident list shows the server name.
|
||||
- Heartbeat detail page: URL panel (after create or rotate), rotate button with confirm, last ping, and duration in the latency chart.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Evaluator panics or bad inventory data: recover per monitor, log, and leave states unchanged.
|
||||
- Mongo errors in the sweep: log and continue to the next monitor. The next tick retries.
|
||||
- Notification failures are logged only, as today.
|
||||
|
||||
## Testing
|
||||
|
||||
- Table tests for every evaluator: breach, clear, mount filter, missing inventory, zero totals.
|
||||
- Sweep tests (Mongo test helper): the for-duration gate, pending to down, stale inventory skipped, a server leaving the selector resolves its incident silently, parent rollup counts.
|
||||
- Heartbeat handler tests: ping/start/fail transitions, duration sample, unknown token 404, rate limit 429, fail body truncation.
|
||||
- Overdue tests: never pinged stays pending, overdue goes down, start without finish goes down, ping recovers.
|
||||
- `applyTransition` regression: existing http monitor incident open/resolve unchanged.
|
||||
- `go test ./internal/api/` including `TestRegisteredRoutesPassBootAssertions`.
|
||||
- Playwright: create a heartbeat, curl a ping, see up.
|
||||
|
||||
## Phases
|
||||
|
||||
1. Shared `applyTransition` refactor, heartbeat type, public endpoints, overdue sweep, rotate route, UI. Ships alone.
|
||||
2. Metric type, `RebootRequiredSince`, `monitor_server_states`, evaluators, sweep, servers route, UI.
|
||||
|
||||
## Documentation
|
||||
|
||||
Update the Monitors section of `CLAUDE.md` and add a user guide page in vantage-docs for both types.
|
||||
Reference in New Issue
Block a user