diff --git a/docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md b/docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md new file mode 100644 index 0000000..ef3fde7 --- /dev/null +++ b/docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md @@ -0,0 +1,184 @@ +# Metric alerts and heartbeat monitors + +Date: 2026-09-17 +Source: competitive gap review, "Alerts on agent metrics" (effort M) and "Heartbeat (push) monitors" (effort S). + +## Goal + +1. Alert on data agents already report (disk, memory, load, failed units, unhealthy containers, reboot pending, agent offline), targeted by tag so one rule covers the fleet. +2. Alert when a push-based job (backup, cron) stops pinging, reports failure, or starts and never finishes. + +Both reuse the existing monitor model, states, incidents, notification channels, groups and status pages. + +## Out of scope + +- Scheduled workflows pinging a heartbeat automatically. +- Metric history graphs (separate gap review item). +- Agent-side evaluation. No agent release is required. + +## Data model + +New constants in `models/monitor.go`: `MonitorMetric = "metric"`, `MonitorHeartbeat = "heartbeat"`. + +`MonitorTarget` gains: + +| Field | Type | Used by | Meaning | +|---|---|---|---| +| `Selector` | `map[string]string` | metric | Server tag selector. Empty means the whole org fleet. | +| `Metric` | `string` | metric | One of the metric kinds below. | +| `Threshold` | `float64` | metric | Breach threshold, unit depends on kind. | +| `Mount` | `string` | metric (disk kinds) | Mountpoint. Empty means any mount breaches. | +| `PeriodSec` | `int` | heartbeat | Expected time between pings. | +| `GraceSec` | `int` | heartbeat | Extra time allowed before overdue, and the max run time after `/start`. | + +`Monitor` gains `ForSec int` (metric): the condition must hold continuously this long before a server goes down. 0 means on first evaluation. + +`Monitor` gains `HeartbeatTokenHash string` (`json:"-"`), SHA-256 of the ping token. The plaintext token is returned only on create and on rotate. + +`MonitorState` gains, for heartbeats: `LastPingAt *time.Time`, `StartedAt *time.Time`. + +`Incident` gains `ServerID string` (`omitempty`). Empty for every non-metric monitor. + +`Inventory` gains `RebootRequiredSince *time.Time`, set by `StoreInventory` when `reboot_required` turns true and unset when it turns false. + +New collection `monitor_server_states`, one document per (metric monitor, matching server): + +```go +type MonitorServerState struct { + InstanceID string // org + MonitorID string + ServerID string + Status string // up | down | pending + BreachSince *time.Time // first evaluation where the condition held; nil when clear + Value float64 // last evaluated value, for display + Message string // e.g. "/var 94.2% used" + UpdatedAt time.Time +} +``` + +Unique index on `(monitor_id, server_id)`. + +### Metric kinds + +| Kind | Source | Breach when | Threshold unit | +|---|---|---|---| +| `disk_pct` | `inventory.partitions` | used/total*100 >= threshold on `Mount` (or any mount) | percent | +| `disk_free_gb` | `inventory.partitions` | (total-used)/1e9 <= threshold | GB | +| `mem_pct` | `inventory.memory` | used/total*100 >= threshold | percent | +| `load_per_core` | `inventory.cpu.load1 / cores` | >= threshold | ratio | +| `unit_failed` | workloads, kind `unit` | any unit state `failed` (threshold unused) | - | +| `container_unhealthy` | workloads, kind `container` | any health `unhealthy` (threshold unused) | - | +| `reboot_pending_days` | `inventory.reboot_required_since` | now - since >= threshold days | days | +| `agent_offline_min` | `server.last_seen` | now - last_seen >= threshold minutes | minutes | + +Validation on create/update: `Metric` is a known kind; `Threshold` > 0 for kinds that use it; `disk_pct` and `mem_pct` threshold <= 100; `Mount`, if set, is an absolute path; `Selector` keys and values follow existing tag rules. `IntervalSec`, `Runner` and `Retries` are ignored for metric and heartbeat monitors and stored as `runner = "server"`, so `ListServerScheduledMonitors` must exclude these two types from the pull scheduler. + +Heartbeat validation: `PeriodSec` >= 60, `GraceSec` >= 0 (default 300 when unset). + +## Shared transition logic + +Extract from `ingestResult` in `services/monitors.go`: + +```go +func applyTransition(ctx context.Context, m *models.Monitor, serverID, prev, next, message string, now time.Time) +``` + +It opens an incident (with `ServerID`) on a change to down, resolves the matching open incident (same `monitor_id` and `server_id`) on down to up, and calls `notifyTransition`. `ingestResult` calls it with `serverID = ""`. Behaviour of existing monitor types does not change. + +`notify.Event` gains `ServerName string`. Channel formatters include it when set ("disk on web-01: /var 94.2% used"). + +## Metric evaluation + +New package `server/internal/metricsched`, started from `main.go` beside `monitorsched`. Like monitors, it runs regardless of licence state. + +Every 30 seconds, for each enabled metric monitor: + +1. Resolve servers with `ListServersFiltered(instanceID, selector)`. +2. Load existing `monitor_server_states` for the monitor. +3. For each server: + - For kinds other than `agent_offline_min`: skip (keep previous state) when `inventory.metrics_at` is older than 5 minutes or missing. A dead agent must not flap other alerts; `agent_offline_min` covers it. + - Evaluate the kind to get `(breach bool, value float64, message string)`. + - Breach: set `BreachSince` if nil. If `now - BreachSince >= ForSec`, next = down, otherwise next = pending when previously up or new. + - No breach: `BreachSince = nil`, next = up. + - Upsert the state doc. On a status change call `applyTransition` with the server ID. A new server's first state is never treated as a transition from down. +4. State docs for servers no longer matched (tag removed, server deleted): resolve any open incident without notifying, and delete the doc. +5. Roll up the parent `Monitor.State`: status = down if any child down, else pending if any pending, else up (no matching servers means up). Message is "N of M servers breaching". `last_check_at = now`. Write one `MonitorSample` (up = no child down, latency 0) and increment the hourly rollup, so uptime graphs and status pages work unchanged. + +Deleting a metric monitor deletes its `monitor_server_states`. + +Evaluators are pure functions `func(kind string, t models.MonitorTarget, srv models.Server, wls []models.Workload, now time.Time) (bool, float64, string)` so they can be table-tested without Mongo. Workloads are loaded only for monitors whose kind needs them. + +## Heartbeats + +### Public endpoints + +Registered outside `/api`, unauthenticated, no scope declarations needed: + +- `GET|POST /hb/:token`: success ping +- `GET|POST /hb/:token/start`: run started +- `GET|POST /hb/:token/fail`: run failed + +Lookup is by SHA-256 of the token. Unknown token or disabled monitor gives 404. Rate limit is one accepted request per second per token (in-process, Redis-backed if a limiter helper already exists); excess gives 429. The response body is `OK`. Request bodies over 1 KB are truncated; only `/fail` uses the body. + +Behaviour: + +- **ping**: `LastPingAt = now`. If `StartedAt` is set, duration = now - StartedAt, write a sample with `LatencyMs = duration`, then clear `StartedAt`. Otherwise write a sample with latency 0. Next status is up; call `applyTransition` if changed. +- **start**: `StartedAt = now`. No status change. +- **fail**: clear `StartedAt`, write a down sample, next status is down immediately with message `reported failure: ` (body defaults to empty). The failure is also recorded as `LastPingAt` so the overdue timer restarts from it. + +All state writes go through one conditional update per request, so concurrent pings don't lose updates. + +### Overdue sweep + +The same 30-second `metricsched` loop checks enabled heartbeat monitors: + +- New monitor with no ping yet: pending, never down. The overdue clock starts at the first ping. +- `now > LastPingAt + PeriodSec + GraceSec`: down, message `no ping since