From 73da4647019e7f1da1fedff87e5a8cf3edc25822 Mon Sep 17 00:00:00 2001 From: mrhid6 Date: Thu, 17 Sep 2026 07:48:25 +0000 Subject: [PATCH] docs(plan): metric alerts and heartbeat monitors implementation plan --- .../2026-09-17-metric-alerts-heartbeats.md | 2397 +++++++++++++++++ 1 file changed, 2397 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-17-metric-alerts-heartbeats.md diff --git a/docs/superpowers/plans/2026-09-17-metric-alerts-heartbeats.md b/docs/superpowers/plans/2026-09-17-metric-alerts-heartbeats.md new file mode 100644 index 0000000..f8ff462 --- /dev/null +++ b/docs/superpowers/plans/2026-09-17-metric-alerts-heartbeats.md @@ -0,0 +1,2397 @@ +# Metric Alerts and Heartbeat Monitors Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a `heartbeat` monitor type (push pings with overdue detection) and a `metric` monitor type (tag-targeted alerts on agent-reported disk, memory, load, units, containers, reboot pending, agent offline). + +**Architecture:** Both types live in the existing `monitors` collection and reuse incidents, channels, samples, rollups and status pages. Neither is run by `monitorsched`. A new leader-scoped loop, `metricsched`, sweeps every 30 seconds: heartbeats for overdue, metric monitors for per-server evaluation against stored inventory and workloads. The incident/notify side of `ingestResult` is extracted into `applyTransition`, which all three paths share. Decision logic is written as pure functions so it can be unit-tested without Mongo; this repo has no Mongo test harness. + +**Tech Stack:** Go 1.x, gin, mongo-driver v2, Redis (go-redis), Next.js/React/TypeScript, Playwright. + +**Spec:** `docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md` + +## Global Constraints + +- Every new route under `/api` needs an entry in `routeScopes` (server/internal/api/scopes.go) AND in `serverScopedRoutes` (server/internal/api/serverscope.go), or the server refuses to boot. Run `go test ./internal/api/` after touching routes. +- **Deviation from spec:** the public ping endpoints are mounted at `/public/hb/:token` (plus `/start` and `/fail`), not `/hb/:token`. `/public/` is already routed to the Go server by nginx (deploy/docker/nginx/vantage.conf), the Helm ingress and web/next.config.ts. A new top-level prefix would need all three changed. Update the spec's Heartbeats section in Task 3. +- Monitors run regardless of licence state. `metricsched` runs inside `bus.RunAsLeader(ctx, "housekeeping", ...)` next to `monitorsched.Start`. +- Heartbeat tokens are stored only as SHA-256 hex hashes (`HeartbeatTokenHash`, `json:"-"`). Plaintext is returned only by create and rotate. +- Heartbeat: `PeriodSec >= 60`; `GraceSec >= 0`, defaulting to 300 when 0 on create. +- Metric stale cutoff: `inventory.metrics_at` older than 5 minutes means skip (except `agent_offline_min`). +- Sweep interval: 30 seconds. +- Ping rate limit: 1 accepted request per second per token. Allow when Redis is unavailable, matching `RateLimitPublicStatus`. +- `/fail` body is truncated to 1024 bytes. +- Go code comments follow the repo's style: explain *why*, full sentences. +- Commit after each task. Commit messages are conventional (`feat(monitors): ...`). +- Run Go tests from `server/`: `go test ./...`. Run web checks from `web/`: `npm run lint && npx tsc --noEmit`. + +--- + +## File Structure + +**Phase 1: heartbeat** +- Modify `server/internal/models/monitor.go`: new constants and fields. +- Modify `server/internal/notify/dispatch.go`: `Event.ServerName` in the title. +- Create `server/internal/services/monitortransition.go`: `decideStatus` (pure) and `applyTransition`. +- Modify `server/internal/services/monitors.go`: use the above; exclude passive types from the pull scheduler; heartbeat validation and token on create; lock the type on update. +- Create `server/internal/services/heartbeats.go`: token helpers, `RecordHeartbeat`, `RotateHeartbeatToken`, `heartbeatVerdict` (pure), `SweepHeartbeats`. +- Create `server/internal/services/heartbeats_test.go`. +- Create `server/internal/api/heartbeats.go`: public ping handler, per-token rate limit, rotate handler. +- Modify `server/internal/api/handlers.go`, `monitors.go`, `scopes.go`, `serverscope.go`. +- Create `server/internal/metricsched/scheduler.go`: the 30-second loop. +- Modify `server/cmd/main.go`. +- Modify `web/lib/api.ts`, `web/components/monitors/MonitorForm.tsx`, `web/app/(app)/monitors/page.tsx`, `web/app/(app)/monitors/[id]/page.tsx`. +- Create `web/components/monitors/HeartbeatUrlPanel.tsx`. + +**Phase 2: metrics** +- Modify `server/internal/services/inventory.go`: `reboot_required_since`. +- Create `server/internal/services/metricrules.go`: `EvaluateMetric` (pure), `nextServerState` (pure), `rollupParent` (pure), validation. +- Create `server/internal/services/metricrules_test.go`. +- Create `server/internal/services/metricsweep.go`: `SweepMetricMonitors`, state collection access, indexes. +- Modify `server/internal/services/migrate_instance.go`: `ScopedCollections` += `monitor_server_states`. +- Modify `server/internal/api/monitors.go`, `scopes.go`, `serverscope.go`: `GET /api/monitors/:id/servers`; incidents filtered by visible server. +- Modify `server/internal/mcp/tools_create.go`. +- Modify web files for the metric form, list and detail page. +- Create `web/components/monitors/MetricServersTable.tsx`. +- Create `web/e2e/heartbeat.spec.ts`. +- Modify `CLAUDE.md`. + +--- + +## Phase 1: Heartbeat monitors + +### Task 1: Model fields, notify server name, shared transition logic + +**Files:** +- Modify: `server/internal/models/monitor.go` +- Modify: `server/internal/notify/dispatch.go:18-50` +- Create: `server/internal/services/monitortransition.go` +- Create: `server/internal/services/monitortransition_test.go` +- Create: `server/internal/notify/dispatch_title_test.go` +- Modify: `server/internal/services/monitors.go` (`ingestResult` at ~325, `notifyTransition` at ~427, `listMonitorsForRunner` at ~103) + +**Interfaces:** +- Produces: `models.MonitorMetric`, `models.MonitorHeartbeat`, `models.IsPassiveMonitor(t string) bool`, new fields listed below, `services.decideStatus(prev string, fails, retries int, up bool) (string, int)`, `services.applyTransition(ctx context.Context, m *models.Monitor, serverID, serverName, prev, next, message string, now time.Time)`, `notify.Event.ServerName`. + +- [ ] **Step 1: Add model constants and fields** + +In `server/internal/models/monitor.go`: + +```go +const ( + MonitorHTTP = "http" + MonitorTCP = "tcp" + MonitorICMP = "icmp" + MonitorTLS = "tls" + MonitorMetric = "metric" + MonitorHeartbeat = "heartbeat" +) + +// IsPassiveMonitor reports whether a monitor type is evaluated from data that +// arrives (a ping, an agent report) rather than by running a check. Passive +// monitors are never handed to monitorsched or to an agent. +func IsPassiveMonitor(t string) bool { + return t == MonitorMetric || t == MonitorHeartbeat +} +``` + +Append to `MonitorTarget`: + +```go + // Metric monitors. + Selector map[string]string `bson:"selector,omitempty" json:"selector,omitempty"` + Metric string `bson:"metric,omitempty" json:"metric,omitempty"` + Threshold float64 `bson:"threshold,omitempty" json:"threshold,omitempty"` + Mount string `bson:"mount,omitempty" json:"mount,omitempty"` + // Heartbeat monitors. + PeriodSec int `bson:"period_sec,omitempty" json:"period_sec,omitempty"` + GraceSec int `bson:"grace_sec,omitempty" json:"grace_sec,omitempty"` +``` + +Append to `MonitorState`: + +```go + LastPingAt *time.Time `bson:"last_ping_at,omitempty" json:"last_ping_at,omitempty"` + StartedAt *time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"` +``` + +Append to `Monitor` (after `ChannelIDs`): + +```go + // ForSec is how long a metric condition must hold before a server is down. + ForSec int `bson:"for_sec,omitempty" json:"for_sec,omitempty"` + // HeartbeatTokenHash is the SHA-256 of the ping token. The token itself is + // shown once, on create or rotate, and never stored. + HeartbeatTokenHash string `bson:"heartbeat_token_hash,omitempty" json:"-"` +``` + +Append to `Incident`: + +```go + // ServerID is set only for metric monitors, which keep one incident per + // breaching server. + ServerID string `bson:"server_id,omitempty" json:"server_id,omitempty"` +``` + +Add the per-server state type: + +```go +// MonitorServerState is one metric monitor's view of one matching server. +type MonitorServerState struct { + InstanceID string `bson:"instance_id" json:"instance_id"` + MonitorID string `bson:"monitor_id" json:"monitor_id"` + ServerID string `bson:"server_id" json:"server_id"` + Hostname string `bson:"-" json:"hostname,omitempty"` + Status string `bson:"status" json:"status"` + BreachSince *time.Time `bson:"breach_since,omitempty" json:"breach_since,omitempty"` + Value float64 `bson:"value" json:"value"` + Message string `bson:"message,omitempty" json:"message,omitempty"` + UpdatedAt time.Time `bson:"updated_at" json:"updated_at"` +} +``` + +Add `RebootRequiredSince *time.Time \`bson:"reboot_required_since,omitempty" json:"reboot_required_since,omitempty"\`` to `Inventory` in `server/internal/models/server.go` (used in Task 6; adding it now keeps model changes in one commit). + +- [ ] **Step 2: Write the failing tests** + +`server/internal/services/monitortransition_test.go`: + +```go +package services + +import ( + "testing" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +// decideStatus is the retry state machine lifted out of ingestResult. These +// cases pin its existing behaviour so the refactor cannot change it. +func TestDecideStatus(t *testing.T) { + cases := []struct { + name string + prev string + fails int + retries int + up bool + wantState string + wantFails int + }{ + {"up resets fails", models.StatusDown, 3, 2, true, models.StatusUp, 0}, + {"first failure below retries is pending", models.StatusPending, 0, 3, false, models.StatusPending, 1}, + {"failure below retries keeps up", models.StatusUp, 0, 3, false, models.StatusUp, 1}, + {"failure reaching retries is down", models.StatusUp, 2, 3, false, models.StatusDown, 3}, + {"retries below one treated as one", models.StatusUp, 0, 0, false, models.StatusDown, 1}, + {"empty prev failing below retries is pending", "", 0, 2, false, models.StatusPending, 1}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got, fails := decideStatus(c.prev, c.fails, c.retries, c.up) + if got != c.wantState || fails != c.wantFails { + t.Fatalf("got (%s,%d), want (%s,%d)", got, fails, c.wantState, c.wantFails) + } + }) + } +} +``` + +`server/internal/notify/dispatch_title_test.go`: + +```go +package notify + +import ( + "strings" + "testing" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +func TestTitleIncludesServerName(t *testing.T) { + ev := Event{MonitorName: "Disk full", Type: models.MonitorMetric, NewStatus: models.StatusDown, ServerName: "web-01", Message: "/var 94.2% used"} + got := ev.title() + want := "[Vantage] Disk full (metric) on web-01 is DOWN: /var 94.2% used" + if got != want { + t.Fatalf("title = %q, want %q", got, want) + } +} + +func TestTitleWithoutServerNameUnchanged(t *testing.T) { + ev := Event{MonitorName: "Site", Type: models.MonitorHTTP, NewStatus: models.StatusUp} + if got := ev.title(); strings.Contains(got, " on ") || got != "[Vantage] Site (http) recovered" { + t.Fatalf("title = %q", got) + } +} +``` + +- [ ] **Step 3: Run tests to verify they fail** + +Run: `cd server && go test ./internal/services/ -run TestDecideStatus && go test ./internal/notify/ -run TestTitle` +Expected: compile errors, `undefined: decideStatus` and `unknown field ServerName`. + +- [ ] **Step 4: Implement `Event.ServerName`** + +In `server/internal/notify/dispatch.go`, add `ServerName string` to `Event` after `MonitorName`. In `title()`, replace the final `else` branch: + +```go + } else { + name := e.MonitorName + if e.ServerName != "" { + name = fmt.Sprintf("%s (%s) on %s", e.MonitorName, e.Type, e.ServerName) + } else { + name = fmt.Sprintf("%s (%s)", e.MonitorName, e.Type) + } + s = fmt.Sprintf("[Vantage] %s %s", name, verb) + } +``` + +Then grep the other dispatchers (`grep -n "MonitorName" server/internal/notify/*.go`). Wherever a formatter builds its own body from `MonitorName` instead of `title()`, add the server name the same way (webhook JSON: add `"server_name": ev.ServerName` with `omitempty` semantics). + +- [ ] **Step 5: Create `monitortransition.go`** + +```go +package services + +import ( + "context" + "log" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/notify" + "github.com/google/uuid" + "go.mongodb.org/mongo-driver/v2/bson" +) + +// decideStatus is the pull-check retry state machine: a success is always up, +// failures below the retry count leave the status alone (or pending when there +// is none yet), and reaching the count is down. +func decideStatus(prev string, fails, retries int, up bool) (string, int) { + if retries < 1 { + retries = 1 + } + if up { + return models.StatusUp, 0 + } + fails++ + if fails >= retries { + return models.StatusDown, fails + } + if prev == "" || prev == models.StatusPending { + return models.StatusPending, fails + } + return prev, fails +} + +// applyTransition is the one place a status change becomes an incident and a +// notification. serverID is empty for every monitor except a metric monitor, +// which keeps one incident per server so each breach opens and resolves on its +// own. +func applyTransition(ctx context.Context, m *models.Monitor, serverID, serverName, prev, next, message string, now time.Time) { + if next == prev { + return + } + switch next { + case models.StatusDown: + inc := models.Incident{ + InstanceID: m.InstanceID, + IncidentID: uuid.NewString(), + MonitorID: m.MonitorID, + ServerID: serverID, + StartedAt: now, + Cause: message, + } + if _, err := db.Col("incidents").InsertOne(ctx, inc); err != nil { + log.Printf("monitors: open incident for %s: %v", m.MonitorID, err) + } + notifyTransition(m, serverName, prev, next, message) + case models.StatusUp: + if prev != models.StatusDown { + return + } + resolveIncident(ctx, m, serverID, now) + notifyTransition(m, serverName, prev, next, message) + } +} + +// resolveIncident closes the open incident for a monitor (and server, for a +// metric monitor) without notifying. It is also used when a server stops +// matching a metric monitor's selector: nothing recovered, so nobody is told. +func resolveIncident(ctx context.Context, m *models.Monitor, serverID string, now time.Time) { + filter := bson.M{"monitor_id": m.MonitorID, "instance_id": m.InstanceID, "resolved_at": nil} + if serverID != "" { + filter["server_id"] = serverID + } else { + filter["server_id"] = bson.M{"$exists": false} + } + if _, err := db.Col("incidents").UpdateMany(ctx, filter, bson.M{"$set": bson.M{"resolved_at": now}}); err != nil { + log.Printf("monitors: resolve incident for %s: %v", m.MonitorID, err) + } +} +``` + +- [ ] **Step 6: Rewire `ingestResult` and `notifyTransition` in `monitors.go`** + +Replace the retry block (from `prev := m.State.Status` to the end of the fails logic) with: + +```go + prev := m.State.Status + newStatus, fails := decideStatus(prev, m.State.Fails, m.Retries, res.Up) +``` + +Replace the entire `if newStatus != prev { switch ... }` block at the end with: + +```go + applyTransition(ctx, m, "", "", prev, newStatus, res.Message, now) + return nil +``` + +Change `notifyTransition` to take the server name and previous status explicitly (the stored state is stale by the time a sweep calls it): + +```go +func notifyTransition(m *models.Monitor, serverName, oldStatus, newStatus, message string) { +``` + +and in the `notify.Event` literal set `ServerName: serverName` and `OldStatus: oldStatus`. Remove the `uuid` import from monitors.go if unused. + +Existing incidents have no `server_id` field, so the `$exists: false` filter in `resolveIncident` matches them. + +- [ ] **Step 7: Exclude passive types from the pull scheduler and agents** + +In `listMonitorsForRunner`: + +```go + filter := bson.M{"runner": runner, "enabled": true, + "type": bson.M{"$nin": []string{models.MonitorMetric, models.MonitorHeartbeat}}} +``` + +- [ ] **Step 8: Run tests** + +Run: `cd server && go build ./... && go vet ./... && go test ./internal/services/ ./internal/notify/` +Expected: PASS. + +- [ ] **Step 9: Commit** + +```bash +git add server/internal/models server/internal/notify server/internal/services/monitortransition.go server/internal/services/monitortransition_test.go server/internal/services/monitors.go +git commit -m "refactor(monitors): extract applyTransition and add passive monitor model fields" +``` + +--- + +### Task 2: Heartbeat service (tokens, ping recording, overdue verdict, rotate) + +**Files:** +- Create: `server/internal/services/heartbeats.go` +- Create: `server/internal/services/heartbeats_test.go` +- Modify: `server/internal/services/monitors.go` (`CreateMonitor`, `UpdateMonitor`) + +**Interfaces:** +- Consumes: `applyTransition`, `models.MonitorHeartbeat`, `MonitorState.LastPingAt/StartedAt`, `Monitor.HeartbeatTokenHash`. +- Produces: + - `const HeartbeatPing, HeartbeatStart, HeartbeatFail = "ping", "start", "fail"` + - `const MaxHeartbeatBody = 1024` + - `var ErrHeartbeatNotFound = errors.New("heartbeat not found")` + - `func NewHeartbeatToken() (token, hash string, err error)` + - `func HashHeartbeatToken(token string) string` + - `func RecordHeartbeat(token, kind, body string, now time.Time) error` + - `func RotateHeartbeatToken(instanceID, monitorID string) (string, error)` + - `func heartbeatVerdict(m models.Monitor, now time.Time) (down bool, message string)` + - `func SweepHeartbeats(now time.Time)` + - `func validateHeartbeat(t *models.MonitorTarget) error` + - `CreateMonitor` now returns the plaintext token via a new field on its result: add `HeartbeatToken string \`bson:"-" json:"heartbeat_token,omitempty"\`` to `models.Monitor`. + +- [ ] **Step 1: Write the failing tests** + +`server/internal/services/heartbeats_test.go`: + +```go +package services + +import ( + "strings" + "testing" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +func hbMonitor(status string, lastPing, started *time.Time) models.Monitor { + return models.Monitor{ + Type: models.MonitorHeartbeat, + Target: models.MonitorTarget{PeriodSec: 3600, GraceSec: 300}, + State: models.MonitorState{Status: status, LastPingAt: lastPing, StartedAt: started}, + } +} + +func TestHeartbeatVerdict(t *testing.T) { + now := time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC) + at := func(d time.Duration) *time.Time { v := now.Add(-d); return &v } + + cases := []struct { + name string + m models.Monitor + wantDown bool + wantMsg string + }{ + {"never pinged stays pending", hbMonitor(models.StatusPending, nil, nil), false, ""}, + {"within period", hbMonitor(models.StatusUp, at(30*time.Minute), nil), false, ""}, + {"inside grace", hbMonitor(models.StatusUp, at(62*time.Minute), nil), false, ""}, + {"overdue", hbMonitor(models.StatusUp, at(66*time.Minute), nil), true, "no ping since"}, + {"started inside grace", hbMonitor(models.StatusUp, at(10*time.Minute), at(4*time.Minute)), false, ""}, + {"started never finished", hbMonitor(models.StatusUp, at(10*time.Minute), at(6*time.Minute)), true, "never finished"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + down, msg := heartbeatVerdict(c.m, now) + if down != c.wantDown || !strings.Contains(msg, c.wantMsg) { + t.Fatalf("got (%v,%q), want (%v, contains %q)", down, msg, c.wantDown, c.wantMsg) + } + }) + } +} + +func TestNewHeartbeatTokenHashes(t *testing.T) { + tok, hash, err := NewHeartbeatToken() + if err != nil { + t.Fatal(err) + } + if len(tok) < 32 { + t.Fatalf("token too short: %d", len(tok)) + } + if hash != HashHeartbeatToken(tok) || hash == tok { + t.Fatal("hash must be the SHA-256 of the token and differ from it") + } +} + +func TestValidateHeartbeat(t *testing.T) { + tg := models.MonitorTarget{PeriodSec: 59} + if err := validateHeartbeat(&tg); err == nil { + t.Fatal("period under 60 must be rejected") + } + tg = models.MonitorTarget{PeriodSec: 60} + if err := validateHeartbeat(&tg); err != nil || tg.GraceSec != 300 { + t.Fatalf("grace should default to 300, got %d err %v", tg.GraceSec, err) + } + tg = models.MonitorTarget{PeriodSec: 60, GraceSec: -1} + if err := validateHeartbeat(&tg); err == nil { + t.Fatal("negative grace must be rejected") + } +} + +func TestTruncateHeartbeatBody(t *testing.T) { + long := strings.Repeat("x", MaxHeartbeatBody+50) + if got := failMessage(long); len(got) != len("reported failure: ")+MaxHeartbeatBody { + t.Fatalf("len = %d", len(got)) + } + if got := failMessage(" "); got != "reported failure" { + t.Fatalf("empty body message = %q", got) + } +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd server && go test ./internal/services/ -run 'Heartbeat'` +Expected: compile error, `undefined: heartbeatVerdict`. + +- [ ] **Step 3: Implement `heartbeats.go`** + +```go +package services + +import ( + "context" + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "errors" + "fmt" + "log" + "strings" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/mongo" + "go.mongodb.org/mongo-driver/v2/mongo/options" +) + +const ( + HeartbeatPing = "ping" + HeartbeatStart = "start" + HeartbeatFail = "fail" +) + +// MaxHeartbeatBody bounds what a /fail request can put into an incident cause +// and a notification. A job's stderr can be megabytes; the first kilobyte is +// what a human reads. +const MaxHeartbeatBody = 1024 + +const defaultHeartbeatGraceSec = 300 + +var ErrHeartbeatNotFound = errors.New("heartbeat not found") + +func NewHeartbeatToken() (string, string, error) { + raw := make([]byte, 24) + if _, err := rand.Read(raw); err != nil { + return "", "", err + } + tok := base64.RawURLEncoding.EncodeToString(raw) + return tok, HashHeartbeatToken(tok), nil +} + +func HashHeartbeatToken(token string) string { + sum := sha256.Sum256([]byte(token)) + return hex.EncodeToString(sum[:]) +} + +func validateHeartbeat(t *models.MonitorTarget) error { + if t.PeriodSec < 60 { + return fmt.Errorf("period_sec must be at least 60") + } + if t.GraceSec < 0 { + return fmt.Errorf("grace_sec must not be negative") + } + if t.GraceSec == 0 { + t.GraceSec = defaultHeartbeatGraceSec + } + return nil +} + +func failMessage(body string) string { + body = strings.TrimSpace(body) + if len(body) > MaxHeartbeatBody { + body = body[:MaxHeartbeatBody] + } + if body == "" { + return "reported failure" + } + return "reported failure: " + body +} + +// heartbeatVerdict decides whether a heartbeat is overdue. A heartbeat that has +// never pinged is never down: the clock starts at the first ping, so creating +// one before the job is deployed does not page anyone. +func heartbeatVerdict(m models.Monitor, now time.Time) (bool, string) { + grace := time.Duration(m.Target.GraceSec) * time.Second + if m.State.StartedAt != nil && now.After(m.State.StartedAt.Add(grace)) { + return true, fmt.Sprintf("started %s, never finished", m.State.StartedAt.UTC().Format(time.RFC3339)) + } + if m.State.LastPingAt == nil { + return false, "" + } + deadline := m.State.LastPingAt.Add(time.Duration(m.Target.PeriodSec)*time.Second + grace) + if now.After(deadline) { + return true, fmt.Sprintf("no ping since %s", m.State.LastPingAt.UTC().Format(time.RFC3339)) + } + return false, "" +} + +// RecordHeartbeat applies one ping. The state change is a single +// FindOneAndUpdate on the token hash, so two pings racing each other cannot +// both read the old state and lose one of the writes. +func RecordHeartbeat(token, kind, body string, now time.Time) error { + ctx, cancel := monCtx() + defer cancel() + + filter := bson.M{"heartbeat_token_hash": HashHeartbeatToken(token), "type": models.MonitorHeartbeat, "enabled": true} + set := bson.M{"state.last_check_at": now} + unset := bson.M{} + var next, message string + switch kind { + case HeartbeatStart: + set["state.started_at"] = now + case HeartbeatPing: + set["state.last_ping_at"] = now + set["state.status"] = models.StatusUp + set["state.message"] = "" + unset["state.started_at"] = "" + next = models.StatusUp + case HeartbeatFail: + message = failMessage(body) + set["state.last_ping_at"] = now + set["state.status"] = models.StatusDown + set["state.message"] = message + unset["state.started_at"] = "" + next = models.StatusDown + default: + return fmt.Errorf("unknown heartbeat kind %q", kind) + } + upd := bson.M{"$set": set} + if len(unset) > 0 { + upd["$unset"] = unset + } + + // ReturnDocument Before: the previous status and started_at are what the + // transition and the duration need. + var before models.Monitor + err := db.Col("monitors").FindOneAndUpdate(ctx, filter, upd, + options.FindOneAndUpdate().SetReturnDocument(options.Before)).Decode(&before) + if errors.Is(err, mongo.ErrNoDocuments) { + return ErrHeartbeatNotFound + } + if err != nil { + return err + } + if kind == HeartbeatStart { + return nil + } + + latency := 0 + if kind == HeartbeatPing && before.State.StartedAt != nil { + latency = int(now.Sub(*before.State.StartedAt).Milliseconds()) + } + if latency > 0 { + db.Col("monitors").UpdateOne(ctx, bson.M{"monitor_id": before.MonitorID}, bson.M{"$set": bson.M{"state.latency_ms": latency}}) + } + recordSample(ctx, &before, kind == HeartbeatPing, latency, now) + applyTransition(ctx, &before, "", "", before.State.Status, next, message, now) + return nil +} + +// SweepHeartbeats marks overdue heartbeats down. Recovery only ever comes from +// a ping, so the sweep never moves anything up. +func SweepHeartbeats(now time.Time) { + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + defer cancel() + cur, err := db.Col("monitors").Find(ctx, bson.M{ + "type": models.MonitorHeartbeat, "enabled": true, + "state.status": bson.M{"$ne": models.StatusDown}, + }) + if err != nil { + log.Printf("heartbeats: list: %v", err) + return + } + var monitors []models.Monitor + if err := cur.All(ctx, &monitors); err != nil { + log.Printf("heartbeats: decode: %v", err) + return + } + for i := range monitors { + m := &monitors[i] + down, msg := heartbeatVerdict(*m, now) + if !down { + continue + } + // Conditional on status so a ping landing between the read and this + // write is not overwritten by a stale "down". + res, err := db.Col("monitors").UpdateOne(ctx, + bson.M{"monitor_id": m.MonitorID, "state.status": m.State.Status}, + bson.M{"$set": bson.M{"state.status": models.StatusDown, "state.message": msg, "state.last_check_at": now}, + "$unset": bson.M{"state.started_at": ""}}) + if err != nil || res.ModifiedCount == 0 { + continue + } + recordSample(ctx, m, false, 0, now) + applyTransition(ctx, m, "", "", m.State.Status, models.StatusDown, msg, now) + } +} + +func RotateHeartbeatToken(instanceID, monitorID string) (string, error) { + ctx, cancel := monCtx() + defer cancel() + tok, hash, err := NewHeartbeatToken() + if err != nil { + return "", err + } + res, err := db.Col("monitors").UpdateOne(ctx, + bson.M{"monitor_id": monitorID, "instance_id": instanceID, "type": models.MonitorHeartbeat}, + bson.M{"$set": bson.M{"heartbeat_token_hash": hash}}) + if err != nil { + return "", err + } + if res.MatchedCount == 0 { + return "", ErrHeartbeatNotFound + } + return tok, nil +} +``` + +- [ ] **Step 4: Extract `recordSample` from `ingestResult`** + +In `monitors.go`, move the sample insert and rollup upsert out of `ingestResult` into: + +```go +// recordSample writes one result at full resolution and folds it into the +// hourly rollup, the two records every history view reads. +func recordSample(ctx context.Context, m *models.Monitor, up bool, latencyMs int, now time.Time) { + u := 0 + if up { + u = 1 + } + db.Col("monitor_samples").InsertOne(ctx, models.MonitorSample{ + InstanceID: m.InstanceID, MonitorID: m.MonitorID, At: now, Up: up, LatencyMs: latencyMs, + }) + db.Col("monitor_rollups").UpdateOne(ctx, + bson.M{"monitor_id": m.MonitorID, "period_start": now.Truncate(time.Hour)}, + bson.M{ + "$inc": bson.M{"checks": 1, "up_count": u, "sum_latency": int64(latencyMs)}, + "$setOnInsert": bson.M{"instance_id": m.InstanceID}, + }, + options.UpdateOne().SetUpsert(true)) +} +``` + +Call `recordSample(ctx, m, res.Up, res.LatencyMs, now)` from `ingestResult` where the removed code was. Keep the existing comment about samples on the helper. + +- [ ] **Step 5: Heartbeat handling in `CreateMonitor` and `UpdateMonitor`** + +In `CreateMonitor`, after the group normalisation: + +```go + if m.Type == models.MonitorHeartbeat { + if err := validateHeartbeat(&m.Target); err != nil { + return nil, err + } + tok, hash, err := NewHeartbeatToken() + if err != nil { + return nil, err + } + m.HeartbeatTokenHash = hash + m.HeartbeatToken = tok + // A heartbeat is never run, so neither where nor how often applies. + m.Runner = models.RunnerServer + m.IntervalSec = 0 + } +``` + +(Metric validation is added in Task 8.) Add to `models.Monitor`: + +```go + // HeartbeatToken is the plaintext token, set only on the create response. + HeartbeatToken string `bson:"-" json:"heartbeat_token,omitempty"` +``` + +In `UpdateMonitor`, at the top, lock the type across the passive boundary. A heartbeat without a token, or an http monitor with one, is a state nothing else expects: + +```go + existing, err := GetMonitor(instanceID, monitorID) + if err != nil { + return err + } + if existing == nil { + return fmt.Errorf("monitor not found") + } + if raw, present := upd["type"]; present { + if t, _ := raw.(string); t != existing.Type && (models.IsPassiveMonitor(t) || models.IsPassiveMonitor(existing.Type)) { + return fmt.Errorf("type cannot be changed to or from %s", map[bool]string{true: t, false: existing.Type}[models.IsPassiveMonitor(t)]) + } + } + if raw, present := upd["target"]; present && existing.Type == models.MonitorHeartbeat { + tg, ok := raw.(models.MonitorTarget) + if !ok { + return fmt.Errorf("target must be an object") + } + if err := validateHeartbeat(&tg); err != nil { + return err + } + upd["target"] = tg + } +``` + +- [ ] **Step 6: Run tests** + +Run: `cd server && go build ./... && go vet ./... && go test ./internal/services/` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add server/internal/services server/internal/models +git commit -m "feat(monitors): heartbeat token, ping recording and overdue verdict" +``` + +--- + +### Task 3: Public ping endpoints, rotate route, sweeper loop + +**Files:** +- Create: `server/internal/api/heartbeats.go` +- Create: `server/internal/api/heartbeats_test.go` +- Modify: `server/internal/api/handlers.go:~66` (next to `/public/status/:pageId`) +- Modify: `server/internal/api/monitors.go` (`registerMonitorRoutes`, `createMonitor`) +- Modify: `server/internal/api/scopes.go:~109` +- Modify: `server/internal/api/serverscope.go:~293` +- Create: `server/internal/metricsched/scheduler.go` +- Modify: `server/cmd/main.go:~261` +- Modify: `docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md` (path change) + +**Interfaces:** +- Consumes: `services.RecordHeartbeat`, `services.RotateHeartbeatToken`, `services.SweepHeartbeats`, `services.ErrHeartbeatNotFound`, `services.MaxHeartbeatBody`. +- Produces: routes `GET|POST /public/hb/:token`, `GET|POST /public/hb/:token/:kind`, `POST /api/monitors/:id/rotate-token` returning `{"heartbeat_token": string}`; `metricsched.Start(ctx)`; exported `metricsched.Sweep func(now time.Time)` hook list used in Task 9. + +- [ ] **Step 1: Write the failing handler test** + +`server/internal/api/heartbeats_test.go` tests the parts that need no database: path parsing and the unknown-kind 404. + +```go +package api + +import ( + "net/http" + "net/http/httptest" + "testing" + + "github.com/gin-gonic/gin" +) + +func TestHeartbeatKindFromPath(t *testing.T) { + cases := map[string]string{"": "ping", "start": "start", "fail": "fail", "bogus": ""} + for in, want := range cases { + if got := heartbeatKind(in); got != want { + t.Errorf("heartbeatKind(%q) = %q, want %q", in, got, want) + } + } +} + +func TestHeartbeatUnknownKindIs404(t *testing.T) { + gin.SetMode(gin.TestMode) + r := gin.New() + r.POST("/public/hb/:token/:kind", handleHeartbeat) + w := httptest.NewRecorder() + r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/public/hb/abc/explode", nil)) + if w.Code != http.StatusNotFound { + t.Fatalf("code = %d, want 404", w.Code) + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd server && go test ./internal/api/ -run Heartbeat` +Expected: compile error, `undefined: heartbeatKind`. + +- [ ] **Step 3: Implement `api/heartbeats.go`** + +```go +package api + +import ( + "errors" + "io" + "log" + "net/http" + "strconv" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" + "github.com/gin-gonic/gin" +) + +func heartbeatKind(seg string) string { + switch seg { + case "": + return services.HeartbeatPing + case "start": + return services.HeartbeatStart + case "fail": + return services.HeartbeatFail + } + return "" +} + +// rateLimitHeartbeat admits one request per token per second. A cron job +// pinging in a loop should not become a write per request, and a leaked URL +// should not be a way to hammer Mongo. Like the status page limiter it allows +// when Redis is down: a missed ping pages someone. +func rateLimitHeartbeat() gin.HandlerFunc { + return func(c *gin.Context) { + rdb := auth.Redis() + if rdb == nil { + c.Next() + return + } + key := "vantage:hbrl:" + services.HashHeartbeatToken(c.Param("token")) + ":" + strconv.FormatInt(time.Now().Unix(), 10) + count, err := rdb.Incr(c.Request.Context(), key).Result() + if err != nil { + c.Next() + return + } + if count == 1 { + rdb.Expire(c.Request.Context(), key, 2*time.Second) + } + if count > 1 { + c.Header("Retry-After", "1") + c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "too many requests", "code": "rate_limited"}) + return + } + c.Next() + } +} + +// handleHeartbeat records a push from a job. It is mounted on the gin root +// under /public for the same reasons as the status page (see +// getPublicStatusPage): no session, no token, no licence gate, and /public is +// already routed to this server by every deployment. +// +// Unknown token, disabled monitor and unknown kind all answer the same 404. +func handleHeartbeat(c *gin.Context) { + kind := heartbeatKind(c.Param("kind")) + if kind == "" { + c.String(http.StatusNotFound, "not found") + return + } + var body string + if kind == services.HeartbeatFail && c.Request.Body != nil { + b, _ := io.ReadAll(io.LimitReader(c.Request.Body, services.MaxHeartbeatBody)) + body = string(b) + } + err := services.RecordHeartbeat(c.Param("token"), kind, body, time.Now()) + if errors.Is(err, services.ErrHeartbeatNotFound) { + c.String(http.StatusNotFound, "not found") + return + } + if err != nil { + log.Printf("heartbeat: %v", err) + c.String(http.StatusInternalServerError, "error") + return + } + c.String(http.StatusOK, "OK") +} + +// rotateHeartbeatToken godoc +// +// @Summary Rotate a heartbeat monitor's ping token +// @Description Issues a new token and invalidates the old ping URL immediately. The token is returned only in this response. +// @Tags monitors +// @Produce json +// @Param id path string true "Monitor ID" +// @Success 200 {object} object{heartbeat_token=string} +// @Failure 404 {object} ErrorResponse +// @Failure 500 {object} ErrorResponse +// @Security cookieAuth +// @Security bearerAuth +// @Router /monitors/{id}/rotate-token [post] +func rotateHeartbeatToken(c *gin.Context) { + tok, err := services.RotateHeartbeatToken(auth.InstanceID(c), c.Param("id")) + if errors.Is(err, services.ErrHeartbeatNotFound) { + c.JSON(http.StatusNotFound, gin.H{"error": "heartbeat monitor not found"}) + return + } + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + c.JSON(http.StatusOK, gin.H{"heartbeat_token": tok}) +} +``` + +- [ ] **Step 4: Register routes and declarations** + +`handlers.go`, directly under the `/public/status/:pageId` line: + +```go + hb := r.Group("/public/hb/:token", rateLimitHeartbeat()) + { + hb.GET("", handleHeartbeat) + hb.POST("", handleHeartbeat) + hb.GET("/:kind", handleHeartbeat) + hb.POST("/:kind", handleHeartbeat) + } +``` + +`monitors.go` `registerMonitorRoutes`: `g.POST("/monitors/:id/rotate-token", rotateHeartbeatToken)`. + +`scopes.go` after `"GET /api/monitors/:id/samples"`: `"POST /api/monitors/:id/rotate-token": "monitors:write",` + +`serverscope.go` next to `DELETE /api/monitors/:id`: + +```go + // Rotating a ping token touches no server and returns only the token. + "POST /api/monitors/:id/rotate-token": fleetWide, +``` + +Also check that `RequireActiveLicense` in `licence.go` gates the rotate POST (deny by default). Leave it gated, matching create/update. + +In `createMonitor` in `api/monitors.go`, map validation errors to 400 instead of 500. Services return `fmt.Errorf` for validation, so wrap them: add `var ErrInvalidMonitor = errors.New("invalid monitor")` in services, return `fmt.Errorf("%w: period_sec must be at least 60", ErrInvalidMonitor)` from `validateHeartbeat` (and later from the metric validators), and in the handler: + +```go + if errors.Is(err, services.ErrInvalidMonitor) { + c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } +``` + +Apply the same mapping in `updateMonitor`. Update `TestValidateHeartbeat` to also assert `errors.Is(err, ErrInvalidMonitor)`. + +- [ ] **Step 5: Create `metricsched`** + +`server/internal/metricsched/scheduler.go`: + +```go +// Package metricsched sweeps passive monitors - heartbeats and, from phase 2, +// metric monitors - on a fixed tick. Nothing here runs a check: it reads what +// pings and agents already delivered and decides what is overdue or breaching. +package metricsched + +import ( + "context" + "log" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" +) + +const tick = 30 * time.Second + +func Start(ctx context.Context) { + go func() { + t := time.NewTicker(tick) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case now := <-t.C: + sweep(now) + } + } + }() +} + +// sweep recovers per sweep so one bad document cannot stop alerting for the +// whole instance until the next deploy. +func sweep(now time.Time) { + defer func() { + if r := recover(); r != nil { + log.Printf("metricsched: sweep panic: %v", r) + } + }() + services.SweepHeartbeats(now) +} +``` + +`cmd/main.go`: add `metricsched.Start(jobCtx)` after `monitorsched.Start(jobCtx)` and import the package. + +- [ ] **Step 6: Update the spec's path** + +In the spec's "Public endpoints" section, replace `/hb/:token` with `/public/hb/:token` in all three bullets and add: "Mounted under /public because every deployment already routes that prefix to the server." + +- [ ] **Step 7: Run tests** + +Run: `cd server && go build ./... && go vet ./... && go test ./internal/api/ ./internal/services/` +Expected: PASS, including `TestRegisteredRoutesPassBootAssertions`. + +- [ ] **Step 8: Commit** + +```bash +git add server docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md +git commit -m "feat(monitors): public heartbeat ping endpoints, token rotation and sweeper" +``` + +--- + +### Task 4: Heartbeat UI + +**Files:** +- Modify: `web/lib/api.ts:48-110, ~1026` +- Modify: `web/components/monitors/MonitorForm.tsx` +- Create: `web/components/monitors/HeartbeatUrlPanel.tsx` +- Modify: `web/app/(app)/monitors/page.tsx` +- Modify: `web/app/(app)/monitors/[id]/page.tsx` +- Modify: `web/app/(app)/monitors/new/page.tsx` + +**Interfaces:** +- Consumes: `POST /api/monitors` response `heartbeat_token`; `POST /api/monitors/:id/rotate-token`; `state.last_ping_at`, `state.started_at`, `target.period_sec`, `target.grace_sec`. +- Produces: `api.rotateHeartbeatToken(monitorId): Promise<{heartbeat_token: string}>`, `heartbeatUrl(token: string): string`, and the `HeartbeatUrlPanel` component `{ token: string }`. + +- [ ] **Step 1: API types** + +In `web/lib/api.ts`: + +```ts +export type MonitorType = "http" | "tcp" | "icmp" | "tls" | "heartbeat" | "metric"; +``` + +Add to `MonitorTarget`: `period_sec?: number; grace_sec?: number; selector?: Record; metric?: MetricKind; threshold?: number; mount?: string;`, and above it: + +```ts +export type MetricKind = + | "disk_pct" | "disk_free_gb" | "mem_pct" | "load_per_core" + | "unit_failed" | "container_unhealthy" | "reboot_pending_days" | "agent_offline_min"; +``` + +Add to `MonitorState`: `last_ping_at?: string; started_at?: string;`. Add to `Monitor` and `MonitorInput`: `for_sec?: number;`. Add to `Monitor`: `heartbeat_token?: string;`. Add to `Incident`: `server_id?: string;`. + +Next to the other monitor calls: + +```ts + rotateHeartbeatToken(monitorId: string) { + return request<{ heartbeat_token: string }>(`/monitors/${monitorId}/rotate-token`, { method: "POST" }); + }, +``` + +Export a helper: + +```ts +/** The ping URL a job calls. Same origin as the app: /public is routed to the server everywhere. */ +export function heartbeatUrl(token: string): string { + return `${window.location.origin}/public/hb/${token}`; +} +``` + +- [ ] **Step 2: `HeartbeatUrlPanel.tsx`** + +```tsx +"use client"; + +import { heartbeatUrl } from "@/lib/api"; +import { Button, useToast } from "@/components/ui"; + +/** Shown once, straight after create or rotate: the token is not retrievable later. */ +export function HeartbeatUrlPanel({ token }: { token: string }) { + const toast = useToast(); + const url = heartbeatUrl(token); + const copy = async () => { + await navigator.clipboard.writeText(url); + toast.success("Ping URL copied"); + }; + return ( +
+

+ Copy this URL now. It is not shown again; rotate the token if you lose it. +

+
+ {url} + +
+ 
+{`# success
+curl -fsS -m 10 --retry 3 ${url}
+# mark start, to measure duration
+curl -fsS -m 10 ${url}/start
+# report failure with output
+your-job 2>&1 | tail -c 1024 | curl -fsS -m 10 --data-binary @- ${url}/fail`}
+
+
+ ); +} +``` + +Check the real `useToast` API in `web/components/ui/Toast.tsx` and the `Button` `size` prop in `Button.tsx`. Adjust the calls to match; don't add new props. + +- [ ] **Step 3: Form support in `MonitorForm.tsx`** + +- Add a `heartbeat` entry to `typeCopy`: `{ title: "Heartbeat", blurb: "Your job calls a URL; alert when it stops.", target: "Ping URL" }`. (The `metric` entry comes in Task 11. Until then, filter `metric` out of the picker: `(Object.keys(typeCopy) as MonitorType[])` only lists keys present, so just don't add it yet. `typeCopy` needs to become `Partial>` or TypeScript will complain.) +- State: `const [periodMin, setPeriodMin] = useState(Math.round((initial?.target.period_sec ?? 3600) / 60));` and `const [graceMin, setGraceMin] = useState(Math.round((initial?.target.grace_sec ?? 300) / 60));` +- In the target builder (line ~121 chain) add `else if (type === "heartbeat") { target.period_sec = periodMin * 60; target.grace_sec = graceMin * 60; }`. +- Render, when `type === "heartbeat"`: two number inputs, "Expected every (minutes)" (min 1) and "Grace (minutes)" (min 0), using the same input components/classes as the existing TLS warn-days field. +- Hide the interval, runner and retries fields when `type === "heartbeat"`. Wrap their JSX in `{type !== "heartbeat" && type !== "metric" && (...)}`. +- When editing (`initial` set) and `initial.type` is heartbeat or metric, disable the type picker (the server rejects the change). + +- [ ] **Step 4: Show the token after create** + +In `web/app/(app)/monitors/new/page.tsx`, after `api.createMonitor` resolves: if `created.heartbeat_token` is set, route to `/monitors/${id}?token=${encodeURIComponent(created.heartbeat_token)}` instead of the usual destination. On the detail page, read `useSearchParams().get("token")`, keep it in state, then call `router.replace` to the same path without the query, so the token doesn't sit in browser history longer than one navigation. + +- [ ] **Step 5: Detail page** + +In `web/app/(app)/monitors/[id]/page.tsx`, when `monitor.type === "heartbeat"`: +- If a token is in state, render `` at the top. +- A card showing "Last ping" (`relativeTime(state.last_ping_at)` from MonitorVisuals, or "Never" when unset), "Expected every" / "Grace" in minutes, and "Running since" when `state.started_at` is set. +- A "Rotate token" button that opens the existing `ConfirmDialog` ("The current URL stops working immediately."), then `api.rotateHeartbeatToken(id)` sets the token state. +- The latency chart's label reads "Duration" for heartbeats. Find where the chart title or axis says latency and switch on type. + +- [ ] **Step 6: List page** + +In `web/app/(app)/monitors/page.tsx`, where a row shows its target (URL or host), show `Last ping ${relativeTime(m.state.last_ping_at)}` or "Waiting for first ping" for heartbeats. + +- [ ] **Step 7: Verify** + +Run: `cd web && npx tsc --noEmit && npm run lint` +Expected: no errors. + +- [ ] **Step 8: Commit** + +```bash +git add web +git commit -m "feat(web): heartbeat monitor form, ping URL panel and token rotation" +``` + +--- + +## Phase 2: Metric alerts + +### Task 5: Track when a reboot became required + +**Files:** +- Modify: `server/internal/services/inventory.go` +- Create: `server/internal/services/inventory_reboot_test.go` + +**Interfaces:** +- Consumes: `models.Inventory.RebootRequiredSince` (Task 1). +- Produces: `inventory.reboot_required_since` kept current; pure `rebootSinceUpdate(prevRequired bool, prevSince *time.Time, nowRequired bool, now time.Time) (set *time.Time, unset bool)`. + +- [ ] **Step 1: Failing test** + +```go +package services + +import ( + "testing" + "time" +) + +func TestRebootSinceUpdate(t *testing.T) { + now := time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC) + earlier := now.Add(-72 * time.Hour) + + if set, unset := rebootSinceUpdate(false, nil, true, now); set == nil || !set.Equal(now) || unset { + t.Fatal("turning on must stamp now") + } + if set, unset := rebootSinceUpdate(true, &earlier, true, now); set != nil || unset { + t.Fatal("staying on must keep the original stamp") + } + if set, unset := rebootSinceUpdate(true, nil, true, now); set == nil { + t.Fatal("on with no stamp (pre-upgrade data) must stamp now") + } + if set, unset := rebootSinceUpdate(true, &earlier, false, now); set != nil || !unset { + t.Fatal("turning off must clear the stamp") + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd server && go test ./internal/services/ -run TestRebootSinceUpdate` +Expected: `undefined: rebootSinceUpdate`. + +- [ ] **Step 3: Implement** + +In `inventory.go`: + +```go +// rebootSinceUpdate decides how reboot_required_since changes. The stamp is +// kept from the first report that needed a reboot, so "pending for 7 days" +// means seven days, not seven days since the last static report. +func rebootSinceUpdate(prevRequired bool, prevSince *time.Time, nowRequired bool, now time.Time) (*time.Time, bool) { + if !nowRequired { + return nil, prevSince != nil || prevRequired + } + if prevSince == nil { + return &now, false + } + return nil, false +} +``` + +In `StoreInventory` inside `if r.IncludeStatic`, before the update: + +```go + var prev struct { + Inventory struct { + RebootRequired bool `bson:"reboot_required"` + RebootRequiredSince *time.Time `bson:"reboot_required_since"` + } `bson:"inventory"` + } + _ = db.Col("servers").FindOne(ctx, bson.M{"server_id": serverID}, + options.FindOne().SetProjection(bson.M{"inventory.reboot_required": 1, "inventory.reboot_required_since": 1})).Decode(&prev) + if since, clear := rebootSinceUpdate(prev.Inventory.RebootRequired, prev.Inventory.RebootRequiredSince, r.RebootRequired, now); since != nil { + set["inventory.reboot_required_since"] = *since + } else if clear { + unset = bson.M{"inventory.reboot_required_since": ""} + } +``` + +Declare `var unset bson.M` before the `if`, and build the update as `upd := bson.M{"$set": set}; if unset != nil { upd["$unset"] = unset }`. Import `options`. + +- [ ] **Step 4: Run tests** + +Run: `cd server && go test ./internal/services/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/services/inventory.go server/internal/services/inventory_reboot_test.go +git commit -m "feat(inventory): record when a reboot became required" +``` + +--- + +### Task 6: Metric evaluators and state decisions (pure) + +**Files:** +- Create: `server/internal/services/metricrules.go` +- Create: `server/internal/services/metricrules_test.go` + +**Interfaces:** +- Produces: + - `const MetricDiskPct, MetricDiskFreeGB, MetricMemPct, MetricLoadPerCore, MetricUnitFailed, MetricContainerUnhealthy, MetricRebootPendingDays, MetricAgentOfflineMin` (string values as in the spec) + - `const metricStaleAfter = 5 * time.Minute` + - `func metricNeedsWorkloads(kind string) bool` + - `func EvaluateMetric(t models.MonitorTarget, srv models.Server, wls []models.Workload, now time.Time) (breach bool, value float64, message string, ok bool)`, where `ok=false` means stale or missing data, so skip + - `func nextServerState(prev *models.MonitorServerState, breach bool, forSec int, now time.Time) (status string, breachSince *time.Time)` + - `func rollupParent(states []models.MonitorServerState) (status, message string)` + - `func validateMetric(m *models.Monitor) error` + +- [ ] **Step 1: Failing tests** + +```go +package services + +import ( + "errors" + "strings" + "testing" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +var evalNow = time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC) + +func fresh() *time.Time { t := evalNow.Add(-time.Minute); return &t } + +func srvWith(inv models.Inventory) models.Server { + if inv.MetricsAt == nil { + inv.MetricsAt = fresh() + } + ls := evalNow.Add(-30 * time.Second) + return models.Server{ServerID: "s1", Hostname: "web-01", Inventory: &inv, LastSeen: &ls} +} + +func TestEvaluateMetric(t *testing.T) { + disk := models.Inventory{Partitions: []models.Partition{ + {Mountpoint: "/", TotalBytes: 100e9, UsedBytes: 50e9}, + {Mountpoint: "/var", TotalBytes: 100e9, UsedBytes: 95e9}, + }} + stale := evalNow.Add(-10 * time.Minute) + since := evalNow.Add(-8 * 24 * time.Hour) + oldSeen := evalNow.Add(-20 * time.Minute) + + cases := []struct { + name string + t models.MonitorTarget + srv models.Server + wls []models.Workload + wantBreach bool + wantOK bool + msgHas string + }{ + {"disk pct any mount breaches", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 90}, srvWith(disk), nil, true, true, "/var"}, + {"disk pct named mount clear", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 90, Mount: "/"}, srvWith(disk), nil, false, true, "/"}, + {"disk pct missing mount skips", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 90, Mount: "/data"}, srvWith(disk), nil, false, false, ""}, + {"disk free gb breaches", models.MonitorTarget{Metric: MetricDiskFreeGB, Threshold: 10}, srvWith(disk), nil, true, true, "GB free"}, + {"zero total partition ignored", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 1}, srvWith(models.Inventory{Partitions: []models.Partition{{Mountpoint: "/proc"}}}), nil, false, false, ""}, + {"mem pct", models.MonitorTarget{Metric: MetricMemPct, Threshold: 80}, srvWith(models.Inventory{Memory: models.MemInfo{TotalBytes: 100, UsedBytes: 85}}), nil, true, true, "memory"}, + {"load per core", models.MonitorTarget{Metric: MetricLoadPerCore, Threshold: 1.5}, srvWith(models.Inventory{CPU: models.CPUInfo{Cores: 4, Load1: 8}}), nil, true, true, "load"}, + {"load no cores skips", models.MonitorTarget{Metric: MetricLoadPerCore, Threshold: 1.5}, srvWith(models.Inventory{CPU: models.CPUInfo{Load1: 8}}), nil, false, false, ""}, + {"stale inventory skips", models.MonitorTarget{Metric: MetricMemPct, Threshold: 1}, srvWith(models.Inventory{MetricsAt: &stale, Memory: models.MemInfo{TotalBytes: 100, UsedBytes: 99}}), nil, false, false, ""}, + {"unit failed", models.MonitorTarget{Metric: MetricUnitFailed}, srvWith(models.Inventory{}), []models.Workload{{Kind: "unit", Name: "backup.service", State: "failed"}}, true, true, "backup.service"}, + {"container unhealthy clear", models.MonitorTarget{Metric: MetricContainerUnhealthy}, srvWith(models.Inventory{}), []models.Workload{{Kind: "container", Name: "db", Health: "healthy"}}, false, true, ""}, + {"reboot pending days", models.MonitorTarget{Metric: MetricRebootPendingDays, Threshold: 7}, srvWith(models.Inventory{RebootRequired: true, RebootRequiredSince: &since}), nil, true, true, "reboot"}, + {"reboot not required", models.MonitorTarget{Metric: MetricRebootPendingDays, Threshold: 7}, srvWith(models.Inventory{}), nil, false, true, ""}, + {"agent offline ignores stale inventory", models.MonitorTarget{Metric: MetricAgentOfflineMin, Threshold: 10}, + func() models.Server { s := srvWith(models.Inventory{MetricsAt: &stale}); s.LastSeen = &oldSeen; return s }(), nil, true, true, "not seen"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + breach, _, msg, ok := EvaluateMetric(c.t, c.srv, c.wls, evalNow) + if breach != c.wantBreach || ok != c.wantOK || !strings.Contains(msg, c.msgHas) { + t.Fatalf("got breach=%v ok=%v msg=%q", breach, ok, msg) + } + }) + } +} + +func TestNextServerState(t *testing.T) { + past := func(d time.Duration) *time.Time { v := evalNow.Add(-d); return &v } + + st, since := nextServerState(nil, true, 300, evalNow) + if st != models.StatusPending || since == nil || !since.Equal(evalNow) { + t.Fatalf("new breach: %s %v", st, since) + } + st, _ = nextServerState(&models.MonitorServerState{Status: models.StatusPending, BreachSince: past(2 * time.Minute)}, true, 300, evalNow) + if st != models.StatusPending { + t.Fatalf("under for_sec should stay pending, got %s", st) + } + st, since = nextServerState(&models.MonitorServerState{Status: models.StatusPending, BreachSince: past(6 * time.Minute)}, true, 300, evalNow) + if st != models.StatusDown || !since.Equal(*past(6 * time.Minute)) { + t.Fatalf("over for_sec should be down keeping since, got %s %v", st, since) + } + st, _ = nextServerState(nil, true, 0, evalNow) + if st != models.StatusDown { + t.Fatalf("for_sec 0 is down immediately, got %s", st) + } + st, since = nextServerState(&models.MonitorServerState{Status: models.StatusDown, BreachSince: past(time.Hour)}, false, 300, evalNow) + if st != models.StatusUp || since != nil { + t.Fatalf("clear should be up with no since, got %s %v", st, since) + } +} + +func TestRollupParent(t *testing.T) { + if s, m := rollupParent(nil); s != models.StatusUp || m != "no matching servers" { + t.Fatalf("empty: %s %q", s, m) + } + states := []models.MonitorServerState{{Status: models.StatusUp}, {Status: models.StatusPending}, {Status: models.StatusDown}} + if s, m := rollupParent(states); s != models.StatusDown || m != "1 of 3 servers breaching" { + t.Fatalf("got %s %q", s, m) + } + if s, _ := rollupParent(states[:2]); s != models.StatusPending { + t.Fatalf("pending wins over up, got %s", s) + } +} + +func TestValidateMetric(t *testing.T) { + bad := []models.MonitorTarget{ + {Metric: "cpu_magic", Threshold: 1}, + {Metric: MetricDiskPct, Threshold: 0}, + {Metric: MetricDiskPct, Threshold: 101}, + {Metric: MetricDiskPct, Threshold: 90, Mount: "var"}, + } + for _, tg := range bad { + m := models.Monitor{Type: models.MonitorMetric, Target: tg} + if err := validateMetric(&m); !errors.Is(err, ErrInvalidMonitor) { + t.Errorf("%+v: want ErrInvalidMonitor, got %v", tg, err) + } + } + ok := models.Monitor{Type: models.MonitorMetric, ForSec: -5, Target: models.MonitorTarget{Metric: MetricUnitFailed}} + if err := validateMetric(&ok); err != nil || ok.ForSec != 0 { + t.Fatalf("unit_failed needs no threshold and negative for_sec clamps to 0: %v %d", err, ok.ForSec) + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd server && go test ./internal/services/ -run 'EvaluateMetric|NextServerState|RollupParent|ValidateMetric'` +Expected: `undefined: EvaluateMetric`. + +- [ ] **Step 3: Implement `metricrules.go`** + +```go +package services + +import ( + "fmt" + "path" + "strings" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +const ( + MetricDiskPct = "disk_pct" + MetricDiskFreeGB = "disk_free_gb" + MetricMemPct = "mem_pct" + MetricLoadPerCore = "load_per_core" + MetricUnitFailed = "unit_failed" + MetricContainerUnhealthy = "container_unhealthy" + MetricRebootPendingDays = "reboot_pending_days" + MetricAgentOfflineMin = "agent_offline_min" +) + +// metricStaleAfter is how old an agent's metrics can be before a rule stops +// judging them. A dead agent's last report must not hold an alert open or +// clear one; agent_offline_min is the rule for a dead agent. +const metricStaleAfter = 5 * time.Minute + +func metricNeedsWorkloads(kind string) bool { + return kind == MetricUnitFailed || kind == MetricContainerUnhealthy +} + +func metricUsesThreshold(kind string) bool { + return !metricNeedsWorkloads(kind) +} + +func validateMetric(m *models.Monitor) error { + t := &m.Target + switch t.Metric { + case MetricDiskPct, MetricDiskFreeGB, MetricMemPct, MetricLoadPerCore, + MetricUnitFailed, MetricContainerUnhealthy, MetricRebootPendingDays, MetricAgentOfflineMin: + default: + return fmt.Errorf("%w: unknown metric %q", ErrInvalidMonitor, t.Metric) + } + if metricUsesThreshold(t.Metric) && t.Threshold <= 0 { + return fmt.Errorf("%w: threshold must be greater than 0", ErrInvalidMonitor) + } + if (t.Metric == MetricDiskPct || t.Metric == MetricMemPct) && t.Threshold > 100 { + return fmt.Errorf("%w: threshold must be 100 or less", ErrInvalidMonitor) + } + if t.Mount != "" && !path.IsAbs(t.Mount) { + return fmt.Errorf("%w: mount must be an absolute path", ErrInvalidMonitor) + } + if m.ForSec < 0 { + m.ForSec = 0 + } + return nil +} + +// EvaluateMetric judges one server against one rule. ok=false means there is +// nothing trustworthy to judge - stale metrics, a mount that does not exist on +// this host - and the caller keeps the previous state rather than guessing. +func EvaluateMetric(t models.MonitorTarget, srv models.Server, wls []models.Workload, now time.Time) (bool, float64, string, bool) { + if t.Metric == MetricAgentOfflineMin { + if srv.LastSeen == nil { + return false, 0, "", false + } + mins := now.Sub(*srv.LastSeen).Minutes() + return mins >= t.Threshold, mins, fmt.Sprintf("agent not seen for %.0f minutes", mins), true + } + inv := srv.Inventory + if inv == nil || inv.MetricsAt == nil || now.Sub(*inv.MetricsAt) > metricStaleAfter { + return false, 0, "", false + } + + switch t.Metric { + case MetricDiskPct, MetricDiskFreeGB: + return evalDisk(t, inv.Partitions) + case MetricMemPct: + if inv.Memory.TotalBytes == 0 { + return false, 0, "", false + } + pct := float64(inv.Memory.UsedBytes) / float64(inv.Memory.TotalBytes) * 100 + return pct >= t.Threshold, pct, fmt.Sprintf("memory %.1f%% used", pct), true + case MetricLoadPerCore: + if inv.CPU.Cores == 0 { + return false, 0, "", false + } + v := inv.CPU.Load1 / float64(inv.CPU.Cores) + return v >= t.Threshold, v, fmt.Sprintf("load %.2f per core", v), true + case MetricRebootPendingDays: + if !inv.RebootRequired || inv.RebootRequiredSince == nil { + return false, 0, "", true + } + days := now.Sub(*inv.RebootRequiredSince).Hours() / 24 + return days >= t.Threshold, days, fmt.Sprintf("reboot pending for %.0f days", days), true + case MetricUnitFailed: + return evalWorkloads(wls, "unit", func(w models.Workload) bool { return w.State == "failed" }, "failed") + case MetricContainerUnhealthy: + return evalWorkloads(wls, "container", func(w models.Workload) bool { return w.Health == "unhealthy" }, "unhealthy") + } + return false, 0, "", false +} + +// evalDisk reports the worst matching partition, so "any mount" names the one +// that is actually full. +func evalDisk(t models.MonitorTarget, parts []models.Partition) (bool, float64, string, bool) { + found := false + var worstBreach bool + var worstVal float64 + var worstMsg string + for _, p := range parts { + if p.TotalBytes == 0 || (t.Mount != "" && p.Mountpoint != t.Mount) { + continue + } + var breach bool + var val float64 + var msg string + if t.Metric == MetricDiskPct { + val = float64(p.UsedBytes) / float64(p.TotalBytes) * 100 + breach = val >= t.Threshold + msg = fmt.Sprintf("%s %.1f%% used", p.Mountpoint, val) + } else { + val = float64(p.TotalBytes-p.UsedBytes) / 1e9 + breach = val <= t.Threshold + msg = fmt.Sprintf("%s %.1f GB free", p.Mountpoint, val) + } + worse := !found || + (t.Metric == MetricDiskPct && val > worstVal) || + (t.Metric == MetricDiskFreeGB && val < worstVal) + if worse { + worstBreach, worstVal, worstMsg = breach, val, msg + } + found = true + } + if !found { + return false, 0, "", false + } + return worstBreach, worstVal, worstMsg, true +} + +func evalWorkloads(wls []models.Workload, kind string, bad func(models.Workload) bool, word string) (bool, float64, string, bool) { + var names []string + for _, w := range wls { + if w.Kind == kind && bad(w) { + names = append(names, w.Name) + } + } + if len(names) == 0 { + return false, 0, "", true + } + return true, float64(len(names)), fmt.Sprintf("%s %s: %s", kind, word, strings.Join(names, ", ")), true +} + +// nextServerState applies the "for N seconds" gate. BreachSince is the first +// sweep that saw the condition, so the gate measures continuous breach and a +// single clear sweep restarts it. +func nextServerState(prev *models.MonitorServerState, breach bool, forSec int, now time.Time) (string, *time.Time) { + if !breach { + return models.StatusUp, nil + } + since := now + if prev != nil && prev.BreachSince != nil { + since = *prev.BreachSince + } + if now.Sub(since) >= time.Duration(forSec)*time.Second { + return models.StatusDown, &since + } + return models.StatusPending, &since +} + +func rollupParent(states []models.MonitorServerState) (string, string) { + if len(states) == 0 { + return models.StatusUp, "no matching servers" + } + down, pending := 0, 0 + for _, s := range states { + switch s.Status { + case models.StatusDown: + down++ + case models.StatusPending: + pending++ + } + } + msg := fmt.Sprintf("%d of %d servers breaching", down, len(states)) + switch { + case down > 0: + return models.StatusDown, msg + case pending > 0: + return models.StatusPending, msg + } + return models.StatusUp, msg +} +``` + +Make the tests match the implementation exactly: the `container unhealthy clear` case expects `msgHas ""` (always true) and `ok=true`. The `reboot not required` case expects `ok=true` and no breach. The `agent offline` case expects `msgHas "not seen"`. + +- [ ] **Step 4: Run tests** + +Run: `cd server && go test ./internal/services/ -run 'EvaluateMetric|NextServerState|RollupParent|ValidateMetric' -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/services/metricrules.go server/internal/services/metricrules_test.go +git commit -m "feat(monitors): metric rule evaluators and per-server state decisions" +``` + +--- + +### Task 7: Metric validation on create/update, tag-scope check, MCP + +**Files:** +- Modify: `server/internal/services/monitors.go` (`CreateMonitor`, `UpdateMonitor`) +- Create: `server/internal/services/metricscope_test.go` +- Modify: `server/internal/mcp/tools_create.go:89-140, 257-265` + +**Interfaces:** +- Consumes: `validateMetric`, `ErrInvalidMonitor`. +- Produces: `func selectorWithinScope(sel, tokenScope map[string]string) bool`; `ErrMonitorOutOfScope = errors.New("selector is outside this credential's tag scope")`. The API maps it to 403. + +- [ ] **Step 1: Failing test** + +```go +package services + +import "testing" + +// A restricted token must not be able to create a rule that watches servers it +// cannot see: the per-server table and incident messages would disclose them. +func TestSelectorWithinScope(t *testing.T) { + scope := map[string]string{"env": "prod"} + cases := []struct { + sel map[string]string + want bool + }{ + {nil, false}, + {map[string]string{"env": "dev"}, false}, + {map[string]string{"role": "web"}, false}, + {map[string]string{"env": "prod"}, true}, + {map[string]string{"env": "prod", "role": "web"}, true}, + } + for _, c := range cases { + if got := selectorWithinScope(c.sel, scope); got != c.want { + t.Errorf("sel %v: got %v want %v", c.sel, got, c.want) + } + } + if !selectorWithinScope(nil, nil) { + t.Error("an unrestricted credential may use any selector, including the whole fleet") + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd server && go test ./internal/services/ -run TestSelectorWithinScope` +Expected: `undefined: selectorWithinScope`. + +- [ ] **Step 3: Implement** + +In `metricrules.go`: + +```go +var ErrMonitorOutOfScope = errors.New("selector is outside this credential's tag scope") + +// selectorWithinScope holds when every server the selector can match is also +// inside the token's scope - true exactly when the selector pins every pair +// the scope does. +func selectorWithinScope(sel, tokenScope map[string]string) bool { + for k, v := range tokenScope { + if sel[k] != v { + return false + } + } + return true +} +``` + +(add the `errors` import). In `CreateMonitor`, next to the heartbeat block: + +```go + if m.Type == models.MonitorMetric { + if err := validateMetric(m); err != nil { + return nil, err + } + if !selectorWithinScope(m.Target.Selector, tokenScope) { + return nil, ErrMonitorOutOfScope + } + m.Runner = models.RunnerServer + m.IntervalSec = 0 + } +``` + +In `UpdateMonitor`, extend the target block so metric monitors are covered too: + +```go + if raw, present := upd["target"]; present && models.IsPassiveMonitor(existing.Type) { + tg, ok := raw.(models.MonitorTarget) + if !ok { + return fmt.Errorf("%w: target must be an object", ErrInvalidMonitor) + } + switch existing.Type { + case models.MonitorHeartbeat: + if err := validateHeartbeat(&tg); err != nil { + return err + } + case models.MonitorMetric: + probe := models.Monitor{Type: existing.Type, Target: tg, ForSec: existing.ForSec} + if err := validateMetric(&probe); err != nil { + return err + } + if !selectorWithinScope(tg.Selector, tokenScope) { + return ErrMonitorOutOfScope + } + } + upd["target"] = tg + } + // Editing any field of a metric monitor the token could not have created is + // refused, so a restricted token cannot rename or disable a fleet-wide rule. + if existing.Type == models.MonitorMetric && !selectorWithinScope(existing.Target.Selector, tokenScope) { + return ErrMonitorOutOfScope + } +``` + +In `api/monitors.go`, add `for_sec` to the `updateMonitor` body struct (`ForSec *int \`json:"for_sec"\``, then `upd["for_sec"]`) and to its godoc object. In both `createMonitor` and `updateMonitor`: + +```go + if errors.Is(err, services.ErrMonitorOutOfScope) { + c.JSON(http.StatusForbidden, gin.H{"error": err.Error()}) + return + } +``` + +Also block delete of an out-of-scope metric monitor. In `services.DeleteMonitor` add a `tokenScope map[string]string` parameter, load the monitor, and return `ErrMonitorOutOfScope` under the same condition. Update the API handler (`auth.ServerScope(c)`) and every other caller (`grep -rn "DeleteMonitor(" server/internal`). While there, extend `DeleteMonitor` to also `DeleteMany` from `monitor_server_states`. + +- [ ] **Step 4: MCP** + +In `buildMonitor` (tools_create.go:89), after the existing fields: + +```go + target.Metric = stringArg(rawTarget, "metric") + target.Mount = stringArg(rawTarget, "mount") + if n, ok := rawTarget["threshold"].(float64); ok { + target.Threshold = n + } + if n, ok := rawTarget["period_sec"].(float64); ok { + target.PeriodSec = int(n) + } + if n, ok := rawTarget["grace_sec"].(float64); ok { + target.GraceSec = int(n) + } + if sel, ok := rawTarget["selector"].(map[string]any); ok { + target.Selector = map[string]string{} + for k, v := range sel { + if s, ok := v.(string); ok { + target.Selector[k] = s + } + } + } +``` + +and read `for_sec` from `args` into `m.ForSec`. Update the tool's `type` description to "http, tcp, icmp, tls, heartbeat or metric". Update the `target` description to add: "heartbeat takes period_sec and optional grace_sec; metric takes metric (disk_pct, disk_free_gb, mem_pct, load_per_core, unit_failed, container_unhealthy, reboot_pending_days, agent_offline_min), threshold, optional mount and selector (tag map)". Add a `for_sec` integer arg. In the handler's return map add `"heartbeat_token": created.HeartbeatToken` when non-empty, with the note "Shown once." + +- [ ] **Step 5: Run tests** + +Run: `cd server && go build ./... && go vet ./... && go test ./internal/services/ ./internal/mcp/ ./internal/api/` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add server +git commit -m "feat(monitors): validate metric monitors and confine selectors to token scope" +``` + +--- + +### Task 8: Metric sweep + +**Files:** +- Create: `server/internal/services/metricsweep.go` +- Modify: `server/internal/services/migrate_instance.go:24-55` +- Modify: `server/internal/services/mfa_scoped_test.go` (add collection to the list, or copy the pattern into a new test) +- Modify: `server/internal/metricsched/scheduler.go` +- Modify: `server/cmd/main.go:~172` + +**Interfaces:** +- Consumes: `EvaluateMetric`, `nextServerState`, `rollupParent`, `metricNeedsWorkloads`, `applyTransition`, `resolveIncident`, `recordSample`, `ListServersFiltered`, `GetWorkloads(instanceID, serverID) (*models.ServerWorkloads, error)`. +- Produces: `func SweepMetricMonitors(now time.Time)`, `func EnsureMonitorServerStateIndexes() error`, `func ListMonitorServerStates(instanceID, monitorID string) ([]models.MonitorServerState, error)`. + +- [ ] **Step 1: Failing scoped-collection test** + +In `mfa_scoped_test.go`, rename nothing. Add a sibling test: + +```go +// Per-server metric state is tenant data and must be purged with its instance. +func TestMonitorServerStatesAreScoped(t *testing.T) { + for _, got := range ScopedCollections { + if got == "monitor_server_states" { + return + } + } + t.Error("monitor_server_states is not in ScopedCollections") +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd server && go test ./internal/services/ -run TestMonitorServerStatesAreScoped` +Expected: FAIL. + +- [ ] **Step 3: Add to `ScopedCollections`** + +Append `"monitor_server_states",` after `"monitor_samples",` in `migrate_instance.go`. Read the comment above the list first. If entries must also appear in a migration step, follow the pattern the MFA change used for `user_mfa` (`grep -rn user_mfa server/internal/services/migrate*`). + +- [ ] **Step 4: Implement `metricsweep.go`** + +```go +package services + +import ( + "context" + "log" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/mongo" + "go.mongodb.org/mongo-driver/v2/mongo/options" +) + +// EnsureMonitorServerStateIndexes declares the one key every read and upsert +// uses. Unique, so a double sweep across a leader handover cannot fork a +// server's state into two documents. +func EnsureMonitorServerStateIndexes() error { + _, err := db.Col("monitor_server_states").Indexes().CreateOne(context.Background(), mongo.IndexModel{ + Keys: bson.D{{Key: "monitor_id", Value: 1}, {Key: "server_id", Value: 1}}, + Options: options.Index().SetUnique(true), + }) + if err != nil { + log.Printf("warning: monitor_server_states indexes: %v", err) + } + return nil +} + +func ListMonitorServerStates(instanceID, monitorID string) ([]models.MonitorServerState, error) { + ctx, cancel := monCtx() + defer cancel() + cur, err := db.Col("monitor_server_states").Find(ctx, bson.M{"instance_id": instanceID, "monitor_id": monitorID}) + if err != nil { + return nil, err + } + out := []models.MonitorServerState{} + if err := cur.All(ctx, &out); err != nil { + return nil, err + } + return out, nil +} + +// ponytail: every metric monitor re-lists its servers each sweep; cache the +// fleet per instance per sweep if large fleets show up in the sweep log. +func SweepMetricMonitors(now time.Time) { + ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second) + defer cancel() + cur, err := db.Col("monitors").Find(ctx, bson.M{"type": models.MonitorMetric, "enabled": true}) + if err != nil { + log.Printf("metrics: list: %v", err) + return + } + var monitors []models.Monitor + if err := cur.All(ctx, &monitors); err != nil { + log.Printf("metrics: decode: %v", err) + return + } + for i := range monitors { + sweepOneMetric(ctx, &monitors[i], now) + } +} + +func sweepOneMetric(ctx context.Context, m *models.Monitor, now time.Time) { + defer func() { + if r := recover(); r != nil { + log.Printf("metrics: monitor %s panic: %v", m.MonitorID, r) + } + }() + + servers, err := ListServersFiltered(m.InstanceID, m.Target.Selector) + if err != nil { + log.Printf("metrics: servers for %s: %v", m.MonitorID, err) + return + } + prevStates, err := ListMonitorServerStates(m.InstanceID, m.MonitorID) + if err != nil { + log.Printf("metrics: states for %s: %v", m.MonitorID, err) + return + } + prevByServer := map[string]*models.MonitorServerState{} + for i := range prevStates { + prevByServer[prevStates[i].ServerID] = &prevStates[i] + } + + col := db.Col("monitor_server_states") + current := make([]models.MonitorServerState, 0, len(servers)) + matched := map[string]bool{} + for _, srv := range servers { + matched[srv.ServerID] = true + prev := prevByServer[srv.ServerID] + + var wls []models.Workload + if metricNeedsWorkloads(m.Target.Metric) { + if sw, err := GetWorkloads(m.InstanceID, srv.ServerID); err == nil && sw != nil { + wls = sw.Workloads + } + } + breach, value, msg, ok := EvaluateMetric(m.Target, srv, wls, now) + if !ok { + if prev != nil { + current = append(current, *prev) + } + continue + } + + status, since := nextServerState(prev, breach, m.ForSec, now) + st := models.MonitorServerState{ + InstanceID: m.InstanceID, MonitorID: m.MonitorID, ServerID: srv.ServerID, + Status: status, BreachSince: since, Value: value, Message: msg, UpdatedAt: now, + } + if _, err := col.ReplaceOne(ctx, bson.M{"monitor_id": m.MonitorID, "server_id": srv.ServerID}, st, + options.Replace().SetUpsert(true)); err != nil { + log.Printf("metrics: save state %s/%s: %v", m.MonitorID, srv.ServerID, err) + continue + } + current = append(current, st) + + // A server seen for the first time has no previous status to leave, so + // it can open an incident but never announce a recovery. + prevStatus := models.StatusPending + if prev != nil { + prevStatus = prev.Status + } + applyTransition(ctx, m, srv.ServerID, srv.Hostname, prevStatus, status, msg, now) + } + + // Servers that left the selector, or the fleet, did not recover: their + // incidents close quietly and their state goes. + for _, prev := range prevStates { + if matched[prev.ServerID] { + continue + } + if prev.Status == models.StatusDown { + resolveIncident(ctx, m, prev.ServerID, now) + } + col.DeleteOne(ctx, bson.M{"monitor_id": m.MonitorID, "server_id": prev.ServerID}) + } + + status, msg := rollupParent(current) + db.Col("monitors").UpdateOne(ctx, bson.M{"monitor_id": m.MonitorID}, bson.M{"$set": bson.M{ + "state.status": status, "state.message": msg, "state.last_check_at": now, + }}) + recordSample(ctx, m, status != models.StatusDown, 0, now) +} +``` + +Check `models.ServerWorkloads` for the field holding the slice (`grep -n "type ServerWorkloads" -A10 server/internal/models/workloads.go`) and use that name in place of `sw.Workloads` if it differs. + +A metric monitor has no notification of its own at the parent level: each server's transition notifies. The parent state is display only, so don't call `applyTransition` for it. + +- [ ] **Step 5: Wire the sweep and index** + +`metricsched/scheduler.go` `sweep`: add `services.SweepMetricMonitors(now)` after `SweepHeartbeats`. `cmd/main.go` next to `EnsureMonitorSampleIndexes`: + +```go + if err := services.EnsureMonitorServerStateIndexes(); err != nil { + log.Printf("warning: %v", err) + } +``` + +(match how the sibling call handles its error). + +- [ ] **Step 6: Run tests** + +Run: `cd server && go build ./... && go vet ./... && go test ./...` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add server +git commit -m "feat(monitors): sweep metric monitors per server with incidents per breach" +``` + +--- + +### Task 9: Per-server states API and scoped incidents + +**Files:** +- Modify: `server/internal/api/monitors.go` +- Modify: `server/internal/api/scopes.go` +- Modify: `server/internal/api/serverscope.go:295-301` +- Create: `server/internal/services/metricfilter_test.go` + +**Interfaces:** +- Consumes: `ListMonitorServerStates`, `VisibleServerIDs`, `ListServers` or `ListServersFiltered` for hostnames. +- Produces: `GET /api/monitors/:id/servers` returning `[]models.MonitorServerState` with `hostname`; `func filterByVisibleServer[T any](items []T, serverID func(T) string, visible map[string]bool, restricted bool) []T`. + +- [ ] **Step 1: Failing test** + +```go +package services + +import ( + "testing" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +func TestFilterByVisibleServer(t *testing.T) { + incs := []models.Incident{{IncidentID: "a"}, {IncidentID: "b", ServerID: "s1"}, {IncidentID: "c", ServerID: "s2"}} + id := func(i models.Incident) string { return i.ServerID } + + if got := filterByVisibleServer(incs, id, nil, false); len(got) != 3 { + t.Fatalf("unrestricted keeps all, got %d", len(got)) + } + got := filterByVisibleServer(incs, id, map[string]bool{"s1": true}, true) + if len(got) != 2 || got[0].IncidentID != "a" || got[1].IncidentID != "b" { + t.Fatalf("restricted keeps serverless and visible only, got %+v", got) + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd server && go test ./internal/services/ -run TestFilterByVisibleServer` +Expected: `undefined: filterByVisibleServer`. + +- [ ] **Step 3: Implement the filter** + +In `tokenscope.go`: + +```go +// FilterByVisibleServer drops items tied to a server the credential cannot +// see. Items with no server (every non-metric incident) are always kept. +func FilterByVisibleServer[T any](items []T, serverID func(T) string, visible map[string]bool, restricted bool) []T { + if !restricted { + return items + } + out := items[:0:0] + for _, it := range items { + if id := serverID(it); id == "" || visible[id] { + out = append(out, it) + } + } + return out +} +``` + +Export it as `FilterByVisibleServer` and call it that in the test too (the test is in package `services`, so rename the call in Step 1's test). + +- [ ] **Step 4: Handlers** + +In `api/monitors.go`, in `getMonitorIncidents` before responding: + +```go + visible, restricted, err := services.VisibleServerIDs(auth.InstanceID(c), auth.ServerScope(c)) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + incidents = services.FilterByVisibleServer(incidents, func(i models.Incident) string { return i.ServerID }, visible, restricted) +``` + +New handler, registered as `g.GET("/monitors/:id/servers", getMonitorServers)`: + +```go +// getMonitorServers godoc +// +// @Summary List a metric monitor's per-server states +// @Tags monitors +// @Produce json +// @Param id path string true "Monitor ID" +// @Success 200 {array} models.MonitorServerState +// @Failure 404 {object} ErrorResponse +// @Failure 500 {object} ErrorResponse +// @Security cookieAuth +// @Security bearerAuth +// @Router /monitors/{id}/servers [get] +func getMonitorServers(c *gin.Context) { + instanceID := auth.InstanceID(c) + m, err := services.GetMonitor(instanceID, c.Param("id")) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + if m == nil || m.Type != models.MonitorMetric { + c.JSON(http.StatusNotFound, gin.H{"error": "metric monitor not found"}) + return + } + states, err := services.ListMonitorServerStates(instanceID, m.MonitorID) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + visible, restricted, err := services.VisibleServerIDs(instanceID, auth.ServerScope(c)) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + states = services.FilterByVisibleServer(states, func(s models.MonitorServerState) string { return s.ServerID }, visible, restricted) + + servers, err := services.ListServersFiltered(instanceID, m.Target.Selector) + if err == nil { + names := map[string]string{} + for _, s := range servers { + names[s.ServerID] = s.Hostname + } + for i := range states { + states[i].Hostname = names[states[i].ServerID] + } + } + c.JSON(http.StatusOK, states) +} +``` + +`scopes.go`: `"GET /api/monitors/:id/servers": "monitors:read",` + +`serverscope.go`: move `"GET /api/monitors/:id/incidents"` from `exempt` to `scoped` and update the comment above it, which says incidents carry no server identifier. That is no longer true for metric monitors, which record `server_id` and filter by visibility. Add: + +```go + // Per-server metric state names servers, so it is filtered to the ones the + // credential can see. + "GET /api/monitors/:id/servers": scoped, +``` + +Read `serverscope_test.go` for any test that pins the incidents classification and update it. + +- [ ] **Step 5: Run tests** + +Run: `cd server && go build ./... && go test ./internal/api/ ./internal/services/` +Expected: PASS, including boot assertions. + +- [ ] **Step 6: Commit** + +```bash +git add server +git commit -m "feat(api): metric monitor per-server states and server-scoped incidents" +``` + +--- + +### Task 10: Metric UI + +**Files:** +- Modify: `web/lib/api.ts` +- Modify: `web/components/monitors/MonitorForm.tsx` +- Create: `web/components/monitors/MetricServersTable.tsx` +- Modify: `web/app/(app)/monitors/page.tsx` +- Modify: `web/app/(app)/monitors/[id]/page.tsx` + +**Interfaces:** +- Consumes: `GET /api/monitors/:id/servers`, `GET /api/servers/tags` (existing, see `listKnownTags` for the response shape), the `MetricKind` type from Task 4. +- Produces: `api.monitorServers(id): Promise`, `metricCopy` (label and unit per kind). + +- [ ] **Step 1: API client** + +```ts +export interface MonitorServerState { + monitor_id: string; + server_id: string; + hostname?: string; + status: MonitorStatus; + breach_since?: string; + value: number; + message?: string; + updated_at: string; +} +``` + +```ts + monitorServers(monitorId: string) { + return request(`/monitors/${monitorId}/servers`); + }, +``` + +And in `web/components/monitors/MonitorVisuals.tsx`, export: + +```ts +export const metricCopy: Record = { + disk_pct: { label: "Disk used", unit: "%", mount: true }, + disk_free_gb: { label: "Disk free below", unit: "GB", mount: true }, + mem_pct: { label: "Memory used", unit: "%" }, + load_per_core: { label: "Load per core", unit: "×" }, + unit_failed: { label: "Systemd unit failed" }, + container_unhealthy: { label: "Container unhealthy" }, + reboot_pending_days: { label: "Reboot pending for", unit: "days" }, + agent_offline_min: { label: "Agent offline for", unit: "minutes" }, +}; +``` + +- [ ] **Step 2: Form** + +In `MonitorForm.tsx`: +- Add `metric: { title: "Server metric", blurb: "Alert on disk, memory, load, units or reboots across tagged servers.", target: "Rule" }` to `typeCopy`. +- State: `metric` (default `"disk_pct"`), `threshold` (default 90), `mount` (""), `forMin` (default 5, from `initial.for_sec / 60`), `selector: Record` (from `initial.target.selector ?? {}`). +- Selector editor: look at how `CreateKeyDialog.tsx` edits `tagSelector` (it's the same `Record` shape, likely with known-tag suggestions from `/servers/tags`). If that editor is inline JSX, extract it into `web/components/servers/TagSelectorInput.tsx` with props `{ value: Record; onChange(v): void }` and use it in both places. If it's already a component, import it. Under the editor, show the helper text "Leave empty to watch every server." +- Kind `