diff --git a/docs/superpowers/plans/2026-07-20-fleet-inventory.md b/docs/superpowers/plans/2026-07-20-fleet-inventory.md index eb436c2..1788583 100644 --- a/docs/superpowers/plans/2026-07-20-fleet-inventory.md +++ b/docs/superpowers/plans/2026-07-20-fleet-inventory.md @@ -645,9 +645,222 @@ git commit -m "fix: fleet inventory verification fixes" --- +# Service Monitoring (uptime-kuma replacement) + +Extends the fleet work: in-app service monitors replacing uptime-kuma. Monitors (HTTP/TCP/ICMP/TLS) run **server-side** (public endpoints) or **agent-side** (agent probes its own host). Both runners feed one server-side ingest pipeline: state → incidents → rollups → notifications. + +**Design:** validated in brainstorm 2026-07-21. Hybrid runners, all 4 check types, latest+incidents+rollups history, multi-channel notify (webhook/SMTP/Discord/Slack/Telegram), dedicated `SyncMonitors`/`ReportChecks` RPCs. + +**Build order — 3 phases, each shippable:** +- **P1 (Tasks 7–10):** data model, checker pkg, server scheduler, ingest pipeline, `/monitors` UI. Server-run only. No agent, no notify. +- **P2 (Tasks 11–12):** `SyncMonitors` + `ReportChecks` RPCs, agent checker + scheduler, agent-run monitors bound to a server. +- **P3 (Tasks 13–14):** notification channels + dispatch + settings UI. + +## Monitoring Global Constraints + +- Same as fleet: no tests this iteration; verify with `go build ./...`, `go vet ./...`, `npm run build`. JSON-codec gRPC — edit both pb files identically, mirror `ReportUpdates` wiring. Separate Go modules, so the checker pkg is **duplicated** in `server/` and `agent/` (same convention as pb files). +- Reuse existing patterns: REST handlers like `server/internal/api`, services like `server/internal/services/servers.go`, `db.Col(...)`, react-query + Tailwind UI like `web/app/servers`. + +--- + +## Task 7: Monitoring data model + checker package (server) + +**Files:** +- Create: `server/internal/models/monitor.go` +- Create: `server/internal/checker/checker.go` (+ `http.go`, `tcp.go`, `icmp.go`, `tls.go`) + +**Interfaces:** +- Produces: `models.Monitor` (+ `MonitorState`, `MonitorTarget`), `models.Incident`, `models.Rollup`. `checker.Run(ctx, models.Monitor) checker.Result` where `Result{Up bool; LatencyMs int; Message string; CertExpiry *time.Time}`. + +- [ ] **Step 1: Model** + +```go +type MonitorTarget struct { + URL string `bson:"url,omitempty" json:"url,omitempty"` + Host string `bson:"host,omitempty" json:"host,omitempty"` + Port int `bson:"port,omitempty" json:"port,omitempty"` + Method string `bson:"method,omitempty" json:"method,omitempty"` + ExpectedStatus int `bson:"expected_status,omitempty" json:"expected_status,omitempty"` + Keyword string `bson:"keyword,omitempty" json:"keyword,omitempty"` + TLSWarnDays int `bson:"tls_warn_days,omitempty" json:"tls_warn_days,omitempty"` +} +type MonitorState struct { + Status string `bson:"status" json:"status"` // up|down|pending + LastCheckAt *time.Time `bson:"last_check_at,omitempty" json:"last_check_at,omitempty"` + LatencyMs int `bson:"latency_ms" json:"latency_ms"` + Message string `bson:"message,omitempty" json:"message,omitempty"` + CertExpiryAt *time.Time `bson:"cert_expiry_at,omitempty" json:"cert_expiry_at,omitempty"` + Fails int `bson:"fails" json:"fails"` // consecutive failures +} +type Monitor struct { + MonitorID string `bson:"monitor_id" json:"monitor_id"` + Name string `bson:"name" json:"name"` + Type string `bson:"type" json:"type"` // http|tcp|icmp|tls + Target MonitorTarget `bson:"target" json:"target"` + IntervalSec int `bson:"interval_sec" json:"interval_sec"` + Runner string `bson:"runner" json:"runner"` // "server" or a server_id + Retries int `bson:"retries" json:"retries"` // consecutive fails before down + Enabled bool `bson:"enabled" json:"enabled"` + ChannelIDs []string `bson:"channel_ids,omitempty" json:"channel_ids,omitempty"` + State MonitorState `bson:"state" json:"state"` + CreatedAt time.Time `bson:"created_at" json:"created_at"` +} +type Incident struct { + IncidentID string `bson:"incident_id" json:"incident_id"` + MonitorID string `bson:"monitor_id" json:"monitor_id"` + StartedAt time.Time `bson:"started_at" json:"started_at"` + ResolvedAt *time.Time `bson:"resolved_at,omitempty" json:"resolved_at,omitempty"` + Cause string `bson:"cause,omitempty" json:"cause,omitempty"` +} +type Rollup struct { + MonitorID string `bson:"monitor_id" json:"monitor_id"` + PeriodStart time.Time `bson:"period_start" json:"period_start"` // hour bucket + Checks int `bson:"checks" json:"checks"` + UpCount int `bson:"up_count" json:"up_count"` + SumLatency int64 `bson:"sum_latency" json:"sum_latency"` +} +``` + +- [ ] **Step 2: Checker package** — `Run(ctx, m)` switches on `m.Type`: + - **http**: `http.Client` GET/HEAD `m.Target.URL`, assert status == ExpectedStatus (default 200), optional `Keyword` body contains; capture TLS peer cert expiry when https. + - **tcp**: `net.DialTimeout("tcp", host:port)`, latency = dial time. + - **icmp**: raw ICMP echo (agent/server run as root). Fall back to `net.Dial("ip4:icmp")`; on permission error return down with message. + - **tls**: `tls.Dial`, read `ConnectionState().PeerCertificates[0].NotAfter` → `CertExpiry`; down if within `TLSWarnDays` or expired. + - All: wrap with per-check timeout (min(IntervalSec, 10s)); `Result.Message` = short reason on failure. + +- [ ] **Step 3: Verify build** — `cd server && go build ./... && go vet ./...` + +- [ ] **Step 4: Commit** — `feat(server): monitor model + checker package` + +--- + +## Task 8: Ingest pipeline + rollups service + +**Files:** +- Create: `server/internal/services/monitors.go` + +**Interfaces:** +- Produces: `IngestResult(monitorID string, res checker.Result) error` — the single entry both runners use. `ListMonitors`, `GetMonitor`, `CreateMonitor`, `UpdateMonitor`, `DeleteMonitor`, `ListIncidents(monitorID)`, `UptimeRollups(monitorID, since)`. + +- [ ] **Step 1: `IngestResult`** — load monitor; compute new status with `Retries` threshold (increment `state.Fails` on failure, flip to `down` only when `Fails >= Retries`; reset + flip `up` on success). On **transition**: open incident (`down`) or resolve open incident (`up`), and enqueue notification (P3 — leave a `// TODO(P3): dispatch` hook now). Always `$set` state fields. Upsert current-hour `Rollup` (`$inc` checks/up_count/sum_latency). Use `db.Col("monitors")`, `db.Col("incidents")`, `db.Col("monitor_rollups")`, `context.WithTimeout`. + +- [ ] **Step 2: CRUD + queries** — standard service funcs mirroring `services/servers.go`. `UptimeRollups` aggregates buckets since a cutoff → uptime % + avg latency series. + +- [ ] **Step 3: Verify build** — `go build ./... && go vet ./...` + +- [ ] **Step 4: Commit** — `feat(server): monitor ingest pipeline, incidents, rollups` + +--- + +## Task 9: Server scheduler + REST API + +**Files:** +- Create: `server/internal/monitorsched/scheduler.go` +- Create: `server/internal/api/monitors.go` +- Modify: server bootstrap (wherever services/gRPC start) to launch the scheduler; router registration where `api` routes are mounted. + +**Interfaces:** +- Produces: a scheduler that ticks enabled `runner=="server"` monitors on their `IntervalSec` and calls `checker.Run` → `services.IngestResult`. REST: `GET/POST /api/monitors`, `GET/PUT/DELETE /api/monitors/:id`, `GET /api/monitors/:id/incidents`, `GET /api/monitors/:id/uptime`. + +- [ ] **Step 1: Scheduler** — on boot load monitors; per-monitor goroutine or a min-heap wheel keyed on next-run. Only `runner=="server"`. Reload on CRUD (simplest: re-read every N sec, or a reload channel fired by the service). Skip disabled. + +- [ ] **Step 2: REST handlers** — mirror an existing `server/internal/api` handler file for style + auth middleware. JSON in/out of `models.Monitor`. + +- [ ] **Step 3: Verify build** — `go build ./... && go vet ./...` + +- [ ] **Step 4: Commit** — `feat(server): server-run monitor scheduler + REST API` + +--- + +## Task 10: Frontend — monitors UI (P1) + +**Files:** +- Modify: `web/lib/api.ts` (Monitor types + bindings) +- Create: `web/app/monitors/page.tsx` (list), `web/app/monitors/[id]/page.tsx` (detail), `web/app/monitors/new/page.tsx` (create/edit form) +- Modify: main nav to add **Monitors** (same place Steps was added) + +**Interfaces:** +- Consumes: `/api/monitors*` (T9). + +- [ ] **Step 1: Types + api bindings** — `Monitor`, `MonitorState`, `Incident`, uptime series; `api.monitors.list/get/create/update/remove/incidents/uptime`. +- [ ] **Step 2: List page** — table: name, type, status badge (up/down/pending), uptime % (24h), latency, last check. `refetchInterval: 30000`. +- [ ] **Step 3: Detail page** — status header, heartbeat/uptime bars (24h + 30d from rollups), latency chart, incident timeline, cert expiry, assigned channels (read-only until P3). +- [ ] **Step 4: Create/edit form** — type-dependent fields (URL vs host/port), interval, retries, runner select (`server` or a registered server for agent-run — server option only wired in P2), enabled. +- [ ] **Step 5: Verify build** — `cd web && npm run build` +- [ ] **Step 6: Commit** — `feat(web): monitors list/detail/form UI` + +--- + +## Task 11: SyncMonitors + ReportChecks RPCs (P2) + +**Files:** +- Modify: `proto/vantage/v1/vantage.proto`, `server/internal/grpc/pb/vantage.pb.go`, `agent/internal/grpc/pb/vantage.pb.go`, `server/internal/grpc/server.go`, `agent/internal/grpc/client.go` + +**Interfaces:** +- Produces: `SyncMonitors(server_id, agent_token) -> repeated MonitorSpec`; `ReportChecks(server_id, agent_token, repeated CheckResult) -> ReportChecksResponse`. `MonitorSpec{monitor_id, type, target fields, interval_sec, retries}`. `CheckResult{monitor_id, up, latency_ms, message, cert_expiry_unix}`. + +- [ ] **Step 1: pb structs + proto** — add messages to both pb files + proto doc. +- [ ] **Step 2: Wire both RPCs** — mirror `ReportUpdates` plumbing (interface, Unimplemented stub, client method, `Vantage_ServiceDesc.Methods`, `_Vantage_*_Handler`) in both pb files. Server handlers on `vantageServer` (after `ReportUpdates` at server.go:78): `SyncMonitors` returns monitors where `runner==req.ServerId && enabled`; `ReportChecks` validates token then loops `services.IngestResult`. Client methods on `*Client` in client.go (after `ReportUpdates` at client.go:117). +- [ ] **Step 3: Verify build** — both modules `go build ./... && go vet ./...` +- [ ] **Step 4: Commit** — `feat(proto): SyncMonitors + ReportChecks RPCs` + +--- + +## Task 12: Agent checker + scheduler (P2) + +**Files:** +- Create: `agent/internal/checker/` (duplicate of server checker pkg) +- Create: `agent/internal/monitors/monitors.go` (poll + run + report loop) +- Modify: agent main loop to start it (alongside the sync loop in `agent/internal/sync` / the inventory ticker from Task 4) + +**Interfaces:** +- Consumes: `client.SyncMonitors`, `client.ReportChecks`, agent `checker`. + +- [ ] **Step 1: Duplicate checker pkg** into agent module (identical logic; imports agent pb). +- [ ] **Step 2: Monitor loop** — poll `SyncMonitors` every 30s for assigned specs; per-spec ticker on `IntervalSec` runs `checker.Run`; batch `CheckResult`s and `ReportChecks`. `serverID`/`agentToken`/`*Client` in scope from the existing loop. +- [ ] **Step 3: Verify build** — `cd agent && go build ./... && go vet ./...` (+ `GOOS=windows go build ./...`; icmp may no-op on Windows). +- [ ] **Step 4: Commit** — `feat(agent): agent-run monitor scheduler` + +--- + +## Task 13: Notification channels + dispatch (P3) + +**Files:** +- Create: `server/internal/models/channel.go`, `server/internal/services/channels.go`, `server/internal/notify/` (`dispatch.go`, `webhook.go`, `smtp.go`, `discord.go`, `slack.go`, `telegram.go`), `server/internal/api/channels.go` +- Modify: `server/internal/services/monitors.go` (replace the P2 `// TODO(P3): dispatch` hook) + +**Interfaces:** +- Produces: `models.NotificationChannel{channel_id, name, type, config map, enabled}`. `notify.Dispatch(channel, event)` where `event` = monitor + old/new status + message. `notify.Test(channel)`. + +- [ ] **Step 1: Model + CRUD service + REST** (`/api/channels*`, incl. `POST /api/channels/:id/test`). +- [ ] **Step 2: Dispatch abstraction** — webhook/discord/slack/telegram are HTTP POST with per-type JSON payload; SMTP via `net/smtp`. Per-monitor routing via `monitor.ChannelIDs`; resend interval so an ongoing `down` re-alerts at most every N min (track `last_notified_at` on monitor state). +- [ ] **Step 3: Fire on transition** — in `IngestResult`, on up/down flip resolve channels and `notify.Dispatch` each (goroutine, best-effort, log failures). +- [ ] **Step 4: Verify build** — `go build ./... && go vet ./...` +- [ ] **Step 5: Commit** — `feat(server): multi-channel monitor notifications` + +--- + +## Task 14: Frontend — notification settings (P3) + +**Files:** +- Modify: `web/lib/api.ts` (channel types + bindings), `web/app/settings/` (add notifications section/page) +- Modify: monitor create/edit form (Task 10) to select channels + +**Interfaces:** +- Consumes: `/api/channels*`. + +- [ ] **Step 1: Channel types + api bindings.** +- [ ] **Step 2: Settings UI** — list/add/edit channels, type-dependent config fields, **Test** button hitting `/api/channels/:id/test`. +- [ ] **Step 3: Wire channel multi-select** into the monitor form. +- [ ] **Step 4: Verify build** — `cd web && npm run build` +- [ ] **Step 5: Commit** — `feat(web): notification channel settings UI` + +--- + ## Self-Review Notes - **Spec coverage:** §3 model → T1; §4 RPC → T1; §5 collectors + scheduler → T3, T4; §6 handler/store → T2; §7 frontend → T5. Split cadence (30s metrics / 15m static) in T4 scheduler; merge rules preserving static in T2 `StoreInventory`. Tests omitted per Global Constraints. - **Startup snapshot:** agent sends `Collect(true)` immediately so static fields populate without waiting 15 min. - **Types consistent:** `InventoryReport` field names identical across proto, both pb files, store service, and TS interface (`usage_pct`, `used_bytes`, `total_bytes`, `swap_*`). - **Follow-ups (out of scope):** time-series history, usage alerting, Windows collectors, servers-list CPU/RAM badges. +- **Monitoring (Tasks 7–14):** hybrid runner service-monitor replacing uptime-kuma, added 2026-07-21. 3 phases — P1 server-run engine+UI (T7–10), P2 agent-run RPCs (T11–12), P3 multi-channel notify (T13–14). Single `IngestResult` pipeline for both runners; checker pkg duplicated per module (pb convention). Design: brainstorm 2026-07-21. Follow-ups out of scope: status pages, maintenance windows, per-check auth headers, ICMP on Windows.