docs: add service monitoring phases to fleet inventory plan
Server Deploy / deploy (push) Successful in 21s
Server Deploy / deploy (push) Successful in 21s
This commit is contained in:
@@ -645,9 +645,222 @@ git commit -m "fix: fleet inventory verification fixes"
|
||||
|
||||
---
|
||||
|
||||
# Service Monitoring (uptime-kuma replacement)
|
||||
|
||||
Extends the fleet work: in-app service monitors replacing uptime-kuma. Monitors (HTTP/TCP/ICMP/TLS) run **server-side** (public endpoints) or **agent-side** (agent probes its own host). Both runners feed one server-side ingest pipeline: state → incidents → rollups → notifications.
|
||||
|
||||
**Design:** validated in brainstorm 2026-07-21. Hybrid runners, all 4 check types, latest+incidents+rollups history, multi-channel notify (webhook/SMTP/Discord/Slack/Telegram), dedicated `SyncMonitors`/`ReportChecks` RPCs.
|
||||
|
||||
**Build order — 3 phases, each shippable:**
|
||||
- **P1 (Tasks 7–10):** data model, checker pkg, server scheduler, ingest pipeline, `/monitors` UI. Server-run only. No agent, no notify.
|
||||
- **P2 (Tasks 11–12):** `SyncMonitors` + `ReportChecks` RPCs, agent checker + scheduler, agent-run monitors bound to a server.
|
||||
- **P3 (Tasks 13–14):** notification channels + dispatch + settings UI.
|
||||
|
||||
## Monitoring Global Constraints
|
||||
|
||||
- Same as fleet: no tests this iteration; verify with `go build ./...`, `go vet ./...`, `npm run build`. JSON-codec gRPC — edit both pb files identically, mirror `ReportUpdates` wiring. Separate Go modules, so the checker pkg is **duplicated** in `server/` and `agent/` (same convention as pb files).
|
||||
- Reuse existing patterns: REST handlers like `server/internal/api`, services like `server/internal/services/servers.go`, `db.Col(...)`, react-query + Tailwind UI like `web/app/servers`.
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Monitoring data model + checker package (server)
|
||||
|
||||
**Files:**
|
||||
- Create: `server/internal/models/monitor.go`
|
||||
- Create: `server/internal/checker/checker.go` (+ `http.go`, `tcp.go`, `icmp.go`, `tls.go`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `models.Monitor` (+ `MonitorState`, `MonitorTarget`), `models.Incident`, `models.Rollup`. `checker.Run(ctx, models.Monitor) checker.Result` where `Result{Up bool; LatencyMs int; Message string; CertExpiry *time.Time}`.
|
||||
|
||||
- [ ] **Step 1: Model**
|
||||
|
||||
```go
|
||||
type MonitorTarget struct {
|
||||
URL string `bson:"url,omitempty" json:"url,omitempty"`
|
||||
Host string `bson:"host,omitempty" json:"host,omitempty"`
|
||||
Port int `bson:"port,omitempty" json:"port,omitempty"`
|
||||
Method string `bson:"method,omitempty" json:"method,omitempty"`
|
||||
ExpectedStatus int `bson:"expected_status,omitempty" json:"expected_status,omitempty"`
|
||||
Keyword string `bson:"keyword,omitempty" json:"keyword,omitempty"`
|
||||
TLSWarnDays int `bson:"tls_warn_days,omitempty" json:"tls_warn_days,omitempty"`
|
||||
}
|
||||
type MonitorState struct {
|
||||
Status string `bson:"status" json:"status"` // up|down|pending
|
||||
LastCheckAt *time.Time `bson:"last_check_at,omitempty" json:"last_check_at,omitempty"`
|
||||
LatencyMs int `bson:"latency_ms" json:"latency_ms"`
|
||||
Message string `bson:"message,omitempty" json:"message,omitempty"`
|
||||
CertExpiryAt *time.Time `bson:"cert_expiry_at,omitempty" json:"cert_expiry_at,omitempty"`
|
||||
Fails int `bson:"fails" json:"fails"` // consecutive failures
|
||||
}
|
||||
type Monitor struct {
|
||||
MonitorID string `bson:"monitor_id" json:"monitor_id"`
|
||||
Name string `bson:"name" json:"name"`
|
||||
Type string `bson:"type" json:"type"` // http|tcp|icmp|tls
|
||||
Target MonitorTarget `bson:"target" json:"target"`
|
||||
IntervalSec int `bson:"interval_sec" json:"interval_sec"`
|
||||
Runner string `bson:"runner" json:"runner"` // "server" or a server_id
|
||||
Retries int `bson:"retries" json:"retries"` // consecutive fails before down
|
||||
Enabled bool `bson:"enabled" json:"enabled"`
|
||||
ChannelIDs []string `bson:"channel_ids,omitempty" json:"channel_ids,omitempty"`
|
||||
State MonitorState `bson:"state" json:"state"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
}
|
||||
type Incident struct {
|
||||
IncidentID string `bson:"incident_id" json:"incident_id"`
|
||||
MonitorID string `bson:"monitor_id" json:"monitor_id"`
|
||||
StartedAt time.Time `bson:"started_at" json:"started_at"`
|
||||
ResolvedAt *time.Time `bson:"resolved_at,omitempty" json:"resolved_at,omitempty"`
|
||||
Cause string `bson:"cause,omitempty" json:"cause,omitempty"`
|
||||
}
|
||||
type Rollup struct {
|
||||
MonitorID string `bson:"monitor_id" json:"monitor_id"`
|
||||
PeriodStart time.Time `bson:"period_start" json:"period_start"` // hour bucket
|
||||
Checks int `bson:"checks" json:"checks"`
|
||||
UpCount int `bson:"up_count" json:"up_count"`
|
||||
SumLatency int64 `bson:"sum_latency" json:"sum_latency"`
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Checker package** — `Run(ctx, m)` switches on `m.Type`:
|
||||
- **http**: `http.Client` GET/HEAD `m.Target.URL`, assert status == ExpectedStatus (default 200), optional `Keyword` body contains; capture TLS peer cert expiry when https.
|
||||
- **tcp**: `net.DialTimeout("tcp", host:port)`, latency = dial time.
|
||||
- **icmp**: raw ICMP echo (agent/server run as root). Fall back to `net.Dial("ip4:icmp")`; on permission error return down with message.
|
||||
- **tls**: `tls.Dial`, read `ConnectionState().PeerCertificates[0].NotAfter` → `CertExpiry`; down if within `TLSWarnDays` or expired.
|
||||
- All: wrap with per-check timeout (min(IntervalSec, 10s)); `Result.Message` = short reason on failure.
|
||||
|
||||
- [ ] **Step 3: Verify build** — `cd server && go build ./... && go vet ./...`
|
||||
|
||||
- [ ] **Step 4: Commit** — `feat(server): monitor model + checker package`
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Ingest pipeline + rollups service
|
||||
|
||||
**Files:**
|
||||
- Create: `server/internal/services/monitors.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `IngestResult(monitorID string, res checker.Result) error` — the single entry both runners use. `ListMonitors`, `GetMonitor`, `CreateMonitor`, `UpdateMonitor`, `DeleteMonitor`, `ListIncidents(monitorID)`, `UptimeRollups(monitorID, since)`.
|
||||
|
||||
- [ ] **Step 1: `IngestResult`** — load monitor; compute new status with `Retries` threshold (increment `state.Fails` on failure, flip to `down` only when `Fails >= Retries`; reset + flip `up` on success). On **transition**: open incident (`down`) or resolve open incident (`up`), and enqueue notification (P3 — leave a `// TODO(P3): dispatch` hook now). Always `$set` state fields. Upsert current-hour `Rollup` (`$inc` checks/up_count/sum_latency). Use `db.Col("monitors")`, `db.Col("incidents")`, `db.Col("monitor_rollups")`, `context.WithTimeout`.
|
||||
|
||||
- [ ] **Step 2: CRUD + queries** — standard service funcs mirroring `services/servers.go`. `UptimeRollups` aggregates buckets since a cutoff → uptime % + avg latency series.
|
||||
|
||||
- [ ] **Step 3: Verify build** — `go build ./... && go vet ./...`
|
||||
|
||||
- [ ] **Step 4: Commit** — `feat(server): monitor ingest pipeline, incidents, rollups`
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Server scheduler + REST API
|
||||
|
||||
**Files:**
|
||||
- Create: `server/internal/monitorsched/scheduler.go`
|
||||
- Create: `server/internal/api/monitors.go`
|
||||
- Modify: server bootstrap (wherever services/gRPC start) to launch the scheduler; router registration where `api` routes are mounted.
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: a scheduler that ticks enabled `runner=="server"` monitors on their `IntervalSec` and calls `checker.Run` → `services.IngestResult`. REST: `GET/POST /api/monitors`, `GET/PUT/DELETE /api/monitors/:id`, `GET /api/monitors/:id/incidents`, `GET /api/monitors/:id/uptime`.
|
||||
|
||||
- [ ] **Step 1: Scheduler** — on boot load monitors; per-monitor goroutine or a min-heap wheel keyed on next-run. Only `runner=="server"`. Reload on CRUD (simplest: re-read every N sec, or a reload channel fired by the service). Skip disabled.
|
||||
|
||||
- [ ] **Step 2: REST handlers** — mirror an existing `server/internal/api` handler file for style + auth middleware. JSON in/out of `models.Monitor`.
|
||||
|
||||
- [ ] **Step 3: Verify build** — `go build ./... && go vet ./...`
|
||||
|
||||
- [ ] **Step 4: Commit** — `feat(server): server-run monitor scheduler + REST API`
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Frontend — monitors UI (P1)
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/lib/api.ts` (Monitor types + bindings)
|
||||
- Create: `web/app/monitors/page.tsx` (list), `web/app/monitors/[id]/page.tsx` (detail), `web/app/monitors/new/page.tsx` (create/edit form)
|
||||
- Modify: main nav to add **Monitors** (same place Steps was added)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `/api/monitors*` (T9).
|
||||
|
||||
- [ ] **Step 1: Types + api bindings** — `Monitor`, `MonitorState`, `Incident`, uptime series; `api.monitors.list/get/create/update/remove/incidents/uptime`.
|
||||
- [ ] **Step 2: List page** — table: name, type, status badge (up/down/pending), uptime % (24h), latency, last check. `refetchInterval: 30000`.
|
||||
- [ ] **Step 3: Detail page** — status header, heartbeat/uptime bars (24h + 30d from rollups), latency chart, incident timeline, cert expiry, assigned channels (read-only until P3).
|
||||
- [ ] **Step 4: Create/edit form** — type-dependent fields (URL vs host/port), interval, retries, runner select (`server` or a registered server for agent-run — server option only wired in P2), enabled.
|
||||
- [ ] **Step 5: Verify build** — `cd web && npm run build`
|
||||
- [ ] **Step 6: Commit** — `feat(web): monitors list/detail/form UI`
|
||||
|
||||
---
|
||||
|
||||
## Task 11: SyncMonitors + ReportChecks RPCs (P2)
|
||||
|
||||
**Files:**
|
||||
- Modify: `proto/vantage/v1/vantage.proto`, `server/internal/grpc/pb/vantage.pb.go`, `agent/internal/grpc/pb/vantage.pb.go`, `server/internal/grpc/server.go`, `agent/internal/grpc/client.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `SyncMonitors(server_id, agent_token) -> repeated MonitorSpec`; `ReportChecks(server_id, agent_token, repeated CheckResult) -> ReportChecksResponse`. `MonitorSpec{monitor_id, type, target fields, interval_sec, retries}`. `CheckResult{monitor_id, up, latency_ms, message, cert_expiry_unix}`.
|
||||
|
||||
- [ ] **Step 1: pb structs + proto** — add messages to both pb files + proto doc.
|
||||
- [ ] **Step 2: Wire both RPCs** — mirror `ReportUpdates` plumbing (interface, Unimplemented stub, client method, `Vantage_ServiceDesc.Methods`, `_Vantage_*_Handler`) in both pb files. Server handlers on `vantageServer` (after `ReportUpdates` at server.go:78): `SyncMonitors` returns monitors where `runner==req.ServerId && enabled`; `ReportChecks` validates token then loops `services.IngestResult`. Client methods on `*Client` in client.go (after `ReportUpdates` at client.go:117).
|
||||
- [ ] **Step 3: Verify build** — both modules `go build ./... && go vet ./...`
|
||||
- [ ] **Step 4: Commit** — `feat(proto): SyncMonitors + ReportChecks RPCs`
|
||||
|
||||
---
|
||||
|
||||
## Task 12: Agent checker + scheduler (P2)
|
||||
|
||||
**Files:**
|
||||
- Create: `agent/internal/checker/` (duplicate of server checker pkg)
|
||||
- Create: `agent/internal/monitors/monitors.go` (poll + run + report loop)
|
||||
- Modify: agent main loop to start it (alongside the sync loop in `agent/internal/sync` / the inventory ticker from Task 4)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `client.SyncMonitors`, `client.ReportChecks`, agent `checker`.
|
||||
|
||||
- [ ] **Step 1: Duplicate checker pkg** into agent module (identical logic; imports agent pb).
|
||||
- [ ] **Step 2: Monitor loop** — poll `SyncMonitors` every 30s for assigned specs; per-spec ticker on `IntervalSec` runs `checker.Run`; batch `CheckResult`s and `ReportChecks`. `serverID`/`agentToken`/`*Client` in scope from the existing loop.
|
||||
- [ ] **Step 3: Verify build** — `cd agent && go build ./... && go vet ./...` (+ `GOOS=windows go build ./...`; icmp may no-op on Windows).
|
||||
- [ ] **Step 4: Commit** — `feat(agent): agent-run monitor scheduler`
|
||||
|
||||
---
|
||||
|
||||
## Task 13: Notification channels + dispatch (P3)
|
||||
|
||||
**Files:**
|
||||
- Create: `server/internal/models/channel.go`, `server/internal/services/channels.go`, `server/internal/notify/` (`dispatch.go`, `webhook.go`, `smtp.go`, `discord.go`, `slack.go`, `telegram.go`), `server/internal/api/channels.go`
|
||||
- Modify: `server/internal/services/monitors.go` (replace the P2 `// TODO(P3): dispatch` hook)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `models.NotificationChannel{channel_id, name, type, config map, enabled}`. `notify.Dispatch(channel, event)` where `event` = monitor + old/new status + message. `notify.Test(channel)`.
|
||||
|
||||
- [ ] **Step 1: Model + CRUD service + REST** (`/api/channels*`, incl. `POST /api/channels/:id/test`).
|
||||
- [ ] **Step 2: Dispatch abstraction** — webhook/discord/slack/telegram are HTTP POST with per-type JSON payload; SMTP via `net/smtp`. Per-monitor routing via `monitor.ChannelIDs`; resend interval so an ongoing `down` re-alerts at most every N min (track `last_notified_at` on monitor state).
|
||||
- [ ] **Step 3: Fire on transition** — in `IngestResult`, on up/down flip resolve channels and `notify.Dispatch` each (goroutine, best-effort, log failures).
|
||||
- [ ] **Step 4: Verify build** — `go build ./... && go vet ./...`
|
||||
- [ ] **Step 5: Commit** — `feat(server): multi-channel monitor notifications`
|
||||
|
||||
---
|
||||
|
||||
## Task 14: Frontend — notification settings (P3)
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/lib/api.ts` (channel types + bindings), `web/app/settings/` (add notifications section/page)
|
||||
- Modify: monitor create/edit form (Task 10) to select channels
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `/api/channels*`.
|
||||
|
||||
- [ ] **Step 1: Channel types + api bindings.**
|
||||
- [ ] **Step 2: Settings UI** — list/add/edit channels, type-dependent config fields, **Test** button hitting `/api/channels/:id/test`.
|
||||
- [ ] **Step 3: Wire channel multi-select** into the monitor form.
|
||||
- [ ] **Step 4: Verify build** — `cd web && npm run build`
|
||||
- [ ] **Step 5: Commit** — `feat(web): notification channel settings UI`
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- **Spec coverage:** §3 model → T1; §4 RPC → T1; §5 collectors + scheduler → T3, T4; §6 handler/store → T2; §7 frontend → T5. Split cadence (30s metrics / 15m static) in T4 scheduler; merge rules preserving static in T2 `StoreInventory`. Tests omitted per Global Constraints.
|
||||
- **Startup snapshot:** agent sends `Collect(true)` immediately so static fields populate without waiting 15 min.
|
||||
- **Types consistent:** `InventoryReport` field names identical across proto, both pb files, store service, and TS interface (`usage_pct`, `used_bytes`, `total_bytes`, `swap_*`).
|
||||
- **Follow-ups (out of scope):** time-series history, usage alerting, Windows collectors, servers-list CPU/RAM badges.
|
||||
- **Monitoring (Tasks 7–14):** hybrid runner service-monitor replacing uptime-kuma, added 2026-07-21. 3 phases — P1 server-run engine+UI (T7–10), P2 agent-run RPCs (T11–12), P3 multi-channel notify (T13–14). Single `IngestResult` pipeline for both runners; checker pkg duplicated per module (pb convention). Design: brainstorm 2026-07-21. Follow-ups out of scope: status pages, maintenance windows, per-check auth headers, ICMP on Windows.
|
||||
|
||||
Reference in New Issue
Block a user