diff --git a/docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md b/docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md new file mode 100644 index 0000000..3cc9609 --- /dev/null +++ b/docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md @@ -0,0 +1,235 @@ +# Server tags and scheduled workflows + +Date: 2026-08-04 + +Two features, designed together because the second is worth much less without +the first. Tags make a target set describable; schedules make it recur. A +nightly job that patches "everything tagged `env:staging`" needs both halves, +and neither half is large on its own. + +--- + +## Part A — Server tags + +### Model + +`models.Server` gains one field: + +```go +Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"` +``` + +Keys and values are lowercase `[a-z0-9_-]`. Keys are capped at 32 characters, +values at 64, and a server holds at most 20 tags. Validation lives in the +service layer rather than the handler, so the tag endpoint, the server-create +path and anything added later cannot disagree about what a valid tag is. + +There is **no `tags` collection.** A tag is a property of a server, not an +entity with a lifecycle: a registry would need reference counting to know when +a tag stopped existing, and garbage collection to act on it, which is work +bought for nothing. The list of known keys and values that the UI offers for +autocomplete is a distinct aggregation over `servers`, cached for 60 seconds — +the same treatment org lookups already get. + +No reserved keys ship in this change. If inventory-derived tags (`os`, `arch`) +are added later they take a `sys:` key prefix, so a user tag written today can +never collide with a system tag invented tomorrow. + +Index: `{instance_id: 1, "tags.$**": 1}` — a wildcard index over the tag +subdocument, because the queried key is chosen by the user at request time and +cannot be named in advance. + +### API + +``` +PUT /api/servers/:id/tags # replace the whole map +GET /api/servers/tags # known keys and values, for pickers +GET /api/servers?tag=env:prod # repeatable; AND across keys +``` + +`PUT` replaces the entire map rather than patching one tag. A tag set is small +enough that sending all of it is free, and last-write-wins over a whole map is +easier to reason about than merge semantics between two people editing the same +server. The audit event records the map before and after. + +`?tag=` is repeatable and ANDs: `?tag=env:prod&tag=role:web` matches servers +carrying both. A malformed value (no colon, unknown characters) is a 400 rather +than a silent empty result — a filter that matches nothing and a filter that is +nonsense look identical in a list, and only one of them is the user's fault. + +### Targeting + +`models.Workflow` gains `TargetTags map[string]string` beside the existing +`TargetServerIDs`. One function in `services` resolves them: + +```go +ResolveTargets(ctx, instanceID string, ids []string, tags map[string]string) ([]Server, error) +``` + +- Result is the **distinct union** of the explicit IDs and the tag matches. +- Tag matching ANDs across keys. +- Offline servers are included. The dispatcher already answers 503 per server, + and a patch run that silently omits an unreachable machine is worse than one + that visibly fails on it. +- Empty IDs **and** empty tags returns `ErrNoTargets` (400). A workflow that + matches nothing must say so rather than report success over zero servers. + +The resolved set is snapshotted into `WorkflowRun.ServerRuns` exactly as today. +History records what actually ran, not what the selector would match when the +run is later read back — the same reason `steps_snapshot` exists. + +### Frontend + +- **Server detail**: tag chips in the header with an inline editor. Keys + autocomplete from `GET /api/servers/tags`, values autocomplete per key. +- **`/servers`**: a filter bar that reads and writes the same `?tag=` query + params the API takes, so a filtered fleet view is a URL someone can send. +- **Workflow designer**: a target section holding both inputs, with a live + "runs on 14 servers" readout that lists them on hover. The union model costs + us the at-a-glance answer to "what will this touch"; this readout buys it + back, and it is the reason the union is acceptable. + +--- + +## Part B — Scheduled workflows + +### Model + +```go +type Schedule struct { + Enabled bool `bson:"enabled" json:"enabled"` + Cron string `bson:"cron" json:"cron"` // 5-field + TZ string `bson:"tz" json:"tz"` // IANA name +} + +type Skip struct { + Reason string `bson:"reason" json:"reason"` // "missed" | "already_running" + Due time.Time `bson:"due" json:"due"` + At time.Time `bson:"at" json:"at"` +} +``` + +On `Workflow`: + +```go +Schedule *Schedule `bson:"schedule,omitempty"` +NextRunAt *time.Time `bson:"next_run_at,omitempty"` // UTC, indexed +LastRunAt *time.Time `bson:"last_run_at,omitempty"` +LastSkipped *Skip `bson:"last_skipped,omitempty"` +``` + +`next_run_at` is **persisted, not held in memory.** A leader handover between +computing the next occurrence and firing it would otherwise either lose the +occurrence or fire it twice. Coordination state has to live where every replica +can see it — the same argument that put `workflow_log_seq` in MongoDB. + +Cron parsing uses `robfig/cron/v3`'s **parser only** — `Parse` and +`Next(time)`. Its scheduler and goroutines are not used; the loop below is ours +and has to be, because it runs under the leader lock. + +**Alpine ships no tzdata.** `server/Dockerfile` builds a slim image, so +`time.LoadLocation("Europe/London")` returns an error and every schedule +falls back to UTC — an hour wrong for half the year, in the direction nobody +notices until a maintenance window lands in business hours. `main` therefore +imports `_ "time/tzdata"`, embedding the database in the binary. Zone names are +also validated at save time, so an unknown zone is a 400 rather than a surprise +at 2am. + +### Scheduler + +A new `server/internal/workflowsched` package, started inside the **existing** +`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, `StartReaper` +and the sweepers. One role, one lock. It takes the same cancellable context and +returns the instant leadership is lost. + +The loop ticks every 30 seconds: + +1. `find({schedule.enabled: true, next_run_at: {$lte: now}})`. +2. **Claim atomically.** `findOneAndUpdate` matching the document *and* its + current `next_run_at`, setting the recomputed next occurrence. A process + that reaches the same document after another has claimed it matches nothing + and does nothing. The claim is what makes this correct; the leader lock only + makes it cheap. +3. **Grace check.** If `now - due > 1h`, record + `last_skipped{reason: "missed"}`, write an audit event, and do not run. A + job missed by ten minutes during a deploy should still run; one missed by + two days should not fire at lunchtime. +4. **Overlap check.** If a run for this workflow is still active, record + `last_skipped{reason: "already_running"}`, audit, and do not run. A patch + workflow must never run twice at once, and a silent skip is how a week goes + by before anyone notices nothing ran. +5. Otherwise start the run through the **same** `RunWorkflow` path a person + uses, with `TriggeredBy: "schedule"`. + +Step 5 is the design. A scheduled run is an ordinary run with a different +trigger: no second dispatch path, no second snapshot format, and the run detail +page needs no changes to display one. + +### API + +``` +PUT /api/workflows/:id/schedule # {enabled, cron, tz} +GET /api/workflows/:id/schedule/preview?cron=…&tz=… # next 3 occurrences +``` + +`PUT` validates the expression and the zone, then computes and stores +`next_run_at`. The preview endpoint exists so the browser and the scheduler +agree on what a cron string means — a client-side cron parser that disagrees +with the server by one field is a bug found in production, at night. + +### Frontend + +- **Workflow page**: a schedule card with preset buttons (hourly, nightly at + HH:MM, weekly on DAY at HH:MM) that write cron underneath, a raw cron field + for anything else, a timezone select, and the next three occurrences rendered + from the preview endpoint in mono. +- **Workflows list**: a schedule chip and the next run as relative time. +- **Skips are surfaced**, not just stored: a warning line reading + "Skipped Sun 02:00 — previous run still active". Recording a reason nobody + reads is the same as not recording one. + +--- + +## Out of scope + +**Notification on scheduled-run failure.** It needs the monitor channel +machinery pointed at workflow outcomes and its own answer to what counts as +failure — a non-zero exit on a step with `on_failure: continue` is not +obviously an alert. Visibility in this change is the run list and the recorded +skip reason. Excluded deliberately, not overlooked. + +**Tag-scoped permissions.** Roles stay instance-wide. Tags describe servers; +they do not yet gate who may act on them. + +**Inventory-derived tags.** Reserved via the `sys:` prefix, not implemented. + +--- + +## Migration and compatibility + +No migration is required. `Tags`, `TargetTags` and `Schedule` are all +`omitempty` and absent means what it meant before: no tags, no selector, no +schedule. Existing workflows keep their explicit server lists and behave +identically. + +The wildcard tag index and the `next_run_at` index are declared by a new +`EnsureServerIndexes`, following the convention `EnsureSecretIndexes` and +`EnsureWorkflowIndexes` already set: it warns rather than aborting boot, +because a missing index degrades +tag filtering to a collection scan on a small collection rather than breaking +the fleet list. + +## Testing + +- `ResolveTargets`: union deduplicates; AND across tag keys; empty/empty + returns `ErrNoTargets`; offline servers are included. +- Tag validation: charset, length caps, tag count cap, malformed `?tag=` is a + 400. +- Schedule validation: bad cron and unknown zone both 400; `next_run_at` is + computed in the stored zone, verified across a DST boundary. +- Scheduler claim: two concurrent claims of the same due workflow start exactly + one run. +- Grace window: due 10 minutes ago runs; due 2 hours ago records `missed`. +- Overlap: an active run yields `already_running` and no second run. +- Preview endpoint and the scheduler agree on the next occurrence for a table + of expressions, including a DST-crossing one.