docs: design for server tags and scheduled workflows
This commit is contained in:
@@ -0,0 +1,235 @@
|
||||
# Server tags and scheduled workflows
|
||||
|
||||
Date: 2026-08-04
|
||||
|
||||
Two features, designed together because the second is worth much less without
|
||||
the first. Tags make a target set describable; schedules make it recur. A
|
||||
nightly job that patches "everything tagged `env:staging`" needs both halves,
|
||||
and neither half is large on its own.
|
||||
|
||||
---
|
||||
|
||||
## Part A — Server tags
|
||||
|
||||
### Model
|
||||
|
||||
`models.Server` gains one field:
|
||||
|
||||
```go
|
||||
Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"`
|
||||
```
|
||||
|
||||
Keys and values are lowercase `[a-z0-9_-]`. Keys are capped at 32 characters,
|
||||
values at 64, and a server holds at most 20 tags. Validation lives in the
|
||||
service layer rather than the handler, so the tag endpoint, the server-create
|
||||
path and anything added later cannot disagree about what a valid tag is.
|
||||
|
||||
There is **no `tags` collection.** A tag is a property of a server, not an
|
||||
entity with a lifecycle: a registry would need reference counting to know when
|
||||
a tag stopped existing, and garbage collection to act on it, which is work
|
||||
bought for nothing. The list of known keys and values that the UI offers for
|
||||
autocomplete is a distinct aggregation over `servers`, cached for 60 seconds —
|
||||
the same treatment org lookups already get.
|
||||
|
||||
No reserved keys ship in this change. If inventory-derived tags (`os`, `arch`)
|
||||
are added later they take a `sys:` key prefix, so a user tag written today can
|
||||
never collide with a system tag invented tomorrow.
|
||||
|
||||
Index: `{instance_id: 1, "tags.$**": 1}` — a wildcard index over the tag
|
||||
subdocument, because the queried key is chosen by the user at request time and
|
||||
cannot be named in advance.
|
||||
|
||||
### API
|
||||
|
||||
```
|
||||
PUT /api/servers/:id/tags # replace the whole map
|
||||
GET /api/servers/tags # known keys and values, for pickers
|
||||
GET /api/servers?tag=env:prod # repeatable; AND across keys
|
||||
```
|
||||
|
||||
`PUT` replaces the entire map rather than patching one tag. A tag set is small
|
||||
enough that sending all of it is free, and last-write-wins over a whole map is
|
||||
easier to reason about than merge semantics between two people editing the same
|
||||
server. The audit event records the map before and after.
|
||||
|
||||
`?tag=` is repeatable and ANDs: `?tag=env:prod&tag=role:web` matches servers
|
||||
carrying both. A malformed value (no colon, unknown characters) is a 400 rather
|
||||
than a silent empty result — a filter that matches nothing and a filter that is
|
||||
nonsense look identical in a list, and only one of them is the user's fault.
|
||||
|
||||
### Targeting
|
||||
|
||||
`models.Workflow` gains `TargetTags map[string]string` beside the existing
|
||||
`TargetServerIDs`. One function in `services` resolves them:
|
||||
|
||||
```go
|
||||
ResolveTargets(ctx, instanceID string, ids []string, tags map[string]string) ([]Server, error)
|
||||
```
|
||||
|
||||
- Result is the **distinct union** of the explicit IDs and the tag matches.
|
||||
- Tag matching ANDs across keys.
|
||||
- Offline servers are included. The dispatcher already answers 503 per server,
|
||||
and a patch run that silently omits an unreachable machine is worse than one
|
||||
that visibly fails on it.
|
||||
- Empty IDs **and** empty tags returns `ErrNoTargets` (400). A workflow that
|
||||
matches nothing must say so rather than report success over zero servers.
|
||||
|
||||
The resolved set is snapshotted into `WorkflowRun.ServerRuns` exactly as today.
|
||||
History records what actually ran, not what the selector would match when the
|
||||
run is later read back — the same reason `steps_snapshot` exists.
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Server detail**: tag chips in the header with an inline editor. Keys
|
||||
autocomplete from `GET /api/servers/tags`, values autocomplete per key.
|
||||
- **`/servers`**: a filter bar that reads and writes the same `?tag=` query
|
||||
params the API takes, so a filtered fleet view is a URL someone can send.
|
||||
- **Workflow designer**: a target section holding both inputs, with a live
|
||||
"runs on 14 servers" readout that lists them on hover. The union model costs
|
||||
us the at-a-glance answer to "what will this touch"; this readout buys it
|
||||
back, and it is the reason the union is acceptable.
|
||||
|
||||
---
|
||||
|
||||
## Part B — Scheduled workflows
|
||||
|
||||
### Model
|
||||
|
||||
```go
|
||||
type Schedule struct {
|
||||
Enabled bool `bson:"enabled" json:"enabled"`
|
||||
Cron string `bson:"cron" json:"cron"` // 5-field
|
||||
TZ string `bson:"tz" json:"tz"` // IANA name
|
||||
}
|
||||
|
||||
type Skip struct {
|
||||
Reason string `bson:"reason" json:"reason"` // "missed" | "already_running"
|
||||
Due time.Time `bson:"due" json:"due"`
|
||||
At time.Time `bson:"at" json:"at"`
|
||||
}
|
||||
```
|
||||
|
||||
On `Workflow`:
|
||||
|
||||
```go
|
||||
Schedule *Schedule `bson:"schedule,omitempty"`
|
||||
NextRunAt *time.Time `bson:"next_run_at,omitempty"` // UTC, indexed
|
||||
LastRunAt *time.Time `bson:"last_run_at,omitempty"`
|
||||
LastSkipped *Skip `bson:"last_skipped,omitempty"`
|
||||
```
|
||||
|
||||
`next_run_at` is **persisted, not held in memory.** A leader handover between
|
||||
computing the next occurrence and firing it would otherwise either lose the
|
||||
occurrence or fire it twice. Coordination state has to live where every replica
|
||||
can see it — the same argument that put `workflow_log_seq` in MongoDB.
|
||||
|
||||
Cron parsing uses `robfig/cron/v3`'s **parser only** — `Parse` and
|
||||
`Next(time)`. Its scheduler and goroutines are not used; the loop below is ours
|
||||
and has to be, because it runs under the leader lock.
|
||||
|
||||
**Alpine ships no tzdata.** `server/Dockerfile` builds a slim image, so
|
||||
`time.LoadLocation("Europe/London")` returns an error and every schedule
|
||||
falls back to UTC — an hour wrong for half the year, in the direction nobody
|
||||
notices until a maintenance window lands in business hours. `main` therefore
|
||||
imports `_ "time/tzdata"`, embedding the database in the binary. Zone names are
|
||||
also validated at save time, so an unknown zone is a 400 rather than a surprise
|
||||
at 2am.
|
||||
|
||||
### Scheduler
|
||||
|
||||
A new `server/internal/workflowsched` package, started inside the **existing**
|
||||
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, `StartReaper`
|
||||
and the sweepers. One role, one lock. It takes the same cancellable context and
|
||||
returns the instant leadership is lost.
|
||||
|
||||
The loop ticks every 30 seconds:
|
||||
|
||||
1. `find({schedule.enabled: true, next_run_at: {$lte: now}})`.
|
||||
2. **Claim atomically.** `findOneAndUpdate` matching the document *and* its
|
||||
current `next_run_at`, setting the recomputed next occurrence. A process
|
||||
that reaches the same document after another has claimed it matches nothing
|
||||
and does nothing. The claim is what makes this correct; the leader lock only
|
||||
makes it cheap.
|
||||
3. **Grace check.** If `now - due > 1h`, record
|
||||
`last_skipped{reason: "missed"}`, write an audit event, and do not run. A
|
||||
job missed by ten minutes during a deploy should still run; one missed by
|
||||
two days should not fire at lunchtime.
|
||||
4. **Overlap check.** If a run for this workflow is still active, record
|
||||
`last_skipped{reason: "already_running"}`, audit, and do not run. A patch
|
||||
workflow must never run twice at once, and a silent skip is how a week goes
|
||||
by before anyone notices nothing ran.
|
||||
5. Otherwise start the run through the **same** `RunWorkflow` path a person
|
||||
uses, with `TriggeredBy: "schedule"`.
|
||||
|
||||
Step 5 is the design. A scheduled run is an ordinary run with a different
|
||||
trigger: no second dispatch path, no second snapshot format, and the run detail
|
||||
page needs no changes to display one.
|
||||
|
||||
### API
|
||||
|
||||
```
|
||||
PUT /api/workflows/:id/schedule # {enabled, cron, tz}
|
||||
GET /api/workflows/:id/schedule/preview?cron=…&tz=… # next 3 occurrences
|
||||
```
|
||||
|
||||
`PUT` validates the expression and the zone, then computes and stores
|
||||
`next_run_at`. The preview endpoint exists so the browser and the scheduler
|
||||
agree on what a cron string means — a client-side cron parser that disagrees
|
||||
with the server by one field is a bug found in production, at night.
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Workflow page**: a schedule card with preset buttons (hourly, nightly at
|
||||
HH:MM, weekly on DAY at HH:MM) that write cron underneath, a raw cron field
|
||||
for anything else, a timezone select, and the next three occurrences rendered
|
||||
from the preview endpoint in mono.
|
||||
- **Workflows list**: a schedule chip and the next run as relative time.
|
||||
- **Skips are surfaced**, not just stored: a warning line reading
|
||||
"Skipped Sun 02:00 — previous run still active". Recording a reason nobody
|
||||
reads is the same as not recording one.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
**Notification on scheduled-run failure.** It needs the monitor channel
|
||||
machinery pointed at workflow outcomes and its own answer to what counts as
|
||||
failure — a non-zero exit on a step with `on_failure: continue` is not
|
||||
obviously an alert. Visibility in this change is the run list and the recorded
|
||||
skip reason. Excluded deliberately, not overlooked.
|
||||
|
||||
**Tag-scoped permissions.** Roles stay instance-wide. Tags describe servers;
|
||||
they do not yet gate who may act on them.
|
||||
|
||||
**Inventory-derived tags.** Reserved via the `sys:` prefix, not implemented.
|
||||
|
||||
---
|
||||
|
||||
## Migration and compatibility
|
||||
|
||||
No migration is required. `Tags`, `TargetTags` and `Schedule` are all
|
||||
`omitempty` and absent means what it meant before: no tags, no selector, no
|
||||
schedule. Existing workflows keep their explicit server lists and behave
|
||||
identically.
|
||||
|
||||
The wildcard tag index and the `next_run_at` index are declared by a new
|
||||
`EnsureServerIndexes`, following the convention `EnsureSecretIndexes` and
|
||||
`EnsureWorkflowIndexes` already set: it warns rather than aborting boot,
|
||||
because a missing index degrades
|
||||
tag filtering to a collection scan on a small collection rather than breaking
|
||||
the fleet list.
|
||||
|
||||
## Testing
|
||||
|
||||
- `ResolveTargets`: union deduplicates; AND across tag keys; empty/empty
|
||||
returns `ErrNoTargets`; offline servers are included.
|
||||
- Tag validation: charset, length caps, tag count cap, malformed `?tag=` is a
|
||||
400.
|
||||
- Schedule validation: bad cron and unknown zone both 400; `next_run_at` is
|
||||
computed in the stored zone, verified across a DST boundary.
|
||||
- Scheduler claim: two concurrent claims of the same due workflow start exactly
|
||||
one run.
|
||||
- Grace window: due 10 minutes ago runs; due 2 hours ago records `missed`.
|
||||
- Overlap: an active run yields `already_running` and no second run.
|
||||
- Preview endpoint and the scheduler agree on the next occurrence for a table
|
||||
of expressions, including a DST-crossing one.
|
||||
Reference in New Issue
Block a user