# Server tags and scheduled workflows

Date: 2026-08-04

Two features, designed together because the second is worth much less without
the first. Tags make a target set describable; schedules make it recur. A
nightly job that patches "everything tagged `env:staging`" needs both halves,
and neither half is large on its own.

---

## Part A — Server tags

### Model

`models.Server` gains one field:

```go
Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"`
```

Keys and values are lowercase `[a-z0-9_-]`. Keys are capped at 32 characters,
values at 64, and a server holds at most 20 tags. Validation lives in the
service layer rather than the handler, so the tag endpoint, the server-create
path and anything added later cannot disagree about what a valid tag is.

There is **no `tags` collection.** A tag is a property of a server, not an
entity with a lifecycle: a registry would need reference counting to know when
a tag stopped existing, and garbage collection to act on it, which is work
bought for nothing. The list of known keys and values that the UI offers for
autocomplete is a distinct aggregation over `servers`, cached for 60 seconds —
the same treatment org lookups already get.

No reserved keys ship in this change. If inventory-derived tags (`os`, `arch`)
are added later they take a `sys:` key prefix, so a user tag written today can
never collide with a system tag invented tomorrow.

Index: `{instance_id: 1, "tags.$**": 1}` — a wildcard index over the tag
subdocument, because the queried key is chosen by the user at request time and
cannot be named in advance.

### API

```
PUT    /api/servers/:id/tags        # replace the whole map
GET    /api/servers/tags            # known keys and values, for pickers
GET    /api/servers?tag=env:prod    # repeatable; AND across keys
```

`PUT` replaces the entire map rather than patching one tag. A tag set is small
enough that sending all of it is free, and last-write-wins over a whole map is
easier to reason about than merge semantics between two people editing the same
server. The audit event records the map before and after.

`?tag=` is repeatable and ANDs: `?tag=env:prod&tag=role:web` matches servers
carrying both. A malformed value (no colon, unknown characters) is a 400 rather
than a silent empty result — a filter that matches nothing and a filter that is
nonsense look identical in a list, and only one of them is the user's fault.

### Targeting

`models.Workflow` gains `TargetTags map[string]string` beside the existing
`TargetServerIDs`. One function in `services` resolves them:

```go
ResolveTargets(ctx, instanceID string, ids []string, tags map[string]string) ([]Server, error)
```

- Result is the **distinct union** of the explicit IDs and the tag matches.
- Tag matching ANDs across keys.
- Offline servers are included. The dispatcher already answers 503 per server,
  and a patch run that silently omits an unreachable machine is worse than one
  that visibly fails on it.
- Empty IDs **and** empty tags returns `ErrNoTargets` (400). A workflow that
  matches nothing must say so rather than report success over zero servers.

The resolved set is snapshotted into `WorkflowRun.ServerRuns` exactly as today.
History records what actually ran, not what the selector would match when the
run is later read back — the same reason `steps_snapshot` exists.

### Frontend

- **Server detail**: tag chips in the header with an inline editor. Keys
  autocomplete from `GET /api/servers/tags`, values autocomplete per key.
- **`/servers`**: a filter bar that reads and writes the same `?tag=` query
  params the API takes, so a filtered fleet view is a URL someone can send.
- **Workflow designer**: a target section holding both inputs, with a live
  "runs on 14 servers" readout that lists them on hover. The union model costs
  us the at-a-glance answer to "what will this touch"; this readout buys it
  back, and it is the reason the union is acceptable.

---

## Part B — Scheduled workflows

### Model

```go
type Schedule struct {
    Enabled bool   `bson:"enabled" json:"enabled"`
    Cron    string `bson:"cron"    json:"cron"`   // 5-field
    TZ      string `bson:"tz"      json:"tz"`     // IANA name
}

type Skip struct {
    Reason string    `bson:"reason" json:"reason"` // "missed" | "already_running"
    Due    time.Time `bson:"due"    json:"due"`
    At     time.Time `bson:"at"     json:"at"`
}
```

On `Workflow`:

```go
Schedule    *Schedule  `bson:"schedule,omitempty"`
NextRunAt   *time.Time `bson:"next_run_at,omitempty"`   // UTC, indexed
LastRunAt   *time.Time `bson:"last_run_at,omitempty"`
LastSkipped *Skip      `bson:"last_skipped,omitempty"`
```

`next_run_at` is **persisted, not held in memory.** A leader handover between
computing the next occurrence and firing it would otherwise either lose the
occurrence or fire it twice. Coordination state has to live where every replica
can see it — the same argument that put `workflow_log_seq` in MongoDB.

Cron parsing uses `robfig/cron/v3`'s **parser only** — `Parse` and
`Next(time)`. Its scheduler and goroutines are not used; the loop below is ours
and has to be, because it runs under the leader lock.

**Alpine ships no tzdata.** `server/Dockerfile` builds a slim image, so
`time.LoadLocation("Europe/London")` returns an error and every schedule
falls back to UTC — an hour wrong for half the year, in the direction nobody
notices until a maintenance window lands in business hours. `main` therefore
imports `_ "time/tzdata"`, embedding the database in the binary. Zone names are
also validated at save time, so an unknown zone is a 400 rather than a surprise
at 2am.

### Scheduler

A new `server/internal/workflowsched` package, started inside the **existing**
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, `StartReaper`
and the sweepers. One role, one lock. It takes the same cancellable context and
returns the instant leadership is lost.

The loop ticks every 30 seconds:

1. `find({schedule.enabled: true, next_run_at: {$lte: now}})`.
2. **Claim atomically.** `findOneAndUpdate` matching the document *and* its
   current `next_run_at`, setting the recomputed next occurrence. A process
   that reaches the same document after another has claimed it matches nothing
   and does nothing. The claim is what makes this correct; the leader lock only
   makes it cheap.
3. **Grace check.** If `now - due > 1h`, record
   `last_skipped{reason: "missed"}`, write an audit event, and do not run. A
   job missed by ten minutes during a deploy should still run; one missed by
   two days should not fire at lunchtime.
4. **Overlap check.** If a run for this workflow is still active, record
   `last_skipped{reason: "already_running"}`, audit, and do not run. A patch
   workflow must never run twice at once, and a silent skip is how a week goes
   by before anyone notices nothing ran.
5. Otherwise start the run through the **same** `RunWorkflow` path a person
   uses, with `TriggeredBy: "schedule"`.

Step 5 is the design. A scheduled run is an ordinary run with a different
trigger: no second dispatch path, no second snapshot format, and the run detail
page needs no changes to display one.

### API

```
PUT /api/workflows/:id/schedule                         # {enabled, cron, tz}
GET /api/workflows/:id/schedule/preview?cron=…&tz=…     # next 3 occurrences
```

`PUT` validates the expression and the zone, then computes and stores
`next_run_at`. The preview endpoint exists so the browser and the scheduler
agree on what a cron string means — a client-side cron parser that disagrees
with the server by one field is a bug found in production, at night.

### Frontend

- **Workflow page**: a schedule card with preset buttons (hourly, nightly at
  HH:MM, weekly on DAY at HH:MM) that write cron underneath, a raw cron field
  for anything else, a timezone select, and the next three occurrences rendered
  from the preview endpoint in mono.
- **Workflows list**: a schedule chip and the next run as relative time.
- **Skips are surfaced**, not just stored: a warning line reading
  "Skipped Sun 02:00 — previous run still active". Recording a reason nobody
  reads is the same as not recording one.

---

## Out of scope

**Notification on scheduled-run failure.** It needs the monitor channel
machinery pointed at workflow outcomes and its own answer to what counts as
failure — a non-zero exit on a step with `on_failure: continue` is not
obviously an alert. Visibility in this change is the run list and the recorded
skip reason. Excluded deliberately, not overlooked.

**Tag-scoped permissions.** Roles stay instance-wide. Tags describe servers;
they do not yet gate who may act on them.

**Inventory-derived tags.** Reserved via the `sys:` prefix, not implemented.

---

## Migration and compatibility

No migration is required. `Tags`, `TargetTags` and `Schedule` are all
`omitempty` and absent means what it meant before: no tags, no selector, no
schedule. Existing workflows keep their explicit server lists and behave
identically.

The wildcard tag index and the `next_run_at` index are declared by a new
`EnsureServerIndexes`, following the convention `EnsureSecretIndexes` and
`EnsureWorkflowIndexes` already set: it warns rather than aborting boot,
because a missing index degrades
tag filtering to a collection scan on a small collection rather than breaking
the fleet list.

## Testing

- `ResolveTargets`: union deduplicates; AND across tag keys; empty/empty
  returns `ErrNoTargets`; offline servers are included.
- Tag validation: charset, length caps, tag count cap, malformed `?tag=` is a
  400.
- Schedule validation: bad cron and unknown zone both 400; `next_run_at` is
  computed in the stored zone, verified across a DST boundary.
- Scheduler claim: two concurrent claims of the same due workflow start exactly
  one run.
- Grace window: due 10 minutes ago runs; due 2 hours ago records `missed`.
- Overlap: an active run yields `already_running` and no second run.
- Preview endpoint and the scheduler agree on the next occurrence for a table
  of expressions, including a DST-crossing one.
