fix(monitors): document 403 and 404 on monitor update and delete; regenerate openapi

Mask base64 and URL-encoded forms of secret values, each line of a
multi-line secret, and secrets loaded by earlier steps of the run.
Replace longer values first so overlapping secrets mask cleanly.

.gitignore

@@ -13,6 +13,10 @@ installer/nssm.zip
 installer/checksums-msi.txt
 .next
 *.tsbuildinfo
+web/test-results
+web/playwright-report
+web/blob-report
+web/playwright/.cache
 graphify-out
 docker-compose.live.yml
 .claude

CLAUDE.md

@@ -173,6 +173,51 @@ Skips are recorded and surfaced, not just logged: past the 1h grace window is
 `missed`, an active run is `already_running`, and a schedule that no longer
 parses is disabled rather than left spinning the loop every 30 seconds forever.
 
+### Scheduled patching
+
+Four collections: `maintenance_windows` (cron start, IANA zone, duration),
+`patch_policies` (selector, window, `all|security`, `never|if_required`,
+concurrency cap, channels), `patch_runs` (one per firing or manual Apply
+updates, one `servers[]` entry per target) and `patch_run_outputs` (one per
+run and server, holding the package manager's output tail). The output lives
+apart from the run because a large run with up to 64KB per server would pass
+MongoDB's 16MB document limit; `GetPatchRun` fills `servers[].output` back in
+memory so the API shape is unchanged, and the tick paths never read it. All
+four are in `ScopedCollections`.
+
+**Runs are driven by database state, not goroutines.** A run can last hours; a
+goroutine-driven run is stranded at `running` when its pod dies. `patchsched`
+ticks every 30s inside the housekeeping leader: it claims due policies with the
+workflowsched `next_run_at` pattern, then advances every running run. Every
+decision is a pure function in `internal/patchrun` (`Advance`, `ApplyResult`,
+`VerifyReboot`, `Finalize`) and every write is guarded by the server run's
+current status, so a result landing mid-tick is never overwritten.
+
+**The window end never kills a package manager.** No server is dispatched in
+the last `patchrun.LatestStartBeforeEnd` (15 minutes) of a window; queued and
+waiting servers close at the window end as before. A server already patching
+may finish past the end: its no-result timeout is its own dispatch time plus
+`ManualTimeout` (2h, the agent's backstop) plus `ResultGrace` (20 minutes), for
+windowed and manual runs alike. `RebootTimeout` is 45 minutes.
+
+**Results do not cross the bus.** The pod holding the agent's stream writes
+`PatchResult` straight into the run, found by `servers.command_id` and the
+agent's own server ID. A reboot is settled by the first static inventory report
+whose boot time differs from (is later than) `boot_time_before`, the
+`inventory.boot_time` recorded when the server moved to rebooting, so host and
+server clock skew does not matter. Without `boot_time_before` the report's boot
+time must be later than `rebooted_at`. A report during the one-minute grace
+does not count either way.
+
+**Old agents must never receive a scope.** An agent before
+`patchrun.MinAgentVersion` ignores `scope` and installs everything, so policy
+runs mark it `agent_too_old` and do not dispatch. A manual Apply updates still
+sends such an agent the empty command and records "no result reported".
+
+The next window starts after `max(now, windowEnd)`, so windows never overlap,
+including across a daylight-saving fall-back. `patchsched` must not import
+`services`; its dependencies are injected from `main.go`.
+
 ### Server tags and workflow targeting
 
 A server carries `tags map[string]string` - lowercase `[a-z0-9_-]`, key ≤32,
@@ -222,7 +267,16 @@ a dark ground.
 
 ### Monitors
 
-HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.
+HTTP, TCP, ICMP and TLS checks are pull-based. Each has a `runner`: `"server"` (executed by `monitorsched`) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.
+
+Two passive types are never run by `monitorsched` or agents; `metricsched` sweeps them every 30s:
+
+- `heartbeat`: jobs call `/public/hb/<token>` (plus `/start`, `/fail`), or send the token in `X-Vantage-Token` to `/public/hb[/start|/fail]`; a URL token wins. Request logs (gin formatter via `api.MaskLogPath`, bundled nginx `log_format vantage`) mask URL tokens. Down when a ping is overdue past period + grace, a start never finishes, or `/fail` is called. The token is stored hashed; plaintext is shown on create and `POST /api/monitors/:id/rotate-token` only.
+- `metric`: a tag `selector` plus a rule (`disk_pct`, `disk_free_gb`, `mem_pct`, `load_per_core`, `unit_failed`, `container_unhealthy`, `reboot_pending_days`, `agent_offline_min`) evaluated against stored inventory and workloads. State is per server in `monitor_server_states`, with one incident per breaching server (`Incident.ServerID`). Inventory older than 5 minutes is skipped. Restricted tokens may only use selectors inside their tag scope.
+
+All three paths share `applyTransition` (services/monitortransition.go) for incidents and notifications.
+
+**Two known gaps, deferred.** Nginx's error log (not the access log) and gin's panic recovery dump can still record a URL token even with `MaskLogPath` masking normal request logs; the `X-Vantage-Token` header avoids putting the token in the URL at all, which is the mitigation until those paths are masked too. And the bundled nginx `vantage.conf` is reproduced in `vantage-docs`' self-hosted install docs, so its `log_format` change must be mirrored there as well - the routing-change rule above already says the file must be, this extends it to the log format.
 
 ### Notification channels
 
@@ -337,10 +391,12 @@ need a PowerShell Gallery install on every host and fails on an air-gapped
 fleet. `CurrentVersion` is empty on Windows and `NewVersion` carries the KB
 article ID: a Windows update is not a version bump of a named package.
 
-**The agent never reboots a host.** `ApplyUpdatesCmd` installs and stops there;
-`inventory.reboot_required` reports that one is owed, set on the static snapshot
-every 15 minutes. Linux fills it too, from `/var/run/reboot-required` or
-`dnf needs-restarting -r`.
+**The agent reboots a host only when a patch command asks and a reboot is
+owed.** `ApplyUpdatesCmd` carries `scope`, `reboot_if_required` and
+`deadline_unix`; an empty command still means "everything, no reboot". The
+agent answers with `PatchResult` and, when rebooting, sends it first and then
+restarts after a one-minute grace. `inventory.reboot_required` is still set on
+the static snapshot every 15 minutes and at agent start.
 
 ### Package inventory and CVE findings
 
@@ -699,6 +755,15 @@ purpose - the collection is `api_tokens`, the prefix is `vt_`, the routes are
 `/api/tokens`, and renaming a published endpoint to match a nav label would
 break every script already written against it.
 
+**Every `/api` route must be declared twice or the server refuses to boot**:
+once in `routeScopes` (or `sessionOnlyRoutes`) and once in `serverScopedRoutes`.
+`sessionOnlyRoutes` holds the MFA routes, which no API token may reach - a
+token that could enrol a passkey or satisfy step-up would be a way around MFA.
+They have no scope on purpose, and `RequireScopes` refuses any unmapped route
+to a token. `TestRegisteredRoutesPassBootAssertions` registers the real routes
+and runs both boot assertions, so a missing declaration fails CI rather than
+production startup.
+
 `server/internal/api/docs/openapi.json` is a **generated, committed** OpenAPI
 3.1 document - `swag v2` reading `@…` annotations off the handlers - served at
 `GET /api/openapi.json` and rendered as a reference page by a vendored Scalar
@@ -710,6 +775,93 @@ reference that lies. Scalar is vendored (`scalar.standalone.js`, served from
 reference page has to work on an air-gapped install with no outbound access at
 all - the same requirement licence verification already meets.
 
+### Multi-factor authentication
+
+Local and `hq`-sourced members can enrol TOTP and WebAuthn passkeys; OIDC users
+are exempt (`auth_source == "oidc"`), since their IdP owns authentication.
+
+**A password that checks out mints a pending-login ticket, not a session with
+an `mfa_pending` flag.** The ticket is a Redis key (`km:mfa:<id>`, 5 min TTL)
+referenced by a separate `km_mfa_pending` cookie; only the `/auth/mfa/*` and
+`/auth/mfa/enrol/*` endpoints accept it, and it exchanges for a `km_session`
+only on a completed second factor or completed enrolment. A flag on `Session`
+would fail open - any route mounted under `auth.Middleware`, today's or a
+future one, could serve a half-authenticated user by forgetting to check it.
+A ticket fails closed: nothing under `auth.Middleware` recognises it at all,
+because it is never a `*Session`. `Session` itself gained `AMR []string`
+(`pwd`, `otp`, `webauthn`, `recovery`, `oidc`) and `StepUpAt *time.Time`; sign-in
+counts as a step-up, so `StepUpAt` is set at session creation.
+
+**The pending ticket's attempt counter is an atomic Redis counter
+(`km:mfa:<id>:attempts`), not a field rewritten on the ticket document.** Two
+requests racing to fail a guess would otherwise both read the same `attempts`
+and both write it back incremented once, undercounting. `INCR` has no such
+race; the fifth failure deletes the ticket.
+
+**The TOTP replay guard is keyed on the time step, not the code**:
+`km:totp:<user_id>:<step>`, `SET NX` with a 90s TTL. Keying on the code itself
+would let the same 6 digits be replayed across two different steps that
+happen to compute it (a 1-in-a-million collision, but a free one to close);
+keying on the step means a given 30-second window can be spent exactly once,
+which is what "single-use" actually means for a TOTP code.
+
+**`user_mfa.totp_pending_enc` holds an unconfirmed TOTP secret and is
+deliberately not in `vantage-shared`'s `backup.ciphertextFields["user_mfa"]`**,
+which lists only `totp_secret_enc`. The confirmed secret is the one that
+authenticates anyone; an abandoned setup attempt (scanned once, never
+confirmed, replaced by the next `POST /me/mfa/totp/setup` call) is not worth
+widening the backup contract's surface for. `vantagectl verify`'s live probe
+therefore never touches it - this is intentional, not the same silent gap the
+ciphertext-field mirror otherwise guards against.
+
+**Two new collections**, both in `ScopedCollections` so an instance purge
+removes them, both with a fatal index builder like `EnsureAuthIndexes`:
+
+- `user_mfa` - one document per user who has started enrolment. Unique index
+  `{instance_id, user_id}`. `totp_confirmed_at: nil` means setup started but
+  TOTP is not active; "has MFA" means that field is set or the user owns a
+  passkey.
+- `webauthn_credentials` - one document per passkey. Unique index
+  `{instance_id, credential_id}`, plus `{instance_id, user_id}`. `sign_count`
+  backs clone detection: a non-increasing non-zero count fails the assertion.
+
+`require_mfa` (`models.Settings.RequireMFA bool`, `bson:"require_mfa"`)
+shipped in `vantage-shared` v0.7.0 - a plain bool because absent must mean off.
+Switching it on does not revoke existing sessions; they end at their normal
+24h TTL, and the next sign-in enforces enrolment for anyone with no factor yet.
+
+**WebAuthn's RP ID is the request host with any port stripped**, resolved
+per-request rather than configured, the same way the org/host guard resolves
+an instance from `<slug>.vantage.<tld>`. A passkey is bound to the host it was
+registered on: moving a self-hosted instance to a new domain, or renaming a
+cloud instance (see "A rename moves the host" above), invalidates every
+passkey on it. TOTP and recovery codes are unaffected, since they carry no
+host binding. The docs say so; there is no migration path for a passkey
+across a host change.
+
+**Step-up** (`auth.RequireStepUp()`) gates three existing sensitive routes -
+`POST /api/secrets/:group/reveal`, `GET /api/keys/:id/private-key`,
+`POST /api/console/connect` - plus the MFA-management endpoints that create or
+remove a factor. It passes when `StepUpAt` is within the last **ten minutes**,
+when the session's `AMR` contains `oidc` (the IdP's own session policy
+governs), or **when the request authenticated with an API token**
+(`TokenID != ""`). That last exemption is a known, accepted gap, not an
+oversight: a token has no human present to prompt for a second factor, so a
+token holding `secrets:read` or `keys:read` can reveal a secret or download a
+private key with no re-authentication at all. The mitigation is scoped,
+short-lived tokens, tracked separately in the gap review, not a code change
+here - a later reviewer should not "fix" this silently. `POST /api/me/step-up`
+takes `{totp}`, `{recovery}` or `{password}` (password only for a user with no
+MFA); `POST /api/me/step-up/webauthn/begin` and `/finish` do the same with a
+passkey. All three, like every unauthenticated MFA endpoint, sit behind
+`RateLimitAuth()` - a fixed Redis window, 20 requests/minute per
+`c.ClientIP()`, answering 429 with `Retry-After` - on the `RateLimitTokens`
+pattern but for sign-in and re-authentication rather than API tokens.
+
+Library versions: `github.com/pquerna/otp` for TOTP, `github.com/go-webauthn/webauthn`
+**v0.18.1** for WebAuthn ceremonies, `qrcode` (npm) to draw the enrolment QR
+client-side so an air-gapped install needs nothing external.
+
 ### The public host
 
 **vantage.hostxtra.co.uk is not served by this repository.** The marketing site
@@ -887,7 +1039,7 @@ service Vantage {
 }
 ```
 
-`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.
+`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, `StepOutputChunk`, or `PatchResult`. `AgentMessage` now also carries `PatchResult`, the answer to `ApplyUpdatesCmd`.
 
 `ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`, `OpenProxyCmd`, `PingCmd`, `RefreshWorkloadsCmd`, `ControlWorkloadCmd`,
 `WorkloadLogsCmd`.
@@ -920,6 +1072,11 @@ POST /auth/bootstrap     /auth/login  /auth/logout
 GET  /auth/me
 GET  /auth/providers                          # {local_enabled, providers:[{id,name,preset}]} - no issuer, client ID or secret
 GET  /api/secrets/:group/values               # bearer token (ESO)
+POST /auth/mfa/totp  /auth/mfa/recovery       # second factor against a pending-login ticket
+POST /auth/mfa/webauthn/begin  /finish
+POST /auth/passkey/begin  /auth/passkey/finish            # passwordless sign-in
+POST /auth/mfa/enrol/totp/setup  /confirm                 # ticket-scoped forced enrolment
+POST /auth/mfa/enrol/passkey/begin  /finish
 ```
 
 Session-authed under `/api`:
@@ -936,7 +1093,8 @@ workflows    GET,POST /steps · PUT,DELETE /steps/:id · GET /steps/:id/export
              GET /runs/:runId · POST /runs/:runId/cancel
              GET /runs/:runId/servers/:serverId/logs[/stream]
 monitors     GET,POST /monitors · GET,PUT,DELETE /monitors/:id
-             GET /monitors/:id/{incidents,uptime}
+             GET /monitors/:id/{incidents,uptime,samples,servers}
+             POST /monitors/:id/rotate-token
 channels     GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:id/test
 secrets      GET,POST /secrets · GET,PUT,DELETE /secrets/:group
              POST /secrets/:group/reveal · DELETE /secrets/:group/:key
@@ -955,11 +1113,22 @@ status-pages GET,POST /status-pages · GET,PUT,DELETE /status-pages/:pageId  (ow
              GET,POST /status-pages/:pageId/incidents
              PUT,DELETE /status-pages/:pageId/incidents/:incidentId
              POST /status-pages/:pageId/incidents/:incidentId/updates
+patching     GET,POST /maintenance-windows · POST /maintenance-windows/preview
+             GET,PUT,DELETE /maintenance-windows/:id                (writes: owner|admin)
+             GET,POST /patch-policies · GET,PUT,DELETE /patch-policies/:id
+             POST /patch-policies/:id/run-now                        (writes: owner|admin)
+             GET /patch-runs · GET /patch-runs/:runId · POST /patch-runs/:runId/cancel
 audit        GET /audit
 agent        GET /agent/latest-version
 settings     GET,PUT /settings · POST /settings/secrets-token       (owner|admin)
 licence      GET /license · POST /license                (POST: self-hosted only)
 org          GET,POST /org/users · PUT /org/users/:id/role · DELETE /org/users/:id
+             DELETE /org/users/:id/mfa                             (owner|admin, step-up)
+mfa          GET /me/mfa · POST /me/mfa/totp/setup                (step-up)
+             POST /me/mfa/totp/confirm · DELETE /me/mfa/totp        (step-up)
+             POST /me/mfa/recovery/regenerate                       (step-up)
+             POST,PATCH,DELETE /me/passkeys[/begin,/finish,/:id]    (step-up, except rename)
+             POST /me/step-up · POST /me/step-up/webauthn/begin  /finish
 providers    GET,POST /auth/providers · PUT,DELETE /auth/providers/:id
              POST /auth/providers/:id/{test,ack-notice} · GET /auth/presets  (owner|admin)
 tokens       GET /tokens · GET /tokens/scopes · POST /tokens · DELETE /tokens/:id
@@ -999,10 +1168,23 @@ plane, each of which this codebase enforces:
 - **`FREE_INSTANCE_REAP_AFTER` must match admin's value.** Admin names the date
   in its warning emails; this side performs the delete, because it is the only
   service that knows which collections carry `instance_id`.
+- **Disputes lock and purge instances.** HQ writes `instances.locked_at` when
+  an account is disputed and `instances.purge_after` only once the dispute
+  fails. `services.InstanceLocked` (60s cache) refuses sessions, API tokens
+  and agents on a locked instance, and `InstanceForHost` and `SoleInstance`
+  resolve it to nothing. The check in `auth.Middleware` is explicit because the
+  host guard only runs when a host resolves. `HostLocked` is the one place that
+  tells "locked" from "unknown", so the login page can say access is suspended
+  rather than draw a form; `resolveLoginInstance` refuses a locked instance on
+  the single-instance fallback too, which otherwise served its sign-in page.
+  A locked instance's public status page stays an ordinary 404. `ReapTerminatedInstances` purges once
+  both fields are present and `purge_after` has passed, independent of
+  `FREE_INSTANCE_REAP_AFTER`, and `ReapFreeInstances` skips locked instances
+  so a restore finds them intact.
 
 ## MongoDB Collections
 
-`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `server_workloads` · `api_tokens` · `status_pages` · `status_incidents` · `migrations`
+`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `monitor_server_states` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `server_workloads` · `api_tokens` · `status_pages` · `status_incidents` · `maintenance_windows` · `patch_policies` · `patch_runs` · `patch_run_outputs` · `user_mfa` · `webauthn_credentials` · `migrations`
 
 Every document except `migrations` carries `org_id`. Struct definitions are the source of truth - see `server/internal/models/`.
 

deploy/docker/nginx/vantage.conf

@@ -12,6 +12,17 @@ map $http_upgrade $connection_upgrade {
     ''      close;
 }
 
+# Heartbeat ping URLs carry a credential. Log them with the token replaced;
+# the header form (X-Vantage-Token) is never logged by this format.
+map $request_uri $vantage_log_uri {
+    "~^/public/hb/(?!start(?:[/?]|$)|fail(?:[/?]|$))[^/?]+(?<hb_rest>.*)$" "/public/hb/***$hb_rest";
+    default $request_uri;
+}
+
+log_format vantage '$remote_addr - $remote_user [$time_local] '
+                   '"$request_method $vantage_log_uri $server_protocol" '
+                   '$status $body_bytes_sent "$http_referer" "$http_user_agent"';
+
 upstream vantage_server {
     server server:8080;
     keepalive 16;
@@ -27,6 +38,8 @@ server {
     listen [::]:80;
     server_name _;
 
+    access_log /var/log/nginx/access.log vantage;
+
     # Step imports and licence pastes are the largest request bodies.
     client_max_body_size 10m;
 

docs/superpowers/specs/2026-09-14-scheduled-patching-design.md

@@ -0,0 +1,539 @@
+# Maintenance windows and scheduled patching
+
+Date: 2026-09-14
+
+## Goal
+
+Let an operator say "install security updates on every `env:prod` server,
+Sundays 02:00 to 04:00 Europe/London, and reboot them if the OS says a reboot
+is owed", and have Vantage do it, report exactly what happened per server, and
+say so loudly when it did not.
+
+Today the only patching path is **Apply updates**, which runs immediately on one
+server, installs everything, never reboots, and reports nothing back: the agent
+logs a failure locally and the control plane never hears of it.
+
+Out of scope, deliberately:
+
+- **Update rings / staged rollout.** A policy runs all its targets within one
+  window, bounded only by `max_concurrent`. Rings are their own feature and
+  build on this one.
+- **Package holds and per-package allow-lists.** Scope is `all` or `security`.
+- **Live-streamed patch output.** A capped output tail is stored per server;
+  `workflow_log_lines` is not reused.
+- **Alert muting during windows.** The window is a standalone object so this can
+  reference it later, but v1 does not mute anything.
+- **Licence gating.** Patching is free on every tier, consistent with
+  `applyUpdates` already being exempt from the licence gate.
+
+## Current state
+
+- `POST /api/servers/:id/apply-updates` calls `services.DispatchApplyUpdates`,
+  which sends an empty `ApplyUpdatesCmd` and returns. No result is awaited or
+  recorded.
+- Agent `handleApplyUpdates` runs `updates.ApplyAll()` and, on success only,
+  sends an empty `ReportUpdates`. Output is discarded (`exec.Cmd.Run()`).
+- The apt path wraps `apt-get update` **and** `apt-get upgrade` in one 5-minute
+  context, so a large upgrade can be killed partway. dnf, yum, zypper, pacman
+  and apk run with no timeout at all.
+- The agent never reboots. `inventory.reboot_required` is set on the 15-minute
+  static snapshot, and at agent start.
+- `workflowsched` already solves scheduling: 5-field cron, IANA timezone,
+  `next_run_at` persisted and claimed atomically, recorded skips, run inside
+  `bus.RunAsLeader("housekeeping", ...)`.
+- `default_steps/` ships `apply_package_updates` and `reboot_server` bash steps.
+  They stay as they are; this feature does not use them.
+
+## Approach
+
+A native patch path, not a workflow convention. The agent gains a result-bearing
+update command with a scope and an opt-in reboot. Three new collections hold
+windows, policies and runs. A new `patchsched` loop fires policies and advances
+runs from state held in MongoDB, so a leader handover mid-window loses nothing.
+The manual button and the vulnerability page move onto the same run model, so
+every patch Vantage performs has a record.
+
+Rejected alternatives:
+
+- **Build on the workflow engine** (script steps per package manager). The patch
+  logic would exist twice, as scripts and as the agent's `updates` package; the
+  engine sends one step list to every target, so a mixed Linux/Windows policy
+  needs per-OS branching it does not have; and the manual button would stay
+  fire-and-forget.
+- **Native command, recorded as a `WorkflowRun`.** Saves one page, but binds
+  patching to `steps_snapshot` and log sequencing that do not describe it, and
+  breaks the rule that a run always shows the script that ran.
+
+## Data model
+
+All three collections carry `instance_id` and are added to
+`services.ScopedCollections`, so instance purge covers them.
+
+### `maintenance_windows`
+
+```go
+type MaintenanceWindow struct {
+    ID         bson.ObjectID `bson:"_id,omitempty" json:"-"`
+    InstanceID string        `bson:"instance_id" json:"instance_id"`
+    WindowID   string        `bson:"window_id" json:"window_id"`
+    Name       string        `bson:"name" json:"name"`
+    Cron       string        `bson:"cron" json:"cron"`         // 5-field, window start
+    TZ         string        `bson:"tz" json:"tz"`             // IANA name
+    DurationMinutes int      `bson:"duration_minutes" json:"duration_minutes"` // 15..720
+    CreatedAt  time.Time     `bson:"created_at" json:"created_at"`
+    UpdatedAt  time.Time     `bson:"updated_at" json:"updated_at"`
+}
+```
+
+A window answers "when" and nothing else: no targets, no behaviour. Cron and TZ
+are validated through `workflowsched.NextOccurrence`, exactly as workflow
+schedules are. Deleting a window referenced by any policy is refused with 409
+`window_in_use`. Editing a window recomputes `next_run_at` on every policy that
+references it, in the same service call.
+
+### `patch_policies`
+
+```go
+type PatchPolicy struct {
+    ID               bson.ObjectID     `bson:"_id,omitempty" json:"-"`
+    InstanceID       string            `bson:"instance_id" json:"instance_id"`
+    PolicyID         string            `bson:"policy_id" json:"policy_id"`
+    Name             string            `bson:"name" json:"name"`
+    Enabled          bool              `bson:"enabled" json:"enabled"`
+    WindowID         string            `bson:"window_id" json:"window_id"`
+    TargetServerIDs  []string          `bson:"target_server_ids" json:"target_server_ids"`
+    TargetTags       map[string]string `bson:"target_tags,omitempty" json:"target_tags,omitempty"`
+    Scope            string            `bson:"scope" json:"scope"`   // "all" | "security"
+    Reboot           string            `bson:"reboot" json:"reboot"` // "never" | "if_required"
+    MaxConcurrent    int               `bson:"max_concurrent" json:"max_concurrent"` // 0 = no cap
+    NotifyChannelIDs []string          `bson:"notify_channel_ids,omitempty" json:"notify_channel_ids,omitempty"`
+    NextRunAt        *time.Time        `bson:"next_run_at,omitempty" json:"next_run_at,omitempty"`
+    LastRunAt        *time.Time        `bson:"last_run_at,omitempty" json:"last_run_at,omitempty"`
+    LastSkipped      *Skip             `bson:"last_skipped,omitempty" json:"last_skipped,omitempty"`
+    DisabledReason   string            `bson:"disabled_reason,omitempty" json:"disabled_reason,omitempty"`
+    CreatedAt        time.Time         `bson:"created_at" json:"created_at"`
+    UpdatedAt        time.Time         `bson:"updated_at" json:"updated_at"`
+}
+```
+
+Targets use the workflow selector semantics unchanged: the distinct union of
+named servers and tag matches, resolved through `services.ResolveTargets` at
+fire time, and an empty selector matches nothing (`ErrNoTargets` on save).
+`Skip` is the existing `models.Skip`.
+
+Index: `{instance_id: 1, enabled: 1, next_run_at: 1}`.
+
+### `patch_runs`
+
+```go
+type PatchRun struct {
+    ID          bson.ObjectID    `bson:"_id,omitempty" json:"-"`
+    InstanceID  string           `bson:"instance_id" json:"instance_id"`
+    RunID       string           `bson:"run_id" json:"run_id"`
+    PolicyID    string           `bson:"policy_id,omitempty" json:"policy_id,omitempty"` // empty for manual
+    PolicyName  string           `bson:"policy_name,omitempty" json:"policy_name,omitempty"`
+    TriggeredBy string           `bson:"triggered_by" json:"triggered_by"` // "schedule" | actor
+    Source      string           `bson:"source" json:"source"`             // schedule | server | vulnerabilities | mcp | run_now
+    CancelledAt *time.Time       `bson:"cancelled_at,omitempty" json:"cancelled_at,omitempty"`
+    Scope       string           `bson:"scope" json:"scope"`               // snapshot
+    Reboot      string           `bson:"reboot" json:"reboot"`             // snapshot
+    MaxConcurrent int            `bson:"max_concurrent" json:"max_concurrent"` // snapshot
+    WindowEnd   *time.Time       `bson:"window_end,omitempty" json:"window_end,omitempty"` // nil for manual
+    Status      string           `bson:"status" json:"status"` // running | succeeded | partial | failed | cancelled
+    StartedAt   time.Time        `bson:"started_at" json:"started_at"`
+    FinishedAt  *time.Time       `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
+    Servers     []PatchServerRun `bson:"servers" json:"servers"`
+}
+
+type PatchServerRun struct {
+    ServerID     string     `bson:"server_id" json:"server_id"`
+    Hostname     string     `bson:"hostname" json:"hostname"`
+    Status       string     `bson:"status" json:"status"`
+    CommandID    string     `bson:"command_id,omitempty" json:"-"`
+    PendingBefore int       `bson:"pending_before" json:"pending_before"`
+    PendingAfter  *int      `bson:"pending_after,omitempty" json:"pending_after,omitempty"`
+    RebootedAt   *time.Time `bson:"rebooted_at,omitempty" json:"rebooted_at,omitempty"`
+    VerifiedAt   *time.Time `bson:"verified_at,omitempty" json:"verified_at,omitempty"`
+    Output       string     `bson:"output,omitempty" json:"output,omitempty"` // tail, max 64KB
+    Error        string     `bson:"error,omitempty" json:"error,omitempty"`
+    StartedAt    *time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
+    FinishedAt   *time.Time `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
+}
+```
+
+Server run statuses:
+
+| Status            | Terminal | Meaning                                                              |
+| ----------------- | -------- | -------------------------------------------------------------------- |
+| `queued`          | no       | Waiting for a concurrency slot                                       |
+| `waiting_offline` | no       | Agent not connected; retried each tick while the window is open      |
+| `patching`        | no       | Command dispatched, no result yet                                    |
+| `rebooting`       | no       | Agent announced a reboot; awaiting a post-boot inventory report      |
+| `succeeded`       | yes      | Patched, and rebooted and verified if a reboot was owed and allowed  |
+| `failed`          | yes      | Package manager failed, no result, or reboot not verified            |
+| `unsupported`     | yes      | Security-only requested on a host with no security metadata         |
+| `agent_too_old`   | yes      | Agent predates patch results; not dispatched                         |
+| `missed_offline`  | yes      | Offline for the whole window                                         |
+| `window_closed`   | yes      | Still queued when the window ended                                   |
+| `cancelled`       | yes      | Run cancelled before this server was dispatched                      |
+
+Scope, reboot and concurrency are snapshotted onto the run so editing a policy
+never changes what a past run shows. `PendingBefore` is the server's
+`available_updates` count at dispatch; `PendingAfter` comes from the result.
+
+Indexes: `{instance_id: 1, policy_id: 1, started_at: -1}`,
+`{instance_id: 1, "servers.server_id": 1, started_at: -1}`,
+`{status: 1}` (the tick's scan), and `{"servers.command_id": 1}` (result lookup).
+Runs are swept under `workflow_log_retention_days` by the existing log sweeper.
+
+## Wire contract
+
+Changed in `vantage-shared` (`grpc/pb` and `proto/vantage/v1/vantage.proto` in
+the same commit):
+
+```proto
+message ApplyUpdatesCmd {
+  string scope = 1;              // "" or "all" | "security"
+  bool   reboot_if_required = 2;
+  int64  deadline_unix = 3;      // 0 = none; the agent caps at 2h
+}
+
+message PatchResult {            // new AgentMessage oneof variant: PatchResult patch_result = 8;
+  string command_id = 1;
+  string status = 2;             // ok | failed | unsupported | busy
+  string message = 3;
+  string output_tail = 4;        // at most 64KB, newest bytes kept
+  int32  pending_after = 5;
+  bool   reboot_required = 6;
+  bool   rebooting = 7;
+}
+
+message InventoryReport {
+  // existing fields ...
+  int64 boot_time_unix = 11;     // every report
+}
+```
+
+The existing comment on `InventoryReport.reboot_required` ("The agent never
+reboots") is reworded in the same commit to match the new reboot rule.
+
+An empty `ApplyUpdatesCmd` means `scope: all`, no reboot, no deadline: exactly
+today's behaviour, so a new agent under an old server is unaffected.
+
+### Old agents
+
+An old agent ignores the new fields. Under `scope: security` it would install
+**everything**, and it sends no `PatchResult`. The server therefore gates on
+`servers.agent_version`:
+
+- `agentSupportsPatchResults(version)` is a semver comparison against the first
+  agent release carrying this feature. Empty, unparseable and dev versions are
+  treated as too old.
+- A policy run marks an older server `agent_too_old` without dispatching.
+- A manual Apply updates on an older server still dispatches the empty command,
+  records the server run as `succeeded` immediately with
+  `error: "no result reported: agent predates patch results"`, and the UI shows
+  it as unverified. The button keeps working through the transition.
+
+## Agent
+
+`internal/updates`:
+
+- `Apply(opts ApplyOptions) (Result, error)` replaces `ApplyAll()`.
+  `ApplyOptions{Scope string; Deadline time.Time}`. `Result{Output []byte;
+  Unsupported bool}`.
+- A package-level mutex: a second `Apply` while one runs returns `ErrBusy`.
+- Index refresh (`apt-get update`, and nothing else) keeps its own 5-minute
+  timeout. The upgrade runs under a context ending at `Deadline`, or 2h from
+  start when no deadline is given. The existing single 5-minute context is
+  removed.
+- Combined stdout and stderr go to a 64KB ring buffer that keeps the newest
+  bytes.
+- Non-interactive everywhere: `DEBIAN_FRONTEND=noninteractive` and
+  `-o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold` for
+  apt, `--non-interactive` for zypper, `--noconfirm` stays for pacman.
+
+Security-only, per package manager:
+
+| Manager   | Command                                                                                              | No security metadata |
+| --------- | ---------------------------------------------------------------------------------------------------- | -------------------- |
+| apt       | temp sources list of entries whose suite ends in `-security`, from `/etc/apt/sources.list`, `*.list` and deb822 `*.sources`; `apt-get update` and `apt-get upgrade -y` with `-o Dir::Etc::SourceList=<tmp> -o Dir::Etc::SourceParts=-` | no such entries: `unsupported` |
+| dnf / yum | `<pm> upgrade --security -y`                                                                          | n/a                  |
+| zypper    | `zypper --non-interactive patch --category security`                                                  | n/a                  |
+| pacman    | none                                                                                                  | always `unsupported` |
+| apk       | none                                                                                                  | always `unsupported` |
+| Windows   | existing COM search, keeping updates whose `Categories` include Security Updates (`0FA1201D-4330-4FA8-8AE9-B877473B6441`) or Critical Updates (`E6CF1350-C01B-414D-A61F-263D14D133B4`) | n/a |
+
+`unsupported` installs nothing. It never falls back to `all`.
+
+apt security source filtering is a pure function over file contents
+(`securitySources(files map[string]string) (list string, ok bool)`) so it is
+testable on any platform.
+
+`internal/sync` `handleApplyUpdates`:
+
+1. `updates.Apply(...)` with the command's scope and deadline.
+2. Re-run `CheckAvailable`, `ReportUpdates` the result (on success **and**
+   failure, so counts are fresh either way), read `RebootRequired()`.
+3. If `reboot_if_required`, a reboot is owed, and at least 5 minutes remain
+   before the deadline: send `PatchResult{status: ok, rebooting: true, ...}` on
+   the command stream, then run `shutdown -r +1` (Linux) or
+   `shutdown /r /t 60 /c "Vantage patch policy"` (Windows). The one-minute
+   grace lets the result leave before the host goes down.
+4. Otherwise send `PatchResult` with the outcome.
+
+`internal/inventory` reports `boot_time_unix` on every report (Linux
+`/proc/stat` `btime`, Windows via the existing collection).
+
+The agent's documented promise changes from "never reboots a host" to "never
+reboots a host unless the command explicitly asks and the OS reports a reboot
+is owed". `vantage-agent/CLAUDE.md` and this repository's CLAUDE.md are updated
+in the same change.
+
+## Scheduler and run lifecycle
+
+### `patchsched`
+
+Runs inside the existing `RunAsLeader("housekeeping", ...)`, ticking every 30s.
+It must not import `services` (the same cycle `workflowsched` avoids); its
+dependencies are injected from `main.go` as `patchsched.Deps`.
+
+Each tick does two things.
+
+**Fire due policies.** For each enabled policy with `next_run_at <= now`, claim
+it with the `workflowsched` pattern: `UpdateOne` matching `policy_id` **and**
+the current `next_run_at`, setting the next occurrence. A zero match means
+another process claimed it. Then `Decide(due, windowEnd, now, running, nTargets)`:
+
+| Condition                                             | Result                                             |
+| ----------------------------------------------------- | -------------------------------------------------- |
+| window missing or no longer parses                    | disable policy, set `disabled_reason`, audit        |
+| `now >= windowEnd` or `now > due + 1h`                | skip `missed`                                       |
+| a run of this policy has `status: running`            | skip `already_running`                              |
+| targets resolve to zero servers                       | skip `no_targets`                                   |
+| otherwise                                             | fire                                               |
+
+Skips set `last_skipped` and write `patch.skipped`, as workflow skips do. Firing
+creates a `patch_run` with every target `queued` (or `agent_too_old`),
+`WindowEnd = due + duration`, sets `last_run_at`, and writes `patch.run_started`
+with actor `schedule`.
+
+**Advance running runs.** For each `patch_run` with `status: running`, load
+connection state for its servers and apply `Advance(run, now, connected)`, a
+pure function returning the transitions:
+
+- While `now < WindowEnd` (or always, for a manual run): move `queued` and
+  `waiting_offline` servers to `patching` up to `MaxConcurrent` in flight
+  (`patching` plus `rebooting`), dispatching `ApplyUpdatesCmd` with
+  `deadline_unix = WindowEnd`. A dispatch that fails (503, agent offline) moves
+  the server to `waiting_offline`.
+- When `now >= WindowEnd`: `queued` to `window_closed`, `waiting_offline` to
+  `missed_offline`.
+- `patching` with no result past `deadline + 10m` (manual: start + 2h + 10m):
+  `failed`, `"no result from agent"`.
+- `rebooting` with no verifying report past `RebootedAt + 20m`: `failed`,
+  `"did not come back within 20 minutes"`.
+- When every server is terminal, `Finalize` sets the run status: `succeeded` if
+  all servers succeeded, `failed` if none did, otherwise `partial`.
+  `unsupported`, `agent_too_old`, `missed_offline` and `window_closed` are not
+  successes. Then notify (below) and write `patch.run_finished`.
+
+Every transition is written with a filter on the server's current status, so a
+result arriving concurrently is never overwritten by a stale tick.
+
+### Results
+
+`CommandStream`, on whichever pod holds the agent's stream, handles
+`PatchResult` by updating the server run found by `servers.command_id`, via
+`ApplyResult(serverRun, result, now)`:
+
+- `ok` without `rebooting`: `succeeded`, `PendingAfter`, output.
+- `ok` with `rebooting`: `rebooting`, `RebootedAt = now`, write `patch.reboot`.
+- `failed` or `busy`: `failed` with the message.
+- `unsupported`: `unsupported`.
+
+Nothing awaits the result on the bus. The run document is the only state, so a
+pod or leader change mid-window loses nothing. A result for an unknown command
+ID is dropped and logged.
+
+### Reboot verification
+
+`ReportInventory` checks, for this server, any server run in `rebooting`, and
+only on reports with `include_static` set: `reboot_required` is only computed
+on static snapshots, so a metrics-only report would read as "no reboot owed".
+The agent sends a static snapshot at start, so the first report after a reboot
+qualifies.
+`VerifyReboot(serverRun, bootTime, rebootRequired, now)`:
+
+- `bootTime <= RebootedAt`: not yet rebooted, no change. This is why boot time
+  is used rather than "a report arrived": a static snapshot sent during the
+  one-minute grace must not count.
+- `bootTime > RebootedAt` and `reboot_required` false: `succeeded`,
+  `VerifiedAt = now`.
+- `bootTime > RebootedAt` and `reboot_required` still true: `failed`,
+  `"still requires a reboot after restarting"`.
+
+### Manual runs
+
+`POST /servers/:id/apply-updates` and the vulnerability page's Apply updates
+create a one-server `patch_run` (`scope: all`, `reboot: never`, no window,
+`TriggeredBy` the actor, `Source` one of `server`, `vulnerabilities` or `mcp`)
+and dispatch immediately in
+the handler rather than waiting for the next tick. A dispatch failure still
+answers 503, and the run is recorded as `failed` so the attempt is not lost.
+Timeouts and finalisation go through the same tick.
+
+### Cancel
+
+`POST /patch-runs/:runId/cancel` moves `queued` and `waiting_offline` servers to
+`cancelled`. Servers already `patching` or `rebooting` continue: interrupting a
+package manager mid-transaction is worse than letting it finish. The run
+finalises as `cancelled` once the in-flight servers settle, unless none were in
+flight, in which case immediately.
+
+### Notifications
+
+When a run finalises as `partial` or `failed`, one summary goes to each channel
+in `NotifyChannelIDs` through the existing `notify` dispatch, as a new event
+type `patch`: `[Vantage] Patch policy "Sunday prod" partial: 38 succeeded,
+2 failed, 1 missed offline (run 3f2a...)`. The webhook payload is the existing
+event shape with that summary as its message; no new payload fields. A clean
+run sends nothing.
+
+## REST API
+
+All under `/api`, each route registered in `routeScopes` (so
+`AssertScopeMapComplete` passes) and annotated for `swag`; `openapi.json` is
+regenerated and committed.
+
+```
+maintenance-windows  GET,POST /maintenance-windows · GET,PUT,DELETE /maintenance-windows/:id
+                     POST /maintenance-windows/preview        {cron,tz,duration_minutes} -> next 3 {start,end}
+patch-policies       GET,POST /patch-policies · GET,PUT,DELETE /patch-policies/:id
+                     POST /patch-policies/:id/run-now          window = now .. now + window duration
+patch-runs           GET /patch-runs?policy_id=&server_id=&limit= · GET /patch-runs/:runId
+                     POST /patch-runs/:runId/cancel
+servers              POST /servers/:id/apply-updates           202 {run_id, message}
+```
+
+- New scope resource **`patching`** (`:read`, `:write`). Windows, policies and
+  runs use it. `apply-updates` stays on `servers:write`.
+- Creating, editing and deleting windows and policies, and run-now, are
+  owner or admin (`RequireRole`). Viewing and cancelling runs are open to every
+  role. Apply updates keeps its current access.
+- Saving a policy through a tag-restricted token is refused if its selector
+  reaches outside the restriction, reusing `validateWorkflowTargetScope`
+  unchanged. Scheduled firing passes a nil token scope, as workflow schedules do.
+- `apply-updates` keeps `message` in its response for existing scripts and adds
+  `run_id`.
+- MCP `apply_updates` returns the `run_id`. No new MCP tools.
+
+Errors: `400` validation (`invalid_cron`, `invalid_tz`, `invalid_duration`,
+`invalid_scope`, `invalid_reboot`, `no_targets`), `404` unknown window, policy
+or run, `409 window_in_use`, `503` agent offline on apply-updates.
+
+## Audit
+
+Every event uses the `patch.` prefix so the audit page groups them under one
+category: `patch.window_created|window_updated|window_deleted`,
+`patch.policy_created|policy_updated|policy_deleted|policy_disabled`,
+`patch.run_started`, `patch.run_finished`, `patch.skipped`, `patch.cancelled`,
+`patch.reboot` (one per server Vantage reboots, naming the policy),
+and the existing `updates.applied` for manual runs, now carrying the run ID.
+
+## Frontend
+
+`web/`, dark tokens only, no hex, pills carry shape and label.
+
+- Sidebar: **Patching** in the Fleet group.
+- `/patching`, three tabs:
+  - **Policies**: name, next window in its own timezone, resolved target count
+    via `web/lib/targets.ts`, scope and reboot chips, last-run pill, enabled
+    toggle. A disabled policy shows its `disabled_reason`.
+  - **Windows**: name, schedule in words, duration, policies using it.
+  - **Runs**: newest first, filter by policy.
+- **Policy editor** (modal): name; window picker with inline "New window";
+  targets with `DualListBox` plus tag rows; scope radio, noting apk and pacman
+  hosts report unsupported for security-only; reboot radio, where "If required"
+  shows "Up to N servers may reboot during this window"; max concurrent;
+  notification channels. Shows "N of M targets need an agent update" linking to
+  the servers when any target is too old.
+- **Window editor** (modal): name, presets writing cron (Nightly 02:00,
+  Sunday 02:00, Saturday 22:00, Monthly 1st 02:00), cron field, timezone,
+  duration; next 3 occurrences from `/maintenance-windows/preview`.
+- `/patching/runs/[runId]`: header with status, trigger, window end and counts
+  per status; table of servers with status pill, updates installed
+  (`PendingBefore - PendingAfter`), rebooted and verified times, error; each row
+  expands to the output tail on the `--well` surface. Cancel while running. The
+  page polls while the run is `running`.
+- **Server detail, OS updates panel**: "Covered by *Sunday prod*, next window
+  Sun 21 Sep 02:00 BST" or "Not covered by any patch policy"; last patch run
+  with link. Apply updates navigates to the new run.
+- **Vulnerabilities**: Apply updates navigates to the new run.
+
+## Documentation
+
+`vantage-docs`:
+
+- New `docs/vantage/patching.md`: windows, policies, scope per package manager
+  (including unsupported), reboot rule and verification, statuses table,
+  offline and window-close behaviour, agent version requirement.
+- `vantage/servers.md`: remove the "Applying updates is not scheduled or staged"
+  warning, describe the run record, update the reboot sentence.
+- `vantage/vulnerabilities.md`: Apply updates creates a run; link to patching.
+- `hq/licensing-and-entitlements.md`: patching is available on every tier.
+- `reference/api-tokens.md`: the `patching` scope.
+
+`vantage-app/CLAUDE.md`: new "Scheduled patching" subsystem section, updated
+"Inventory and OS updates", collection list, REST list, `ServerCommand` and
+`AgentMessage` variants. `vantage-agent/CLAUDE.md`: the reboot promise.
+
+## Testing
+
+The repository tests pure functions without a database, so the logic is shaped
+for that and the Mongo layer is a thin shell.
+
+Server:
+
+- `patchsched.Decide`: on time, late within grace, past grace, past window
+  end, already running, zero targets, and a DST case (`Europe/London`, last
+  Sunday of October, 01:30 start occurring twice; `NextOccurrence` behaviour is
+  asserted, not assumed).
+- `patchrun.Advance`: concurrency cap never exceeded counting `rebooting`;
+  offline then online inside the window dispatches; offline to window end is
+  `missed_offline`; queued at window end is `window_closed`; nothing dispatched
+  after window end; no-result timeout; reboot timeout; cancelled runs dispatch
+  nothing; manual runs ignore window rules.
+- `patchrun.ApplyResult` for each result status.
+- `patchrun.VerifyReboot`: boot time before, equal to and after `RebootedAt`,
+  with reboot still owed and cleared.
+- `patchrun.Finalize`: every mix of terminal statuses.
+- `agentSupportsPatchResults`: older, equal, newer, pre-release, empty,
+  unparseable.
+- Validation of windows and policies.
+- `scopes_test` covers the new routes; a token tag-restriction case in the
+  style of `workflow_target_scope_test`.
+
+Agent:
+
+- `securitySources`: Debian and Ubuntu `.list`, Ubuntu 24.04 deb822 `.sources`,
+  commented lines, a file set with no security suites (`ok == false`).
+- Ring buffer keeps the newest 64KB.
+- Windows security category filtering, as a parser test beside `winparse_test`.
+- Manual verification on Debian 12, Ubuntu 24.04, Rocky 9 and Windows Server
+  2022: security-only and all, reboot and verification, window close
+  mid-queue, agent busy.
+
+## Rollout
+
+1. `vantage-shared`: `ApplyUpdatesCmd` fields, `PatchResult`,
+   `InventoryReport.boot_time_unix`, proto and `pb` together. Release a tag.
+2. `vantage-app`: bump the pin; ship collections, indexes, `patchsched`,
+   results handling, API, UI, MCP change and docs. With no new agents yet,
+   policies show every target as `agent_too_old` and the manual button works as
+   before, now with a run record.
+3. `vantage-agent`: bump the pin, ship the agent changes, tag `agent/v*`. Set
+   that version as the gate constant in the server in step 2 (the gate names a
+   version that does not exist yet until step 3 ships, which is harmless: every
+   agent reads as too old until it does).
+4. `vantage-docs`: publish the patching page.

docs/superpowers/specs/2026-09-15-mfa-local-signin-design.md

@@ -0,0 +1,342 @@
+# MFA for local sign-in (TOTP and passkeys) - design
+
+Date: 2026-09-15
+Gap review entry: `#r-mfa` ("Build next", effort S - revised upward, see Scope)
+
+## Problem
+
+Local and HQ-projected members sign in with an email address and a password and
+nothing else. MFA exists only through SSO, which is a paid feature. One phished
+password on a Free instance gives an attacker root script execution across the
+fleet, root consoles and decrypted private keys. MFA is a hard requirement on
+most security questionnaires and must ship on every tier, with no licence gate.
+
+## Scope
+
+In:
+
+- TOTP as a second factor, with 10 single-use recovery codes.
+- WebAuthn passkeys, both as a second factor after a password and as
+  passwordless sign-in (discoverable credentials, user verification preferred).
+- An owner setting, `require_mfa`, that forces enrolment for password users.
+- Owner/admin reset of another member's MFA.
+- Step-up re-authentication before three sensitive actions: secret reveal,
+  private key download, console connect.
+- A per-IP rate limit on every unauthenticated sign-in endpoint.
+
+Out:
+
+- MFA owned by Vantage HQ and projected to instances. HQ users enrol per
+  instance, like local users.
+- Step-up for API tokens (see Known limitations).
+- Forcing OIDC users through their IdP again (`prompt=login`) for step-up.
+- A grace period for `require_mfa`.
+- SMS or email codes.
+
+## Who MFA applies to
+
+| `auth_source` | Can enrol | Covered by `require_mfa` | Step-up factor              |
+| ------------- | --------- | ------------------------ | --------------------------- |
+| `local`       | yes       | yes                      | MFA if enrolled, else password |
+| `hq`          | yes       | yes                      | MFA if enrolled, else password |
+| `oidc`        | no        | no                       | none - passes through       |
+
+OIDC users are exempt because their IdP owns authentication; their session
+carries `amr: ["oidc"]`.
+
+## Approach: pending-login ticket
+
+A password that checks out no longer mints a session when the user has MFA or
+must enrol. It mints a **pending-login ticket** instead: a Redis key referenced
+by a separate short-lived cookie. Only the MFA endpoints accept the ticket, and
+only a completed second factor (or completed enrolment) exchanges it for a
+`km_session`.
+
+The reason for this shape over "a session with an `mfa_pending` flag" is that a
+half-authenticated user never becomes a `*Session` at all, so no route mounted
+under `auth.Middleware` - today's or a future one - can serve them by
+forgetting a check. The flag design fails open; this one fails closed.
+
+## Data model
+
+### `user_mfa` (control plane, new)
+
+One document per user who has started enrolment.
+
+| Field               | Type        | Notes |
+| ------------------- | ----------- | ----- |
+| `instance_id`       | string      | tenant scope |
+| `user_id`           | string      | unique index `{instance_id, user_id}` |
+| `webauthn_handle`   | binary(64)  | random, never the `user_id` (WebAuthn user handle) |
+| `totp_secret_enc`   | string      | AES-256-GCM via `services.encryptString`; empty when TOTP is off |
+| `totp_confirmed_at` | *time       | nil means setup started but not confirmed - TOTP not active |
+| `recovery_codes`    | []{hash, used_at} | 10 codes, SHA-256 of the normalised code |
+| `updated_at`        | time        | |
+
+A user "has MFA" when `totp_confirmed_at` is set or they own at least one
+passkey. A pending unconfirmed TOTP secret is replaced on the next setup call.
+
+### `webauthn_credentials` (control plane, new)
+
+One document per passkey.
+
+| Field           | Type     | Notes |
+| --------------- | -------- | ----- |
+| `instance_id`   | string   | |
+| `user_id`       | string   | index `{instance_id, user_id}` |
+| `credential_id` | binary   | unique index `{instance_id, credential_id}` |
+| `public_key`    | binary   | COSE key; not secret |
+| `sign_count`    | uint32   | clone detection: a non-increasing non-zero count fails the assertion |
+| `aaguid`        | binary   | |
+| `transports`    | []string | |
+| `name`          | string   | user-editable, default from AAGUID or "Passkey" |
+| `created_at`, `last_used_at` | time | |
+
+Both collections are added to `ScopedCollections`, with a test asserting it,
+so an instance purge removes them. Their index builder is fatal on failure,
+like `EnsureAuthIndexes` - the unique indexes are a security property.
+
+### `vantage-shared` changes (one release, then bump `server`'s pin)
+
+- `models.Settings.RequireMFA bool` (`bson:"require_mfa"`). A plain bool is
+  correct here: absent must mean off.
+- `backup.ciphertextFields["user_mfa"] = []string{"totp_secret_enc"}`. Without
+  it `vantagectl verify` silently skips these secrets.
+
+### Redis keys
+
+| Key                         | TTL   | Holds |
+| --------------------------- | ----- | ----- |
+| `km:mfa:<id>`               | 5 min | pending ticket: `user_id`, `instance_id`, `methods`, `enrol_only`, `attempts` |
+| `km:wa:<id>`                | 5 min | WebAuthn session data (challenge) for one ceremony |
+| `km:totp:<user_id>:<step>`  | 90 s  | `SET NX` replay guard: a TOTP code works once |
+| `km:rl:auth:<ip>`           | 1 min | fixed-window counter for the sign-in rate limit |
+
+The pending ticket cookie is `km_mfa_pending`: HttpOnly, SameSite=Lax, Secure
+under the same rule as `SetSessionCookie`, Path `/`, MaxAge 300. It is cleared
+when exchanged or when the ticket is deleted.
+
+### `Session` additions
+
+- `AMR []string` - how the session was authenticated: `pwd`, `otp`, `webauthn`,
+  `recovery`, `oidc`. An HQ user's password sign-in is `pwd` like a local one.
+- `StepUpAt *time.Time` - last successful step-up. Sign-in itself counts as a
+  step-up, so `StepUpAt` is set at session creation.
+
+Updating `StepUpAt` rewrites the Redis value under the same session ID, keeping
+the remaining TTL (`KEEPTTL`).
+
+## Sign-in flows
+
+### Password, then second factor
+
+1. `POST /auth/login` runs unchanged up to `VerifyPassword`.
+2. Then:
+   - user is `oidc`: unreachable (no local password).
+   - user has MFA: create a ticket with `methods` (`totp`, `webauthn`,
+     `recovery` as applicable), set `km_mfa_pending`, answer
+     `200 {mfa_required: true, methods: [...]}`.
+   - user has no MFA and `require_mfa` is on: create a ticket with
+     `enrol_only: true`, answer `200 {enrol_required: true}`.
+   - otherwise: mint the session as today, `amr: ["pwd"]`.
+3. The client calls one of:
+   - `POST /auth/mfa/totp {code}`
+   - `POST /auth/mfa/recovery {code}`
+   - `POST /auth/mfa/webauthn/begin` then `POST /auth/mfa/webauthn/finish`
+4. On success the ticket is deleted, the session is minted with
+   `amr: ["pwd", <factor>]`, `TouchLastLogin` runs, and the response matches
+   today's `{ok: true}`.
+
+Each failure increments `attempts`; the fifth deletes the ticket.
+
+### Forced enrolment
+
+An `enrol_only` ticket is accepted only by:
+
+- `POST /auth/mfa/enrol/totp/setup` → `{secret, otpauth_uri}`
+- `POST /auth/mfa/enrol/totp/confirm {code}` → `{recovery_codes}` and the session
+- `POST /auth/mfa/enrol/passkey/begin|finish` → `{recovery_codes}` and the session
+
+It is refused by the plain `/auth/mfa/*` verify endpoints and vice versa, so a
+user cannot skip enrolment by presenting the ticket elsewhere.
+
+Existing sessions are not revoked when `require_mfa` is switched on; they end
+at their 24h TTL and the next sign-in enforces enrolment.
+
+### Passwordless passkey
+
+1. `POST /auth/passkey/begin` - resolves the instance with
+   `resolveLoginInstance`, applies the locked-instance and
+   `LocalLoginPermitted` checks exactly as `/auth/login` does, and returns
+   assertion options with an empty `allowCredentials` and
+   `userVerification: "required"`.
+2. `POST /auth/passkey/finish` - looks the credential up by
+   `{instance_id, credential_id}`, verifies the assertion including the UV flag,
+   and mints the session with `amr: ["webauthn"]`.
+
+User verification is preferred, not required. Requiring it made password
+managers such as NordPass prompt for their master password on every use, so it
+was relaxed on 2026-09-16 at the product owner's decision. A passkey used
+without verification is possession-only; passwordless sign-in and step-up then
+rest on the device or vault being unlocked, and still satisfy `require_mfa`.
+
+The discoverable lookup is scoped by `instance_id` from the host, never by the
+credential alone - the same rule that makes `users` lookups instance-scoped.
+
+### WebAuthn relying party
+
+- RP ID: the request host with any port removed. On cloud that is the
+  instance subdomain; self-hosted, the install's own host. The reverse proxy
+  must preserve `Host`, which the instance host guard already requires.
+- Expected origin: `https://<host>` (or `http://` only when the request itself
+  is plain HTTP on `localhost`, for development).
+- A passkey is bound to its host. Moving an instance to a new address
+  invalidates its passkeys; TOTP and recovery codes still work. The docs say so.
+
+## Step-up
+
+`auth.RequireStepUp()` is new middleware, mounted after `auth.Middleware` on:
+
+- `POST /api/secrets/:group/reveal`
+- `GET /api/keys/:id/private-key`
+- `POST /api/console/connect`
+- the MFA-management endpoints marked "step-up" below
+
+It passes when any of these holds:
+
+- the request authenticated with an API token (`TokenID != ""`);
+- `AMR` contains `oidc`;
+- `StepUpAt` is within the last 10 minutes.
+
+Otherwise it answers `403 {error: "re-authentication required", code:
+"step_up_required", methods: [...]}`, where `methods` is the user's MFA factors,
+or `["password"]` when they have none.
+
+`POST /api/me/step-up` accepts `{totp}`, `{recovery}` or `{password}` (password
+only for a user with no MFA), and `POST /api/me/step-up/webauthn/begin|finish`
+does the same with a passkey. Success sets `StepUpAt`. Failures are rate
+limited by the same per-IP limiter and audited.
+
+## Account endpoints (session-authenticated, every role)
+
+| Method and path                         | Step-up | Notes |
+| --------------------------------------- | ------- | ----- |
+| `GET /api/me/mfa`                       | no      | TOTP on/off, passkeys, recovery codes remaining, `require_mfa` |
+| `POST /api/me/mfa/totp/setup`           | yes     | returns secret and `otpauth://` URI; replaces an unconfirmed one |
+| `POST /api/me/mfa/totp/confirm`         | no      | code must verify; returns recovery codes when none exist yet |
+| `DELETE /api/me/mfa/totp`               | yes     | refused if last factor and `require_mfa` |
+| `POST /api/me/mfa/recovery/regenerate`  | yes     | invalidates the old set, returns 10 new codes once |
+| `POST /api/me/passkeys/begin`, `/finish`| yes     | registers a discoverable credential, UV required |
+| `PATCH /api/me/passkeys/:id`            | no      | rename |
+| `DELETE /api/me/passkeys/:id`           | yes     | refused if last factor and `require_mfa` |
+| `DELETE /api/org/users/:id/mfa`         | yes     | owner or admin; an admin cannot reset an owner; clears TOTP, passkeys, recovery codes |
+
+The `/me/*` endpoints refuse OIDC users with 409 `mfa_not_applicable`.
+The first factor enrolled from `/account/security` issues recovery codes;
+later factors do not regenerate them.
+
+`require_mfa` is set through the existing `PUT /api/settings`, owner only for
+that field (admins may save other settings).
+
+Every handler carries swag annotations; `openapi.json` is regenerated.
+
+## Rate limiting
+
+There is no rate limit on `/auth/login` today. A new `RateLimitAuth()` fixed
+window, on the `RateLimitTokens` pattern, applies to `/auth/login`,
+`/auth/mfa/*`, `/auth/passkey/*` and `/api/me/step-up*`: 20 requests per minute
+per `c.ClientIP()`, answering 429 with `Retry-After`. The ticket's 5-attempt
+cap still bounds guesses per ticket; the IP limit bounds tickets per attacker.
+
+## Audit events
+
+`mfa.enrolled` (factor), `mfa.removed` (factor), `mfa.reset` (target user),
+`mfa.recovery_used`, `mfa.recovery_regenerated`, `mfa.failed` (factor, at
+sign-in), `step_up.ok`, `step_up.failed`, `settings.require_mfa` (on/off).
+Details never contain codes, secrets or credential IDs.
+
+## UI (`web/`)
+
+- **`/login`**
+  - "Sign in with passkey" button when `window.PublicKeyCredential` exists and
+    local login is enabled.
+  - After the password: a second-factor step with a 6-digit code input,
+    "Use passkey", and "Use a recovery code".
+  - `enrol_required`: an inline wizard - choose TOTP (QR code, secret as text,
+    confirm code) or passkey, then recovery codes shown once with copy and an
+    "I have saved these" checkbox before continuing.
+  - `mfa_ticket_expired` returns the form to the password step with a message.
+- **`/account/security`** - new page, every role, linked from the sidebar user
+  menu. TOTP status and setup/remove, passkey list with rename and remove,
+  recovery codes remaining and regenerate. Hidden content with an explanation
+  for OIDC users.
+- **`/settings`** - owner-only "Require MFA for password sign-in" toggle; the
+  users table gains an MFA column and a "Reset MFA" action.
+- **`StepUpModal`** - `request()` in `web/lib/api.ts` catches 403 with
+  `code: "step_up_required"`, opens the modal with the offered methods, and on
+  success retries the original request once. Callers of the three sensitive
+  actions need no changes.
+- QR codes are drawn client-side (the `qrcode` npm package); nothing is fetched
+  from outside, so air-gapped installs work.
+
+## Errors
+
+| Situation                          | Answer |
+| ---------------------------------- | ------ |
+| missing or expired ticket          | 401 `mfa_ticket_expired` |
+| wrong code                         | 401 `invalid_code`, `attempts_left` |
+| fifth wrong code                   | ticket deleted, 401 `mfa_ticket_expired` |
+| replayed TOTP code                 | 401 `invalid_code` |
+| WebAuthn verification failure      | 401 `invalid_assertion` |
+| origin or RP ID mismatch           | 400 `origin_mismatch` |
+| removing last factor under policy  | 409 `mfa_required_by_policy` |
+| rate limited                       | 429 with `Retry-After` |
+| Redis unavailable                  | 503; sign-in already depends on Redis |
+
+TOTP: SHA-1, 6 digits, 30 s, ±1 step, issuer = instance name, account = email.
+
+## Libraries
+
+- `github.com/pquerna/otp` - TOTP generation and validation.
+- `github.com/go-webauthn/webauthn` - WebAuthn ceremonies.
+- `qrcode` (npm) - QR rendering in the browser.
+
+## Testing
+
+Go unit tests:
+
+- TOTP verify, skew window, replay guard.
+- Recovery codes: normalisation, single use, regenerate invalidates the old set.
+- Ticket state machine: TTL, attempt cap, `enrol_only` scope in both directions.
+- Login branching: no MFA, MFA, `require_mfa` without MFA, OIDC-exempt.
+- `RequireStepUp`: inside and outside the window, token exemption, OIDC exemption.
+- Last-factor refusal under `require_mfa`; admin cannot reset an owner.
+- Passwordless passkey honours locked instances and `LocalLoginPermitted`.
+- `user_mfa` and `webauthn_credentials` present in `ScopedCollections`.
+- WebAuthn ceremonies against a software authenticator fixture.
+
+Playwright:
+
+- TOTP enrol and sign-in, with codes generated in the test.
+- Passkey enrol, second-factor sign-in, passwordless sign-in and step-up using
+  Chrome's virtual authenticator (`WebAuthn.addVirtualAuthenticator` over CDP).
+- Forced enrolment when `require_mfa` is on.
+- Step-up modal on secret reveal, then no prompt within 10 minutes.
+
+## Known limitations
+
+- **API tokens bypass step-up.** A token with `secrets:read` or `keys:read` can
+  reveal secrets and download private keys without re-authentication. Tokens
+  have no human present to prompt. Scoped, short-lived tokens are the mitigation
+  and are tracked separately in the gap review.
+- OIDC users are never prompted for step-up; their IdP's session policy governs.
+- HQ users on several instances enrol once per instance.
+- Passkeys stop working if an instance changes host.
+
+## Documentation
+
+- `CLAUDE.md`: a "Multi-factor authentication" subsystem section covering the
+  ticket design, step-up, the shared-module changes and `ciphertextFields`.
+- `vantage-docs`: enrolling, recovery codes, owner reset, `require_mfa`,
+  passkeys bound to the host.
+- Gap review: mark `#r-mfa` shipped, and the MFA row in the comparison table.

docs/superpowers/specs/2026-09-17-metric-alerts-heartbeats-design.md

@@ -0,0 +1,186 @@
+# Metric alerts and heartbeat monitors
+
+Date: 2026-09-17
+Source: competitive gap review, "Alerts on agent metrics" (effort M) and "Heartbeat (push) monitors" (effort S).
+
+## Goal
+
+1. Alert on data agents already report (disk, memory, load, failed units, unhealthy containers, reboot pending, agent offline), targeted by tag so one rule covers the fleet.
+2. Alert when a push-based job (backup, cron) stops pinging, reports failure, or starts and never finishes.
+
+Both reuse the existing monitor model, states, incidents, notification channels, groups and status pages.
+
+## Out of scope
+
+- Scheduled workflows pinging a heartbeat automatically.
+- Metric history graphs (separate gap review item).
+- Agent-side evaluation. No agent release is required.
+
+## Data model
+
+New constants in `models/monitor.go`: `MonitorMetric = "metric"`, `MonitorHeartbeat = "heartbeat"`.
+
+`MonitorTarget` gains:
+
+| Field | Type | Used by | Meaning |
+|---|---|---|---|
+| `Selector` | `map[string]string` | metric | Server tag selector. Empty means the whole org fleet. |
+| `Metric` | `string` | metric | One of the metric kinds below. |
+| `Threshold` | `float64` | metric | Breach threshold, unit depends on kind. |
+| `Mount` | `string` | metric (disk kinds) | Mountpoint. Empty means any mount breaches. |
+| `PeriodSec` | `int` | heartbeat | Expected time between pings. |
+| `GraceSec` | `int` | heartbeat | Extra time allowed before overdue, and the max run time after `/start`. |
+
+`Monitor` gains `ForSec int` (metric): the condition must hold continuously this long before a server goes down. 0 means on first evaluation.
+
+`Monitor` gains `HeartbeatTokenHash string` (`json:"-"`), SHA-256 of the ping token. The plaintext token is returned only on create and on rotate.
+
+`MonitorState` gains, for heartbeats: `LastPingAt *time.Time`, `StartedAt *time.Time`.
+
+`Incident` gains `ServerID string` (`omitempty`). Empty for every non-metric monitor.
+
+`Inventory` gains `RebootRequiredSince *time.Time`, set by `StoreInventory` when `reboot_required` turns true and unset when it turns false.
+
+New collection `monitor_server_states`, one document per (metric monitor, matching server):
+
+```go
+type MonitorServerState struct {
+    InstanceID  string     // org
+    MonitorID   string
+    ServerID    string
+    Status      string     // up | down | pending
+    BreachSince *time.Time // first evaluation where the condition held; nil when clear
+    Value       float64    // last evaluated value, for display
+    Message     string     // e.g. "/var 94.2% used"
+    UpdatedAt   time.Time
+}
+```
+
+Unique index on `(monitor_id, server_id)`.
+
+### Metric kinds
+
+| Kind | Source | Breach when | Threshold unit |
+|---|---|---|---|
+| `disk_pct` | `inventory.partitions` | used/total*100 >= threshold on `Mount` (or any mount) | percent |
+| `disk_free_gb` | `inventory.partitions` | (total-used)/1e9 <= threshold | GB |
+| `mem_pct` | `inventory.memory` | used/total*100 >= threshold | percent |
+| `load_per_core` | `inventory.cpu.load1 / cores` | >= threshold | ratio |
+| `unit_failed` | workloads, kind `unit` | any unit state `failed` (threshold unused) | - |
+| `container_unhealthy` | workloads, kind `container` | any health `unhealthy` (threshold unused) | - |
+| `reboot_pending_days` | `inventory.reboot_required_since` | now - since >= threshold days | days |
+| `agent_offline_min` | `server.last_seen` | now - last_seen >= threshold minutes | minutes |
+
+Validation on create/update: `Metric` is a known kind; `Threshold` > 0 for kinds that use it; `disk_pct` and `mem_pct` threshold <= 100; `Mount`, if set, is an absolute path; `Selector` keys and values follow existing tag rules. `IntervalSec`, `Runner` and `Retries` are ignored for metric and heartbeat monitors and stored as `runner = "server"`, so `ListServerScheduledMonitors` must exclude these two types from the pull scheduler.
+
+Heartbeat validation: `PeriodSec` >= 60, `GraceSec` >= 0 (default 300 when unset).
+
+## Shared transition logic
+
+Extract from `ingestResult` in `services/monitors.go`:
+
+```go
+func applyTransition(ctx context.Context, m *models.Monitor, serverID, prev, next, message string, now time.Time)
+```
+
+It opens an incident (with `ServerID`) on a change to down, resolves the matching open incident (same `monitor_id` and `server_id`) on down to up, and calls `notifyTransition`. `ingestResult` calls it with `serverID = ""`. Behaviour of existing monitor types does not change.
+
+`notify.Event` gains `ServerName string`. Channel formatters include it when set ("disk on web-01: /var 94.2% used").
+
+## Metric evaluation
+
+New package `server/internal/metricsched`, started from `main.go` beside `monitorsched`. Like monitors, it runs regardless of licence state.
+
+Every 30 seconds, for each enabled metric monitor:
+
+1. Resolve servers with `ListServersFiltered(instanceID, selector)`.
+2. Load existing `monitor_server_states` for the monitor.
+3. For each server:
+   - For kinds other than `agent_offline_min`: skip (keep previous state) when `inventory.metrics_at` is older than 5 minutes or missing. A dead agent must not flap other alerts; `agent_offline_min` covers it.
+   - Evaluate the kind to get `(breach bool, value float64, message string)`.
+   - Breach: set `BreachSince` if nil. If `now - BreachSince >= ForSec`, next = down, otherwise next = pending when previously up or new.
+   - No breach: `BreachSince = nil`, next = up.
+   - Upsert the state doc. On a status change call `applyTransition` with the server ID. A new server's first state is never treated as a transition from down.
+4. State docs for servers no longer matched (tag removed, server deleted): resolve any open incident without notifying, and delete the doc.
+5. Roll up the parent `Monitor.State`: status = down if any child down, else pending if any pending, else up (no matching servers means up). Message is "N of M servers breaching". `last_check_at = now`. Write one `MonitorSample` (up = no child down, latency 0) and increment the hourly rollup, so uptime graphs and status pages work unchanged.
+
+Deleting a metric monitor deletes its `monitor_server_states`.
+
+Evaluators are pure functions `func(kind string, t models.MonitorTarget, srv models.Server, wls []models.Workload, now time.Time) (bool, float64, string)` so they can be table-tested without Mongo. Workloads are loaded only for monitors whose kind needs them.
+
+## Heartbeats
+
+### Public endpoints
+
+Registered outside `/api`, unauthenticated, no scope declarations needed:
+
+- `GET|POST /public/hb/:token`: success ping
+- `GET|POST /public/hb/:token/start`: run started
+- `GET|POST /public/hb/:token/fail`: run failed
+
+Mounted under /public because every deployment already routes that prefix to the server. The token may instead be sent in the `X-Vantage-Token` header to `/public/hb`, `/public/hb/start` or `/public/hb/fail`; a URL token wins when both are present. The server's request log and the bundled nginx access log mask the URL token.
+
+Lookup is by SHA-256 of the token. Unknown token or disabled monitor gives 404. Rate limit is one accepted request per second per token (in-process, Redis-backed if a limiter helper already exists); excess gives 429. The response body is `OK`. Request bodies over 1 KB are truncated; only `/fail` uses the body.
+
+Behaviour:
+
+- **ping**: `LastPingAt = now`. If `StartedAt` is set, duration = now - StartedAt, write a sample with `LatencyMs = duration`, then clear `StartedAt`. Otherwise write a sample with latency 0. Next status is up; call `applyTransition` if changed.
+- **start**: `StartedAt = now`. No status change.
+- **fail**: clear `StartedAt`, write a down sample, next status is down immediately with message `reported failure: <body>` (body defaults to empty). The failure is also recorded as `LastPingAt` so the overdue timer restarts from it.
+
+All state writes go through one conditional update per request, so concurrent pings don't lose updates.
+
+### Overdue sweep
+
+The same 30-second `metricsched` loop checks enabled heartbeat monitors:
+
+- New monitor with no ping yet: pending, never down. The overdue clock starts at the first ping.
+- `now > LastPingAt + PeriodSec + GraceSec`: down, message `no ping since <time>`.
+- `StartedAt != nil && now > StartedAt + GraceSec`: down, message `started <time>, never finished`.
+
+Recovery happens only through a ping.
+
+### Token management
+
+- Create returns `heartbeat_token` and `heartbeat_url` once.
+- `POST /api/monitors/:id/rotate-token` issues a new token and invalidates the old one. It needs `routeScopes` (`monitors:write`) and `serverScopedRoutes` entries.
+- The UI shows the URL only right after create or rotate, with a copy button and curl examples for ping, start and fail.
+
+## API
+
+- Existing create/update monitor endpoints accept the new types and fields. OpenAPI annotations are updated.
+- `GET /api/monitors/:id/servers` returns `monitor_server_states` for a metric monitor, with hostnames. It needs `routeScopes` (`monitors:read`) and `serverScopedRoutes` entries, and filters rows to servers visible to the acting token.
+- A tag-scoped token may create or update a metric monitor only when its selector includes every key/value of the token's scope. Otherwise the response is 403.
+- The MCP `create_monitor` tool accepts the new types.
+
+## Web UI
+
+- Monitor form: a type picker adds Metric and Heartbeat. Metric shows a selector editor (reuse the tag selector component), a kind dropdown, a threshold with a unit label, mount (disk kinds only) and "for N minutes". Heartbeat shows period and grace.
+- Monitors list: metric rows show "N of M servers breaching". Heartbeat rows show last ping as relative time.
+- Metric detail page: a per-server table with server, status, value, since and message, linking to the server page. The incident list shows the server name.
+- Heartbeat detail page: URL panel (after create or rotate), rotate button with confirm, last ping, and duration in the latency chart.
+
+## Error handling
+
+- Evaluator panics or bad inventory data: recover per monitor, log, and leave states unchanged.
+- Mongo errors in the sweep: log and continue to the next monitor. The next tick retries.
+- Notification failures are logged only, as today.
+
+## Testing
+
+- Table tests for every evaluator: breach, clear, mount filter, missing inventory, zero totals.
+- Sweep tests (Mongo test helper): the for-duration gate, pending to down, stale inventory skipped, a server leaving the selector resolves its incident silently, parent rollup counts.
+- Heartbeat handler tests: ping/start/fail transitions, duration sample, unknown token 404, rate limit 429, fail body truncation.
+- Overdue tests: never pinged stays pending, overdue goes down, start without finish goes down, ping recovers.
+- `applyTransition` regression: existing http monitor incident open/resolve unchanged.
+- `go test ./internal/api/` including `TestRegisteredRoutesPassBootAssertions`.
+- Playwright: create a heartbeat, curl a ping, see up.
+
+## Phases
+
+1. Shared `applyTransition` refactor, heartbeat type, public endpoints, overdue sweep, rotate route, UI. Ships alone.
+2. Metric type, `RebootRequiredSince`, `monitor_server_states`, evaluators, sweep, servers route, UI.
+
+## Documentation
+
+Update the Monitors section of `CLAUDE.md` and add a user guide page in vantage-docs for both types.

server/cmd/main.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"errors"
+	"fmt"
 	"log"
 	"net/http"
 	"os"
@@ -23,7 +24,10 @@ import (
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
 	grpcserver "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/mcp"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/metricsched"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/monitorsched"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchsched"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulnsched"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
@@ -136,6 +140,10 @@ func runSchemaSetup() {
 		log.Fatalf("failed to ensure auth indexes: %v", err)
 	}
 
+	if err := services.EnsureMFAIndexes(); err != nil {
+		log.Fatalf("failed to ensure mfa indexes: %v", err)
+	}
+
 	if err := services.EnsureAPITokenIndexes(); err != nil {
 		log.Fatalf("api token indexes: %v", err)
 	}
@@ -167,6 +175,10 @@ func runSchemaSetup() {
 		log.Printf("warning: failed to ensure monitor sample indexes: %v", err)
 	}
 
+	if err := services.EnsureMonitorServerStateIndexes(); err != nil {
+		log.Printf("warning: %v", err)
+	}
+
 	if err := services.EnsureVulnIndexes(); err != nil {
 		log.Printf("warning: failed to ensure vuln indexes: %v", err)
 	}
@@ -179,6 +191,10 @@ func runSchemaSetup() {
 		log.Printf("warning: failed to ensure status page indexes: %v", err)
 	}
 
+	if err := services.EnsurePatchIndexes(); err != nil {
+		log.Printf("warning: patch indexes: %v", err)
+	}
+
 	if err := services.EnsureAuditIndexes(); err != nil {
 		log.Printf("warning: failed to ensure audit indexes: %v", err)
 	}
@@ -215,6 +231,7 @@ func serve() {
 		log.Fatalf("failed to connect to Redis: %v", err)
 	}
 	log.Println("connected to Redis")
+	services.RedisClient = auth.Redis()
 
 	// The bus carries agent commands and step results between replicas. It is
 	// not optional even on a single-replica deployment: dispatch takes the same
@@ -248,11 +265,23 @@ func serve() {
 		services.StartAuditSweeper(jobCtx)
 		services.StartReaper(jobCtx)
 		monitorsched.Start(jobCtx)
+		metricsched.Start(jobCtx)
 		workflowsched.Start(jobCtx, workflowsched.Deps{
 			TriggerWorkflow: services.TriggerWorkflow,
 			LogEvent:        services.LogEvent,
 		})
 
+		patchsched.Start(jobCtx, patchsched.Deps{
+			LookupWindow: services.LookupWindow,
+			CountTargets: services.CountPolicyTargets,
+			StartPolicyRun: func(p models.PatchPolicy, windowEnd time.Time) error {
+				_, err := services.StartPolicyRun(p, windowEnd, models.PatchSourceSchedule, "schedule")
+				return err
+			},
+			AdvanceRuns: services.AdvancePatchRuns,
+			LogEvent:    services.LogEvent,
+		})
+
 		vulnsched.Start(jobCtx, vulnsched.Deps{
 			LogEvent:   services.LogEvent,
 			SendDigest: services.SendVulnDigest,
@@ -283,7 +312,16 @@ func serve() {
 		log.Fatalf("trusted proxies: %v", err)
 	}
 	r.Use(gin.Recovery())
-	r.Use(gin.LoggerWithConfig(gin.LoggerConfig{SkipPaths: []string{"/api/console/tunnel"}}))
+	r.Use(gin.LoggerWithConfig(gin.LoggerConfig{
+		SkipPaths: []string{"/api/console/tunnel"},
+		// Heartbeat URLs carry a credential; the request log must not.
+		Formatter: func(p gin.LogFormatterParams) string {
+			return fmt.Sprintf("[GIN] %v | %3d | %13v | %15s | %-7s %#v\n%s",
+				p.TimeStamp.Format("2006/01/02 - 15:04:05"),
+				p.StatusCode, p.Latency, p.ClientIP, p.Method,
+				api.MaskLogPath(p.Path), p.ErrorMessage)
+		},
+	}))
 	r.Use(corsMiddleware())
 	services.SetStatusRedis(auth.Redis())
 

server/go.mod

@@ -6,12 +6,14 @@ require (
 	github.com/aquasecurity/trivy-db v0.0.0-20260813095258-0e0340a01b57
 	github.com/coreos/go-oidc/v3 v3.21.0
 	github.com/gin-gonic/gin v1.12.0
+	github.com/go-webauthn/webauthn v0.18.1
 	github.com/google/uuid v1.6.0
 	github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f
 	github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23
 	github.com/knqyf263/go-rpm-version v0.0.0-20260811110310-1815e1f1b790
 	github.com/modelcontextprotocol/go-sdk v1.7.0
 	github.com/opencontainers/image-spec v1.1.1
+	github.com/pquerna/otp v1.5.0
 	github.com/redis/go-redis/v9 v9.22.0
 	github.com/robfig/cron/v3 v3.0.1
 	github.com/wwt/guac v1.3.2
@@ -23,12 +25,19 @@ require (
 )
 
 require (
+	github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
 	github.com/bytedance/gopkg v0.1.4 // indirect
+	github.com/fxamacker/cbor/v2 v2.9.3 // indirect
+	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
+	github.com/go-webauthn/x v0.3.1 // indirect
 	github.com/goccy/go-yaml v1.19.2 // indirect
+	github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
+	github.com/google/go-tpm v0.9.8 // indirect
 	github.com/google/jsonschema-go v0.4.3 // indirect
 	github.com/hyperboloide/lk v0.0.0-20251220053519-b291812e3216 // indirect
 	github.com/oklog/ulid/v2 v2.1.2 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
+	github.com/philhofer/fwd v1.2.0 // indirect
 	github.com/quic-go/qpack v0.6.0 // indirect
 	github.com/quic-go/quic-go v0.62.0 // indirect
 	github.com/samber/lo v1.53.0 // indirect
@@ -37,7 +46,10 @@ require (
 	github.com/segmentio/encoding v0.5.4 // indirect
 	github.com/stretchr/objx v0.5.3 // indirect
 	github.com/stretchr/testify v1.12.1 // indirect
+	github.com/tinylib/msgp v1.6.4 // indirect
+	github.com/x448/float16 v0.8.4 // indirect
 	github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
+	github.com/yuin/goldmark v1.8.6 // indirect
 	go.etcd.io/bbolt v1.5.0 // indirect
 	go.opentelemetry.io/otel v1.46.0 // indirect
 	go.opentelemetry.io/otel/trace v1.46.0 // indirect
@@ -46,7 +58,7 @@ require (
 )
 
 require (
-	gitea.hostxtra.co.uk/vantage/vantage-shared v0.2.2
+	gitea.hostxtra.co.uk/vantage/vantage-shared v0.7.0
 	github.com/bytedance/sonic v1.15.3 // indirect
 	github.com/bytedance/sonic/loader v0.5.2 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect

server/go.sum

@@ -1,11 +1,11 @@
-gitea.hostxtra.co.uk/vantage/vantage-shared v0.2.1 h1:rPzXSRwU+4+F2pdkmDrIxKsIzqz3S6feJEWalGmKqfU=
-gitea.hostxtra.co.uk/vantage/vantage-shared v0.2.1/go.mod h1:dWjeOFLltQ8sv9Pnn1xRxGfWGgqa2fkG0esuaJLoPXQ=
-gitea.hostxtra.co.uk/vantage/vantage-shared v0.2.2 h1:ibUp4zoh3GQgM298o5Mxmfv2cFM09uwWAPIDrE9yUjI=
-gitea.hostxtra.co.uk/vantage/vantage-shared v0.2.2/go.mod h1:dWjeOFLltQ8sv9Pnn1xRxGfWGgqa2fkG0esuaJLoPXQ=
+gitea.hostxtra.co.uk/vantage/vantage-shared v0.7.0 h1:wwXvHfDKZB44EEj6BXl9O68hLC3kfPz3eak9wvupIRA=
+gitea.hostxtra.co.uk/vantage/vantage-shared v0.7.0/go.mod h1:Zo66XhqF8No3dveIowLCepvMxVg8KnhsNMz0k0Xpuck=
 github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986 h1:2a30xLN2sUZcMXl50hg+PJCIDdJgIvIbVcKqLJ/ZrtM=
 github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986/go.mod h1:NT+jyeCzXk6vXR5MTkdn4z64TgGfE5HMLC8qfj5unl8=
 github.com/aquasecurity/trivy-db v0.0.0-20260813095258-0e0340a01b57 h1:A3Lz/9ip/qigafSxqBWcu7S8i+tJbQS7DB2V0XibOKs=
 github.com/aquasecurity/trivy-db v0.0.0-20260813095258-0e0340a01b57/go.mod h1:iIEV2oGuZScvfyX2SMIn78iVMNnepgo0QuJJh/srgVI=
+github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc h1:biVzkmvwrH8WK8raXaxBx6fRVTlJILwEwQGL1I/ByEI=
+github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
 github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
 github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
 github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
@@ -25,6 +25,8 @@ github.com/coreos/go-oidc/v3 v3.21.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/fxamacker/cbor/v2 v2.9.3 h1:oQBnFATpNdY8gJHTndDDv5Xl4QqNaz51G5LLEPhng3Q=
+github.com/fxamacker/cbor/v2 v2.9.3/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
 github.com/gabriel-vasile/mimetype v1.4.15 h1:05iP/CYtZ/w455R/KZM6rZ5ieAdh99UPtd+d3YzLmaI=
 github.com/gabriel-vasile/mimetype v1.4.15/go.mod h1:azpTcoLcDZRNgFou5j+APrqQx9HqVPWa6ijYQIIVswQ=
 github.com/gin-contrib/sse v1.1.2 h1:MU2fgl1RrdYTMcgJLtz2kJF+vPg3xrqaaKfUUU18tCo=
@@ -45,6 +47,12 @@ github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJn
 github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
 github.com/go-playground/validator/v10 v10.30.4 h1:9Rcod2ZPO6mOEG6b4GqyoHE/H6//Ze0RuhOo1hT1x0w=
 github.com/go-playground/validator/v10 v10.30.4/go.mod h1:numpT+RPLE91R9oYWMY/R9zRgJBewr3IXHko4OISPpk=
+github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
+github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/go-webauthn/webauthn v0.18.1 h1:KaQw6M+ODLvxHwddyeo6zFhhmicfLup/BClhigB6A+0=
+github.com/go-webauthn/webauthn v0.18.1/go.mod h1:s4rZTQnKHWxIh6G3yEGqlxvtiLceA1jigll8FpqSgQ8=
+github.com/go-webauthn/x v0.3.1 h1:1ff37z3XfmTTomkhlURgGizLIDyOvPgTt2t9nlzKLRo=
+github.com/go-webauthn/x v0.3.1/go.mod h1:ZInxAynYXfBPvvm5gzKZ7geBlL23K71xASMgohHl/Rg=
 github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
 github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
 github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
@@ -55,6 +63,10 @@ github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek
 github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
+github.com/google/go-tpm-tools v0.3.13-0.20230620182252-4639ecce2aba h1:qJEJcuLzH5KDR0gKc0zcktin6KSAwL7+jWKBYceddTc=
+github.com/google/go-tpm-tools v0.3.13-0.20230620182252-4639ecce2aba/go.mod h1:EFYHy8/1y2KfgTAsx7Luu7NGhoxtuVHnNo8jE7FikKc=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/jsonschema-go v0.4.3 h1:/DBOLZTfDow7pe2GmaJNhltueGTtDKICi8V8p+DQPd0=
 github.com/google/jsonschema-go v0.4.3/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE=
@@ -99,7 +111,11 @@ github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgr
 github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
 github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
 github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
+github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
+github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs=
+github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
 github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
 github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
 github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
@@ -137,12 +153,16 @@ github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXl
 github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
 github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
 github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
+github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
+github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
 github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
 github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
 github.com/ugorji/go/codec v1.3.2 h1:zkEASHHyEClGeURfgNT9PJZVfAbs9oEX9QXggwWNJbc=
 github.com/ugorji/go/codec v1.3.2/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4=
 github.com/wwt/guac v1.3.2 h1:sH6OFGa/1tBs7ieWBVlZe7t6F5JAOWBry/tqQL/Vup4=
 github.com/wwt/guac v1.3.2/go.mod h1:eKm+NrnK7A88l4UBEcYNpZQGMpZRryYKoz4D/0/n1C0=
+github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
+github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
 github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
 github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
 github.com/xdg-go/scram v1.2.0 h1:bYKF2AEwG5rqd1BumT4gAnvwU/M9nBp2pTSxeZw7Wvs=
@@ -154,6 +174,8 @@ github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT0
 github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=
 github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfSfmXjznFBSZNN13rSJjlIOI1fUNAtF7rmI=
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
+github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
+github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
 github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
 github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
 go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
@@ -182,16 +204,12 @@ golang.org/x/arch v0.31.0 h1:22MlEb14/O/EPCYHFxsDdv5TuLD5dMjT5e2QeJw4ULk=
 golang.org/x/arch v0.31.0/go.mod h1:KcJSod3cqT2dKcjBxqTyGfbumNikqU9p5tHJinPJnuY=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y=
-golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I=
 golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
 golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
-golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
-golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
 golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
 golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
 golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
@@ -214,8 +232,6 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
-golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
-golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE=
@@ -223,9 +239,8 @@ golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
-golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
-golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
+golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
 gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=

server/internal/api/bootassert_test.go

@@ -0,0 +1,42 @@
+package api
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/gin-gonic/gin"
+)
+
+// The two boot assertions only run against a real engine in main.go, so a
+// route added without its declarations compiled, passed every unit test and
+// then refused to start in production. This registers the real routes and
+// runs both assertions, so that failure lands in CI instead.
+func TestRegisteredRoutesPassBootAssertions(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+	r := gin.New()
+	RegisterRoutes(r)
+
+	if err := AssertScopeMapComplete(r); err != nil {
+		t.Fatalf("scope map: %v", err)
+	}
+
+	var routes []string
+	for _, route := range r.Routes() {
+		if strings.HasPrefix(route.Path, "/api/") {
+			routes = append(routes, route.Method+" "+route.Path)
+		}
+	}
+	if err := AssertServerScopeMapComplete(routes); err != nil {
+		t.Fatalf("server scope map: %v", err)
+	}
+}
+
+// A route cannot be both session-only and token-reachable: an entry in both
+// maps would silently hand the MFA routes to any token holding that scope.
+func TestSessionOnlyRoutesHaveNoTokenScope(t *testing.T) {
+	for route := range sessionOnlyRoutes {
+		if scope, ok := routeScopes[route]; ok {
+			t.Errorf("%s is session-only but routeScopes grants it to tokens with %q", route, scope)
+		}
+	}
+}

server/internal/api/handlers.go

@@ -42,7 +42,25 @@ func RegisterRoutes(r *gin.Engine) {
 
 	r.GET("/auth/bootstrap-status", auth.HandleBootstrapStatus)
 	r.POST("/auth/bootstrap", auth.HandleBootstrap)
-	r.POST("/auth/login", auth.HandleLocalLogin)
+
+	// Every unauthenticated sign-in and enrolment step lives behind
+	// RateLimitAuth: without it, the five-attempt cap on a single ticket is
+	// trivially sidestepped by starting a fresh sign-in each time.
+	authGroup := r.Group("", RateLimitAuth())
+	{
+		authGroup.POST("/auth/login", auth.HandleLocalLogin)
+		authGroup.POST("/auth/mfa/totp", auth.HandleMFATOTP)
+		authGroup.POST("/auth/mfa/recovery", auth.HandleMFARecovery)
+		authGroup.POST("/auth/mfa/webauthn/begin", auth.HandleMFAWebAuthnBegin)
+		authGroup.POST("/auth/mfa/webauthn/finish", auth.HandleMFAWebAuthnFinish)
+		authGroup.POST("/auth/passkey/begin", auth.HandlePasskeyLoginBegin)
+		authGroup.POST("/auth/passkey/finish", auth.HandlePasskeyLoginFinish)
+		authGroup.POST("/auth/mfa/enrol/totp/setup", auth.HandleEnrolTOTPSetup)
+		authGroup.POST("/auth/mfa/enrol/totp/confirm", auth.HandleEnrolTOTPConfirm)
+		authGroup.POST("/auth/mfa/enrol/passkey/begin", auth.HandleEnrolPasskeyBegin)
+		authGroup.POST("/auth/mfa/enrol/passkey/finish", auth.HandleEnrolPasskeyFinish)
+	}
+
 	r.POST("/auth/logout", auth.HandleLogout)
 	r.GET("/auth/me", auth.HandleMe)
 	r.GET("/auth/oidc/:providerId/start", auth.HandleSSOStart)
@@ -53,6 +71,14 @@ func RegisterRoutes(r *gin.Engine) {
 	// rather than under /api precisely so that none of those apply.
 	r.GET("/public/status/:pageId", RateLimitPublicStatus(), getPublicStatusPage)
 
+	// Ping endpoints for heartbeat monitors. The token is in the URL or the
+	// X-Vantage-Token header; see resolveHeartbeat for the shapes. Rate
+	// limiting is per token inside the handler.
+	for _, p := range []string{"/public/hb", "/public/hb/:a", "/public/hb/:a/:b"} {
+		r.GET(p, handleHeartbeat)
+		r.POST(p, handleHeartbeat)
+	}
+
 	apiGroup := r.Group("/api")
 	apiGroup.Use(auth.Middleware())
 	// Scope enforcement sits between authentication and the licence gate, and
@@ -91,6 +117,20 @@ func RegisterRoutes(r *gin.Engine) {
 		apiGroup.POST("/tokens", createToken)
 		apiGroup.DELETE("/tokens/:id", revokeToken)
 
+		apiGroup.GET("/me/mfa", getMyMFA)
+		apiGroup.POST("/me/mfa/totp/setup", auth.RequireStepUp(), setupTOTP)
+		apiGroup.POST("/me/mfa/totp/confirm", confirmTOTP)
+		apiGroup.DELETE("/me/mfa/totp", auth.RequireStepUp(), removeTOTP)
+		apiGroup.POST("/me/mfa/recovery/regenerate", auth.RequireStepUp(), regenerateRecoveryCodes)
+		apiGroup.POST("/me/passkeys/begin", auth.RequireStepUp(), auth.HandleRegisterPasskeyBegin)
+		apiGroup.POST("/me/passkeys/finish", auth.RequireStepUp(), auth.HandleRegisterPasskeyFinish)
+		apiGroup.PATCH("/me/passkeys/:id", renamePasskey)
+		apiGroup.DELETE("/me/passkeys/:id", auth.RequireStepUp(), deletePasskey)
+		apiGroup.POST("/me/step-up", RateLimitAuth(), stepUp)
+		apiGroup.POST("/me/step-up/webauthn/begin", RateLimitAuth(), auth.HandleStepUpWebAuthnBegin)
+		apiGroup.POST("/me/step-up/webauthn/finish", RateLimitAuth(), auth.HandleStepUpWebAuthnFinish)
+		apiGroup.DELETE("/org/users/:id/mfa", auth.RequireRole("owner", "admin"), auth.RequireStepUp(), resetUserMFA)
+
 		apiGroup.GET("/openapi.json", getOpenAPI)
 		apiGroup.GET("/docs", getAPIDocs)
 		apiGroup.GET("/docs/scalar.js", getScalarJS)
@@ -107,19 +147,19 @@ func RegisterRoutes(r *gin.Engine) {
 		apiGroup.POST("/secrets", createSecretGroup)
 		apiGroup.GET("/secrets/:group", getSecretGroup)
 		apiGroup.PUT("/secrets/:group", putSecretGroup)
-		apiGroup.POST("/secrets/:group/reveal", revealSecret)
+		apiGroup.POST("/secrets/:group/reveal", auth.RequireStepUp(), revealSecret)
 		apiGroup.DELETE("/secrets/:group", deleteSecretGroup)
 		apiGroup.DELETE("/secrets/:group/:key", deleteSecretKey)
 
 		apiGroup.GET("/keys", listKeys)
 		apiGroup.POST("/keys", createKey)
 		apiGroup.GET("/keys/:id", getKey)
-		apiGroup.GET("/keys/:id/private-key", getPrivateKey)
+		apiGroup.GET("/keys/:id/private-key", auth.RequireStepUp(), getPrivateKey)
 		apiGroup.DELETE("/keys/:id", deleteKey)
 		apiGroup.POST("/keys/:id/assign", assignKey)
 		apiGroup.DELETE("/keys/:id/assign/:serverId", revokeAssignment)
 
-		apiGroup.POST("/console/connect", RequireFeature("console"), consoleConnect)
+		apiGroup.POST("/console/connect", auth.RequireStepUp(), RequireFeature("console"), consoleConnect)
 		apiGroup.GET("/console/tunnel", RequireFeature("console"), consoleTunnel)
 
 		// MCP is mounted inside /api so that bearer auth, rate limiting, licence
@@ -187,6 +227,7 @@ func RegisterRoutes(r *gin.Engine) {
 		apiGroup.GET("/servers/:id/workloads/:wid/logs", auth.RequireRole("owner", "admin"), getWorkloadLogs)
 
 		registerStatusPageRoutes(apiGroup)
+		registerPatchingRoutes(apiGroup)
 	}
 }
 
@@ -748,30 +789,41 @@ func updateAgent(c *gin.Context) {
 // applyUpdates godoc
 //
 //	@Summary		Apply pending OS updates on a server
-//	@Description	Dispatches ApplyUpdatesCmd. Exempt from the licence gate: security patching is never paywalled.
+//	@Description	Starts a manual patch run (all updates, no reboot) and returns its ID. Exempt from the licence gate: security patching is never paywalled.
 //	@Tags			servers
 //	@Produce		json
-//	@Param			id	path		string	true	"Server ID"
-//	@Success		202	{object}	MessageResponse
-//	@Failure		404	{object}	ErrorResponse
-//	@Failure		503	{object}	ErrorResponse
+//	@Param			id		path		string	true	"Server ID"
+//	@Param			source	query		string	false	"vulnerabilities when started from the vulnerabilities page"
+//	@Success		202		{object}	ApplyUpdatesResponse
+//	@Failure		404		{object}	ErrorResponse
+//	@Failure		503		{object}	ApplyUpdatesErrorResponse	"agent offline; the attempt is recorded as run_id"
 //	@Security		cookieAuth
 //	@Security		bearerAuth
 //	@Router			/servers/{id}/apply-updates [post]
 func applyUpdates(c *gin.Context) {
-	id := c.Param("id")
-	s, err := services.GetServerScoped(auth.InstanceID(c), id, auth.ServerScope(c))
+	instanceID := auth.InstanceID(c)
+	s, err := services.GetServerScoped(instanceID, c.Param("id"), auth.ServerScope(c))
 	if err != nil {
 		c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
 		return
 	}
-
-	if err := services.DispatchApplyUpdates(s.ServerID); err != nil {
-		c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error()})
+	source := models.PatchSourceServer
+	if c.Query("source") == models.PatchSourceVulnerabilities {
+		source = models.PatchSourceVulnerabilities
+	}
+	run, err := services.StartManualRun(instanceID, s, actorFromCtx(c), source)
+	if errors.Is(err, services.ErrAgentOffline) {
+		// The attempt is still recorded, so it has a run ID to show.
+		c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error(), "run_id": run.RunID})
 		return
 	}
-	services.LogEvent(auth.InstanceID(c), "updates.applied", actorFromCtx(c), s.ServerID, "", fmt.Sprintf("package update command dispatched to %s", s.Hostname))
-	c.JSON(http.StatusAccepted, MessageResponse{Message: "apply updates command sent to agent"})
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	services.LogEvent(instanceID, "updates.applied", actorFromCtx(c), s.ServerID, "",
+		fmt.Sprintf("package update run %s started on %s", run.RunID, s.Hostname))
+	c.JSON(http.StatusAccepted, ApplyUpdatesResponse{Message: "apply updates command sent to agent", RunID: run.RunID})
 }
 
 // handleUpdateScript serves a dynamically generated shell script that
@@ -914,6 +966,7 @@ func saveSettings(c *gin.Context) {
 		Alerts                   models.AlertSettings `json:"alerts"`
 		WorkflowLogRetentionDays *int                 `json:"workflow_log_retention_days"`
 		LocalLoginEnabled        *bool                `json:"local_login_enabled"`
+		RequireMFA               *bool                `json:"require_mfa"`
 		APITokenMaxDays          *int                 `json:"api_token_max_days"`
 	}
 	if err := c.ShouldBindJSON(&body); err != nil {
@@ -924,7 +977,13 @@ func saveSettings(c *gin.Context) {
 		c.JSON(http.StatusBadRequest, gin.H{"error": "api_token_max_days cannot be negative"})
 		return
 	}
-	if err := services.SaveSettings(auth.InstanceID(c), body.Alerts, body.WorkflowLogRetentionDays, body.LocalLoginEnabled, body.APITokenMaxDays); err != nil {
+	// The MFA requirement gates every future sign-in, so only an owner may
+	// change it; an admin can still save the rest of this endpoint's settings.
+	if body.RequireMFA != nil && auth.Role(c) != models.RoleOwner {
+		c.JSON(http.StatusForbidden, gin.H{"error": "only an owner can change the MFA requirement"})
+		return
+	}
+	if err := services.SaveSettings(auth.InstanceID(c), body.Alerts, body.WorkflowLogRetentionDays, body.LocalLoginEnabled, body.RequireMFA, body.APITokenMaxDays); err != nil {
 		if errors.Is(err, services.ErrLockout) {
 			c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "local_login_required"})
 			return
@@ -937,6 +996,10 @@ func saveSettings(c *gin.Context) {
 		services.LogEvent(auth.InstanceID(c), "settings.token_policy_updated", actorFromCtx(c), "", "",
 			fmt.Sprintf("API token maximum lifetime set to %d day(s); 0 means no cap", *body.APITokenMaxDays))
 	}
+	if body.RequireMFA != nil {
+		services.LogEvent(auth.InstanceID(c), "settings.require_mfa", actorFromCtx(c), "", "",
+			fmt.Sprintf("enabled=%v", *body.RequireMFA))
+	}
 	c.JSON(http.StatusOK, SavedResponse{Saved: true})
 }
 

server/internal/api/heartbeats.go

@@ -0,0 +1,176 @@
+package api
+
+import (
+	"errors"
+	"io"
+	"log"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+)
+
+// HeartbeatTokenHeader carries the ping token for callers who want it out of
+// URLs, and so out of their own proxies' access logs.
+const HeartbeatTokenHeader = "X-Vantage-Token"
+
+const heartbeatPathPrefix = "/public/hb/"
+
+func heartbeatKind(seg string) string {
+	switch seg {
+	case "":
+		return services.HeartbeatPing
+	case "start":
+		return services.HeartbeatStart
+	case "fail":
+		return services.HeartbeatFail
+	}
+	return ""
+}
+
+// resolveHeartbeat maps the three route shapes onto a token and a kind. a and b
+// are the first and second path segments after /public/hb. A token in the URL
+// wins over the header, so a URL copied from the UI behaves the same no matter
+// what headers a client adds.
+func resolveHeartbeat(header, a, b string) (string, string, bool) {
+	var token, kindSeg string
+	switch {
+	// a is a URL token unless a header is present, no second segment follows
+	// and a is itself a kind word (the header routes /public/hb/start|fail).
+	case a != "" && (header == "" || b != "" || heartbeatKind(a) == ""):
+		token, kindSeg = a, b
+	case header != "":
+		token, kindSeg = header, a
+	default:
+		return "", "", false
+	}
+	kind := heartbeatKind(kindSeg)
+	if token == "" || kind == "" {
+		return "", "", false
+	}
+	return token, kind, true
+}
+
+// MaskLogPath hides a heartbeat token in a request path before it is logged.
+// "start" and "fail" directly under the prefix are the header-token routes and
+// carry no secret, so they are left readable.
+func MaskLogPath(path string) string {
+	rest, found := strings.CutPrefix(path, heartbeatPathPrefix)
+	if !found {
+		return path
+	}
+	end := strings.IndexAny(rest, "/?")
+	if end < 0 {
+		end = len(rest)
+	}
+	seg := rest[:end]
+	if seg == "" || seg == "start" || seg == "fail" {
+		return path
+	}
+	return heartbeatPathPrefix + "***" + rest[end:]
+}
+
+// heartbeatLimitKey buckets requests by token, kind and wall-clock second. The
+// kind is part of the key so a fast job's success ping is not rejected for
+// landing in the same second as its /start, which would leave the run marked
+// started and page as never finished. The token is hashed so Redis never holds
+// the secret.
+func heartbeatLimitKey(token, kind string, now time.Time) string {
+	return "vantage:hbrl:" + services.HashHeartbeatToken(token) + ":" + kind + ":" + strconv.FormatInt(now.Unix(), 10)
+}
+
+// readHeartbeatBody keeps up to MaxHeartbeatBody bytes of a /fail body as the
+// incident message. Other kinds carry no message, so their bodies are ignored.
+func readHeartbeatBody(kind string, r io.Reader) string {
+	if kind != services.HeartbeatFail || r == nil {
+		return ""
+	}
+	b, _ := io.ReadAll(io.LimitReader(r, services.MaxHeartbeatBody))
+	return string(b)
+}
+
+func writeHeartbeatLimited(c *gin.Context) {
+	c.Header("Retry-After", "1")
+	c.String(http.StatusTooManyRequests, "too many requests")
+}
+
+// heartbeatAllowed admits one request per token and kind per second. A cron job pinging
+// in a loop should not become a write per request, and a leaked URL should not
+// be a way to hammer Mongo. Like the status page limiter it allows when Redis
+// is down: a missed ping pages someone. It runs inside the handler rather than
+// as middleware because the token may come from a header.
+func heartbeatAllowed(c *gin.Context, token, kind string) bool {
+	rdb := auth.Redis()
+	if rdb == nil {
+		return true
+	}
+	key := heartbeatLimitKey(token, kind, time.Now())
+	count, err := rdb.Incr(c.Request.Context(), key).Result()
+	if err != nil {
+		return true
+	}
+	if count == 1 {
+		rdb.Expire(c.Request.Context(), key, 2*time.Second)
+	}
+	return count <= 1
+}
+
+// handleHeartbeat records a push from a job. It is mounted on the gin root
+// under /public for the same reasons as the status page (see
+// getPublicStatusPage): no session, no token, no licence gate, and /public is
+// already routed to this server by every deployment.
+//
+// Unknown token, disabled monitor and an unresolvable path all answer the same
+// 404.
+func handleHeartbeat(c *gin.Context) {
+	token, kind, ok := resolveHeartbeat(c.GetHeader(HeartbeatTokenHeader), c.Param("a"), c.Param("b"))
+	if !ok {
+		c.String(http.StatusNotFound, "not found")
+		return
+	}
+	if !heartbeatAllowed(c, token, kind) {
+		writeHeartbeatLimited(c)
+		return
+	}
+	err := services.RecordHeartbeat(token, kind, readHeartbeatBody(kind, c.Request.Body), time.Now())
+	if errors.Is(err, services.ErrHeartbeatNotFound) {
+		c.String(http.StatusNotFound, "not found")
+		return
+	}
+	if err != nil {
+		log.Printf("heartbeat: %v", err)
+		c.String(http.StatusInternalServerError, "error")
+		return
+	}
+	c.String(http.StatusOK, "OK")
+}
+
+// rotateHeartbeatToken godoc
+//
+//	@Summary		Rotate a heartbeat monitor's ping token
+//	@Description	Issues a new token and invalidates the old ping URL immediately. The token is returned only in this response.
+//	@Tags			monitors
+//	@Produce		json
+//	@Param			id	path		string	true	"Monitor ID"
+//	@Success		200	{object}	object{heartbeat_token=string}
+//	@Failure		404	{object}	ErrorResponse
+//	@Failure		500	{object}	ErrorResponse
+//	@Security		cookieAuth
+//	@Security		bearerAuth
+//	@Router			/monitors/{id}/rotate-token [post]
+func rotateHeartbeatToken(c *gin.Context) {
+	tok, err := services.RotateHeartbeatToken(auth.InstanceID(c), c.Param("id"))
+	if errors.Is(err, services.ErrHeartbeatNotFound) {
+		c.JSON(http.StatusNotFound, gin.H{"error": "heartbeat monitor not found"})
+		return
+	}
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"heartbeat_token": tok})
+}

server/internal/api/heartbeats_test.go

@@ -0,0 +1,112 @@
+package api
+
+import (
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/gin-gonic/gin"
+)
+
+func TestResolveHeartbeat(t *testing.T) {
+	cases := []struct {
+		name, header, a, b string
+		token, kind        string
+		ok                 bool
+	}{
+		{"url ping", "", "tok", "", "tok", "ping", true},
+		{"url start", "", "tok", "start", "tok", "start", true},
+		{"url fail", "", "tok", "fail", "tok", "fail", true},
+		{"url bad kind", "", "tok", "explode", "", "", false},
+		{"header ping", "htok", "", "", "htok", "ping", true},
+		{"header start", "htok", "start", "", "htok", "start", true},
+		{"header fail", "htok", "fail", "", "htok", "fail", true},
+		{"url token wins over header", "htok", "tok", "", "tok", "ping", true},
+		{"url token and kind win over header", "htok", "tok", "fail", "tok", "fail", true},
+		{"nothing", "", "", "", "", "", false},
+		// Without a header, /public/hb/start is a token called "start": it
+		// resolves, and the lookup answers 404 like any unknown token.
+		{"bare start without header", "", "start", "", "start", "ping", true},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			tok, kind, ok := resolveHeartbeat(c.header, c.a, c.b)
+			if tok != c.token || kind != c.kind || ok != c.ok {
+				t.Fatalf("got (%q,%q,%v), want (%q,%q,%v)", tok, kind, ok, c.token, c.kind, c.ok)
+			}
+		})
+	}
+}
+
+// A ping URL is a credential. Anything that logs request paths must see a
+// masked one.
+func TestMaskLogPath(t *testing.T) {
+	cases := map[string]string{
+		"/public/hb/abc123":        "/public/hb/***",
+		"/public/hb/abc123/fail":   "/public/hb/***/fail",
+		"/public/hb/abc123?x=1":    "/public/hb/***?x=1",
+		"/public/hb/start":         "/public/hb/start",
+		"/public/hb/fail":          "/public/hb/fail",
+		"/public/hb":               "/public/hb",
+		"/public/status/page":      "/public/status/page",
+		"/api/monitors/abc/uptime": "/api/monitors/abc/uptime",
+	}
+	for in, want := range cases {
+		if got := MaskLogPath(in); got != want {
+			t.Errorf("MaskLogPath(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+func TestHeartbeatUnresolvableIs404(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+	r := gin.New()
+	r.POST("/public/hb/:a/:b", handleHeartbeat)
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/public/hb/abc/explode", nil))
+	if w.Code != http.StatusNotFound {
+		t.Fatalf("code = %d, want 404", w.Code)
+	}
+}
+
+func TestHeartbeatLimitKey(t *testing.T) {
+	now := time.Unix(1700000000, 0)
+	start := heartbeatLimitKey("tok", services.HeartbeatStart, now)
+	ping := heartbeatLimitKey("tok", services.HeartbeatPing, now)
+	if start == ping {
+		t.Fatalf("start and ping share a limiter key: %s", start)
+	}
+	if ping != heartbeatLimitKey("tok", services.HeartbeatPing, now.Add(500*time.Millisecond)) {
+		t.Fatal("same kind in the same second should share a key")
+	}
+	if strings.Contains(ping, "tok") {
+		t.Fatalf("key leaks the plaintext token: %s", ping)
+	}
+}
+
+func TestReadHeartbeatBody(t *testing.T) {
+	long := strings.Repeat("x", services.MaxHeartbeatBody+500)
+	if got := readHeartbeatBody(services.HeartbeatFail, strings.NewReader(long)); len(got) != services.MaxHeartbeatBody {
+		t.Fatalf("fail body not capped: %d", len(got))
+	}
+	if got := readHeartbeatBody(services.HeartbeatPing, strings.NewReader("hi")); got != "" {
+		t.Fatalf("ping body should be ignored, got %q", got)
+	}
+	if got := readHeartbeatBody(services.HeartbeatFail, nil); got != "" {
+		t.Fatalf("nil body: %q", got)
+	}
+}
+
+func TestHeartbeatRateLimitedWritesRetryAfter(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+	w := httptest.NewRecorder()
+	c, _ := gin.CreateTestContext(w)
+	writeHeartbeatLimited(c)
+	if w.Code != http.StatusTooManyRequests || w.Header().Get("Retry-After") != "1" {
+		t.Fatalf("got %d retry-after %q", w.Code, w.Header().Get("Retry-After"))
+	}
+}

server/internal/api/instance.go

@@ -20,13 +20,32 @@ import (
 //	@Security		cookieAuth
 //	@Security		bearerAuth
 //	@Router			/instance/users [get]
+// instanceUserResponse wraps a member with whether they hold an MFA factor,
+// for the settings page's member column and reset action. A wrapper rather
+// than a field on models.User because User is shared with Vantage HQ.
+type instanceUserResponse struct {
+	models.User `bson:",inline"`
+	MFAEnabled  bool `json:"mfa_enabled"`
+}
+
 func listInstanceUsers(c *gin.Context) {
 	users, err := services.ListUsers(auth.InstanceID(c))
 	if err != nil {
 		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
 		return
 	}
-	c.JSON(http.StatusOK, users)
+	// One aggregate over two small collections beats N round trips for a
+	// member list that renders on every settings page load.
+	enabled, err := services.UsersWithMFA(auth.InstanceID(c))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	out := make([]instanceUserResponse, 0, len(users))
+	for _, u := range users {
+		out = append(out, instanceUserResponse{User: u, MFAEnabled: enabled[u.UserID]})
+	}
+	c.JSON(http.StatusOK, out)
 }
 
 func actorMayGrantOwner(c *gin.Context) bool {

server/internal/api/mfa.go

@@ -0,0 +1,294 @@
+package api
+
+import (
+	"errors"
+	"net/http"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+)
+
+// oidcUser refuses MFA management for a user whose IdP owns authentication.
+func oidcUser(c *gin.Context) bool {
+	u, err := services.GetUserInInstance(auth.InstanceID(c), auth.UserID(c))
+	return err == nil && u.AuthSource == models.AuthOIDC
+}
+
+// getMyMFA reports this user's factors.
+//
+//	@Summary	Get my MFA status
+//	@Tags		mfa
+//	@Produce	json
+//	@Success	200	{object}	object{totp_enabled=bool,passkeys=[]models.WebAuthnCredential,recovery_remaining=int,require_mfa=bool,applicable=bool}
+//	@Router		/me/mfa [get]
+func getMyMFA(c *gin.Context) {
+	instanceID, userID := auth.InstanceID(c), auth.UserID(c)
+	m, err := services.GetUserMFA(instanceID, userID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	passkeys, err := services.ListPasskeys(instanceID, userID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{
+		"totp_enabled":       m != nil && m.TOTPConfirmedAt != nil,
+		"passkeys":           passkeys,
+		"recovery_remaining": services.RecoveryCodesRemaining(m),
+		"require_mfa":        services.RequireMFAForInstance(instanceID),
+		"applicable":         !oidcUser(c),
+	})
+}
+
+// setupTOTP issues a new unconfirmed secret.
+//
+//	@Summary	Start TOTP setup
+//	@Tags		mfa
+//	@Produce	json
+//	@Success	200	{object}	object{secret=string,otpauth_uri=string}
+//	@Failure	409	{object}	object{error=string,code=string}
+//	@Router		/me/mfa/totp/setup [post]
+func setupTOTP(c *gin.Context) {
+	if oidcUser(c) {
+		c.JSON(http.StatusConflict, gin.H{"error": "your identity provider manages sign-in", "code": "mfa_not_applicable"})
+		return
+	}
+	instanceID, userID := auth.InstanceID(c), auth.UserID(c)
+	issuer := "Vantage"
+	if inst, err := services.GetInstance(instanceID); err == nil && inst != nil && inst.Name != "" {
+		issuer = inst.Name
+	}
+	secret, uri, err := services.StartTOTPSetup(instanceID, userID, issuer, auth.GetSessionFromContext(c).Email)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"secret": secret, "otpauth_uri": uri})
+}
+
+// confirmTOTP activates the pending secret and issues recovery codes if this
+// is the user's first factor.
+//
+//	@Summary	Confirm TOTP setup
+//	@Tags		mfa
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{code=string}	true	"Six-digit code"
+//	@Success	200		{object}	object{ok=bool,recovery_codes=[]string}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/me/mfa/totp/confirm [post]
+func confirmTOTP(c *gin.Context) {
+	instanceID, userID := auth.InstanceID(c), auth.UserID(c)
+	var body struct {
+		Code string `json:"code"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.Code == "" {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "code required"})
+		return
+	}
+	if err := services.ConfirmTOTP(instanceID, userID, body.Code); err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "that code is not valid", "code": "invalid_code"})
+		return
+	}
+	services.LogEvent(instanceID, "mfa.enrolled", actorFromCtx(c), "", "", "factor=totp")
+
+	m, _ := services.GetUserMFA(instanceID, userID)
+	if services.RecoveryCodesRemaining(m) > 0 {
+		c.JSON(http.StatusOK, gin.H{"ok": true})
+		return
+	}
+	codes, err := services.IssueRecoveryCodes(instanceID, userID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"ok": true, "recovery_codes": codes})
+}
+
+// removeTOTP drops the TOTP factor. Step-up guarded at the route.
+//
+//	@Summary	Remove TOTP
+//	@Tags		mfa
+//	@Produce	json
+//	@Success	204
+//	@Failure	409	{object}	object{error=string,code=string}
+//	@Router		/me/mfa/totp [delete]
+func removeTOTP(c *gin.Context) {
+	instanceID, userID := auth.InstanceID(c), auth.UserID(c)
+	if err := services.CheckCanRemoveFactor(instanceID, userID, services.FactorTOTP); err != nil {
+		c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "mfa_required_by_policy"})
+		return
+	}
+	if err := services.RemoveTOTP(instanceID, userID); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	services.LogEvent(instanceID, "mfa.removed", actorFromCtx(c), "", "", "factor=totp")
+	c.Status(http.StatusNoContent)
+}
+
+// regenerateRecoveryCodes invalidates the old set. Step-up guarded.
+//
+//	@Summary	Regenerate recovery codes
+//	@Tags		mfa
+//	@Produce	json
+//	@Success	200	{object}	object{recovery_codes=[]string}
+//	@Router		/me/mfa/recovery/regenerate [post]
+func regenerateRecoveryCodes(c *gin.Context) {
+	instanceID, userID := auth.InstanceID(c), auth.UserID(c)
+	codes, err := services.IssueRecoveryCodes(instanceID, userID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	services.LogEvent(instanceID, "mfa.recovery_regenerated", actorFromCtx(c), "", "", "")
+	c.JSON(http.StatusOK, gin.H{"recovery_codes": codes})
+}
+
+// renamePasskey and deletePasskey work on the hex credential ID.
+//
+//	@Summary	Rename a passkey
+//	@Tags		mfa
+//	@Accept		json
+//	@Produce	json
+//	@Param		id		path	string				true	"Credential ID"
+//	@Param		body	body	object{name=string}	true	"New name"
+//	@Success	204
+//	@Router		/me/passkeys/{id} [patch]
+func renamePasskey(c *gin.Context) {
+	var body struct {
+		Name string `json:"name"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.Name == "" {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "name required"})
+		return
+	}
+	err := services.RenamePasskey(auth.InstanceID(c), auth.UserID(c), c.Param("id"), body.Name)
+	if errors.Is(err, services.ErrNoPasskey) {
+		c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
+		return
+	}
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	c.Status(http.StatusNoContent)
+}
+
+// @Summary	Delete a passkey
+// @Tags		mfa
+// @Produce	json
+// @Param		id	path	string	true	"Credential ID"
+// @Success	204
+// @Failure	409	{object}	object{error=string,code=string}
+// @Router		/me/passkeys/{id} [delete]
+func deletePasskey(c *gin.Context) {
+	instanceID, userID := auth.InstanceID(c), auth.UserID(c)
+	if err := services.CheckCanRemoveFactor(instanceID, userID, services.FactorWebAuthn); err != nil {
+		c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "mfa_required_by_policy"})
+		return
+	}
+	err := services.DeletePasskey(instanceID, userID, c.Param("id"))
+	if errors.Is(err, services.ErrNoPasskey) {
+		c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
+		return
+	}
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	services.LogEvent(instanceID, "mfa.removed", actorFromCtx(c), "", "", "factor=webauthn")
+	c.Status(http.StatusNoContent)
+}
+
+// resetUserMFA lets an owner or admin clear somebody else's factors.
+//
+//	@Summary	Reset another member's MFA
+//	@Tags		mfa
+//	@Produce	json
+//	@Param		id	path	string	true	"User ID"
+//	@Success	204
+//	@Failure	403	{object}	object{error=string}
+//	@Router		/org/users/{id}/mfa [delete]
+func resetUserMFA(c *gin.Context) {
+	instanceID := auth.InstanceID(c)
+	target, err := services.GetUserInInstance(instanceID, c.Param("id"))
+	if err != nil {
+		c.JSON(http.StatusNotFound, gin.H{"error": "no such member"})
+		return
+	}
+	// An admin resetting an owner's MFA would be a promotion path: clear the
+	// factor, phish the password, hold the instance.
+	if auth.Role(c) != models.RoleOwner && target.Role == models.RoleOwner {
+		c.JSON(http.StatusForbidden, gin.H{"error": "only an owner can reset an owner's MFA"})
+		return
+	}
+	if err := services.ClearMFA(instanceID, target.UserID); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	services.LogEvent(instanceID, "mfa.reset", actorFromCtx(c), "", "", "target="+target.Email)
+	c.Status(http.StatusNoContent)
+}
+
+// stepUp re-authenticates the current session.
+//
+//	@Summary	Re-authenticate before a sensitive action
+//	@Tags		mfa
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{totp=string,recovery=string,password=string}	true	"One factor"
+//	@Success	200		{object}	object{ok=bool}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/me/step-up [post]
+func stepUp(c *gin.Context) {
+	sess := auth.GetSessionFromContext(c)
+	var body struct {
+		TOTP     string `json:"totp"`
+		Recovery string `json:"recovery"`
+		Password string `json:"password"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "a factor is required"})
+		return
+	}
+
+	var err error
+	switch {
+	case body.TOTP != "":
+		err = services.VerifyTOTPCode(sess.InstanceID, sess.UserID, body.TOTP)
+	case body.Recovery != "":
+		err = services.UseRecoveryCode(sess.InstanceID, sess.UserID, body.Recovery)
+	case body.Password != "":
+		// Password is offered only to a user with no MFA at all; accepting it
+		// from an enrolled user would demote step-up to what they already did.
+		has, herr := services.HasMFA(sess.InstanceID, sess.UserID)
+		if herr != nil || has {
+			err = services.ErrBadCode
+		} else {
+			u, uerr := services.GetUserInInstance(sess.InstanceID, sess.UserID)
+			if uerr != nil || !services.VerifyPassword(u, body.Password) {
+				err = services.ErrBadCode
+			}
+		}
+	default:
+		c.JSON(http.StatusBadRequest, gin.H{"error": "a factor is required"})
+		return
+	}
+
+	if err != nil {
+		services.LogEvent(sess.InstanceID, "step_up.failed", actorFromCtx(c), "", "", "")
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "that did not verify", "code": "invalid_code"})
+		return
+	}
+	if err := auth.TouchStepUpFromRequest(c); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not record re-authentication"})
+		return
+	}
+	services.LogEvent(sess.InstanceID, "step_up.ok", actorFromCtx(c), "", "", "")
+	c.JSON(http.StatusOK, gin.H{"ok": true})
+}

server/internal/api/monitors.go

@@ -1,6 +1,7 @@
 package api
 
 import (
+	"errors"
 	"net/http"
 	"strconv"
 	"time"
@@ -21,6 +22,8 @@ func registerMonitorRoutes(g *gin.RouterGroup) {
 	g.GET("/monitors/:id/incidents", getMonitorIncidents)
 	g.GET("/monitors/:id/uptime", getMonitorUptime)
 	g.GET("/monitors/:id/samples", getMonitorSamples)
+	g.GET("/monitors/:id/servers", getMonitorServers)
+	g.POST("/monitors/:id/rotate-token", rotateHeartbeatToken)
 }
 
 // listMonitors godoc
@@ -85,6 +88,14 @@ func createMonitor(c *gin.Context) {
 	}
 	created, err := services.CreateMonitor(auth.InstanceID(c), &m, auth.ServerScope(c))
 	if err != nil {
+		if errors.Is(err, services.ErrInvalidMonitor) {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+		if errors.Is(err, services.ErrMonitorOutOfScope) {
+			c.JSON(http.StatusForbidden, gin.H{"error": err.Error()})
+			return
+		}
 		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
 		return
 	}
@@ -131,9 +142,11 @@ func getMonitor(c *gin.Context) {
 //	@Accept			json
 //	@Produce		json
 //	@Param			id		path	string												true	"Monitor ID"
-//	@Param			body	body	object{name=string,group=string,type=string,target=models.MonitorTarget,interval_sec=int,runner=string,retries=int,enabled=bool,channel_ids=[]string}	true	"Fields to update"
+//	@Param			body	body	object{name=string,group=string,type=string,target=models.MonitorTarget,interval_sec=int,runner=string,retries=int,enabled=bool,channel_ids=[]string,for_sec=int}	true	"Fields to update"
 //	@Success		204
 //	@Failure		400	{object}	ErrorResponse
+//	@Failure		403	{object}	ErrorResponse
+//	@Failure		404	{object}	ErrorResponse
 //	@Failure		500	{object}	ErrorResponse
 //	@Security		cookieAuth
 //	@Security		bearerAuth
@@ -149,6 +162,7 @@ func updateMonitor(c *gin.Context) {
 		Retries     *int                  `json:"retries"`
 		Enabled     *bool                 `json:"enabled"`
 		ChannelIDs  *[]string             `json:"channel_ids"`
+		ForSec      *int                  `json:"for_sec"`
 	}
 	if err := c.ShouldBindJSON(&body); err != nil {
 		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
@@ -182,11 +196,26 @@ func updateMonitor(c *gin.Context) {
 	if body.ChannelIDs != nil {
 		upd["channel_ids"] = *body.ChannelIDs
 	}
+	if body.ForSec != nil {
+		upd["for_sec"] = *body.ForSec
+	}
 	if len(upd) == 0 {
 		c.JSON(http.StatusBadRequest, gin.H{"error": "no fields to update"})
 		return
 	}
 	if err := services.UpdateMonitor(auth.InstanceID(c), c.Param("id"), upd, auth.ServerScope(c)); err != nil {
+		if errors.Is(err, services.ErrInvalidMonitor) {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+		if errors.Is(err, services.ErrMonitorOutOfScope) {
+			c.JSON(http.StatusForbidden, gin.H{"error": err.Error()})
+			return
+		}
+		if errors.Is(err, services.ErrMonitorNotFound) {
+			c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
+			return
+		}
 		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
 		return
 	}
@@ -199,12 +228,17 @@ func updateMonitor(c *gin.Context) {
 //	@Tags			monitors
 //	@Param			id	path	string	true	"Monitor ID"
 //	@Success		204
+//	@Failure		403	{object}	ErrorResponse
 //	@Failure		500	{object}	ErrorResponse
 //	@Security		cookieAuth
 //	@Security		bearerAuth
 //	@Router			/monitors/{id} [delete]
 func deleteMonitor(c *gin.Context) {
-	if err := services.DeleteMonitor(auth.InstanceID(c), c.Param("id")); err != nil {
+	if err := services.DeleteMonitor(auth.InstanceID(c), c.Param("id"), auth.ServerScope(c)); err != nil {
+		if errors.Is(err, services.ErrMonitorOutOfScope) {
+			c.JSON(http.StatusForbidden, gin.H{"error": err.Error()})
+			return
+		}
 		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
 		return
 	}
@@ -238,9 +272,63 @@ func getMonitorIncidents(c *gin.Context) {
 		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
 		return
 	}
+	visible, restricted, err := services.VisibleServerIDs(auth.InstanceID(c), auth.ServerScope(c))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	incidents = services.FilterByVisibleServer(incidents, func(i models.Incident) string { return i.ServerID }, visible, restricted)
 	c.JSON(http.StatusOK, incidents)
 }
 
+// getMonitorServers godoc
+//
+//	@Summary		List a metric monitor's per-server states
+//	@Tags			monitors
+//	@Produce		json
+//	@Param			id	path		string	true	"Monitor ID"
+//	@Success		200	{array}		models.MonitorServerState
+//	@Failure		404	{object}	ErrorResponse
+//	@Failure		500	{object}	ErrorResponse
+//	@Security		cookieAuth
+//	@Security		bearerAuth
+//	@Router			/monitors/{id}/servers [get]
+func getMonitorServers(c *gin.Context) {
+	instanceID := auth.InstanceID(c)
+	m, err := services.GetMonitor(instanceID, c.Param("id"))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	if m == nil || m.Type != models.MonitorMetric {
+		c.JSON(http.StatusNotFound, gin.H{"error": "metric monitor not found"})
+		return
+	}
+	states, err := services.ListMonitorServerStates(instanceID, m.MonitorID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	visible, restricted, err := services.VisibleServerIDs(instanceID, auth.ServerScope(c))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	states = services.FilterByVisibleServer(states, func(s models.MonitorServerState) string { return s.ServerID }, visible, restricted)
+
+	servers, err := services.ListServersFiltered(instanceID, m.Target.Selector)
+	if err == nil {
+		names := map[string]string{}
+		for _, s := range servers {
+			names[s.ServerID] = s.Hostname
+		}
+		for i := range states {
+			states[i].Hostname = names[states[i].ServerID]
+		}
+	}
+	c.JSON(http.StatusOK, states)
+}
+
 // getMonitorSamples godoc
 //
 //	@Summary		Get a monitor's individual check results

server/internal/api/patching.go

@@ -0,0 +1,455 @@
+package api
+
+import (
+	"errors"
+	"fmt"
+	"net/http"
+	"strconv"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+)
+
+// registerPatchingRoutes mounts maintenance windows, patch policies and patch
+// runs. Free on every tier: security patching is never paywalled, so no
+// RequireFeature here. Writes that decide what reboots and when are owner or
+// admin; watching and cancelling a run is open to every role.
+func registerPatchingRoutes(g *gin.RouterGroup) {
+	admin := auth.RequireRole("owner", "admin")
+
+	g.GET("/maintenance-windows", listWindows)
+	g.POST("/maintenance-windows", admin, createWindow)
+	g.POST("/maintenance-windows/preview", previewWindow)
+	g.GET("/maintenance-windows/:id", getWindow)
+	g.PUT("/maintenance-windows/:id", admin, updateWindow)
+	g.DELETE("/maintenance-windows/:id", admin, deleteWindow)
+
+	g.GET("/patch-policies", listPolicies)
+	g.POST("/patch-policies", admin, createPolicy)
+	g.GET("/patch-policies/:id", getPolicy)
+	g.PUT("/patch-policies/:id", admin, updatePolicy)
+	g.DELETE("/patch-policies/:id", admin, deletePolicy)
+	g.POST("/patch-policies/:id/run-now", admin, runPolicyNow)
+
+	g.GET("/patch-runs", listPatchRuns)
+	g.GET("/patch-runs/:runId", getPatchRun)
+	g.POST("/patch-runs/:runId/cancel", cancelPatchRun)
+}
+
+// patchError maps every patching service error once.
+func patchError(c *gin.Context, err error) {
+	switch {
+	case errors.Is(err, services.ErrWindowNotFound), errors.Is(err, services.ErrPolicyNotFound), errors.Is(err, services.ErrPatchRunNotFound):
+		c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
+	case errors.Is(err, services.ErrWindowInUse), errors.Is(err, services.ErrPatchRunActive), errors.Is(err, services.ErrPatchRunFinished):
+		c.JSON(http.StatusConflict, gin.H{"error": err.Error()})
+	case errors.Is(err, services.ErrWindowInvalid), errors.Is(err, services.ErrPolicyInvalid),
+		errors.Is(err, services.ErrNoTargets), errors.Is(err, services.ErrInvalidTag),
+		errors.Is(err, services.ErrWorkflowTargetOutOfScope):
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+	default:
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+	}
+}
+
+// listWindows godoc
+//
+//	@Summary	List maintenance windows
+//	@Tags		patching
+//	@Produce	json
+//	@Success	200	{array}		models.MaintenanceWindow
+//	@Failure	500	{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/maintenance-windows [get]
+func listWindows(c *gin.Context) {
+	ws, err := services.ListWindows(auth.InstanceID(c))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	c.JSON(http.StatusOK, ws)
+}
+
+// createWindow godoc
+//
+//	@Summary	Create a maintenance window
+//	@Tags		patching
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		models.MaintenanceWindow	true	"Window"
+//	@Success	201		{object}	models.MaintenanceWindow
+//	@Failure	400		{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/maintenance-windows [post]
+func createWindow(c *gin.Context) {
+	var body models.MaintenanceWindow
+	if err := c.ShouldBindJSON(&body); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	w, err := services.CreateWindow(auth.InstanceID(c), body)
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(auth.InstanceID(c), "patch.window_created", actorFromCtx(c), "", "",
+		fmt.Sprintf("maintenance window %s: %s %s for %d minutes", w.Name, w.Cron, w.TZ, w.DurationMinutes))
+	c.JSON(http.StatusCreated, w)
+}
+
+// previewWindow godoc
+//
+//	@Summary		Preview the next three maintenance windows
+//	@Description	Computed by the scheduler's own code, so the editor and the scheduler agree.
+//	@Tags			patching
+//	@Accept			json
+//	@Produce		json
+//	@Param			body	body		WindowPreviewRequest	true	"Schedule"
+//	@Success		200		{array}		services.WindowSpan
+//	@Failure		400		{object}	ErrorResponse
+//	@Security		cookieAuth
+//	@Security		bearerAuth
+//	@Router			/maintenance-windows/preview [post]
+func previewWindow(c *gin.Context) {
+	var body WindowPreviewRequest
+	if err := c.ShouldBindJSON(&body); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	spans, err := services.PreviewWindow(body.Cron, body.TZ, body.DurationMinutes, time.Now(), 3)
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	c.JSON(http.StatusOK, spans)
+}
+
+// getWindow godoc
+//
+//	@Summary	Get a maintenance window
+//	@Tags		patching
+//	@Produce	json
+//	@Param		id	path		string	true	"Window ID"
+//	@Success	200	{object}	models.MaintenanceWindow
+//	@Failure	404	{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/maintenance-windows/{id} [get]
+func getWindow(c *gin.Context) {
+	w, err := services.GetWindow(auth.InstanceID(c), c.Param("id"))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	c.JSON(http.StatusOK, w)
+}
+
+// updateWindow godoc
+//
+//	@Summary		Update a maintenance window
+//	@Description	Moves the next run of every enabled policy using it.
+//	@Tags			patching
+//	@Accept			json
+//	@Produce		json
+//	@Param			id		path		string						true	"Window ID"
+//	@Param			body	body		models.MaintenanceWindow	true	"Window"
+//	@Success		200		{object}	models.MaintenanceWindow
+//	@Failure		400		{object}	ErrorResponse
+//	@Failure		404		{object}	ErrorResponse
+//	@Security		cookieAuth
+//	@Security		bearerAuth
+//	@Router			/maintenance-windows/{id} [put]
+func updateWindow(c *gin.Context) {
+	var body models.MaintenanceWindow
+	if err := c.ShouldBindJSON(&body); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	if err := services.CheckWindowScope(auth.InstanceID(c), c.Param("id"), auth.ServerScope(c)); err != nil {
+		patchError(c, err)
+		return
+	}
+	w, err := services.UpdateWindow(auth.InstanceID(c), c.Param("id"), body)
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(auth.InstanceID(c), "patch.window_updated", actorFromCtx(c), "", "",
+		fmt.Sprintf("maintenance window %s: %s %s for %d minutes", w.Name, w.Cron, w.TZ, w.DurationMinutes))
+	c.JSON(http.StatusOK, w)
+}
+
+// deleteWindow godoc
+//
+//	@Summary	Delete a maintenance window
+//	@Tags		patching
+//	@Param		id	path	string	true	"Window ID"
+//	@Success	204
+//	@Failure	404	{object}	ErrorResponse
+//	@Failure	409	{object}	ErrorResponse	"window_in_use"
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/maintenance-windows/{id} [delete]
+func deleteWindow(c *gin.Context) {
+	if err := services.CheckWindowScope(auth.InstanceID(c), c.Param("id"), auth.ServerScope(c)); err != nil {
+		patchError(c, err)
+		return
+	}
+	if err := services.DeleteWindow(auth.InstanceID(c), c.Param("id")); err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(auth.InstanceID(c), "patch.window_deleted", actorFromCtx(c), "", "", "maintenance window "+c.Param("id")+" deleted")
+	c.Status(http.StatusNoContent)
+}
+
+// listPolicies godoc
+//
+//	@Summary	List patch policies
+//	@Tags		patching
+//	@Produce	json
+//	@Success	200	{array}		models.PatchPolicy
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-policies [get]
+func listPolicies(c *gin.Context) {
+	ps, err := services.ListPolicies(auth.InstanceID(c))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	c.JSON(http.StatusOK, ps)
+}
+
+// createPolicy godoc
+//
+//	@Summary	Create a patch policy
+//	@Tags		patching
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		models.PatchPolicy	true	"Policy"
+//	@Success	201		{object}	models.PatchPolicy
+//	@Failure	400		{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-policies [post]
+func createPolicy(c *gin.Context) {
+	var body models.PatchPolicy
+	if err := c.ShouldBindJSON(&body); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	p, err := services.CreatePolicy(auth.InstanceID(c), body, auth.ServerScope(c))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(auth.InstanceID(c), "patch.policy_created", actorFromCtx(c), "", "",
+		fmt.Sprintf("patch policy %s: scope %s, reboot %s, enabled %v", p.Name, p.Scope, p.Reboot, p.Enabled))
+	c.JSON(http.StatusCreated, p)
+}
+
+// getPolicy godoc
+//
+//	@Summary	Get a patch policy
+//	@Tags		patching
+//	@Produce	json
+//	@Param		id	path		string	true	"Policy ID"
+//	@Success	200	{object}	models.PatchPolicy
+//	@Failure	404	{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-policies/{id} [get]
+func getPolicy(c *gin.Context) {
+	p, err := services.GetPolicy(auth.InstanceID(c), c.Param("id"))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	c.JSON(http.StatusOK, p)
+}
+
+// updatePolicy godoc
+//
+//	@Summary	Update a patch policy
+//	@Tags		patching
+//	@Accept		json
+//	@Produce	json
+//	@Param		id		path		string				true	"Policy ID"
+//	@Param		body	body		models.PatchPolicy	true	"Policy"
+//	@Success	200		{object}	models.PatchPolicy
+//	@Failure	400		{object}	ErrorResponse
+//	@Failure	404		{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-policies/{id} [put]
+func updatePolicy(c *gin.Context) {
+	var body models.PatchPolicy
+	if err := c.ShouldBindJSON(&body); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	p, err := services.UpdatePolicy(auth.InstanceID(c), c.Param("id"), body, auth.ServerScope(c))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(auth.InstanceID(c), "patch.policy_updated", actorFromCtx(c), "", "",
+		fmt.Sprintf("patch policy %s: scope %s, reboot %s, enabled %v", p.Name, p.Scope, p.Reboot, p.Enabled))
+	c.JSON(http.StatusOK, p)
+}
+
+// deletePolicy godoc
+//
+//	@Summary	Delete a patch policy
+//	@Tags		patching
+//	@Param		id	path	string	true	"Policy ID"
+//	@Success	204
+//	@Failure	404	{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-policies/{id} [delete]
+func deletePolicy(c *gin.Context) {
+	instanceID := auth.InstanceID(c)
+	p, err := services.GetPolicy(instanceID, c.Param("id"))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	if err := services.CheckPolicyScope(instanceID, *p, auth.ServerScope(c)); err != nil {
+		patchError(c, err)
+		return
+	}
+	if err := services.DeletePolicy(instanceID, p.PolicyID); err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(instanceID, "patch.policy_deleted", actorFromCtx(c), "", "", "patch policy "+p.Name+" deleted")
+	c.Status(http.StatusNoContent)
+}
+
+// runPolicyNow godoc
+//
+//	@Summary		Run a patch policy now
+//	@Description	Opens a window of the policy's usual length starting now.
+//	@Tags			patching
+//	@Produce		json
+//	@Param			id	path		string	true	"Policy ID"
+//	@Success		202	{object}	models.PatchRun
+//	@Failure		404	{object}	ErrorResponse
+//	@Failure		409	{object}	ErrorResponse
+//	@Security		cookieAuth
+//	@Security		bearerAuth
+//	@Router			/patch-policies/{id}/run-now [post]
+func runPolicyNow(c *gin.Context) {
+	run, err := services.StartRunNow(auth.InstanceID(c), c.Param("id"), actorFromCtx(c), auth.ServerScope(c))
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	c.JSON(http.StatusAccepted, run)
+}
+
+// listPatchRuns godoc
+//
+//	@Summary	List patch runs
+//	@Tags		patching
+//	@Produce	json
+//	@Param		policy_id	query		string	false	"Filter by policy"
+//	@Param		server_id	query		string	false	"Filter by server"
+//	@Param		limit		query		int		false	"At most 200, default 50"
+//	@Success	200			{array}		models.PatchRun
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-runs [get]
+func listPatchRuns(c *gin.Context) {
+	instanceID := auth.InstanceID(c)
+	limit, _ := strconv.ParseInt(c.Query("limit"), 10, 64)
+	runs, err := services.ListPatchRuns(instanceID, c.Query("policy_id"), c.Query("server_id"), limit)
+	if err != nil {
+		patchError(c, err)
+		return
+	}
+	out := runs[:0]
+	for i := range runs {
+		before := len(runs[i].Servers)
+		if err := services.ScopePatchRun(instanceID, &runs[i], auth.ServerScope(c)); err != nil {
+			patchError(c, err)
+			return
+		}
+		if before == 0 || len(runs[i].Servers) > 0 {
+			out = append(out, runs[i])
+		}
+	}
+	c.JSON(http.StatusOK, out)
+}
+
+// getPatchRun godoc
+//
+//	@Summary	Get a patch run, with per-server output
+//	@Tags		patching
+//	@Produce	json
+//	@Param		runId	path		string	true	"Run ID"
+//	@Success	200		{object}	models.PatchRun
+//	@Failure	404		{object}	ErrorResponse
+//	@Security	cookieAuth
+//	@Security	bearerAuth
+//	@Router		/patch-runs/{runId} [get]
+func getPatchRun(c *gin.Context) {
+	run, ok := scopedRun(c)
+	if ok {
+		c.JSON(http.StatusOK, run)
+	}
+}
+
+// cancelPatchRun godoc
+//
+//	@Summary		Cancel a patch run
+//	@Description	Stops further dispatch. Servers already patching finish.
+//	@Tags			patching
+//	@Param			runId	path	string	true	"Run ID"
+//	@Success		204
+//	@Failure		404	{object}	ErrorResponse
+//	@Failure		409	{object}	ErrorResponse
+//	@Security		cookieAuth
+//	@Security		bearerAuth
+//	@Router			/patch-runs/{runId}/cancel [post]
+func cancelPatchRun(c *gin.Context) {
+	run, ok := scopedRun(c)
+	if !ok {
+		return
+	}
+	if err := services.CancelPatchRun(auth.InstanceID(c), run.RunID); err != nil {
+		patchError(c, err)
+		return
+	}
+	services.LogEvent(auth.InstanceID(c), "patch.cancelled", actorFromCtx(c), "", "", "patch run "+run.RunID+" cancelled")
+	c.Status(http.StatusNoContent)
+}
+
+// scopedRun loads a run for a caller. A tag-restricted token that cannot see
+// every server in the run gets a 404, the same answer as a run that does not
+// exist, rather than a partial record it could act on.
+func scopedRun(c *gin.Context) (*models.PatchRun, bool) {
+	instanceID := auth.InstanceID(c)
+	run, err := services.GetPatchRun(instanceID, c.Param("runId"))
+	if err != nil {
+		patchError(c, err)
+		return nil, false
+	}
+	before := len(run.Servers)
+	if err := services.ScopePatchRun(instanceID, run, auth.ServerScope(c)); err != nil {
+		patchError(c, err)
+		return nil, false
+	}
+	if len(run.Servers) != before {
+		patchError(c, services.ErrPatchRunNotFound)
+		return nil, false
+	}
+	return run, true
+}

server/internal/api/patching_scope_test.go

@@ -0,0 +1,34 @@
+package api
+
+import "testing"
+
+// Every patching route must carry a scope and a server-scope declaration, or
+// boot fails. Asserting the exact scope here keeps a copy-paste of
+// "patching:read" onto a write route from slipping through.
+func TestPatchingRouteScopes(t *testing.T) {
+	want := map[string]string{
+		"GET /api/maintenance-windows":          "patching:read",
+		"POST /api/maintenance-windows":         "patching:write",
+		"POST /api/maintenance-windows/preview": "patching:read",
+		"GET /api/maintenance-windows/:id":      "patching:read",
+		"PUT /api/maintenance-windows/:id":      "patching:write",
+		"DELETE /api/maintenance-windows/:id":   "patching:write",
+		"GET /api/patch-policies":               "patching:read",
+		"POST /api/patch-policies":              "patching:write",
+		"GET /api/patch-policies/:id":           "patching:read",
+		"PUT /api/patch-policies/:id":           "patching:write",
+		"DELETE /api/patch-policies/:id":        "patching:write",
+		"POST /api/patch-policies/:id/run-now":  "patching:write",
+		"GET /api/patch-runs":                   "patching:read",
+		"GET /api/patch-runs/:runId":            "patching:read",
+		"POST /api/patch-runs/:runId/cancel":    "patching:write",
+	}
+	for route, scope := range want {
+		if got := routeScopes[route]; got != scope {
+			t.Errorf("%s: scope %q, want %q", route, got, scope)
+		}
+		if _, ok := serverScopedRoutes[route]; !ok {
+			t.Errorf("%s: missing from serverScopedRoutes", route)
+		}
+	}
+}

server/internal/api/ratelimit_auth.go

@@ -0,0 +1,48 @@
+package api
+
+import (
+	"fmt"
+	"net/http"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+)
+
+// authRateLimit is per client address per minute. It bounds how many tickets an
+// attacker can start; the ticket's own five-attempt cap bounds guesses inside
+// one. Neither alone is enough.
+const authRateLimit = 20
+
+// RateLimitAuth guards every unauthenticated sign-in endpoint. Without it, the
+// per-ticket cap is trivially sidestepped by starting a new sign-in each time.
+func RateLimitAuth() gin.HandlerFunc {
+	return func(c *gin.Context) {
+		rdb := services.RedisClient
+		if rdb == nil {
+			c.Next()
+			return
+		}
+		window := time.Now().Unix() / 60
+		key := fmt.Sprintf("km:rl:auth:%s:%d", c.ClientIP(), window)
+		ctx := c.Request.Context()
+		n, err := rdb.Incr(ctx, key).Result()
+		if err != nil {
+			// A limiter that cannot reach Redis must not lock out sign-in: fail
+			// open rather than turn a Redis blip into a second outage.
+			c.Next()
+			return
+		}
+		if n == 1 {
+			rdb.Expire(ctx, key, time.Minute)
+		}
+		if n > authRateLimit {
+			c.Header("Retry-After", "60")
+			c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
+				"error": "too many sign-in attempts; try again in a minute",
+			})
+			return
+		}
+		c.Next()
+	}
+}

server/internal/api/scopes.go

@@ -99,14 +99,16 @@ var routeScopes = map[string]string{
 	"GET /api/runs/:runId/servers/:serverId/logs/stream": "workflows:read",
 
 	// Monitor and incident routes, registered by registerMonitorRoutes.
-	"GET /api/monitors":               "monitors:read",
-	"POST /api/monitors":              "monitors:write",
-	"GET /api/monitors/:id":           "monitors:read",
-	"PUT /api/monitors/:id":           "monitors:write",
-	"DELETE /api/monitors/:id":        "monitors:write",
-	"GET /api/monitors/:id/incidents": "monitors:read",
-	"GET /api/monitors/:id/uptime":    "monitors:read",
-	"GET /api/monitors/:id/samples":   "monitors:read",
+	"GET /api/monitors":                   "monitors:read",
+	"POST /api/monitors":                  "monitors:write",
+	"GET /api/monitors/:id":               "monitors:read",
+	"PUT /api/monitors/:id":               "monitors:write",
+	"DELETE /api/monitors/:id":            "monitors:write",
+	"GET /api/monitors/:id/incidents":     "monitors:read",
+	"GET /api/monitors/:id/uptime":        "monitors:read",
+	"GET /api/monitors/:id/samples":       "monitors:read",
+	"GET /api/monitors/:id/servers":       "monitors:read",
+	"POST /api/monitors/:id/rotate-token": "monitors:write",
 
 	// Channel routes, registered by registerChannelRoutes. Channels exist to
 	// serve alerts, so they share the monitors scope rather than getting their
@@ -175,6 +177,23 @@ var routeScopes = map[string]string{
 	"PUT /api/status-pages/:pageId/incidents/:incidentId":          "status:write",
 	"DELETE /api/status-pages/:pageId/incidents/:incidentId":       "status:write",
 	"POST /api/status-pages/:pageId/incidents/:incidentId/updates": "status:write",
+
+	// Scheduled patching: maintenance windows, patch policies and patch runs.
+	"GET /api/maintenance-windows":          "patching:read",
+	"POST /api/maintenance-windows":         "patching:write",
+	"POST /api/maintenance-windows/preview": "patching:read",
+	"GET /api/maintenance-windows/:id":      "patching:read",
+	"PUT /api/maintenance-windows/:id":      "patching:write",
+	"DELETE /api/maintenance-windows/:id":   "patching:write",
+	"GET /api/patch-policies":               "patching:read",
+	"POST /api/patch-policies":              "patching:write",
+	"GET /api/patch-policies/:id":           "patching:read",
+	"PUT /api/patch-policies/:id":           "patching:write",
+	"DELETE /api/patch-policies/:id":        "patching:write",
+	"POST /api/patch-policies/:id/run-now":  "patching:write",
+	"GET /api/patch-runs":                   "patching:read",
+	"GET /api/patch-runs/:runId":            "patching:read",
+	"POST /api/patch-runs/:runId/cancel":    "patching:write",
 }
 
 // RequireScopes enforces routeScopes for token-authenticated requests and does
@@ -221,6 +240,30 @@ var routesOutsideAPIGroup = map[string]bool{
 	"GET /api/secrets/:group/values": true,
 }
 
+// sessionOnlyRoutes are /api routes that an API token must never reach, so they
+// deliberately have no entry in routeScopes. RequireScopes already refuses any
+// unmapped route to a token; this set is what tells AssertScopeMapComplete the
+// absence is a decision rather than an omission.
+//
+// Every entry manages the caller's own second factors or proves the caller is
+// present. A token that could enrol a passkey, regenerate recovery codes or
+// satisfy step-up would turn one leaked token into a way around MFA itself.
+var sessionOnlyRoutes = map[string]bool{
+	"GET /api/me/mfa":                      true,
+	"POST /api/me/mfa/totp/setup":          true,
+	"POST /api/me/mfa/totp/confirm":        true,
+	"DELETE /api/me/mfa/totp":              true,
+	"POST /api/me/mfa/recovery/regenerate": true,
+	"POST /api/me/passkeys/begin":          true,
+	"POST /api/me/passkeys/finish":         true,
+	"PATCH /api/me/passkeys/:id":           true,
+	"DELETE /api/me/passkeys/:id":          true,
+	"POST /api/me/step-up":                 true,
+	"POST /api/me/step-up/webauthn/begin":  true,
+	"POST /api/me/step-up/webauthn/finish": true,
+	"DELETE /api/org/users/:id/mfa":        true,
+}
+
 // AssertScopeMapComplete fails boot when a registered /api route has no scope.
 //
 // Without it, adding a route silently makes it unreachable by every token, and
@@ -236,6 +279,10 @@ func AssertScopeMapComplete(r *gin.Engine) error {
 		if routesOutsideAPIGroup[route.Method+" "+route.Path] {
 			continue
 		}
+		// Session-only routes are refused to every token by having no scope.
+		if sessionOnlyRoutes[route.Method+" "+route.Path] {
+			continue
+		}
 		if _, ok := routeScopes[route.Method+" "+route.Path]; !ok {
 			missing = append(missing, route.Method+" "+route.Path)
 		}

server/internal/api/serverscope.go

@@ -292,13 +292,25 @@ var serverScopedRoutes = map[string]scopeDecl{
 	// then act on.
 	"DELETE /api/monitors/:id": fleetWide,
 
-	// A monitor's incidents, uptime rollups and recent samples are all about
-	// the monitored endpoint - status, latency, timestamps - and carry no
-	// server identifier at all; the runner is a field of the monitor
-	// document, which these do not return.
-	"GET /api/monitors/:id/incidents": exempt,
-	"GET /api/monitors/:id/uptime":    exempt,
-	"GET /api/monitors/:id/samples":   exempt,
+	// Rotating a ping token touches no server and returns only the token.
+	"POST /api/monitors/:id/rotate-token": fleetWide,
+
+	// Uptime rollups and recent samples are about the monitored endpoint -
+	// status, latency, timestamps - and carry no server identifier at all;
+	// the runner is a field of the monitor document, which these do not
+	// return.
+	"GET /api/monitors/:id/uptime":  exempt,
+	"GET /api/monitors/:id/samples": exempt,
+
+	// A metric monitor's incidents carry server_id, so a restricted token
+	// must not see an incident for a server outside its scope. Handled by
+	// services.FilterByVisibleServer rather than by the route classification
+	// itself.
+	"GET /api/monitors/:id/incidents": scoped,
+
+	// Per-server metric state names servers, so it is filtered to the ones the
+	// credential can see.
+	"GET /api/monitors/:id/servers": scoped,
 
 	// ---- notification channels ----
 
@@ -398,6 +410,49 @@ var serverScopedRoutes = map[string]scopeDecl{
 	// Each tool touching server data applies the caller's selector itself.
 	"POST /api/mcp": exempt,
 	"GET /api/mcp":  exempt,
+
+	// Maintenance windows are a cron expression, a zone and a duration. They
+	// name no server and return no server data, so reading and creating one
+	// is exempt. Changing or deleting one moves or stops the patching of
+	// every policy using it, so each is refused when any of those policies
+	// targets servers outside the token's tag restriction.
+	"GET /api/maintenance-windows":          exempt,
+	"POST /api/maintenance-windows":         exempt,
+	"POST /api/maintenance-windows/preview": exempt,
+	"GET /api/maintenance-windows/:id":      exempt,
+	"PUT /api/maintenance-windows/:id":      scoped,
+	"DELETE /api/maintenance-windows/:id":   scoped,
+	// Reading a policy returns its selector (server IDs and tag pairs) and no
+	// hostname, inventory or state, the same data a workflow's targets carry.
+	"GET /api/patch-policies":     exempt,
+	"GET /api/patch-policies/:id": exempt,
+	// Writes and run-now act on the policy's targets, so each is refused when
+	// those targets reach outside the token's tag restriction.
+	"POST /api/patch-policies":             scoped,
+	"PUT /api/patch-policies/:id":          scoped,
+	"DELETE /api/patch-policies/:id":       scoped,
+	"POST /api/patch-policies/:id/run-now": scoped,
+	// Runs name hostnames; ScopePatchRun removes servers the token cannot see.
+	"GET /api/patch-runs":                scoped,
+	"GET /api/patch-runs/:runId":         scoped,
+	"POST /api/patch-runs/:runId/cancel": scoped,
+
+	// Multi-factor authentication. These act on the caller's own factors (or,
+	// for the reset, on another member's), never on a server, and no API token
+	// can reach them at all - see sessionOnlyRoutes.
+	"GET /api/me/mfa":                      exempt,
+	"POST /api/me/mfa/totp/setup":          exempt,
+	"POST /api/me/mfa/totp/confirm":        exempt,
+	"DELETE /api/me/mfa/totp":              exempt,
+	"POST /api/me/mfa/recovery/regenerate": exempt,
+	"POST /api/me/passkeys/begin":          exempt,
+	"POST /api/me/passkeys/finish":         exempt,
+	"PATCH /api/me/passkeys/:id":           exempt,
+	"DELETE /api/me/passkeys/:id":          exempt,
+	"POST /api/me/step-up":                 exempt,
+	"POST /api/me/step-up/webauthn/begin":  exempt,
+	"POST /api/me/step-up/webauthn/finish": exempt,
+	"DELETE /api/org/users/:id/mfa":        exempt,
 }
 
 // AssertServerScopeMapComplete refuses to boot when any registered /api route

server/internal/api/serverscope_test.go

@@ -16,7 +16,7 @@ func TestServerScopeMapCoversEveryScopedRoute(t *testing.T) {
 		if _, ok := routeScopes[r]; ok {
 			continue
 		}
-		if routesOutsideAPIGroup[r] {
+		if routesOutsideAPIGroup[r] || sessionOnlyRoutes[r] {
 			continue
 		}
 		t.Errorf("route %q is declared in serverScopedRoutes but is not a registered route", r)

server/internal/api/types.go

@@ -297,3 +297,26 @@ type StatusIncidentUpdateRequest struct {
 	Status string `json:"status" binding:"required"`
 	Body   string `json:"body" binding:"required"`
 }
+
+// --- patching ---
+
+// ApplyUpdatesResponse keeps the message existing scripts read and adds the
+// run that records what happened.
+type ApplyUpdatesResponse struct {
+	Message string `json:"message"`
+	RunID   string `json:"run_id,omitempty"`
+}
+
+// ApplyUpdatesErrorResponse is the 503 body of apply-updates: the agent is
+// offline, and the attempt is still recorded as a run.
+type ApplyUpdatesErrorResponse struct {
+	Error string `json:"error"`
+	RunID string `json:"run_id,omitempty"`
+}
+
+// WindowPreviewRequest is the body of POST /maintenance-windows/preview.
+type WindowPreviewRequest struct {
+	Cron            string `json:"cron"`
+	TZ              string `json:"tz"`
+	DurationMinutes int    `json:"duration_minutes"`
+}

server/internal/auth/instancehost.go

@@ -68,17 +68,40 @@ func InstanceFromHost(c *gin.Context) (*models.Instance, bool) {
 	return InstanceForHost(c.Request.Host)
 }
 
+// instanceBySlug is a variable so tests can stub the Mongo read.
+var instanceBySlug = services.GetInstanceBySlugIncludingLocked
+
 // InstanceForHost is InstanceFromHost with the host supplied explicitly.
+//
+// An instance Vantage HQ has locked under a dispute resolves to nothing, so
+// every caller refuses it without knowing locks exist. HostLocked is the one
+// place that may tell the two apart, for the login page's message.
 func InstanceForHost(host string) (*models.Instance, bool) {
-	slug := hostSlug(host)
-	if slug == "" {
+	inst := slugInstance(hostSlug(host))
+	if inst == nil || inst.LockedAt != nil {
 		return nil, false
 	}
-	if inst, hit := cachedInstanceFor(slug); hit {
-		return inst, inst != nil
-	}
+	return inst, true
+}
 
-	inst, err := services.GetInstanceBySlug(slug)
+// HostLocked reports whether the host names an instance Vantage HQ has locked.
+// It shares InstanceForHost's cache, so a lock and an unlock are seen within
+// the same 60 seconds by both.
+func HostLocked(host string) bool {
+	inst := slugInstance(hostSlug(host))
+	return inst != nil && inst.LockedAt != nil
+}
+
+// slugInstance reads the instance a slug names, locked or not, through the
+// cache.
+func slugInstance(slug string) *models.Instance {
+	if slug == "" {
+		return nil
+	}
+	if inst, hit := cachedInstanceFor(slug); hit {
+		return inst
+	}
+	inst, err := instanceBySlug(slug)
 	if err != nil || inst == nil {
 		// Negative entries are cached too. Without them an unknown but
 		// well-formed host costs a Mongo query per anonymous request, which
@@ -86,10 +109,10 @@ func InstanceForHost(host string) (*models.Instance, bool) {
 		// round trip is itself a timing oracle separating "no such instance"
 		// from "instance exists, page does not".
 		storeInstance(slug, nil)
-		return nil, false
+		return nil
 	}
 	storeInstance(slug, inst)
-	return inst, true
+	return inst
 }
 
 // SoleInstance resolves the one instance of a deployment that has exactly one.
@@ -99,7 +122,7 @@ func InstanceForHost(host string) (*models.Instance, bool) {
 // more than one instance exists.
 func SoleInstance() (*models.Instance, bool) {
 	if inst, hit := cachedInstanceFor(soleInstanceCacheKey); hit {
-		return inst, inst != nil
+		return inst, inst != nil && inst.LockedAt == nil
 	}
 	n, err := services.CountInstances()
 	if err != nil || n != 1 {
@@ -112,7 +135,8 @@ func SoleInstance() (*models.Instance, bool) {
 		return nil, false
 	}
 	storeInstance(soleInstanceCacheKey, inst)
-	return inst, true
+	// A locked sole instance is refused like any other locked instance.
+	return inst, inst.LockedAt == nil
 }
 
 func cachedInstanceFor(key string) (*models.Instance, bool) {

server/internal/auth/instancehost_locked_test.go

@@ -0,0 +1,56 @@
+package auth
+
+import (
+	"errors"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+// stubSlugs replaces the slug lookup and empties the host cache for one test.
+func stubSlugs(t *testing.T, bySlug map[string]*models.Instance) {
+	t.Helper()
+	prev := instanceBySlug
+	instanceBySlug = func(slug string) (*models.Instance, error) {
+		if inst, ok := bySlug[slug]; ok {
+			return inst, nil
+		}
+		return nil, errors.New("not found")
+	}
+	instanceCacheMu.Lock()
+	instanceCache = map[string]cachedInstance{}
+	instanceCacheMu.Unlock()
+	t.Cleanup(func() {
+		instanceBySlug = prev
+		instanceCacheMu.Lock()
+		instanceCache = map[string]cachedInstance{}
+		instanceCacheMu.Unlock()
+	})
+}
+
+func TestLockedHostResolvesToNothingButReportsLocked(t *testing.T) {
+	at := time.Date(2026, 9, 11, 8, 0, 0, 0, time.UTC)
+	stubSlugs(t, map[string]*models.Instance{
+		"acme": {InstanceID: "i-acme", Slug: "acme", LockedAt: &at},
+		"open": {InstanceID: "i-open", Slug: "open"},
+	})
+
+	if _, ok := InstanceForHost("acme.vantage.example.com"); ok {
+		t.Fatal("a locked instance must not resolve")
+	}
+	if !HostLocked("acme.vantage.example.com") {
+		t.Fatal("a locked instance's host must report locked")
+	}
+
+	if inst, ok := InstanceForHost("open.vantage.example.com"); !ok || inst.InstanceID != "i-open" {
+		t.Fatalf("unlocked instance: got %v, %v", inst, ok)
+	}
+	if HostLocked("open.vantage.example.com") {
+		t.Fatal("an unlocked instance is not locked")
+	}
+
+	if HostLocked("missing.vantage.example.com") || HostLocked("example.com") {
+		t.Fatal("a host naming no instance is not locked")
+	}
+}

server/internal/auth/local.go

@@ -1,6 +1,7 @@
 package auth
 
 import (
+	"errors"
 	"fmt"
 	"net/http"
 
@@ -31,10 +32,18 @@ func SetSessionCookie(c *gin.Context, sessionID string) {
 //
 // Anything else is refused rather than guessed. Picking an instance on someone's
 // behalf is how you sign them into the wrong tenant.
+//
+// An instance Vantage HQ has locked answers ErrInstanceLocked on both paths.
+// Without the check on the fallback, a single-instance deployment whose host
+// resolver hides the locked instance fell through to "the sole instance" and
+// served its sign-in page anyway.
 func resolveLoginInstance(c *gin.Context) (string, error) {
 	if inst, ok := InstanceFromHost(c); ok {
 		return inst.InstanceID, nil
 	}
+	if HostLocked(c.Request.Host) {
+		return "", ErrInstanceLocked
+	}
 	n, err := services.CountInstances()
 	if err != nil {
 		return "", err
@@ -48,9 +57,22 @@ func resolveLoginInstance(c *gin.Context) (string, error) {
 	if err != nil {
 		return "", err
 	}
+	if inst.LockedAt != nil {
+		return "", ErrInstanceLocked
+	}
 	return inst.InstanceID, nil
 }
 
+// ErrInstanceLocked is a sign-in to an instance Vantage HQ has locked under an
+// account dispute. The login page shows it as a message in place of the form.
+var ErrInstanceLocked = errors.New("access to this instance is suspended")
+
+// loginLocked reports whether this request's sign-in target is locked.
+func loginLocked(c *gin.Context) bool {
+	_, err := resolveLoginInstance(c)
+	return errors.Is(err, ErrInstanceLocked)
+}
+
 func HandleLocalLogin(c *gin.Context) {
 	var body struct {
 		Email    string `json:"email"`
@@ -61,6 +83,10 @@ func HandleLocalLogin(c *gin.Context) {
 		return
 	}
 	instanceID, err := resolveLoginInstance(c)
+	if errors.Is(err, ErrInstanceLocked) {
+		c.JSON(http.StatusForbidden, gin.H{"error": err.Error(), "locked": true})
+		return
+	}
 	if err != nil {
 		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
 		return
@@ -76,16 +102,48 @@ func HandleLocalLogin(c *gin.Context) {
 		c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid credentials"})
 		return
 	}
-	sessionID, err := SaveSession(c.Request.Context(), &Session{
-		UserID: u.UserID, InstanceID: u.InstanceID, Role: u.Role, Email: u.Email,
-	})
+	requireMFA := services.RequireMFAForInstance(instanceID)
+	hasMFA, err := services.HasMFA(u.InstanceID, u.UserID)
 	if err != nil {
-		c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not read MFA state"})
 		return
 	}
-	_ = services.TouchLastLogin(u.UserID)
-	SetSessionCookie(c, sessionID)
-	c.JSON(http.StatusOK, gin.H{"ok": true})
+
+	switch loginDecision(hasMFA, requireMFA) {
+	case "session":
+		if err := mintSession(c, u, []string{"pwd"}); err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+			return
+		}
+		c.JSON(http.StatusOK, gin.H{"ok": true})
+
+	case "verify":
+		methods, err := services.MFAMethods(u.InstanceID, u.UserID)
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": "could not read MFA state"})
+			return
+		}
+		id, err := CreateTicket(c.Request.Context(), &Ticket{
+			UserID: u.UserID, InstanceID: u.InstanceID, Email: u.Email, Methods: methods,
+		})
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+			return
+		}
+		SetPendingCookie(c, id)
+		c.JSON(http.StatusOK, gin.H{"mfa_required": true, "methods": methods})
+
+	case "enrol":
+		id, err := CreateTicket(c.Request.Context(), &Ticket{
+			UserID: u.UserID, InstanceID: u.InstanceID, Email: u.Email, EnrolOnly: true,
+		})
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+			return
+		}
+		SetPendingCookie(c, id)
+		c.JSON(http.StatusOK, gin.H{"enrol_required": true})
+	}
 }
 
 // HandleListPublicProviders is unauthenticated: it is what the login page reads
@@ -101,6 +159,10 @@ func HandleListPublicProviders(c *gin.Context) {
 	out := []publicProvider{}
 
 	instanceID, err := resolveLoginInstance(c)
+	if errors.Is(err, ErrInstanceLocked) {
+		c.JSON(http.StatusOK, gin.H{"local_enabled": false, "providers": out, "locked": true})
+		return
+	}
 	if err != nil {
 		// An unresolvable instance is not an error the login page can act on:
 		// it still has to render a password form. Answer the safe shape.
@@ -127,6 +189,12 @@ func HandleBootstrapStatus(c *gin.Context) {
 		err      error
 		instName string
 	)
+	// Checked first: a locked instance must not fall through to the global
+	// user count, which answers needs_setup false and draws the sign-in form.
+	if loginLocked(c) {
+		c.JSON(http.StatusOK, gin.H{"needs_setup": false, "locked": true})
+		return
+	}
 	if inst, ok := InstanceFromHost(c); ok {
 		n, err = services.CountInstanceUsers(inst.InstanceID)
 		instName = inst.Name
@@ -191,14 +259,10 @@ func HandleBootstrap(c *gin.Context) {
 		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
 		return
 	}
-	sessionID, err := SaveSession(c.Request.Context(), &Session{
-		UserID: u.UserID, InstanceID: u.InstanceID, Role: u.Role, Email: u.Email,
-	})
-	if err != nil {
+	if err := mintSession(c, u, []string{"pwd"}); err != nil {
 		c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
 		return
 	}
-	SetSessionCookie(c, sessionID)
 	c.JSON(http.StatusCreated, gin.H{
 		"instance":    inst,
 		"slug":        inst.Slug,

server/internal/auth/locked_test.go

@@ -0,0 +1,34 @@
+package auth
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/gin-gonic/gin"
+)
+
+func TestRefuseLocked(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+	prev := instanceLocked
+	t.Cleanup(func() { instanceLocked = prev })
+	instanceLocked = func(id string) bool { return id == "locked" }
+
+	w := httptest.NewRecorder()
+	c, _ := gin.CreateTestContext(w)
+	if !refuseLocked(c, "locked") {
+		t.Fatal("a locked instance must be refused")
+	}
+	if w.Code != http.StatusUnauthorized {
+		t.Fatalf("status = %d, want 401", w.Code)
+	}
+
+	w = httptest.NewRecorder()
+	c, _ = gin.CreateTestContext(w)
+	if refuseLocked(c, "open") {
+		t.Fatal("an unlocked instance must pass")
+	}
+	if c.IsAborted() {
+		t.Fatal("an unlocked instance must not abort")
+	}
+}

server/internal/auth/mfa_enrol.go

@@ -0,0 +1,230 @@
+package auth
+
+import (
+	"bytes"
+	"encoding/json"
+	"net/http"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+	"github.com/go-webauthn/webauthn/protocol"
+	"github.com/go-webauthn/webauthn/webauthn"
+)
+
+// enrolTicketStillValid refuses an enrol-only ticket once the user has any
+// factor. Otherwise someone holding only the password who started sign-in
+// before the real user enrolled could, within the ticket's lifetime, add
+// their own passkey and replace the user's recovery codes. Answered as an
+// expired ticket, the single indistinguishable error, and the ticket is
+// destroyed. A lookup failure is treated the same way: fail closed.
+//
+// It runs before the factor is written, not in finishEnrolment: by then the
+// factor being enrolled already counts, so HasMFA would always be true.
+func enrolTicketStillValid(c *gin.Context, t *Ticket, ticketID string) bool {
+	has, err := services.HasMFA(t.InstanceID, t.UserID)
+	if err != nil || has {
+		_ = DeleteTicket(c.Request.Context(), ticketID)
+		abortTicketExpired(c)
+		return false
+	}
+	return true
+}
+
+// HandleEnrolTOTPSetup starts enrolment for a user the instance requires MFA
+// from, before they hold a session. Only an enrol-only ticket reaches it.
+//
+//	@Summary	Start forced TOTP enrolment during sign-in
+//	@Tags		auth
+//	@Produce	json
+//	@Success	200	{object}	object{secret=string,otpauth_uri=string}
+//	@Failure	401	{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/enrol/totp/setup [post]
+func HandleEnrolTOTPSetup(c *gin.Context) {
+	t, ticketID, ok := ticketFromRequest(c, scopeEnrol)
+	if !ok || !enrolTicketStillValid(c, t, ticketID) {
+		return
+	}
+	inst, err := services.GetInstance(t.InstanceID)
+	issuer := "Vantage"
+	if err == nil && inst != nil && inst.Name != "" {
+		issuer = inst.Name
+	}
+	secret, uri, err := services.StartTOTPSetup(t.InstanceID, t.UserID, issuer, t.Email)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start enrolment"})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"secret": secret, "otpauth_uri": uri})
+}
+
+// HandleEnrolTOTPConfirm finishes forced enrolment and signs the user in.
+//
+//	@Summary	Confirm forced TOTP enrolment and sign in
+//	@Tags		auth
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{code=string}	true	"Six-digit code"
+//	@Success	200		{object}	object{ok=bool,recovery_codes=[]string}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/enrol/totp/confirm [post]
+func HandleEnrolTOTPConfirm(c *gin.Context) {
+	t, ticketID, ok := ticketFromRequest(c, scopeEnrol)
+	if !ok || !enrolTicketStillValid(c, t, ticketID) {
+		return
+	}
+	var body struct {
+		Code string `json:"code"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.Code == "" {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "code required"})
+		return
+	}
+	if err := services.ConfirmTOTP(t.InstanceID, t.UserID, body.Code); err != nil {
+		left, ferr := FailTicket(c.Request.Context(), ticketID)
+		if ferr != nil || left == 0 {
+			abortTicketExpired(c)
+			return
+		}
+		c.JSON(http.StatusUnauthorized, gin.H{
+			"error": "that code is not valid", "code": "invalid_code", "attempts_left": left,
+		})
+		return
+	}
+	finishEnrolment(c, t, ticketID, services.FactorTOTP)
+}
+
+// finishEnrolment issues recovery codes, mints the session and audits, so the
+// TOTP and passkey enrolment paths cannot drift apart.
+func finishEnrolment(c *gin.Context, t *Ticket, ticketID, factor string) {
+	codes, err := services.IssueRecoveryCodes(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not issue recovery codes"})
+		return
+	}
+	u, err := services.GetUserInInstance(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid credentials"})
+		return
+	}
+	if err := mintSession(c, u, []string{"pwd", factor}); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+		return
+	}
+	_ = DeleteTicket(c.Request.Context(), ticketID)
+	services.LogEvent(t.InstanceID, "mfa.enrolled", u.Email, "", "", "factor="+factor)
+	c.JSON(http.StatusOK, gin.H{"ok": true, "recovery_codes": codes})
+}
+
+// HandleEnrolPasskeyBegin starts forced passkey enrolment during sign-in.
+// Identical to HandleRegisterPasskeyBegin except the user comes from the
+// enrol-only ticket rather than a session, since none exists yet.
+//
+//	@Summary	Begin forced passkey enrolment during sign-in
+//	@Tags		auth
+//	@Produce	json
+//	@Success	200	{object}	object{publicKey=object,ceremony_id=string}
+//	@Failure	401	{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/enrol/passkey/begin [post]
+func HandleEnrolPasskeyBegin(c *gin.Context) {
+	t, ticketID, ok := ticketFromRequest(c, scopeEnrol)
+	if !ok || !enrolTicketStillValid(c, t, ticketID) {
+		return
+	}
+	handle, err := services.WebAuthnHandle(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start enrolment"})
+		return
+	}
+	existing, err := services.ListPasskeys(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start enrolment"})
+		return
+	}
+	lib := make([]webauthn.Credential, 0, len(existing))
+	for _, cr := range existing {
+		lib = append(lib, toLibCredential(cr))
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start enrolment"})
+		return
+	}
+	options, sessionData, err := w.BeginRegistration(
+		waUser{handle: handle, name: t.Email, credentials: lib},
+		webauthn.WithExclusions(webauthn.Credentials(lib).CredentialDescriptors()),
+	)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start enrolment"})
+		return
+	}
+	id, err := saveCeremony(c.Request.Context(), sessionData)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start enrolment"})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"publicKey": options.Response, "ceremony_id": id})
+}
+
+// HandleEnrolPasskeyFinish stores the new credential and finishes forced
+// enrolment, minting the session that HandleEnrolTOTPConfirm also produces.
+//
+//	@Summary	Complete forced passkey enrolment and sign in
+//	@Tags		auth
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{ceremony_id=string,name=string,credential=object}	true	"Attestation"
+//	@Success	200		{object}	object{ok=bool,recovery_codes=[]string}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/enrol/passkey/finish [post]
+func HandleEnrolPasskeyFinish(c *gin.Context) {
+	t, ticketID, ok := ticketFromRequest(c, scopeEnrol)
+	if !ok || !enrolTicketStillValid(c, t, ticketID) {
+		return
+	}
+	var body struct {
+		CeremonyID string          `json:"ceremony_id"`
+		Name       string          `json:"name"`
+		Credential json.RawMessage `json:"credential"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.CeremonyID == "" {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "attestation required"})
+		return
+	}
+	sessionData, err := loadCeremony(c.Request.Context(), body.CeremonyID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "that enrolment expired", "code": "mfa_ticket_expired"})
+		return
+	}
+	parsed, err := protocol.ParseCredentialCreationResponseBody(bytes.NewReader(body.Credential))
+	if err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "that passkey could not be read"})
+		return
+	}
+	handle, err := services.WebAuthnHandle(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not finish enrolment"})
+		return
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not finish enrolment"})
+		return
+	}
+	cred, err := w.CreateCredential(waUser{handle: handle, name: t.Email}, *sessionData, parsed)
+	if err != nil {
+		logWebAuthnFailure(c, "enrolment registration", err)
+		c.JSON(http.StatusBadRequest, gin.H{"error": "that passkey could not be verified"})
+		return
+	}
+	transports := make([]string, 0, len(parsed.Response.Transports))
+	for _, tr := range parsed.Response.Transports {
+		transports = append(transports, string(tr))
+	}
+	if err := services.SavePasskey(t.InstanceID, t.UserID, body.Name,
+		cred.ID, cred.PublicKey, cred.Authenticator.AAGUID, cred.Authenticator.SignCount,
+		transports, cred.Flags.BackupEligible, cred.Flags.BackupState); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not save the passkey"})
+		return
+	}
+	finishEnrolment(c, t, ticketID, services.FactorWebAuthn)
+}

server/internal/auth/mfa_login.go

@@ -0,0 +1,136 @@
+package auth
+
+import (
+	"net/http"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+)
+
+// loginDecision is the branch a verified password takes.
+func loginDecision(hasMFA, requireMFA bool) string {
+	switch {
+	case hasMFA:
+		return "verify"
+	case requireMFA:
+		return "enrol"
+	default:
+		return "session"
+	}
+}
+
+// newSession builds the session a sign-in produces. Sign-in counts as a
+// step-up, so StepUpAt is always the moment of sign-in.
+func newSession(u *models.User, amr []string, now time.Time) *Session {
+	return &Session{
+		UserID: u.UserID, InstanceID: u.InstanceID, Role: u.Role, Email: u.Email,
+		AMR: amr, StepUpAt: &now,
+	}
+}
+
+// oidcSession is the session an SSO callback mints. AMR "oidc" is what exempts
+// it from step-up: the IdP owns authentication policy, and these users have
+// no password or local factor to step up with.
+func oidcSession(u *models.User, name string, now time.Time) *Session {
+	s := newSession(u, []string{"oidc"}, now)
+	s.Name = name
+	return s
+}
+
+// mintSession is the single place a session is created from a user, so every
+// path records amr and step-up freshness the same way.
+func mintSession(c *gin.Context, u *models.User, amr []string) error {
+	return saveSignIn(c, newSession(u, amr, time.Now()))
+}
+
+// saveSignIn persists a sign-in session and sets its cookie. The pending-MFA
+// cookie is cleared because a completed sign-in supersedes any ticket.
+func saveSignIn(c *gin.Context, sess *Session) error {
+	sessionID, err := SaveSession(c.Request.Context(), sess)
+	if err != nil {
+		return err
+	}
+	_ = services.TouchLastLogin(sess.UserID)
+	ClearPendingCookie(c)
+	SetSessionCookie(c, sessionID)
+	return nil
+}
+
+// HandleMFATOTP completes a sign-in with a TOTP code.
+//
+//	@Summary	Complete sign-in with a TOTP code
+//	@Tags		auth
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{code=string}	true	"Six-digit code"
+//	@Success	200		{object}	object{ok=bool}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/totp [post]
+func HandleMFATOTP(c *gin.Context) {
+	handleMFAVerify(c, services.FactorTOTP)
+}
+
+// HandleMFARecovery completes a sign-in with a recovery code.
+//
+//	@Summary	Complete sign-in with a recovery code
+//	@Tags		auth
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{code=string}	true	"Recovery code"
+//	@Success	200		{object}	object{ok=bool}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/recovery [post]
+func HandleMFARecovery(c *gin.Context) {
+	handleMFAVerify(c, services.FactorRecovery)
+}
+
+func handleMFAVerify(c *gin.Context, factor string) {
+	t, ticketID, ok := ticketFromRequest(c, scopeVerify)
+	if !ok {
+		return
+	}
+	var body struct {
+		Code string `json:"code"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.Code == "" {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "code required"})
+		return
+	}
+
+	var err error
+	switch factor {
+	case services.FactorTOTP:
+		err = services.VerifyTOTPCode(t.InstanceID, t.UserID, body.Code)
+	case services.FactorRecovery:
+		err = services.UseRecoveryCode(t.InstanceID, t.UserID, body.Code)
+	}
+	if err != nil {
+		left, ferr := FailTicket(c.Request.Context(), ticketID)
+		services.LogEvent(t.InstanceID, "mfa.failed", t.Email, "", "", "factor="+factor)
+		if ferr != nil || left == 0 {
+			abortTicketExpired(c)
+			return
+		}
+		c.JSON(http.StatusUnauthorized, gin.H{
+			"error": "that code is not valid", "code": "invalid_code", "attempts_left": left,
+		})
+		return
+	}
+
+	u, err := services.GetUserInInstance(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid credentials"})
+		return
+	}
+	if err := mintSession(c, u, []string{"pwd", factor}); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+		return
+	}
+	_ = DeleteTicket(c.Request.Context(), ticketID)
+	if factor == services.FactorRecovery {
+		services.LogEvent(t.InstanceID, "mfa.recovery_used", u.Email, "", "", "")
+	}
+	c.JSON(http.StatusOK, gin.H{"ok": true})
+}

server/internal/auth/mfa_login_test.go

@@ -0,0 +1,27 @@
+package auth
+
+import "testing"
+
+// The whole point of the feature is in this table: a user with MFA is never
+// handed a session by a password alone, and require_mfa turns "no factor" into
+// forced enrolment rather than a free pass.
+func TestLoginDecision(t *testing.T) {
+	cases := []struct {
+		name       string
+		hasMFA     bool
+		requireMFA bool
+		want       string
+	}{
+		{"no mfa, not required", false, false, "session"},
+		{"no mfa, required", false, true, "enrol"},
+		{"has mfa, not required", true, false, "verify"},
+		{"has mfa, required", true, true, "verify"},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := loginDecision(tc.hasMFA, tc.requireMFA); got != tc.want {
+				t.Fatalf("loginDecision(%v,%v) = %q, want %q", tc.hasMFA, tc.requireMFA, got, tc.want)
+			}
+		})
+	}
+}

server/internal/auth/mfaticket.go

@@ -0,0 +1,180 @@
+package auth
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"time"
+
+	"github.com/gin-gonic/gin"
+	"github.com/redis/go-redis/v9"
+)
+
+const (
+	ticketTTL         = 5 * time.Minute
+	maxTicketAttempts = 5
+	ticketPrefix      = "km:mfa:"
+	pendingCookieName = "km_mfa_pending"
+)
+
+// ErrTicketExpired covers every unusable ticket - missing, timed out, or
+// destroyed by too many wrong codes. They are one message on purpose: which of
+// the three it was tells an attacker whether the password was right.
+var ErrTicketExpired = errors.New("this sign-in attempt has expired; start again")
+
+// Ticket is a password that verified but has not yet become a session. It is
+// deliberately NOT a Session: nothing half-authenticated may reach a route
+// under Middleware, and the way to guarantee that is for it never to be the
+// type those routes read.
+type Ticket struct {
+	UserID     string   `json:"user_id"`
+	InstanceID string   `json:"instance_id"`
+	Email      string   `json:"email"`
+	Methods    []string `json:"methods"`
+	EnrolOnly  bool     `json:"enrol_only"`
+}
+
+type ticketScope int
+
+const (
+	scopeVerify ticketScope = iota
+	scopeEnrol
+)
+
+func (t *Ticket) allows(s ticketScope) bool {
+	if t.EnrolOnly {
+		return s == scopeEnrol
+	}
+	return s == scopeVerify
+}
+
+func attemptsLeft(attempts int) int {
+	if attempts >= maxTicketAttempts {
+		return 0
+	}
+	return maxTicketAttempts - attempts
+}
+
+// attemptsKey is the counter backing a ticket's brute-force cap. It is a
+// separate key rather than a field on the ticket JSON so INCR can make the
+// count atomic: two requests racing on the same ticket must each cost one
+// attempt, not both read the same count and both write count+1.
+func attemptsKey(id string) string {
+	return ticketPrefix + id + ":attempts"
+}
+
+func CreateTicket(ctx context.Context, t *Ticket) (string, error) {
+	id, err := randomHex(32)
+	if err != nil {
+		return "", err
+	}
+	data, err := json.Marshal(t)
+	if err != nil {
+		return "", err
+	}
+	if err := rdb.Set(ctx, ticketPrefix+id, data, ticketTTL).Err(); err != nil {
+		return "", err
+	}
+	return id, nil
+}
+
+func LoadTicket(ctx context.Context, id string) (*Ticket, error) {
+	data, err := rdb.Get(ctx, ticketPrefix+id).Bytes()
+	if errors.Is(err, redis.Nil) {
+		return nil, ErrTicketExpired
+	}
+	if err != nil {
+		return nil, err
+	}
+	var t Ticket
+	if err := json.Unmarshal(data, &t); err != nil {
+		return nil, ErrTicketExpired
+	}
+	return &t, nil
+}
+
+// FailTicket records a wrong code and returns how many attempts remain. The
+// count is kept in its own INCR-backed key rather than the ticket JSON: a
+// read-modify-write on the JSON lets requests racing on the same ticket all
+// read the same count and all write count+1, which bypasses the cap instead
+// of costing one attempt each. At zero the ticket is destroyed rather than
+// left to time out.
+func FailTicket(ctx context.Context, id string) (int, error) {
+	// Confirms the ticket exists first, so a missing/expired/destroyed ticket
+	// still answers with the one indistinguishable ErrTicketExpired rather
+	// than incrementing a counter for an id nobody holds.
+	if _, err := LoadTicket(ctx, id); err != nil {
+		return 0, err
+	}
+	key := attemptsKey(id)
+	n, err := rdb.Incr(ctx, key).Result()
+	if err != nil {
+		return 0, err
+	}
+	if n == 1 {
+		// Only the creator of the counter sets its expiry, so a later
+		// increment never extends it past the ticket's own window.
+		if err := rdb.Expire(ctx, key, ticketTTL).Err(); err != nil {
+			return 0, err
+		}
+	}
+	left := attemptsLeft(int(n))
+	if left == 0 {
+		_ = DeleteTicket(ctx, id)
+		return 0, nil
+	}
+	return left, nil
+}
+
+func DeleteTicket(ctx context.Context, id string) error {
+	return rdb.Del(ctx, ticketPrefix+id, attemptsKey(id)).Err()
+}
+
+func SetPendingCookie(c *gin.Context, id string) {
+	secure := c.Request.TLS != nil || c.GetHeader("X-Forwarded-Proto") == "https"
+	http.SetCookie(c.Writer, &http.Cookie{
+		Name:     pendingCookieName,
+		Value:    id,
+		Path:     "/",
+		HttpOnly: true,
+		Secure:   secure,
+		SameSite: http.SameSiteLaxMode,
+		MaxAge:   int(ticketTTL.Seconds()),
+	})
+}
+
+func ClearPendingCookie(c *gin.Context) {
+	secure := c.Request.TLS != nil || c.GetHeader("X-Forwarded-Proto") == "https"
+	http.SetCookie(c.Writer, &http.Cookie{
+		Name: pendingCookieName, Value: "", Path: "/",
+		HttpOnly: true, Secure: secure, SameSite: http.SameSiteLaxMode, MaxAge: -1,
+	})
+}
+
+// ticketFromRequest resolves the pending ticket and enforces its scope. It
+// writes the response and returns false when the ticket is unusable.
+func ticketFromRequest(c *gin.Context, scope ticketScope) (*Ticket, string, bool) {
+	cookie, err := c.Request.Cookie(pendingCookieName)
+	if err != nil || cookie.Value == "" {
+		abortTicketExpired(c)
+		return nil, "", false
+	}
+	t, err := LoadTicket(c.Request.Context(), cookie.Value)
+	if err != nil {
+		abortTicketExpired(c)
+		return nil, "", false
+	}
+	if !t.allows(scope) {
+		abortTicketExpired(c)
+		return nil, "", false
+	}
+	return t, cookie.Value, true
+}
+
+func abortTicketExpired(c *gin.Context) {
+	ClearPendingCookie(c)
+	c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{
+		"error": ErrTicketExpired.Error(), "code": "mfa_ticket_expired",
+	})
+}

server/internal/auth/mfaticket_test.go

@@ -0,0 +1,50 @@
+package auth
+
+import "testing"
+
+// The ticket's attempt cap is the only per-ticket brute-force guard: five
+// wrong codes must destroy it rather than let an attacker keep guessing
+// against a single stolen password.
+func TestAttemptsLeftCountsDownAndHitsZero(t *testing.T) {
+	cases := []struct {
+		attempts int
+		want     int
+	}{
+		{0, 5}, {1, 4}, {4, 1}, {5, 0}, {9, 0},
+	}
+	for _, tc := range cases {
+		if got := attemptsLeft(tc.attempts); got != tc.want {
+			t.Errorf("attemptsLeft(%d) = %d, want %d", tc.attempts, got, tc.want)
+		}
+	}
+}
+
+// attemptsKey must derive deterministically from the ticket id and stay
+// distinct from the ticket's own key, since FailTicket relies on INCR against
+// it being the sole writer of the attempt count.
+func TestAttemptsKeyIsDerivedFromTicketID(t *testing.T) {
+	got := attemptsKey("abc123")
+	want := "km:mfa:abc123:attempts"
+	if got != want {
+		t.Errorf("attemptsKey(%q) = %q, want %q", "abc123", got, want)
+	}
+	if attemptsKey("abc123") == ticketPrefix+"abc123" {
+		t.Error("attempts key must not collide with the ticket's own key")
+	}
+}
+
+// An enrol-only ticket exists because the instance requires MFA the user does
+// not have. It must not satisfy a verification endpoint, and a verification
+// ticket must not reach the enrolment endpoints - each would skip the other's
+// purpose.
+func TestTicketScopeIsEnforcedInBothDirections(t *testing.T) {
+	verify := &Ticket{Methods: []string{"totp"}}
+	enrol := &Ticket{EnrolOnly: true}
+
+	if !verify.allows(scopeVerify) || verify.allows(scopeEnrol) {
+		t.Error("a verification ticket must allow only verification")
+	}
+	if !enrol.allows(scopeEnrol) || enrol.allows(scopeVerify) {
+		t.Error("an enrolment ticket must allow only enrolment")
+	}
+}

server/internal/auth/middleware.go

@@ -12,6 +12,20 @@ import (
 
 const ctxSessionKey = "km_session"
 
+// instanceLocked is a variable so tests can stub the Mongo-backed check.
+var instanceLocked = services.InstanceLocked
+
+// refuseLocked answers 401 for a session or token on an instance Vantage HQ has
+// locked under a dispute. Same body as an expired session: a locked instance
+// is not announced as locked to whoever holds a credential for it.
+func refuseLocked(c *gin.Context, instanceID string) bool {
+	if !instanceLocked(instanceID) {
+		return false
+	}
+	c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "session expired"})
+	return true
+}
+
 func GetSessionFromContext(c *gin.Context) *Session {
 	v, _ := c.Get(ctxSessionKey)
 	sess, _ := v.(*Session)
@@ -81,6 +95,13 @@ func Middleware() gin.HandlerFunc {
 			return
 		}
 
+		// Explicit, because the host guard below cannot do this: the resolver
+		// hides a locked instance, so its host resolves to nothing and that
+		// guard is skipped rather than tripped.
+		if refuseLocked(c, sess.InstanceID) {
+			return
+		}
+
 		c.Set(ctxSessionKey, sess)
 
 		// The host guard applies to both credential kinds. A token carries an

server/internal/auth/oidc.go

@@ -6,6 +6,7 @@ import (
 	"net/http"
 	"strings"
 	"sync"
+	"time"
 
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
@@ -88,6 +89,10 @@ func loadProvider(instanceID, providerID string) (*models.AuthProvider, string,
 func HandleSSOStart(c *gin.Context) {
 	inst, ok := InstanceFromHost(c)
 	if !ok {
+		if HostLocked(c.Request.Host) {
+			c.Redirect(http.StatusFound, "/login?error=instance_locked")
+			return
+		}
 		c.Redirect(http.StatusFound, "/login?error=unknown_host")
 		return
 	}
@@ -221,15 +226,10 @@ func completeSSOLogin(c *gin.Context, instanceID, email, name string) {
 		}
 	}
 
-	sessionID, err := SaveSession(c.Request.Context(), &Session{
-		UserID: u.UserID, InstanceID: u.InstanceID, Role: u.Role, Email: u.Email, Name: name,
-	})
-	if err != nil {
+	if err := saveSignIn(c, oidcSession(u, name, time.Now())); err != nil {
 		c.Redirect(http.StatusFound, "/login?error=session_failed")
 		return
 	}
-	_ = services.TouchLastLogin(u.UserID)
-	SetSessionCookie(c, sessionID)
 	c.Redirect(http.StatusFound, "/")
 }
 

server/internal/auth/passkey_login.go

@@ -0,0 +1,110 @@
+package auth
+
+import (
+	"errors"
+	"net/http"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+	"github.com/go-webauthn/webauthn/protocol"
+	"github.com/go-webauthn/webauthn/webauthn"
+)
+
+// HandlePasskeyLoginBegin starts a passwordless sign-in.
+//
+// It repeats every gate /auth/login applies - instance resolution, the locked
+// instance refusal, and the local-login setting - because this is a second
+// front door, and a front door that skips the locks is not a shortcut.
+//
+//	@Summary	Begin passwordless passkey sign-in
+//	@Tags		auth
+//	@Produce	json
+//	@Success	200	{object}	object{publicKey=object,ceremony_id=string}
+//	@Failure	403	{object}	object{error=string}
+//	@Router		/auth/passkey/begin [post]
+func HandlePasskeyLoginBegin(c *gin.Context) {
+	instanceID, err := resolveLoginInstance(c)
+	if errors.Is(err, ErrInstanceLocked) {
+		c.JSON(http.StatusForbidden, gin.H{"error": err.Error(), "locked": true})
+		return
+	}
+	if err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	if !services.LocalLoginPermitted(instanceID) {
+		c.JSON(http.StatusForbidden, gin.H{"error": "password sign-in is disabled for this instance"})
+		return
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start sign-in"})
+		return
+	}
+	// Discoverable login: no allowCredentials, so the authenticator offers
+	// whichever resident credential it holds for this RP ID.
+	options, sessionData, err := w.BeginDiscoverableLogin(
+		webauthn.WithUserVerification(protocol.VerificationPreferred))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start sign-in"})
+		return
+	}
+	id, err := saveCeremony(c.Request.Context(), sessionData)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start sign-in"})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"publicKey": options.Response, "ceremony_id": id})
+}
+
+// HandlePasskeyLoginFinish verifies a discoverable assertion and mints a
+// session. A user-verified passkey is possession plus a PIN or biometric, so it
+// satisfies require_mfa on its own.
+//
+//	@Summary	Complete passwordless passkey sign-in
+//	@Tags		auth
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{ceremony_id=string,credential=object}	true	"Assertion"
+//	@Success	200		{object}	object{ok=bool}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/auth/passkey/finish [post]
+func HandlePasskeyLoginFinish(c *gin.Context) {
+	instanceID, err := resolveLoginInstance(c)
+	if err != nil {
+		c.JSON(http.StatusForbidden, gin.H{"error": "sign-in is not available here"})
+		return
+	}
+	if !services.LocalLoginPermitted(instanceID) {
+		c.JSON(http.StatusForbidden, gin.H{"error": "password sign-in is disabled for this instance"})
+		return
+	}
+	// finishAssertion is given no userID or email, so it resolves the owning
+	// user from the credential ID scoped to this instance and validates with
+	// the library's discoverable path, refusing a user handle that is not
+	// that owner's.
+	cred, err := finishAssertion(c, instanceID, "", "")
+	if err != nil {
+		logWebAuthnFailure(c, "passwordless assertion", err)
+		c.JSON(http.StatusUnauthorized, gin.H{
+			"error": "that passkey could not be verified", "code": "invalid_assertion",
+		})
+		return
+	}
+	stored, err := services.GetPasskeyByCredentialID(instanceID, cred.ID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid credentials"})
+		return
+	}
+	u, err := services.GetUserInInstance(instanceID, stored.UserID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid credentials"})
+		return
+	}
+	if err := mintSession(c, u, []string{services.FactorWebAuthn}); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+		return
+	}
+	_ = services.TouchPasskey(instanceID, cred.ID, cred.Authenticator.SignCount, cred.Flags.BackupEligible, cred.Flags.BackupState)
+	c.JSON(http.StatusOK, gin.H{"ok": true})
+}

server/internal/auth/session.go

@@ -31,6 +31,14 @@ type Session struct {
 	TokenName  string            `json:"-"`
 	Scopes     []string          `json:"-"`
 	TokenScope map[string]string `json:"-"`
+
+	// AMR records how this session authenticated: pwd, otp, webauthn,
+	// recovery, oidc. Step-up reads it to exempt OIDC sessions, whose IdP owns
+	// authentication policy.
+	AMR []string `json:"amr,omitempty"`
+
+	// StepUpAt is the last successful re-authentication. Sign-in counts as one.
+	StepUpAt *time.Time `json:"step_up_at,omitempty"`
 }
 
 var rdb *redis.Client
@@ -103,3 +111,15 @@ func GetSession(ctx context.Context, id string) (*Session, error) {
 func DeleteSession(ctx context.Context, id string) error {
 	return rdb.Del(ctx, sessionPrefix+id).Err()
 }
+
+// TouchStepUp records a fresh re-authentication without disturbing the
+// session's remaining lifetime.
+func TouchStepUp(ctx context.Context, id string, sess *Session) error {
+	now := time.Now()
+	sess.StepUpAt = &now
+	data, err := json.Marshal(sess)
+	if err != nil {
+		return err
+	}
+	return rdb.Set(ctx, sessionPrefix+id, data, redis.KeepTTL).Err()
+}

server/internal/auth/stepup.go

@@ -0,0 +1,70 @@
+package auth
+
+import (
+	"net/http"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+)
+
+// StepUpWindow is how long one re-authentication covers. Ten minutes is long
+// enough to open several consoles in a row and short enough that a walked-away
+// laptop is not a fleet-wide credential.
+const StepUpWindow = 10 * time.Minute
+
+func stepUpFresh(sess *Session, now time.Time) bool {
+	if sess == nil {
+		return false
+	}
+	// An API token authenticates per request and has no human to prompt.
+	if sess.TokenID != "" {
+		return true
+	}
+	for _, a := range sess.AMR {
+		if a == "oidc" {
+			return true
+		}
+	}
+	return sess.StepUpAt != nil && now.Sub(*sess.StepUpAt) < StepUpWindow
+}
+
+// RequireStepUp guards the actions that hand out credentials rather than
+// describe them: secret reveal, private key download, console connect.
+//
+// It answers a machine-readable code rather than a bare 403 so web/ can open
+// the re-authentication modal and retry the original request.
+func RequireStepUp() gin.HandlerFunc {
+	return func(c *gin.Context) {
+		sess := GetSessionFromContext(c)
+		if stepUpFresh(sess, time.Now()) {
+			c.Next()
+			return
+		}
+		if sess == nil {
+			c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "not authenticated"})
+			return
+		}
+		methods, err := services.MFAMethods(sess.InstanceID, sess.UserID)
+		if err != nil || len(methods) == 0 {
+			methods = []string{services.FactorPassword}
+		}
+		c.AbortWithStatusJSON(http.StatusForbidden, gin.H{
+			"error":   "re-authentication required",
+			"code":    "step_up_required",
+			"methods": methods,
+		})
+	}
+}
+
+// TouchStepUpFromRequest records a fresh step-up against the session the
+// request arrived on. It lives in auth because the cookie name and the Redis
+// key are this package's business.
+func TouchStepUpFromRequest(c *gin.Context) error {
+	cookie, err := c.Request.Cookie(sessionCookieName)
+	if err != nil {
+		return err
+	}
+	sess := GetSessionFromContext(c)
+	return TouchStepUp(c.Request.Context(), cookie.Value, sess)
+}

server/internal/auth/stepup_test.go

@@ -0,0 +1,76 @@
+package auth
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+
+	"github.com/gin-gonic/gin"
+)
+
+func TestStepUpFresh(t *testing.T) {
+	now := time.Now()
+	ago := func(d time.Duration) *time.Time { v := now.Add(-d); return &v }
+
+	cases := []struct {
+		name string
+		sess *Session
+		want bool
+	}{
+		{"just signed in", &Session{StepUpAt: ago(time.Minute)}, true},
+		{"nine minutes ago", &Session{StepUpAt: ago(9 * time.Minute)}, true},
+		{"eleven minutes ago", &Session{StepUpAt: ago(11 * time.Minute)}, false},
+		{"never", &Session{}, false},
+		// An API token has no human to prompt; the spec exempts it and records
+		// the bypass as a known limitation.
+		{"api token", &Session{TokenID: "tok_1"}, true},
+		// An OIDC session's IdP owns authentication policy.
+		{"oidc session", &Session{AMR: []string{"oidc"}, StepUpAt: ago(time.Hour)}, true},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := stepUpFresh(tc.sess, now); got != tc.want {
+				t.Fatalf("stepUpFresh = %v, want %v", got, tc.want)
+			}
+		})
+	}
+}
+
+// A missing session must refuse rather than panic dereferencing sess.
+// auth.Middleware guarantees a non-nil session on every route today, but
+// RequireStepUp must not rely on that holding forever.
+func TestRequireStepUpNilSessionRefusesWithoutPanic(t *testing.T) {
+	gin.SetMode(gin.TestMode)
+	w := httptest.NewRecorder()
+	c, _ := gin.CreateTestContext(w)
+	c.Request = httptest.NewRequest(http.MethodPost, "/console/connect", nil)
+
+	RequireStepUp()(c)
+
+	if w.Code != http.StatusUnauthorized {
+		t.Fatalf("status = %d, want %d", w.Code, http.StatusUnauthorized)
+	}
+}
+
+func TestOIDCSessionIsExemptFromStepUp(t *testing.T) {
+	now := time.Now()
+	u := &models.User{UserID: "u1", InstanceID: "i1", Role: "member", Email: "a@example.com"}
+	s := oidcSession(u, "Ann", now.Add(-24*time.Hour))
+	if !stepUpFresh(s, now) {
+		t.Fatal("an OIDC session must be exempt from step-up however old its sign-in")
+	}
+	if s.StepUpAt == nil || s.Name != "Ann" || s.UserID != "u1" || s.InstanceID != "i1" {
+		t.Fatalf("oidc session missing fields: %+v", s)
+	}
+}
+
+func TestSignInSessionIsFreshAtSignIn(t *testing.T) {
+	now := time.Now()
+	s := newSession(&models.User{UserID: "u1"}, []string{"pwd"}, now)
+	if !stepUpFresh(s, now) {
+		t.Fatal("a session is fresh at the moment of sign-in")
+	}
+}

server/internal/auth/webauthn.go

@@ -0,0 +1,507 @@
+package auth
+
+import (
+	"bytes"
+	"context"
+	"crypto/subtle"
+	"encoding/json"
+	"errors"
+	"log"
+	"net"
+	"net/http"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+	"github.com/gin-gonic/gin"
+	"github.com/go-webauthn/webauthn/protocol"
+	"github.com/go-webauthn/webauthn/webauthn"
+	"github.com/redis/go-redis/v9"
+)
+
+const (
+	ceremonyPrefix = "km:wa:"
+	ceremonyTTL    = 5 * time.Minute
+)
+
+// rpConfig derives the relying party from the request. The RP ID is the host
+// without its port - WebAuthn forbids a port there - while the origin keeps it.
+//
+// This is why the reverse proxy must preserve Host: a proxy rewriting it makes
+// every passkey on the instance fail to verify, with no error that says so.
+func rpConfig(c *gin.Context) (string, string) {
+	host := c.Request.Host
+	rpID := host
+	if h, _, err := net.SplitHostPort(host); err == nil {
+		rpID = h
+	}
+	scheme := "https"
+	if c.Request.TLS == nil && c.GetHeader("X-Forwarded-Proto") != "https" {
+		// Only development is ever plain HTTP; WebAuthn permits it on localhost.
+		scheme = "http"
+	}
+	return rpID, scheme + "://" + host
+}
+
+// logWebAuthnFailure records why a ceremony was refused. The response stays
+// deliberately vague, so without this the only evidence of a misconfigured
+// relying party - most often a proxy that terminates TLS without passing
+// X-Forwarded-Proto: https - is a user reporting that their passkey "could not
+// be verified". Nothing logged here is secret: RP ID, origins, and the
+// library's error, whose DevInfo names expected and received values.
+func logWebAuthnFailure(c *gin.Context, stage string, err error) {
+	rpID, origin := rpConfig(c)
+	detail := err.Error()
+	var perr *protocol.Error
+	if errors.As(err, &perr) && perr.DevInfo != "" {
+		detail += " (" + perr.DevInfo + ")"
+	}
+	log.Printf("webauthn: %s refused: rp_id=%q expected_origin=%q request_origin=%q x_forwarded_proto=%q: %s",
+		stage, rpID, origin, c.GetHeader("Origin"), c.GetHeader("X-Forwarded-Proto"), detail)
+}
+
+func webAuthnFor(c *gin.Context) (*webauthn.WebAuthn, error) {
+	rpID, origin := rpConfig(c)
+	return webauthn.New(&webauthn.Config{
+		RPDisplayName: "Vantage",
+		RPID:          rpID,
+		RPOrigins:     []string{origin},
+		AuthenticatorSelection: protocol.AuthenticatorSelection{
+			ResidentKey: protocol.ResidentKeyRequirementRequired,
+			// Preferred, not required, and not checked on the result. Requiring
+			// it made password managers such as NordPass ask for their master
+			// password on every use, which other sites avoid by preferring it.
+			// The cost is deliberate: an authenticator that skips verification
+			// makes a passkey possession-only, so passwordless sign-in and
+			// step-up then rest on the device or vault being unlocked.
+			UserVerification: protocol.VerificationPreferred,
+		},
+	})
+}
+
+// waUser adapts our records to the library's interface. The handle is random
+// and per-user: a resident credential hands its user handle to any origin that
+// asks, so the user ID must not be it.
+type waUser struct {
+	handle      []byte
+	name        string
+	credentials []webauthn.Credential
+}
+
+func (u waUser) WebAuthnID() []byte                         { return u.handle }
+func (u waUser) WebAuthnName() string                       { return u.name }
+func (u waUser) WebAuthnDisplayName() string                { return u.name }
+func (u waUser) WebAuthnCredentials() []webauthn.Credential { return u.credentials }
+
+func toLibCredential(c models.WebAuthnCredential) webauthn.Credential {
+	return webauthn.Credential{
+		ID:              c.CredentialID,
+		PublicKey:       c.PublicKey,
+		AttestationType: "none",
+		Flags: webauthn.CredentialFlags{
+			BackupEligible: c.BackupEligible != nil && *c.BackupEligible,
+			BackupState:    c.BackupState,
+		},
+		Authenticator: webauthn.Authenticator{
+			AAGUID:    c.AAGUID,
+			SignCount: c.SignCount,
+		},
+	}
+}
+
+// backupEligibleFor is the eligibility to validate an assertion against.
+//
+// A recorded value always stands, so the library still refuses a credential
+// whose eligibility changed. A row registered before the flag was stored has no
+// baseline at all, and comparing against a zero value refused every synced
+// passkey; for those the flag in the signed authenticator data is adopted, and
+// TouchPasskey records it after the assertion verifies.
+func backupEligibleFor(stored *bool, observed bool) bool {
+	if stored != nil {
+		return *stored
+	}
+	return observed
+}
+
+func saveCeremony(ctx context.Context, data *webauthn.SessionData) (string, error) {
+	id, err := randomHex(32)
+	if err != nil {
+		return "", err
+	}
+	blob, err := json.Marshal(data)
+	if err != nil {
+		return "", err
+	}
+	if err := rdb.Set(ctx, ceremonyPrefix+id, blob, ceremonyTTL).Err(); err != nil {
+		return "", err
+	}
+	return id, nil
+}
+
+// loadCeremony consumes the challenge: a WebAuthn challenge is single use, so
+// it is deleted as it is read.
+func loadCeremony(ctx context.Context, id string) (*webauthn.SessionData, error) {
+	blob, err := rdb.GetDel(ctx, ceremonyPrefix+id).Bytes()
+	if errors.Is(err, redis.Nil) {
+		return nil, ErrTicketExpired
+	}
+	if err != nil {
+		return nil, err
+	}
+	var data webauthn.SessionData
+	if err := json.Unmarshal(blob, &data); err != nil {
+		return nil, ErrTicketExpired
+	}
+	return &data, nil
+}
+
+// HandleMFAWebAuthnBegin offers an assertion challenge to a pending sign-in.
+//
+//	@Summary	Begin passkey verification during sign-in
+//	@Tags		auth
+//	@Produce	json
+//	@Success	200	{object}	object{publicKey=object,ceremony_id=string}
+//	@Failure	401	{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/webauthn/begin [post]
+func HandleMFAWebAuthnBegin(c *gin.Context) {
+	t, _, ok := ticketFromRequest(c, scopeVerify)
+	if !ok {
+		return
+	}
+	creds, err := services.ListPasskeys(t.InstanceID, t.UserID)
+	if err != nil || len(creds) == 0 {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "no passkey is registered"})
+		return
+	}
+	handle, err := services.WebAuthnHandle(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	lib := make([]webauthn.Credential, 0, len(creds))
+	for _, cr := range creds {
+		lib = append(lib, toLibCredential(cr))
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	options, sessionData, err := w.BeginLogin(waUser{handle: handle, name: t.Email, credentials: lib},
+		webauthn.WithUserVerification(protocol.VerificationPreferred))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	id, err := saveCeremony(c.Request.Context(), sessionData)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"publicKey": options.Response, "ceremony_id": id})
+}
+
+// HandleMFAWebAuthnFinish verifies the assertion and signs the user in.
+//
+//	@Summary	Complete sign-in with a passkey
+//	@Tags		auth
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{ceremony_id=string,credential=object}	true	"Assertion"
+//	@Success	200		{object}	object{ok=bool}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/auth/mfa/webauthn/finish [post]
+func HandleMFAWebAuthnFinish(c *gin.Context) {
+	t, ticketID, ok := ticketFromRequest(c, scopeVerify)
+	if !ok {
+		return
+	}
+	cred, err := finishAssertion(c, t.InstanceID, t.UserID, t.Email)
+	if err != nil {
+		logWebAuthnFailure(c, "second-factor assertion", err)
+		left, ferr := FailTicket(c.Request.Context(), ticketID)
+		services.LogEvent(t.InstanceID, "mfa.failed", t.Email, "", "", "factor=webauthn")
+		if ferr != nil || left == 0 {
+			abortTicketExpired(c)
+			return
+		}
+		c.JSON(http.StatusUnauthorized, gin.H{
+			"error": "that passkey could not be verified", "code": "invalid_assertion", "attempts_left": left,
+		})
+		return
+	}
+	_ = services.TouchPasskey(t.InstanceID, cred.ID, cred.Authenticator.SignCount, cred.Flags.BackupEligible, cred.Flags.BackupState)
+	u, err := services.GetUserInInstance(t.InstanceID, t.UserID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid credentials"})
+		return
+	}
+	if err := mintSession(c, u, []string{"pwd", services.FactorWebAuthn}); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "session save failed"})
+		return
+	}
+	_ = DeleteTicket(c.Request.Context(), ticketID)
+	c.JSON(http.StatusOK, gin.H{"ok": true})
+}
+
+// HandleRegisterPasskeyBegin starts registration for the signed-in user.
+//
+//	@Summary	Begin passkey registration
+//	@Tags		mfa
+//	@Produce	json
+//	@Success	200	{object}	object{publicKey=object,ceremony_id=string}
+//	@Router		/me/passkeys/begin [post]
+func HandleRegisterPasskeyBegin(c *gin.Context) {
+	sess := GetSessionFromContext(c)
+	handle, err := services.WebAuthnHandle(sess.InstanceID, sess.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start registration"})
+		return
+	}
+	existing, err := services.ListPasskeys(sess.InstanceID, sess.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start registration"})
+		return
+	}
+	lib := make([]webauthn.Credential, 0, len(existing))
+	for _, cr := range existing {
+		lib = append(lib, toLibCredential(cr))
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start registration"})
+		return
+	}
+	options, sessionData, err := w.BeginRegistration(
+		waUser{handle: handle, name: sess.Email, credentials: lib},
+		webauthn.WithExclusions(webauthn.Credentials(lib).CredentialDescriptors()),
+	)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start registration"})
+		return
+	}
+	id, err := saveCeremony(c.Request.Context(), sessionData)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start registration"})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"publicKey": options.Response, "ceremony_id": id})
+}
+
+// HandleRegisterPasskeyFinish stores the new credential.
+//
+//	@Summary	Complete passkey registration
+//	@Tags		mfa
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{ceremony_id=string,name=string,credential=object}	true	"Attestation"
+//	@Success	200		{object}	object{ok=bool,recovery_codes=[]string}
+//	@Router		/me/passkeys/finish [post]
+func HandleRegisterPasskeyFinish(c *gin.Context) {
+	sess := GetSessionFromContext(c)
+	var body struct {
+		CeremonyID string          `json:"ceremony_id"`
+		Name       string          `json:"name"`
+		Credential json.RawMessage `json:"credential"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.CeremonyID == "" {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "attestation required"})
+		return
+	}
+	sessionData, err := loadCeremony(c.Request.Context(), body.CeremonyID)
+	if err != nil {
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "that registration expired", "code": "mfa_ticket_expired"})
+		return
+	}
+	parsed, err := protocol.ParseCredentialCreationResponseBody(bytes.NewReader(body.Credential))
+	if err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "that passkey could not be read"})
+		return
+	}
+	handle, err := services.WebAuthnHandle(sess.InstanceID, sess.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not finish registration"})
+		return
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not finish registration"})
+		return
+	}
+	cred, err := w.CreateCredential(waUser{handle: handle, name: sess.Email}, *sessionData, parsed)
+	if err != nil {
+		logWebAuthnFailure(c, "registration", err)
+		c.JSON(http.StatusBadRequest, gin.H{"error": "that passkey could not be verified"})
+		return
+	}
+	transports := make([]string, 0, len(parsed.Response.Transports))
+	for _, t := range parsed.Response.Transports {
+		transports = append(transports, string(t))
+	}
+	if err := services.SavePasskey(sess.InstanceID, sess.UserID, body.Name,
+		cred.ID, cred.PublicKey, cred.Authenticator.AAGUID, cred.Authenticator.SignCount,
+		transports, cred.Flags.BackupEligible, cred.Flags.BackupState); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not save the passkey"})
+		return
+	}
+	services.LogEvent(sess.InstanceID, "mfa.enrolled", sess.Email, "", "", "factor=webauthn")
+
+	// A first factor earns recovery codes; later ones do not reissue them.
+	m, _ := services.GetUserMFA(sess.InstanceID, sess.UserID)
+	if services.RecoveryCodesRemaining(m) == 0 {
+		codes, err := services.IssueRecoveryCodes(sess.InstanceID, sess.UserID)
+		if err == nil {
+			c.JSON(http.StatusOK, gin.H{"ok": true, "recovery_codes": codes})
+			return
+		}
+	}
+	c.JSON(http.StatusOK, gin.H{"ok": true})
+}
+
+// HandleStepUpWebAuthnBegin offers an assertion challenge for step-up
+// re-authentication: the signed-in user proving it is still them before a
+// guarded action, rather than a pending ticket proving it during sign-in.
+//
+//	@Summary	Begin passkey step-up
+//	@Tags		mfa
+//	@Produce	json
+//	@Success	200	{object}	object{publicKey=object,ceremony_id=string}
+//	@Failure	400	{object}	object{error=string}
+//	@Router		/me/step-up/webauthn/begin [post]
+func HandleStepUpWebAuthnBegin(c *gin.Context) {
+	sess := GetSessionFromContext(c)
+	creds, err := services.ListPasskeys(sess.InstanceID, sess.UserID)
+	if err != nil || len(creds) == 0 {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "no passkey is registered"})
+		return
+	}
+	handle, err := services.WebAuthnHandle(sess.InstanceID, sess.UserID)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	lib := make([]webauthn.Credential, 0, len(creds))
+	for _, cr := range creds {
+		lib = append(lib, toLibCredential(cr))
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	options, sessionData, err := w.BeginLogin(waUser{handle: handle, name: sess.Email, credentials: lib},
+		webauthn.WithUserVerification(protocol.VerificationPreferred))
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	id, err := saveCeremony(c.Request.Context(), sessionData)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not start verification"})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"publicKey": options.Response, "ceremony_id": id})
+}
+
+// HandleStepUpWebAuthnFinish verifies the assertion and records a fresh
+// step-up for the current session. Unlike HandleMFAWebAuthnFinish this does
+// not mint a session: the caller is already signed in, this only proves they
+// still hold the passkey. finishAssertion is called with the session's own
+// user ID, so a credential belonging to somebody else is refused rather than
+// stepping up this session on their behalf.
+//
+//	@Summary	Complete passkey step-up
+//	@Tags		mfa
+//	@Accept		json
+//	@Produce	json
+//	@Param		body	body		object{ceremony_id=string,credential=object}	true	"Assertion"
+//	@Success	200		{object}	object{ok=bool}
+//	@Failure	401		{object}	object{error=string,code=string}
+//	@Router		/me/step-up/webauthn/finish [post]
+func HandleStepUpWebAuthnFinish(c *gin.Context) {
+	sess := GetSessionFromContext(c)
+	cred, err := finishAssertion(c, sess.InstanceID, sess.UserID, sess.Email)
+	if err != nil {
+		logWebAuthnFailure(c, "step-up assertion", err)
+		services.LogEvent(sess.InstanceID, "step_up.failed", sess.Email, "", "", "factor=webauthn")
+		c.JSON(http.StatusUnauthorized, gin.H{"error": "that passkey could not be verified", "code": "invalid_assertion"})
+		return
+	}
+	_ = services.TouchPasskey(sess.InstanceID, cred.ID, cred.Authenticator.SignCount, cred.Flags.BackupEligible, cred.Flags.BackupState)
+	if err := TouchStepUpFromRequest(c); err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "could not record re-authentication"})
+		return
+	}
+	services.LogEvent(sess.InstanceID, "step_up.ok", sess.Email, "", "", "factor=webauthn")
+	c.JSON(http.StatusOK, gin.H{"ok": true})
+}
+
+// finishAssertion is shared by second-factor sign-in, passwordless sign-in and
+// step-up, so the verification rules (backup eligibility, clone detection,
+// instance scope) exist once.
+func finishAssertion(c *gin.Context, instanceID, userID, email string) (*webauthn.Credential, error) {
+	var body struct {
+		CeremonyID string          `json:"ceremony_id"`
+		Credential json.RawMessage `json:"credential"`
+	}
+	if err := c.ShouldBindJSON(&body); err != nil || body.CeremonyID == "" {
+		return nil, errors.New("assertion required")
+	}
+	sessionData, err := loadCeremony(c.Request.Context(), body.CeremonyID)
+	if err != nil {
+		return nil, err
+	}
+	parsed, err := protocol.ParseCredentialRequestResponseBody(bytes.NewReader(body.Credential))
+	if err != nil {
+		return nil, err
+	}
+	stored, err := services.GetPasskeyByCredentialID(instanceID, parsed.RawID)
+	if err != nil {
+		return nil, err
+	}
+	if userID != "" && stored.UserID != userID {
+		return nil, errors.New("credential belongs to another user")
+	}
+	handle, err := services.WebAuthnHandle(instanceID, stored.UserID)
+	if err != nil {
+		return nil, err
+	}
+	w, err := webAuthnFor(c)
+	if err != nil {
+		return nil, err
+	}
+	libCred := toLibCredential(*stored)
+	libCred.Flags.BackupEligible = backupEligibleFor(stored.BackupEligible,
+		parsed.Response.AuthenticatorData.Flags.HasBackupEligible())
+	user := waUser{handle: handle, name: email, credentials: []webauthn.Credential{libCred}}
+	var cred *webauthn.Credential
+	if userID == "" {
+		// Discoverable (passwordless) ceremony: the session carries no user,
+		// so the library asks us to resolve the authenticator's user handle.
+		// The only acceptable answer is the credential's own stored owner in
+		// this instance; any other handle is refused.
+		cred, err = w.ValidateDiscoverableLogin(func(_, userHandle []byte) (webauthn.User, error) {
+			if !discoverableHandleMatches(handle, userHandle) {
+				return nil, errors.New("user handle does not match the credential owner")
+			}
+			return user, nil
+		}, *sessionData, parsed)
+	} else {
+		cred, err = w.ValidateLogin(user, *sessionData, parsed)
+	}
+	if err != nil {
+		return nil, err
+	}
+	// A counter that fails to advance is the library's clone signal. Zero on
+	// both sides means the authenticator does not keep one, which is normal.
+	if cred.Authenticator.CloneWarning {
+		return nil, errors.New("authenticator may be cloned")
+	}
+	return cred, nil
+}
+
+// discoverableHandleMatches reports whether the user handle an authenticator
+// returned belongs to the credential's stored owner. Empty never matches.
+func discoverableHandleMatches(ownerHandle, userHandle []byte) bool {
+	return len(ownerHandle) > 0 && subtle.ConstantTimeCompare(ownerHandle, userHandle) == 1
+}

server/internal/auth/webauthn_test.go

@@ -0,0 +1,90 @@
+package auth
+
+import (
+	"bytes"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/gin-gonic/gin"
+)
+
+// A passkey is bound to its RP ID. Getting this wrong does not fail loudly - it
+// silently makes every existing passkey unusable - so the port-stripping and
+// scheme rules are pinned here.
+func TestRPConfig(t *testing.T) {
+	cases := []struct {
+		name, host, proto string
+		wantID, wantOrig  string
+	}{
+		{"plain host", "acme.vantage.example.com", "https", "acme.vantage.example.com", "https://acme.vantage.example.com"},
+		{"host with port", "vantage.acme.com:8443", "https", "vantage.acme.com", "https://vantage.acme.com:8443"},
+		{"localhost dev", "localhost:3000", "", "localhost", "http://localhost:3000"},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			c, _ := gin.CreateTestContext(httptest.NewRecorder())
+			c.Request = httptest.NewRequest("POST", "/auth/passkey/begin", nil)
+			c.Request.Host = tc.host
+			if tc.proto != "" {
+				c.Request.Header.Set("X-Forwarded-Proto", tc.proto)
+			}
+			id, origin := rpConfig(c)
+			if id != tc.wantID || origin != tc.wantOrig {
+				t.Fatalf("rpConfig = (%q, %q), want (%q, %q)", id, origin, tc.wantID, tc.wantOrig)
+			}
+		})
+	}
+}
+
+func TestDiscoverableHandleMatches(t *testing.T) {
+	owner := bytes.Repeat([]byte{7}, 64)
+	if !discoverableHandleMatches(owner, bytes.Repeat([]byte{7}, 64)) {
+		t.Fatal("the owner's own handle must match")
+	}
+	other := bytes.Repeat([]byte{7}, 64)
+	other[63] = 8
+	if discoverableHandleMatches(owner, other) {
+		t.Fatal("another user's handle must not match")
+	}
+	if discoverableHandleMatches(nil, nil) || discoverableHandleMatches(owner, nil) {
+		t.Fatal("an empty handle must never match")
+	}
+}
+
+// Synced passkeys (iCloud Keychain, Google Password Manager, 1Password) report
+// backup eligibility, and go-webauthn refuses any assertion whose flag differs
+// from the stored credential's. Not storing it made every synced passkey fail
+// with "Backup Eligible flag inconsistency".
+func TestLibCredentialCarriesStoredBackupFlags(t *testing.T) {
+	yes := true
+	lc := toLibCredential(models.WebAuthnCredential{BackupEligible: &yes, BackupState: true})
+	if !lc.Flags.BackupEligible || !lc.Flags.BackupState {
+		t.Fatalf("flags not carried: %+v", lc.Flags)
+	}
+}
+
+func TestBackupEligibleFor(t *testing.T) {
+	yes, no := true, false
+	cases := []struct {
+		name     string
+		stored   *bool
+		observed bool
+		want     bool
+	}{
+		// Registered before the flag was recorded: no baseline, adopt it.
+		{"legacy row, synced passkey", nil, true, true},
+		{"legacy row, hardware key", nil, false, false},
+		// Recorded: the stored value stands, so the library still refuses a
+		// credential whose eligibility genuinely changed.
+		{"recorded eligible", &yes, true, true},
+		{"recorded not eligible, now claims eligible", &no, true, false},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := backupEligibleFor(tc.stored, tc.observed); got != tc.want {
+				t.Fatalf("backupEligibleFor = %v, want %v", got, tc.want)
+			}
+		})
+	}
+}

server/internal/grpc/server.go

@@ -103,6 +103,7 @@ func (s *vantageServer) ReportUpdates(ctx context.Context, req *pb.ReportUpdates
 			Name:           u.Name,
 			CurrentVersion: u.CurrentVersion,
 			NewVersion:     u.NewVersion,
+			Phased:         u.Phased,
 		}
 	}
 	if err := services.StoreAvailableUpdates(srv.ServerID, pkgs); err != nil {
@@ -236,6 +237,12 @@ func (s *vantageServer) ReportInventory(ctx context.Context, req *pb.InventoryRe
 	if err := services.StoreInventory(srv.ServerID, req); err != nil {
 		log.Printf("store inventory for %s: %v", srv.ServerID, err)
 	}
+	// Only static snapshots compute reboot_required, so only they can settle a
+	// reboot. The agent sends one at start, so the first report after a
+	// reboot qualifies.
+	if req.IncludeStatic && req.BootTimeUnix > 0 {
+		services.VerifyPatchReboots(srv.InstanceID, srv.ServerID, time.Unix(req.BootTimeUnix, 0), req.RebootRequired)
+	}
 	return &pb.InventoryReportResponse{}, nil
 }
 
@@ -330,6 +337,9 @@ func (s *vantageServer) CommandStream(stream pb.Vantage_CommandStreamServer) err
 			if m.WorkloadLogsResult != nil {
 				services.WorkloadResults.Deliver(m.WorkloadLogsResult)
 			}
+			if m.PatchResult != nil {
+				services.RecordPatchResult(srv.InstanceID, srv.ServerID, m.PatchResult)
+			}
 			if m.StepResult != nil {
 				services.StepResults.Deliver(m.StepResult)
 			}

server/internal/mcp/tools_create.go

@@ -118,6 +118,25 @@ func buildMonitor(args map[string]any) (models.Monitor, error) {
 	if b, ok := rawTarget["insecure"].(bool); ok {
 		target.Insecure = b
 	}
+	target.Metric = stringArg(rawTarget, "metric")
+	target.Mount = stringArg(rawTarget, "mount")
+	if n, ok := rawTarget["threshold"].(float64); ok {
+		target.Threshold = n
+	}
+	if n, ok := rawTarget["period_sec"].(float64); ok {
+		target.PeriodSec = int(n)
+	}
+	if n, ok := rawTarget["grace_sec"].(float64); ok {
+		target.GraceSec = int(n)
+	}
+	if sel, ok := rawTarget["selector"].(map[string]any); ok {
+		target.Selector = map[string]string{}
+		for k, v := range sel {
+			if s, ok := v.(string); ok {
+				target.Selector[k] = s
+			}
+		}
+	}
 
 	switch monitorType {
 	case models.MonitorHTTP, models.MonitorTLS:
@@ -131,6 +150,9 @@ func buildMonitor(args map[string]any) (models.Monitor, error) {
 		if monitorType == models.MonitorTCP && target.Port == 0 {
 			return models.Monitor{}, fmt.Errorf("target.port is required for a tcp monitor")
 		}
+	case models.MonitorHeartbeat, models.MonitorMetric:
+		// Full validation (validateHeartbeat / validateMetric) runs inside
+		// services.CreateMonitor; nothing further is required here.
 	default:
 		return models.Monitor{}, fmt.Errorf("unknown monitor type %q", monitorType)
 	}
@@ -150,6 +172,10 @@ func buildMonitor(args map[string]any) (models.Monitor, error) {
 	if n, ok := args["interval_sec"].(float64); ok && int(n) > 0 {
 		interval = int(n)
 	}
+	forSec := 0
+	if n, ok := args["for_sec"].(float64); ok && int(n) > 0 {
+		forSec = int(n)
+	}
 
 	return models.Monitor{
 		Name:        name,
@@ -157,6 +183,7 @@ func buildMonitor(args map[string]any) (models.Monitor, error) {
 		Type:        monitorType,
 		Target:      target,
 		IntervalSec: interval,
+		ForSec:      forSec,
 		// Never armed on creation. A monitor that started enabled would begin
 		// alerting real people the moment a model invented it, and creating
 		// must stay a separate decision from acting.
@@ -257,10 +284,11 @@ func init() {
 		Name: "create_monitor",
 		Args: []ToolArg{
 			{Name: "name", Type: ArgString, Description: "Name for the monitor.", Required: true},
-			{Name: "type", Type: ArgString, Description: "Check type: http, tcp, icmp or tls.", Required: true},
-			{Name: "target", Type: ArgObject, Description: "What to check. http/tls take url; tcp/icmp take host, and tcp also port. Optional: method, keyword, expected_status, tls_warn_days, insecure.", Required: true},
+			{Name: "type", Type: ArgString, Description: "Check type: http, tcp, icmp, tls, heartbeat or metric.", Required: true},
+			{Name: "target", Type: ArgObject, Description: "What to check. http/tls take url; tcp/icmp take host, and tcp also port. Optional: method, keyword, expected_status, tls_warn_days, insecure. heartbeat takes period_sec and optional grace_sec; metric takes metric (disk_pct, disk_free_gb, mem_pct, load_per_core, unit_failed, container_unhealthy, reboot_pending_days, agent_offline_min), threshold, optional mount and selector (tag map).", Required: true},
 			{Name: "group", Type: ArgString, Description: "Optional group name to file the monitor under."},
 			{Name: "interval_sec", Type: ArgInteger, Description: "Seconds between checks; defaults to 60."},
+			{Name: "for_sec", Type: ArgInteger, Description: "For a metric monitor, how long the condition must hold before a server counts as down."},
 		},
 		Write: true,
 		Scope: "monitors:write",
@@ -277,12 +305,17 @@ func init() {
 				return nil, fmt.Errorf("could not create the monitor: %w", err)
 			}
 			LogCreated(c, "monitor", created.MonitorID, created.Name)
-			return map[string]any{
+			out := map[string]any{
 				"monitor_id": created.MonitorID,
 				"name":       created.Name,
 				"enabled":    false,
 				"note":       "Created disabled. Enable it in Vantage to start checking.",
-			}, nil
+			}
+			if created.HeartbeatToken != "" {
+				out["heartbeat_token"] = created.HeartbeatToken
+				out["heartbeat_token_note"] = "Shown once."
+			}
+			return out, nil
 		},
 	})
 }

server/internal/mcp/tools_health.go

@@ -187,6 +187,12 @@ func init() {
 				}
 			}
 
+			// Metric incidents name a server, so a server-scoped token sees only
+			// the ones on servers it can see, matching the REST endpoint.
+			visible, restricted, err := services.VisibleServerIDs(c.InstanceID, c.TokenScope)
+			if err != nil {
+				return nil, fmt.Errorf("could not resolve visible servers: %w", err)
+			}
 			out := []incidentSummary{}
 			for _, mid := range monitorIDs {
 				if len(out) >= int(limit) {
@@ -196,6 +202,7 @@ func init() {
 				if err != nil {
 					return nil, fmt.Errorf("could not list incidents: %w", err)
 				}
+				incidents = services.FilterByVisibleServer(incidents, func(i models.Incident) string { return i.ServerID }, visible, restricted)
 				name := mid
 				if monitorNames != nil {
 					if n, ok := monitorNames[mid]; ok {

server/internal/mcp/tools_work.go

@@ -80,6 +80,8 @@ type pendingUpdate struct {
 	Package        string `json:"package"`
 	CurrentVersion string `json:"current_version,omitempty"`
 	NewVersion     string `json:"new_version"`
+	// Phased is an Ubuntu phased update apt defers until the host is selected.
+	Phased bool `json:"phased,omitempty"`
 }
 
 type listPendingUpdatesResult struct {
@@ -390,7 +392,7 @@ func init() {
 					}
 					out = append(out, pendingUpdate{
 						ServerID: srv.ServerID, Hostname: srv.Hostname,
-						Package: u.Name, CurrentVersion: u.CurrentVersion, NewVersion: u.NewVersion,
+						Package: u.Name, CurrentVersion: u.CurrentVersion, NewVersion: u.NewVersion, Phased: u.Phased,
 					})
 				}
 				return listPendingUpdatesResult{Updates: out, Shown: len(out)}, nil
@@ -411,7 +413,7 @@ func init() {
 					}
 					out = append(out, pendingUpdate{
 						ServerID: srv.ServerID, Hostname: srv.Hostname,
-						Package: u.Name, CurrentVersion: u.CurrentVersion, NewVersion: u.NewVersion,
+						Package: u.Name, CurrentVersion: u.CurrentVersion, NewVersion: u.NewVersion, Phased: u.Phased,
 					})
 				}
 			}

server/internal/mcp/tools_write.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
 )
 
@@ -156,16 +157,16 @@ type updateBatchResult struct {
 	Servers   int               `json:"servers"`
 	Succeeded []string          `json:"succeeded"`
 	Failed    map[string]string `json:"failed,omitempty"`
+	RunIDs    map[string]string `json:"run_ids,omitempty"` // server ID -> patch run ID
 }
 
 // apply_updates. The REST route (internal/api/handlers.go's applyUpdates) is
 // per-server: POST /servers/:id/apply-updates resolves one server with
-// services.GetServerScoped and calls services.DispatchApplyUpdates(serverID).
-// There is no fleet-wide variant of that service call to invoke once, so this
-// tool resolves the requested targets through ResolveTargetsScoped exactly as
-// the brief describes, then calls the same DispatchApplyUpdates the REST route
-// calls, once per resolved server - the identical dispatch, just looped
-// instead of hardcoded to one server_id from the URL.
+// services.GetServerScoped and calls services.StartManualRun. There is no
+// fleet-wide variant of that service call to invoke once, so this tool
+// resolves the requested targets through ResolveTargetsScoped, then calls the
+// same StartManualRun once per resolved server and writes the same
+// updates.applied audit event per server with its run ID.
 func init() {
 	All().Register(Tool{
 		Name: "apply_updates",
@@ -178,9 +179,9 @@ func init() {
 		Write:          true,
 		Scope:          "servers:write",
 		Description: "Apply pending OS package updates on real servers, selected by " +
-			"server_ids and/or tags. This installs packages on real machines right now and " +
-			"cannot be undone from here. A server may need a reboot afterward, which this " +
-			"tool does not do.",
+			"server_ids and/or tags. Starts one manual patch run per server. This installs " +
+			"packages on real machines right now and cannot be undone from here. A server " +
+			"may need a reboot afterward, which this tool does not do.",
 		Handler: func(ctx context.Context, c Caller, args map[string]any) (any, error) {
 			ids := stringSliceArg(args, "server_ids")
 			targets, err := services.ResolveTargetsScoped(c.InstanceID, ids, tagArg(args), c.TokenScope)
@@ -192,12 +193,20 @@ func init() {
 			}
 
 			result := updateBatchResult{Servers: len(targets), Failed: map[string]string{}}
-			for _, srv := range targets {
-				if err := services.DispatchApplyUpdates(srv.ServerID); err != nil {
+			for i := range targets {
+				srv := targets[i]
+				run, err := services.StartManualRun(c.InstanceID, &srv, "mcp:"+c.TokenName, models.PatchSourceMCP)
+				if err != nil {
 					result.Failed[srv.ServerID] = err.Error()
 					continue
 				}
+				services.LogEvent(c.InstanceID, "updates.applied", "mcp:"+c.TokenName, srv.ServerID, "",
+					fmt.Sprintf("package update run %s started on %s", run.RunID, srv.Hostname))
 				result.Succeeded = append(result.Succeeded, srv.ServerID)
+				if result.RunIDs == nil {
+					result.RunIDs = map[string]string{}
+				}
+				result.RunIDs[srv.ServerID] = run.RunID
 			}
 			if len(result.Failed) == 0 {
 				result.Failed = nil

server/internal/metricsched/scheduler.go

@@ -0,0 +1,41 @@
+// Package metricsched sweeps passive monitors - heartbeats and, from phase 2,
+// metric monitors - on a fixed tick. Nothing here runs a check: it reads what
+// pings and agents already delivered and decides what is overdue or breaching.
+package metricsched
+
+import (
+	"context"
+	"log"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
+)
+
+const tick = 30 * time.Second
+
+func Start(ctx context.Context) {
+	go func() {
+		t := time.NewTicker(tick)
+		defer t.Stop()
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case now := <-t.C:
+				sweep(now)
+			}
+		}
+	}()
+}
+
+// sweep recovers per sweep so one bad document cannot stop alerting for the
+// whole instance until the next deploy.
+func sweep(now time.Time) {
+	defer func() {
+		if r := recover(); r != nil {
+			log.Printf("metricsched: sweep panic: %v", r)
+		}
+	}()
+	services.SweepHeartbeats(now)
+	services.SweepMetricMonitors(now)
+}

server/internal/models/monitor.go

@@ -7,12 +7,21 @@ import (
 )
 
 const (
-	MonitorHTTP = "http"
-	MonitorTCP  = "tcp"
-	MonitorICMP = "icmp"
-	MonitorTLS  = "tls"
+	MonitorHTTP      = "http"
+	MonitorTCP       = "tcp"
+	MonitorICMP      = "icmp"
+	MonitorTLS       = "tls"
+	MonitorMetric    = "metric"
+	MonitorHeartbeat = "heartbeat"
 )
 
+// IsPassiveMonitor reports whether a monitor type is evaluated from data that
+// arrives (a ping, an agent report) rather than by running a check. Passive
+// monitors are never handed to monitorsched or to an agent.
+func IsPassiveMonitor(t string) bool {
+	return t == MonitorMetric || t == MonitorHeartbeat
+}
+
 const (
 	StatusUp      = "up"
 	StatusDown    = "down"
@@ -38,6 +47,14 @@ type MonitorTarget struct {
 	Keyword        string `bson:"keyword,omitempty" json:"keyword,omitempty"`
 	TLSWarnDays    int    `bson:"tls_warn_days,omitempty" json:"tls_warn_days,omitempty"`
 	Insecure       bool   `bson:"insecure,omitempty" json:"insecure,omitempty"`
+	// Metric monitors.
+	Selector  map[string]string `bson:"selector,omitempty" json:"selector,omitempty"`
+	Metric    string            `bson:"metric,omitempty" json:"metric,omitempty"`
+	Threshold float64           `bson:"threshold,omitempty" json:"threshold,omitempty"`
+	Mount     string            `bson:"mount,omitempty" json:"mount,omitempty"`
+	// Heartbeat monitors.
+	PeriodSec int `bson:"period_sec,omitempty" json:"period_sec,omitempty"`
+	GraceSec  int `bson:"grace_sec,omitempty" json:"grace_sec,omitempty"`
 }
 
 type MonitorState struct {
@@ -48,6 +65,8 @@ type MonitorState struct {
 	CertExpiryAt   *time.Time `bson:"cert_expiry_at,omitempty" json:"cert_expiry_at,omitempty"`
 	Fails          int        `bson:"fails" json:"fails"`
 	LastNotifiedAt *time.Time `bson:"last_notified_at,omitempty" json:"last_notified_at,omitempty"`
+	LastPingAt     *time.Time `bson:"last_ping_at,omitempty" json:"last_ping_at,omitempty"`
+	StartedAt      *time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
 }
 
 type Monitor struct {
@@ -68,6 +87,13 @@ type Monitor struct {
 	ChannelIDs  []string      `bson:"channel_ids,omitempty" json:"channel_ids,omitempty"`
 	State       MonitorState  `bson:"state" json:"state"`
 	CreatedAt   time.Time     `bson:"created_at" json:"created_at"`
+	// ForSec is how long a metric condition must hold before a server is down.
+	ForSec int `bson:"for_sec,omitempty" json:"for_sec,omitempty"`
+	// HeartbeatTokenHash is the SHA-256 of the ping token. The token itself is
+	// shown once, on create or rotate, and never stored.
+	HeartbeatTokenHash string `bson:"heartbeat_token_hash,omitempty" json:"-"`
+	// HeartbeatToken is the plaintext token, set only on the create response.
+	HeartbeatToken string `bson:"-" json:"heartbeat_token,omitempty"`
 }
 
 type Incident struct {
@@ -77,6 +103,22 @@ type Incident struct {
 	StartedAt  time.Time  `bson:"started_at" json:"started_at"`
 	ResolvedAt *time.Time `bson:"resolved_at,omitempty" json:"resolved_at,omitempty"`
 	Cause      string     `bson:"cause,omitempty" json:"cause,omitempty"`
+	// ServerID is set only for metric monitors, which keep one incident per
+	// breaching server.
+	ServerID string `bson:"server_id,omitempty" json:"server_id,omitempty"`
+}
+
+// MonitorServerState is one metric monitor's view of one matching server.
+type MonitorServerState struct {
+	InstanceID  string     `bson:"instance_id" json:"instance_id"`
+	MonitorID   string     `bson:"monitor_id" json:"monitor_id"`
+	ServerID    string     `bson:"server_id" json:"server_id"`
+	Hostname    string     `bson:"-" json:"hostname,omitempty"`
+	Status      string     `bson:"status" json:"status"`
+	BreachSince *time.Time `bson:"breach_since,omitempty" json:"breach_since,omitempty"`
+	Value       float64    `bson:"value" json:"value"`
+	Message     string     `bson:"message,omitempty" json:"message,omitempty"`
+	UpdatedAt   time.Time  `bson:"updated_at" json:"updated_at"`
 }
 
 // MonitorSample is one check result, kept only long enough to draw the

server/internal/models/patch.go

@@ -0,0 +1,116 @@
+package models
+
+import (
+	"time"
+
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+const (
+	PatchScopeAll      = "all"
+	PatchScopeSecurity = "security"
+
+	PatchRebootNever      = "never"
+	PatchRebootIfRequired = "if_required"
+
+	PatchRunRunning   = "running"
+	PatchRunSucceeded = "succeeded"
+	PatchRunPartial   = "partial"
+	PatchRunFailed    = "failed"
+	PatchRunCancelled = "cancelled"
+
+	PatchSrvQueued         = "queued"
+	PatchSrvWaitingOffline = "waiting_offline"
+	PatchSrvPatching       = "patching"
+	PatchSrvRebooting      = "rebooting"
+	PatchSrvSucceeded      = "succeeded"
+	PatchSrvFailed         = "failed"
+	PatchSrvUnsupported    = "unsupported"
+	PatchSrvAgentTooOld    = "agent_too_old"
+	PatchSrvMissedOffline  = "missed_offline"
+	PatchSrvWindowClosed   = "window_closed"
+	PatchSrvCancelled      = "cancelled"
+
+	PatchSourceSchedule        = "schedule"
+	PatchSourceRunNow          = "run_now"
+	PatchSourceServer          = "server"
+	PatchSourceVulnerabilities = "vulnerabilities"
+	PatchSourceMCP             = "mcp"
+)
+
+// MaintenanceWindow answers "when" and nothing else. Policies reference it by
+// ID, so one window can later serve alert muting and status page maintenance
+// without a second definition of the same Sunday morning.
+type MaintenanceWindow struct {
+	ID              bson.ObjectID `bson:"_id,omitempty" json:"-"`
+	InstanceID      string        `bson:"instance_id" json:"instance_id"`
+	WindowID        string        `bson:"window_id" json:"window_id"`
+	Name            string        `bson:"name" json:"name"`
+	Cron            string        `bson:"cron" json:"cron"` // 5-field, window start
+	TZ              string        `bson:"tz" json:"tz"`     // IANA name
+	DurationMinutes int           `bson:"duration_minutes" json:"duration_minutes"`
+	CreatedAt       time.Time     `bson:"created_at" json:"created_at"`
+	UpdatedAt       time.Time     `bson:"updated_at" json:"updated_at"`
+}
+
+type PatchPolicy struct {
+	ID               bson.ObjectID     `bson:"_id,omitempty" json:"-"`
+	InstanceID       string            `bson:"instance_id" json:"instance_id"`
+	PolicyID         string            `bson:"policy_id" json:"policy_id"`
+	Name             string            `bson:"name" json:"name"`
+	Enabled          bool              `bson:"enabled" json:"enabled"`
+	WindowID         string            `bson:"window_id" json:"window_id"`
+	TargetServerIDs  []string          `bson:"target_server_ids" json:"target_server_ids"`
+	TargetTags       map[string]string `bson:"target_tags,omitempty" json:"target_tags,omitempty"`
+	Scope            string            `bson:"scope" json:"scope"`
+	Reboot           string            `bson:"reboot" json:"reboot"`
+	MaxConcurrent    int               `bson:"max_concurrent" json:"max_concurrent"` // 0 = no cap
+	NotifyChannelIDs []string          `bson:"notify_channel_ids,omitempty" json:"notify_channel_ids,omitempty"`
+	NextRunAt        *time.Time        `bson:"next_run_at,omitempty" json:"next_run_at,omitempty"`
+	LastRunAt        *time.Time        `bson:"last_run_at,omitempty" json:"last_run_at,omitempty"`
+	LastSkipped      *Skip             `bson:"last_skipped,omitempty" json:"last_skipped,omitempty"`
+	DisabledReason   string            `bson:"disabled_reason,omitempty" json:"disabled_reason,omitempty"`
+	CreatedAt        time.Time         `bson:"created_at" json:"created_at"`
+	UpdatedAt        time.Time         `bson:"updated_at" json:"updated_at"`
+}
+
+// PatchRun is one firing of a policy, or one manual Apply updates. Scope,
+// reboot and concurrency are copied from the policy at fire time so editing
+// the policy never rewrites what a past run shows.
+type PatchRun struct {
+	ID            bson.ObjectID    `bson:"_id,omitempty" json:"-"`
+	InstanceID    string           `bson:"instance_id" json:"instance_id"`
+	RunID         string           `bson:"run_id" json:"run_id"`
+	PolicyID      string           `bson:"policy_id,omitempty" json:"policy_id,omitempty"`
+	PolicyName    string           `bson:"policy_name,omitempty" json:"policy_name,omitempty"`
+	TriggeredBy   string           `bson:"triggered_by" json:"triggered_by"`
+	Source        string           `bson:"source" json:"source"`
+	Scope         string           `bson:"scope" json:"scope"`
+	Reboot        string           `bson:"reboot" json:"reboot"`
+	MaxConcurrent int              `bson:"max_concurrent" json:"max_concurrent"`
+	WindowEnd     *time.Time       `bson:"window_end,omitempty" json:"window_end,omitempty"`
+	Status        string           `bson:"status" json:"status"`
+	CancelledAt   *time.Time       `bson:"cancelled_at,omitempty" json:"cancelled_at,omitempty"`
+	StartedAt     time.Time        `bson:"started_at" json:"started_at"`
+	FinishedAt    *time.Time       `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
+	Servers       []PatchServerRun `bson:"servers" json:"servers"`
+}
+
+type PatchServerRun struct {
+	ServerID      string     `bson:"server_id" json:"server_id"`
+	Hostname      string     `bson:"hostname" json:"hostname"`
+	Status        string     `bson:"status" json:"status"`
+	CommandID     string     `bson:"command_id,omitempty" json:"-"`
+	PendingBefore int        `bson:"pending_before" json:"pending_before"`
+	PendingAfter  *int       `bson:"pending_after,omitempty" json:"pending_after,omitempty"`
+	RebootedAt    *time.Time `bson:"rebooted_at,omitempty" json:"rebooted_at,omitempty"`
+	VerifiedAt    *time.Time `bson:"verified_at,omitempty" json:"verified_at,omitempty"`
+	// BootTimeBefore is the host's reported boot time when the reboot was
+	// announced. A later report with a different boot time proves the
+	// restart without comparing the host clock to the server clock.
+	BootTimeBefore *time.Time `bson:"boot_time_before,omitempty" json:"-"`
+	Output         string     `bson:"output,omitempty" json:"output,omitempty"`
+	Error          string     `bson:"error,omitempty" json:"error,omitempty"`
+	StartedAt      *time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
+	FinishedAt     *time.Time `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
+}

server/internal/models/server.go

@@ -10,6 +10,9 @@ type PackageUpdate struct {
 	Name           string `bson:"name" json:"name"`
 	CurrentVersion string `bson:"current_version,omitempty" json:"current_version,omitempty"`
 	NewVersion     string `bson:"new_version" json:"new_version"`
+	// Phased is an Ubuntu phased update the host is not yet selected for: apt
+	// lists it but an upgrade defers it, so pending counts leave it out.
+	Phased bool `bson:"phased,omitempty" json:"phased,omitempty"`
 }
 
 type CPUInfo struct {
@@ -42,6 +45,11 @@ type Inventory struct {
 	RebootRequired bool        `bson:"reboot_required,omitempty" json:"reboot_required,omitempty"`
 	MetricsAt      *time.Time  `bson:"metrics_at,omitempty" json:"metrics_at,omitempty"`
 	StaticAt       *time.Time  `bson:"static_at,omitempty" json:"static_at,omitempty"`
+	// BootTime is the host's last reported boot time, stored on every report
+	// that carries one so a patch reboot can be proven by a changed boot.
+	BootTime *time.Time `bson:"boot_time,omitempty" json:"boot_time,omitempty"`
+	// RebootRequiredSince is when the host first reported a pending reboot.
+	RebootRequiredSince *time.Time `bson:"reboot_required_since,omitempty" json:"reboot_required_since,omitempty"`
 }
 
 type Server struct {

server/internal/models/settings.go

@@ -11,3 +11,7 @@ type (
 // APITokenMaxDays re-exports shared.APITokenMaxDays so server/internal/services
 // can read the token lifetime cap without importing shared/models directly.
 func APITokenMaxDays(s *Settings) int { return shared.APITokenMaxDays(s) }
+
+// RequireMFA re-exports shared.RequireMFA so services can read the MFA policy
+// without importing shared/models directly.
+func RequireMFA(s *Settings) bool { return shared.RequireMFA(s) }

server/internal/models/user_mfa.go

@@ -0,0 +1,73 @@
+package models
+
+import (
+	"time"
+
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+// UserMFA is one user's multi-factor enrolment. It is a separate collection
+// rather than fields on User because User lives in vantage-shared, which Vantage
+// HQ also writes: MFA is a control-plane concern per instance.
+type UserMFA struct {
+	ID         bson.ObjectID `bson:"_id,omitempty" json:"-"`
+	InstanceID string        `bson:"instance_id" json:"-"`
+	UserID     string        `bson:"user_id" json:"-"`
+
+	// WebAuthnHandle is a random 64 bytes, never the user ID: the handle is
+	// returned to any origin that asks a resident credential for it.
+	WebAuthnHandle []byte `bson:"webauthn_handle" json:"-"`
+
+	// TOTPSecretEnc is AES-256-GCM hex via services.encryptString. Mirrored in
+	// vantage-shared's backup.ciphertextFields - change one, change the other.
+	TOTPSecretEnc string `bson:"totp_secret_enc,omitempty" json:"-"`
+
+	// TOTPPendingEnc holds a secret from an unconfirmed StartTOTPSetup call.
+	// Transient: never mirrored into vantage-shared's backup.ciphertextFields.
+	TOTPPendingEnc string `bson:"totp_pending_enc,omitempty" json:"-"`
+
+	// TOTPConfirmedAt nil means setup was started but never confirmed, which
+	// does not count as an enrolled factor.
+	TOTPConfirmedAt *time.Time `bson:"totp_confirmed_at,omitempty" json:"totp_confirmed_at,omitempty"`
+
+	RecoveryCodes []RecoveryCode `bson:"recovery_codes,omitempty" json:"-"`
+
+	UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
+}
+
+// RecoveryCode stores only a SHA-256 hash: a leaked database yields no working
+// codes, exactly as api_tokens and agent tokens do.
+type RecoveryCode struct {
+	Hash   string     `bson:"hash"`
+	UsedAt *time.Time `bson:"used_at,omitempty"`
+}
+
+// WebAuthnCredential is one passkey. Nothing here is secret - a public key is
+// public - so no field is encrypted.
+type WebAuthnCredential struct {
+	ID         bson.ObjectID `bson:"_id,omitempty" json:"-"`
+	InstanceID string        `bson:"instance_id" json:"-"`
+	UserID     string        `bson:"user_id" json:"-"`
+
+	CredentialID []byte   `bson:"credential_id" json:"-"`
+	PublicKey    []byte   `bson:"public_key" json:"-"`
+	SignCount    uint32   `bson:"sign_count" json:"-"`
+	AAGUID       []byte   `bson:"aaguid" json:"-"`
+	Transports   []string `bson:"transports,omitempty" json:"transports,omitempty"`
+
+	// BackupEligible is whether the credential can sync between devices. It
+	// never changes for a real credential, and go-webauthn refuses an assertion
+	// whose flag differs from this one, so it must be stored. Nil only on rows
+	// registered before it was recorded; see auth.backupEligibleFor.
+	BackupEligible *bool `bson:"backup_eligible,omitempty" json:"-"`
+	// BackupState is whether it is currently synced. It may change.
+	BackupState bool `bson:"backup_state" json:"-"`
+
+	// CredentialIDHex is the browser-facing identifier for rename and delete.
+	// The raw bytes never reach a URL.
+	CredentialIDHex string `bson:"credential_id_hex" json:"id"`
+
+	Name       string     `bson:"name" json:"name"`
+	CreatedAt  time.Time  `bson:"created_at" json:"created_at"`
+	LastUsedAt *time.Time `bson:"last_used_at,omitempty" json:"last_used_at,omitempty"`
+}

server/internal/notify/dispatch.go

@@ -11,8 +11,13 @@ import (
 // monitor check. MonitorName carries the hostname in that case.
 const TypeServer = "server"
 
+// TypePatch marks a patch run summary. Like a vulnerability digest it is a
+// headline, not a transition, so title() adds no verb.
+const TypePatch = "patch"
+
 type Event struct {
 	MonitorName string
+	ServerName  string
 	Type        string
 	OldStatus   string
 	NewStatus   string
@@ -26,7 +31,7 @@ func (e Event) title() string {
 		verb = "is DOWN"
 	}
 	var s string
-	if e.Type == TypeVuln {
+	if e.Type == TypeVuln || e.Type == TypePatch {
 		// A digest is not a transition. MonitorName already carries the whole
 		// headline ("12 new critical across 4 servers"), so no verb applies.
 		s = fmt.Sprintf("[Vantage] %s", e.MonitorName)
@@ -38,7 +43,13 @@ func (e Event) title() string {
 		}
 		s = fmt.Sprintf("[Vantage] Server %s %s", e.MonitorName, verb)
 	} else {
-		s = fmt.Sprintf("[Vantage] %s (%s) %s", e.MonitorName, e.Type, verb)
+		name := e.MonitorName
+		if e.ServerName != "" {
+			name = fmt.Sprintf("%s (%s) on %s", e.MonitorName, e.Type, e.ServerName)
+		} else {
+			name = fmt.Sprintf("%s (%s)", e.MonitorName, e.Type)
+		}
+		s = fmt.Sprintf("[Vantage] %s %s", name, verb)
 	}
 	if e.Message != "" {
 		s += ": " + e.Message

server/internal/notify/dispatch_title_test.go

@@ -0,0 +1,24 @@
+package notify
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+func TestTitleIncludesServerName(t *testing.T) {
+	ev := Event{MonitorName: "Disk full", Type: models.MonitorMetric, NewStatus: models.StatusDown, ServerName: "web-01", Message: "/var 94.2% used"}
+	got := ev.title()
+	want := "[Vantage] Disk full (metric) on web-01 is DOWN: /var 94.2% used"
+	if got != want {
+		t.Fatalf("title = %q, want %q", got, want)
+	}
+}
+
+func TestTitleWithoutServerNameUnchanged(t *testing.T) {
+	ev := Event{MonitorName: "Site", Type: models.MonitorHTTP, NewStatus: models.StatusUp}
+	if got := ev.title(); strings.Contains(got, " on ") || got != "[Vantage] Site (http) recovered" {
+		t.Fatalf("title = %q", got)
+	}
+}

server/internal/notify/http.go

@@ -33,14 +33,18 @@ func dispatchWebhook(ch models.NotificationChannel, ev Event) error {
 	if target == "" {
 		return fmt.Errorf("webhook: missing url")
 	}
-	return postJSON(target, map[string]any{
+	payload := map[string]any{
 		"monitor":    ev.MonitorName,
 		"type":       ev.Type,
 		"old_status": ev.OldStatus,
 		"new_status": ev.NewStatus,
 		"message":    ev.Message,
 		"time":       ev.Time.Format(time.RFC3339),
-	})
+	}
+	if ev.ServerName != "" {
+		payload["server_name"] = ev.ServerName
+	}
+	return postJSON(target, payload)
 }
 
 func dispatchDiscord(ch models.NotificationChannel, ev Event) error {

server/internal/notify/patch_test.go

@@ -0,0 +1,17 @@
+package notify
+
+import (
+	"testing"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+// A patch summary is not a transition: it must not read "is DOWN" or
+// "recovered".
+func TestPatchEventTitle(t *testing.T) {
+	ev := Event{MonitorName: `Patch policy "Sunday prod" partial`, Type: TypePatch, NewStatus: models.PatchRunPartial, Message: "38 succeeded, 2 failed"}
+	want := `[Vantage] Patch policy "Sunday prod" partial: 38 succeeded, 2 failed`
+	if got := ev.title(); got != want {
+		t.Fatalf("got %q, want %q", got, want)
+	}
+}

server/internal/notify/smtp.go

@@ -27,7 +27,7 @@ func dispatchSMTP(ch models.NotificationChannel, ev Event) error {
 	}
 
 	return sender.SendMonitorAlert(to, mail.MonitorEvent{
-		MonitorName: ev.MonitorName,
+		MonitorName: smtpAlertName(ev),
 		Type:        ev.Type,
 		OldStatus:   ev.OldStatus,
 		NewStatus:   ev.NewStatus,
@@ -36,3 +36,13 @@ func dispatchSMTP(ch models.NotificationChannel, ev Event) error {
 		Down:        ev.NewStatus == models.StatusDown,
 	})
 }
+
+// smtpAlertName is the name shown in the alert email. mail.MonitorEvent (in
+// vantage-shared) has no ServerName field, so the server is folded into the
+// name the same way title() folds it into the notification title.
+func smtpAlertName(ev Event) string {
+	if ev.ServerName == "" {
+		return ev.MonitorName
+	}
+	return fmt.Sprintf("%s on %s", ev.MonitorName, ev.ServerName)
+}

server/internal/notify/smtp_test.go

@@ -0,0 +1,17 @@
+package notify
+
+import "testing"
+
+func TestSMTPAlertNameIncludesServerName(t *testing.T) {
+	ev := Event{MonitorName: "Disk full", ServerName: "web-01"}
+	if got, want := smtpAlertName(ev), "Disk full on web-01"; got != want {
+		t.Fatalf("smtpAlertName = %q, want %q", got, want)
+	}
+}
+
+func TestSMTPAlertNameWithoutServerNameUnchanged(t *testing.T) {
+	ev := Event{MonitorName: "Site"}
+	if got, want := smtpAlertName(ev), "Site"; got != want {
+		t.Fatalf("smtpAlertName = %q, want %q", got, want)
+	}
+}

server/internal/patchrun/patchrun.go

@@ -0,0 +1,275 @@
+// Package patchrun is the patch run state machine as pure functions. Nothing
+// here touches the database: the services layer loads a run, asks this
+// package what should change, and writes that change guarded by the status it
+// expected. That split is what makes the rules testable without MongoDB.
+package patchrun
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
+)
+
+// MinAgentVersion is the first agent release that honours ApplyUpdatesCmd's
+// scope and answers with a PatchResult. An older agent ignores the scope and
+// installs everything, so a policy must never dispatch to one.
+const MinAgentVersion = "1.4.0"
+
+const (
+	// ResultGrace covers what the agent does after the upgrade and before it
+	// answers: a pending-update re-check of up to 10 minutes (Windows Update
+	// search) plus a 2 minute reboot check, with room to spare.
+	ResultGrace = 20 * time.Minute
+	// ManualTimeout is the agent's backstop for one started upgrade, counted
+	// from its own start. The window end never stops a running upgrade.
+	ManualTimeout = 2 * time.Hour
+	// RebootTimeout is how long a rebooting server has to send a post-boot
+	// inventory report. Windows cumulative updates routinely take over 20.
+	RebootTimeout = 45 * time.Minute
+	// LatestStartBeforeEnd is the tail of a window in which no server starts
+	// patching: a late start would run long past the window end.
+	LatestStartBeforeEnd = 15 * time.Minute
+)
+
+// AgentSupportsPatchResults compares major.minor.patch. Empty, "dev" and
+// anything unparseable count as too old, and a pre-release of exactly the
+// minimum version counts as older than it, as semver orders them.
+func AgentSupportsPatchResults(version string) bool {
+	have, pre, ok := parseVersion(version)
+	if !ok {
+		return false
+	}
+	want, _, _ := parseVersion(MinAgentVersion)
+	for i := 0; i < 3; i++ {
+		if have[i] != want[i] {
+			return have[i] > want[i]
+		}
+	}
+	return !pre
+}
+
+func parseVersion(v string) ([3]int, bool, bool) {
+	var out [3]int
+	v = strings.TrimPrefix(strings.TrimSpace(v), "v")
+	v, _, _ = strings.Cut(v, "+")
+	core, pre, hasPre := strings.Cut(v, "-")
+	parts := strings.Split(core, ".")
+	if len(parts) != 3 {
+		return out, false, false
+	}
+	for i, p := range parts {
+		n, err := strconv.Atoi(p)
+		if err != nil || n < 0 {
+			return out, false, false
+		}
+		out[i] = n
+	}
+	return out, hasPre && pre != "", true
+}
+
+// IsTerminal reports whether a server run has finished.
+func IsTerminal(status string) bool {
+	switch status {
+	case models.PatchSrvQueued, models.PatchSrvWaitingOffline, models.PatchSrvPatching, models.PatchSrvRebooting:
+		return false
+	}
+	return true
+}
+
+// Transition is one change Advance wants made. The caller writes it guarded
+// by From, so a result that arrived meanwhile is never overwritten. Dispatch
+// means: send ApplyUpdatesCmd, and set To only once the command is on its way.
+type Transition struct {
+	ServerID string
+	From     string
+	To       string
+	Error    string
+	Dispatch bool
+}
+
+// Advance returns what should change on this tick. connected says which
+// agents hold a command stream right now.
+func Advance(run models.PatchRun, now time.Time, connected map[string]bool) []Transition {
+	if run.Status != models.PatchRunRunning {
+		return nil
+	}
+	windowOpen := run.WindowEnd == nil || now.Before(*run.WindowEnd)
+	// In the last LatestStartBeforeEnd of a window nothing new starts: queued
+	// and waiting servers simply wait, and close at WindowEnd as usual.
+	mayStart := run.WindowEnd == nil || now.Before(run.WindowEnd.Add(-LatestStartBeforeEnd))
+
+	inFlight := 0
+	for _, s := range run.Servers {
+		if s.Status == models.PatchSrvPatching || s.Status == models.PatchSrvRebooting {
+			inFlight++
+		}
+	}
+
+	var out []Transition
+	for _, s := range run.Servers {
+		switch s.Status {
+		case models.PatchSrvQueued, models.PatchSrvWaitingOffline:
+			switch {
+			case run.CancelledAt != nil:
+				out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvCancelled})
+			case !windowOpen:
+				to := models.PatchSrvWindowClosed
+				if s.Status == models.PatchSrvWaitingOffline {
+					to = models.PatchSrvMissedOffline
+				}
+				out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: to})
+			case !mayStart:
+				// The window tail: no dispatch, no transition.
+			case run.MaxConcurrent > 0 && inFlight >= run.MaxConcurrent:
+				// No slot this tick.
+			case !connected[s.ServerID]:
+				if s.Status == models.PatchSrvQueued {
+					out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvWaitingOffline})
+				}
+			default:
+				out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvPatching, Dispatch: true})
+				inFlight++
+			}
+		case models.PatchSrvPatching:
+			if now.After(resultDeadline(run, s)) {
+				out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvFailed, Error: "no result from agent"})
+			}
+		case models.PatchSrvRebooting:
+			if s.RebootedAt != nil && now.After(s.RebootedAt.Add(RebootTimeout)) {
+				out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvFailed, Error: fmt.Sprintf("did not come back within %d minutes", int(RebootTimeout.Minutes()))})
+			}
+		}
+	}
+	return out
+}
+
+// resultDeadline is the same for windowed and manual runs: the agent lets a
+// started upgrade finish past the window end, so the window end says nothing
+// about when a result is due. The base is the server's dispatch time.
+func resultDeadline(run models.PatchRun, s models.PatchServerRun) time.Time {
+	start := run.StartedAt
+	if s.StartedAt != nil {
+		start = *s.StartedAt
+	}
+	return start.Add(ManualTimeout + ResultGrace)
+}
+
+// ApplyResult folds an agent's PatchResult into the server run. It only acts
+// on a server that is patching; anything else is a late or duplicate result.
+func ApplyResult(s models.PatchServerRun, r *pb.PatchResult, now time.Time) (models.PatchServerRun, bool) {
+	if s.Status != models.PatchSrvPatching {
+		return s, false
+	}
+	s.Output = r.OutputTail
+	if r.PendingAfter >= 0 {
+		v := int(r.PendingAfter)
+		s.PendingAfter = &v
+	}
+	switch r.Status {
+	case pb.PatchStatusOK:
+		if r.Rebooting {
+			s.Status = models.PatchSrvRebooting
+			s.RebootedAt = &now
+			return s, true
+		}
+		s.Status = models.PatchSrvSucceeded
+	case pb.PatchStatusUnsupported:
+		s.Status = models.PatchSrvUnsupported
+		s.Error = r.Message
+	default:
+		s.Status = models.PatchSrvFailed
+		s.Error = r.Message
+		if s.Error == "" {
+			s.Error = "agent reported a failure"
+		}
+	}
+	s.FinishedAt = &now
+	return s, true
+}
+
+// VerifyReboot settles a rebooting server from a static inventory report.
+// When the boot time reported before the reboot is known, a later boot time
+// is the proof: both come from the host clock, so skew against the server
+// clock does not matter. Otherwise only a boot time later than the reboot
+// command counts. Either way a snapshot sent during the one-minute grace
+// period, before the host went down, does not.
+func VerifyReboot(s models.PatchServerRun, bootTime time.Time, rebootRequired bool, now time.Time) (models.PatchServerRun, bool) {
+	if s.Status != models.PatchSrvRebooting || s.RebootedAt == nil {
+		return s, false
+	}
+	proven := bootTime.After(*s.RebootedAt)
+	if s.BootTimeBefore != nil {
+		proven = bootTime.After(s.BootTimeBefore.Add(BootTimeTolerance))
+	}
+	if !proven {
+		return s, false
+	}
+	if rebootRequired {
+		s.Status = models.PatchSrvFailed
+		s.Error = "still requires a reboot after restarting"
+	} else {
+		s.Status = models.PatchSrvSucceeded
+		s.VerifiedAt = &now
+	}
+	s.FinishedAt = &now
+	return s, true
+}
+
+// BootTimeTolerance is how far a reported boot time must move before it counts
+// as a reboot. Windows reports boot time as now minus uptime, which drifts by a
+// second or so between reports; a real reboot moves it by far more than this.
+const BootTimeTolerance = 2 * time.Minute
+
+// Finalize says whether the run is over and how it ended. Only succeeded
+// counts as success: unsupported, agent_too_old and the window outcomes did
+// not patch anything.
+func Finalize(run models.PatchRun) (string, bool) {
+	ok := 0
+	for _, s := range run.Servers {
+		if !IsTerminal(s.Status) {
+			return "", false
+		}
+		if s.Status == models.PatchSrvSucceeded {
+			ok++
+		}
+	}
+	switch {
+	case run.CancelledAt != nil:
+		return models.PatchRunCancelled, true
+	case len(run.Servers) > 0 && ok == len(run.Servers):
+		return models.PatchRunSucceeded, true
+	case ok == 0:
+		return models.PatchRunFailed, true
+	default:
+		return models.PatchRunPartial, true
+	}
+}
+
+var summaryOrder = []struct{ status, label string }{
+	{models.PatchSrvSucceeded, "succeeded"},
+	{models.PatchSrvFailed, "failed"},
+	{models.PatchSrvUnsupported, "unsupported"},
+	{models.PatchSrvAgentTooOld, "need an agent update"},
+	{models.PatchSrvMissedOffline, "missed offline"},
+	{models.PatchSrvWindowClosed, "window closed"},
+	{models.PatchSrvCancelled, "cancelled"},
+}
+
+// Summary is the one-line count used in alerts, e.g. "38 succeeded, 2 failed".
+func Summary(run models.PatchRun) string {
+	counts := map[string]int{}
+	for _, s := range run.Servers {
+		counts[s.Status]++
+	}
+	var parts []string
+	for _, o := range summaryOrder {
+		if n := counts[o.status]; n > 0 {
+			parts = append(parts, fmt.Sprintf("%d %s", n, o.label))
+		}
+	}
+	return strings.Join(parts, ", ")
+}

server/internal/patchrun/patchrun_test.go

@@ -0,0 +1,341 @@
+package patchrun
+
+import (
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
+)
+
+var t0 = time.Date(2026, 9, 20, 2, 0, 0, 0, time.UTC)
+
+func tp(t time.Time) *time.Time { return &t }
+
+func srv(id, status string) models.PatchServerRun {
+	return models.PatchServerRun{ServerID: id, Hostname: id, Status: status}
+}
+
+func windowRun(max int, servers ...models.PatchServerRun) models.PatchRun {
+	return models.PatchRun{Status: models.PatchRunRunning, StartedAt: t0, WindowEnd: tp(t0.Add(2 * time.Hour)), MaxConcurrent: max, Servers: servers}
+}
+
+func find(ts []Transition, id string) *Transition {
+	for i := range ts {
+		if ts[i].ServerID == id {
+			return &ts[i]
+		}
+	}
+	return nil
+}
+
+func TestAgentSupportsPatchResults(t *testing.T) {
+	cases := map[string]bool{
+		"1.4.0": true, "v1.4.0": true, "1.4.1": true, "1.10.0": true, "2.0.0": true,
+		"1.3.5": false, "1.4.0-rc1": false, "1.5.0-rc1": true,
+		"": false, "dev": false, "1.4": false, "x.y.z": false,
+	}
+	for v, want := range cases {
+		if got := AgentSupportsPatchResults(v); got != want {
+			t.Errorf("%q: got %v, want %v", v, got, want)
+		}
+	}
+}
+
+func TestAdvanceRespectsConcurrencyCountingRebooting(t *testing.T) {
+	run := windowRun(2,
+		srv("a", models.PatchSrvRebooting),
+		srv("b", models.PatchSrvQueued),
+		srv("c", models.PatchSrvQueued),
+	)
+	run.Servers[0].RebootedAt = tp(t0)
+	ts := Advance(run, t0.Add(time.Minute), map[string]bool{"b": true, "c": true})
+	if tr := find(ts, "b"); tr == nil || !tr.Dispatch || tr.To != models.PatchSrvPatching {
+		t.Fatalf("b should dispatch, got %+v", tr)
+	}
+	if tr := find(ts, "c"); tr != nil {
+		t.Fatalf("c must wait for a slot, got %+v", tr)
+	}
+}
+
+func TestAdvanceUnlimitedWhenZero(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvQueued), srv("b", models.PatchSrvQueued))
+	ts := Advance(run, t0, map[string]bool{"a": true, "b": true})
+	if len(ts) != 2 || !ts[0].Dispatch || !ts[1].Dispatch {
+		t.Fatalf("both should dispatch: %+v", ts)
+	}
+}
+
+func TestAdvanceOfflineWaitsThenDispatches(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvQueued))
+	ts := Advance(run, t0, map[string]bool{})
+	if len(ts) != 1 || ts[0].To != models.PatchSrvWaitingOffline || ts[0].Dispatch {
+		t.Fatalf("offline should wait: %+v", ts)
+	}
+	run.Servers[0].Status = models.PatchSrvWaitingOffline
+	if ts := Advance(run, t0.Add(time.Minute), map[string]bool{}); len(ts) != 0 {
+		t.Fatalf("still offline must be a no-op, got %+v", ts)
+	}
+	ts = Advance(run, t0.Add(2*time.Minute), map[string]bool{"a": true})
+	if len(ts) != 1 || !ts[0].Dispatch {
+		t.Fatalf("back online should dispatch: %+v", ts)
+	}
+}
+
+func TestAdvanceWindowCloses(t *testing.T) {
+	run := windowRun(1, srv("q", models.PatchSrvQueued), srv("w", models.PatchSrvWaitingOffline))
+	ts := Advance(run, t0.Add(2*time.Hour), map[string]bool{"q": true, "w": true})
+	if tr := find(ts, "q"); tr == nil || tr.To != models.PatchSrvWindowClosed || tr.Dispatch {
+		t.Fatalf("queued at window end: %+v", tr)
+	}
+	if tr := find(ts, "w"); tr == nil || tr.To != models.PatchSrvMissedOffline {
+		t.Fatalf("waiting at window end: %+v", tr)
+	}
+}
+
+// A windowed run times out from the server's own dispatch time, not from the
+// window end: a server dispatched late in the window may finish past it.
+func TestAdvanceNoResultTimeoutFromDispatch(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvPatching))
+	run.Servers[0].StartedAt = tp(t0.Add(90 * time.Minute))
+	deadline := t0.Add(90*time.Minute + ManualTimeout + ResultGrace)
+	if ts := Advance(run, run.WindowEnd.Add(ResultGrace+time.Minute), nil); len(ts) != 0 {
+		t.Fatalf("past WindowEnd+grace but inside the dispatch timeout: %+v", ts)
+	}
+	if ts := Advance(run, deadline.Add(-time.Minute), nil); len(ts) != 0 {
+		t.Fatalf("inside the dispatch timeout: %+v", ts)
+	}
+	ts := Advance(run, deadline.Add(time.Minute), nil)
+	if len(ts) != 1 || ts[0].To != models.PatchSrvFailed || ts[0].Error == "" {
+		t.Fatalf("past the dispatch timeout: %+v", ts)
+	}
+}
+
+// Without a server StartedAt the run's own start is the base.
+func TestAdvanceNoResultTimeoutFallsBackToRunStart(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvPatching))
+	if ts := Advance(run, t0.Add(2*time.Hour+19*time.Minute), nil); len(ts) != 0 {
+		t.Fatalf("inside timeout: %+v", ts)
+	}
+	if ts := Advance(run, t0.Add(2*time.Hour+21*time.Minute), nil); len(ts) != 1 || ts[0].To != models.PatchSrvFailed {
+		t.Fatalf("past timeout: %+v", ts)
+	}
+}
+
+func TestAdvanceManualRunTimeout(t *testing.T) {
+	run := models.PatchRun{Status: models.PatchRunRunning, StartedAt: t0, Servers: []models.PatchServerRun{srv("a", models.PatchSrvPatching)}}
+	run.Servers[0].StartedAt = tp(t0)
+	if ts := Advance(run, t0.Add(2*time.Hour+19*time.Minute), nil); len(ts) != 0 {
+		t.Fatalf("manual inside timeout: %+v", ts)
+	}
+	if ts := Advance(run, t0.Add(2*time.Hour+21*time.Minute), nil); len(ts) != 1 || ts[0].To != models.PatchSrvFailed {
+		t.Fatalf("manual past timeout: %+v", ts)
+	}
+}
+
+func TestTimingConstants(t *testing.T) {
+	if ResultGrace != 20*time.Minute || RebootTimeout != 45*time.Minute || LatestStartBeforeEnd != 15*time.Minute {
+		t.Fatalf("ResultGrace=%v RebootTimeout=%v LatestStartBeforeEnd=%v", ResultGrace, RebootTimeout, LatestStartBeforeEnd)
+	}
+}
+
+// No server starts patching in the last 15 minutes of a window: it would
+// either be cut short or run long past the window end.
+func TestAdvanceNoDispatchInWindowTail(t *testing.T) {
+	run := windowRun(0, srv("q", models.PatchSrvQueued), srv("w", models.PatchSrvWaitingOffline), srv("o", models.PatchSrvQueued))
+	cutoff := run.WindowEnd.Add(-LatestStartBeforeEnd)
+	online := map[string]bool{"q": true, "w": true}
+	for _, at := range []time.Time{cutoff, cutoff.Add(time.Minute), run.WindowEnd.Add(-time.Second)} {
+		if ts := Advance(run, at, online); len(ts) != 0 {
+			t.Fatalf("at %v: nothing may change in the window tail, got %+v", at.Sub(t0), ts)
+		}
+	}
+}
+
+func TestAdvanceDispatchJustBeforeWindowTail(t *testing.T) {
+	run := windowRun(0, srv("q", models.PatchSrvQueued))
+	at := run.WindowEnd.Add(-LatestStartBeforeEnd - time.Second)
+	ts := Advance(run, at, map[string]bool{"q": true})
+	if len(ts) != 1 || !ts[0].Dispatch {
+		t.Fatalf("dispatch must be allowed just before the tail: %+v", ts)
+	}
+}
+
+// A manual run has no window and no tail.
+func TestAdvanceManualRunHasNoTail(t *testing.T) {
+	run := models.PatchRun{Status: models.PatchRunRunning, StartedAt: t0, Servers: []models.PatchServerRun{srv("a", models.PatchSrvQueued)}}
+	if ts := Advance(run, t0.Add(10*time.Hour), map[string]bool{"a": true}); len(ts) != 1 || !ts[0].Dispatch {
+		t.Fatalf("manual run must dispatch: %+v", ts)
+	}
+}
+
+func TestAdvanceRebootTimeout(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvRebooting))
+	run.Servers[0].RebootedAt = tp(t0)
+	if ts := Advance(run, t0.Add(44*time.Minute), nil); len(ts) != 0 {
+		t.Fatalf("inside reboot timeout: %+v", ts)
+	}
+	ts := Advance(run, t0.Add(46*time.Minute), nil)
+	if len(ts) != 1 || ts[0].To != models.PatchSrvFailed || ts[0].Error != "did not come back within 45 minutes" {
+		t.Fatalf("past reboot timeout: %+v", ts)
+	}
+}
+
+func TestAdvanceCancelledDispatchesNothing(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvQueued), srv("b", models.PatchSrvWaitingOffline), srv("c", models.PatchSrvPatching))
+	run.CancelledAt = tp(t0)
+	ts := Advance(run, t0.Add(time.Minute), map[string]bool{"a": true, "b": true})
+	for _, id := range []string{"a", "b"} {
+		if tr := find(ts, id); tr == nil || tr.To != models.PatchSrvCancelled || tr.Dispatch {
+			t.Errorf("%s: %+v", id, tr)
+		}
+	}
+	if find(ts, "c") != nil {
+		t.Error("an in-flight server must be left to finish")
+	}
+}
+
+func TestAdvanceIgnoresFinishedRun(t *testing.T) {
+	run := windowRun(0, srv("a", models.PatchSrvQueued))
+	run.Status = models.PatchRunSucceeded
+	if ts := Advance(run, t0, map[string]bool{"a": true}); ts != nil {
+		t.Fatalf("got %+v", ts)
+	}
+}
+
+func TestApplyResult(t *testing.T) {
+	now := t0.Add(10 * time.Minute)
+	cases := []struct {
+		r    pb.PatchResult
+		want string
+	}{
+		{pb.PatchResult{Status: pb.PatchStatusOK, PendingAfter: 0}, models.PatchSrvSucceeded},
+		{pb.PatchResult{Status: pb.PatchStatusOK, Rebooting: true}, models.PatchSrvRebooting},
+		{pb.PatchResult{Status: pb.PatchStatusFailed, Message: "apt broke"}, models.PatchSrvFailed},
+		{pb.PatchResult{Status: pb.PatchStatusBusy, Message: "busy"}, models.PatchSrvFailed},
+		{pb.PatchResult{Status: pb.PatchStatusUnsupported, Message: "no metadata"}, models.PatchSrvUnsupported},
+	}
+	for _, c := range cases {
+		got, ok := ApplyResult(srv("a", models.PatchSrvPatching), &c.r, now)
+		if !ok || got.Status != c.want {
+			t.Errorf("%s: got %s ok=%v, want %s", c.r.Status, got.Status, ok, c.want)
+		}
+		if c.want == models.PatchSrvRebooting && (got.RebootedAt == nil || got.FinishedAt != nil) {
+			t.Errorf("rebooting must set RebootedAt and leave FinishedAt nil: %+v", got)
+		}
+	}
+	if _, ok := ApplyResult(srv("a", models.PatchSrvSucceeded), &pb.PatchResult{Status: pb.PatchStatusOK}, now); ok {
+		t.Error("a result for a server not patching must be ignored")
+	}
+	got, _ := ApplyResult(srv("a", models.PatchSrvPatching), &pb.PatchResult{Status: pb.PatchStatusOK, PendingAfter: -1}, now)
+	if got.PendingAfter != nil {
+		t.Error("PendingAfter -1 means unknown and must stay nil")
+	}
+}
+
+func TestVerifyReboot(t *testing.T) {
+	s := srv("a", models.PatchSrvRebooting)
+	s.RebootedAt = tp(t0)
+	now := t0.Add(5 * time.Minute)
+	if _, ok := VerifyReboot(s, t0.Add(-time.Hour), false, now); ok {
+		t.Error("boot before the reboot command is not proof")
+	}
+	if _, ok := VerifyReboot(s, t0, false, now); ok {
+		t.Error("boot equal to the reboot command is not proof")
+	}
+	got, ok := VerifyReboot(s, t0.Add(2*time.Minute), false, now)
+	if !ok || got.Status != models.PatchSrvSucceeded || got.VerifiedAt == nil {
+		t.Errorf("clean reboot: %+v", got)
+	}
+	got, ok = VerifyReboot(s, t0.Add(2*time.Minute), true, now)
+	if !ok || got.Status != models.PatchSrvFailed || got.Error == "" {
+		t.Errorf("still owed: %+v", got)
+	}
+	if _, ok := VerifyReboot(srv("a", models.PatchSrvPatching), t0.Add(time.Hour), false, now); ok {
+		t.Error("only rebooting servers verify")
+	}
+}
+
+// With the boot time recorded before the reboot, a changed boot time is the
+// proof, whatever the skew between the host clock and the server clock.
+func TestVerifyRebootChangedBoot(t *testing.T) {
+	s := srv("a", models.PatchSrvRebooting)
+	s.RebootedAt = tp(t0)
+	s.BootTimeBefore = tp(t0.Add(-10 * 24 * time.Hour))
+	now := t0.Add(5 * time.Minute)
+	// The host clock runs 10 minutes slow: its new boot time reads earlier
+	// than the server's RebootedAt, yet the boot did change.
+	got, ok := VerifyReboot(s, t0.Add(-8*time.Minute), false, now)
+	if !ok || got.Status != models.PatchSrvSucceeded {
+		t.Fatalf("changed boot behind a slow clock must be proven: %+v ok=%v", got, ok)
+	}
+	// A fast host clock with an unchanged boot is not proof.
+	if _, ok := VerifyReboot(s, *s.BootTimeBefore, false, now); ok {
+		t.Error("an unchanged boot time is not proof")
+	}
+	if _, ok := VerifyReboot(s, s.BootTimeBefore.Add(-time.Minute), false, now); ok {
+		t.Error("an earlier boot time is not proof")
+	}
+}
+
+// Windows derives boot time from now minus uptime, so it drifts by a second or
+// so between reports. A report sent in the grace period before the reboot
+// must not read as a reboot, or a server still owing one is marked failed.
+func TestVerifyRebootIgnoresBootTimeDrift(t *testing.T) {
+	s := srv("a", models.PatchSrvRebooting)
+	s.RebootedAt = tp(t0)
+	s.BootTimeBefore = tp(t0.Add(-10 * 24 * time.Hour))
+	now := t0.Add(30 * time.Second)
+	for _, drift := range []time.Duration{time.Second, BootTimeTolerance} {
+		if _, ok := VerifyReboot(s, s.BootTimeBefore.Add(drift), true, now); ok {
+			t.Errorf("a boot time drifted by %s is not proof of a reboot", drift)
+		}
+	}
+	got, ok := VerifyReboot(s, s.BootTimeBefore.Add(BootTimeTolerance+time.Second), false, now)
+	if !ok || got.Status != models.PatchSrvSucceeded {
+		t.Fatalf("a boot time past the tolerance is a reboot: %+v ok=%v", got, ok)
+	}
+}
+
+func TestFinalize(t *testing.T) {
+	mk := func(statuses ...string) models.PatchRun {
+		r := windowRun(0)
+		for i, s := range statuses {
+			r.Servers = append(r.Servers, srv(string(rune('a'+i)), s))
+		}
+		return r
+	}
+	cases := []struct {
+		run  models.PatchRun
+		want string
+		done bool
+	}{
+		{mk(models.PatchSrvSucceeded, models.PatchSrvSucceeded), models.PatchRunSucceeded, true},
+		{mk(models.PatchSrvSucceeded, models.PatchSrvFailed), models.PatchRunPartial, true},
+		{mk(models.PatchSrvSucceeded, models.PatchSrvUnsupported), models.PatchRunPartial, true},
+		{mk(models.PatchSrvSucceeded, models.PatchSrvAgentTooOld), models.PatchRunPartial, true},
+		{mk(models.PatchSrvMissedOffline, models.PatchSrvWindowClosed), models.PatchRunFailed, true},
+		{mk(models.PatchSrvSucceeded, models.PatchSrvPatching), "", false},
+		{mk(), models.PatchRunFailed, true},
+	}
+	for i, c := range cases {
+		got, done := Finalize(c.run)
+		if got != c.want || done != c.done {
+			t.Errorf("case %d: got %q/%v, want %q/%v", i, got, done, c.want, c.done)
+		}
+	}
+	cancelled := mk(models.PatchSrvSucceeded, models.PatchSrvCancelled)
+	cancelled.CancelledAt = tp(t0)
+	if got, done := Finalize(cancelled); got != models.PatchRunCancelled || !done {
+		t.Errorf("cancelled: %q/%v", got, done)
+	}
+}
+
+func TestSummary(t *testing.T) {
+	r := windowRun(0,
+		srv("a", models.PatchSrvSucceeded), srv("b", models.PatchSrvSucceeded),
+		srv("c", models.PatchSrvFailed), srv("d", models.PatchSrvMissedOffline))
+	if got := Summary(r); got != "2 succeeded, 1 failed, 1 missed offline" {
+		t.Fatalf("got %q", got)
+	}
+}

server/internal/patchsched/decide.go

@@ -0,0 +1,55 @@
+// Package patchsched fires patch policies at the start of their maintenance
+// window and advances running patch runs. Like workflowsched it runs under the
+// housekeeping leader lock and must not import services: services imports
+// this package for NextStart and WindowEnd.
+package patchsched
+
+import (
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
+)
+
+type Decision string
+
+const (
+	Fire          Decision = "fire"
+	SkipMissed    Decision = "missed"
+	SkipRunning   Decision = "already_running"
+	SkipNoTargets Decision = "no_targets"
+)
+
+// Decide is the whole fire or skip rule for one due policy. Missed comes
+// first, as in workflowsched: a stale occurrence is recorded as missed
+// whatever else is true. A window that has already closed is missed even
+// inside the hour of grace, because there is no time left to patch in.
+func Decide(due, windowEnd, now time.Time, runActive bool, targets int) Decision {
+	if !now.Before(windowEnd) || now.Sub(due) > workflowsched.GraceWindow {
+		return SkipMissed
+	}
+	if runActive {
+		return SkipRunning
+	}
+	if targets == 0 {
+		return SkipNoTargets
+	}
+	return Fire
+}
+
+func WindowEnd(start time.Time, durationMinutes int) time.Time {
+	return start.Add(time.Duration(durationMinutes) * time.Minute)
+}
+
+// NextStart is the first window start strictly after from. Callers pass
+// Later(now, currentWindowEnd) so windows never overlap, including across a
+// daylight-saving fall-back where the same wall-clock time occurs twice.
+func NextStart(cron, tz string, from time.Time) (time.Time, error) {
+	return workflowsched.NextOccurrence(cron, tz, from)
+}
+
+func Later(a, b time.Time) time.Time {
+	if a.After(b) {
+		return a
+	}
+	return b
+}

server/internal/patchsched/decide_test.go

@@ -0,0 +1,66 @@
+package patchsched
+
+import (
+	"testing"
+	"time"
+)
+
+var due = time.Date(2026, 9, 20, 2, 0, 0, 0, time.UTC)
+
+func TestDecide(t *testing.T) {
+	end := due.Add(2 * time.Hour)
+	cases := []struct {
+		name    string
+		now     time.Time
+		end     time.Time
+		running bool
+		targets int
+		want    Decision
+	}{
+		{"on time", due, end, false, 3, Fire},
+		{"late within grace", due.Add(59 * time.Minute), end, false, 3, Fire},
+		{"past grace", due.Add(61 * time.Minute), end, false, 3, SkipMissed},
+		{"window already over", due.Add(20 * time.Minute), due.Add(15 * time.Minute), false, 3, SkipMissed},
+		{"missed wins over running", due.Add(2 * time.Hour), end, true, 3, SkipMissed},
+		{"previous run active", due, end, true, 3, SkipRunning},
+		{"no targets", due, end, false, 0, SkipNoTargets},
+	}
+	for _, c := range cases {
+		if got := Decide(due, c.end, c.now, c.running, c.targets); got != c.want {
+			t.Errorf("%s: got %s, want %s", c.name, got, c.want)
+		}
+	}
+}
+
+// Europe/London falls back on 25 October 2026, so 01:30 happens twice. The
+// next window is computed from the end of the current one, so a two-hour
+// window starting at the first 01:30 cannot fire again at the second.
+func TestNextStartAcrossFallBack(t *testing.T) {
+	loc, _ := time.LoadLocation("Europe/London")
+	from := time.Date(2026, 10, 24, 12, 0, 0, 0, time.UTC)
+	first, err := NextStart("30 1 * * 0", "Europe/London", from)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if d := first.In(loc); d.Day() != 25 || d.Month() != time.October || d.Hour() != 1 || d.Minute() != 30 {
+		t.Fatalf("first = %s", d)
+	}
+	end := WindowEnd(first, 120)
+	next, err := NextStart("30 1 * * 0", "Europe/London", Later(first.Add(time.Minute), end))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if d := next.In(loc); d.Day() != 1 || d.Month() != time.November || d.Hour() != 1 || d.Minute() != 30 {
+		t.Fatalf("next = %s, want 2026-11-01 01:30 London", d)
+	}
+}
+
+func TestWindowEndAndLater(t *testing.T) {
+	if got := WindowEnd(due, 90); !got.Equal(due.Add(90 * time.Minute)) {
+		t.Fatalf("WindowEnd = %s", got)
+	}
+	a, b := due, due.Add(time.Second)
+	if !Later(a, b).Equal(b) || !Later(b, a).Equal(b) {
+		t.Fatal("Later must return the later time")
+	}
+}

server/internal/patchsched/sched.go

@@ -0,0 +1,159 @@
+package patchsched
+
+import (
+	"context"
+	"errors"
+	"log"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+const tickInterval = 30 * time.Second
+
+// Deps are injected from main.go: services imports this package, so this
+// package cannot import services.
+type Deps struct {
+	// LookupWindow returns nil, nil when the window no longer exists.
+	LookupWindow   func(instanceID, windowID string) (*models.MaintenanceWindow, error)
+	CountTargets   func(p models.PatchPolicy) (int, error)
+	StartPolicyRun func(p models.PatchPolicy, windowEnd time.Time) error
+	AdvanceRuns    func(ctx context.Context)
+	LogEvent       func(instanceID, eventType, actor, serverID, keyID, details string)
+}
+
+// Start runs until ctx is cancelled, inside bus.RunAsLeader("housekeeping").
+// Each tick fires due policies, then advances every running run.
+func Start(ctx context.Context, deps Deps) {
+	go func() {
+		t := time.NewTicker(tickInterval)
+		defer t.Stop()
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case <-t.C:
+				fireDue(ctx, deps, time.Now())
+				deps.AdvanceRuns(ctx)
+			}
+		}
+	}()
+}
+
+func fireDue(ctx context.Context, deps Deps, now time.Time) {
+	cur, err := db.Col("patch_policies").Find(ctx, bson.M{"enabled": true, "next_run_at": bson.M{"$lte": now}})
+	if err != nil {
+		log.Printf("patchsched: find due: %v", err)
+		return
+	}
+	var due []models.PatchPolicy
+	if err := cur.All(ctx, &due); err != nil {
+		log.Printf("patchsched: decode due: %v", err)
+		return
+	}
+	for _, p := range due {
+		if ctx.Err() != nil {
+			return
+		}
+		process(ctx, deps, p, now)
+	}
+}
+
+func process(ctx context.Context, deps Deps, p models.PatchPolicy, now time.Time) {
+	if p.NextRunAt == nil {
+		return
+	}
+	due := *p.NextRunAt
+
+	w, err := deps.LookupWindow(p.InstanceID, p.WindowID)
+	if err != nil {
+		log.Printf("patchsched: policy %s: load window: %v", p.PolicyID, err)
+		return // a database error is retried next tick, not treated as "gone"
+	}
+	if w == nil {
+		disable(ctx, deps, p, "its maintenance window no longer exists")
+		return
+	}
+	end := WindowEnd(due, w.DurationMinutes)
+	next, err := NextStart(w.Cron, w.TZ, Later(now, end))
+	if err != nil {
+		disable(ctx, deps, p, "its maintenance window schedule is no longer valid: "+err.Error())
+		return
+	}
+
+	// The claim, as in workflowsched: matching the current next_run_at means a
+	// second process reaching this policy matches nothing.
+	res, err := db.Col("patch_policies").UpdateOne(ctx,
+		bson.M{"policy_id": p.PolicyID, "next_run_at": due},
+		bson.M{"$set": bson.M{"next_run_at": next}})
+	if err != nil || res.MatchedCount == 0 {
+		return
+	}
+
+	n, err := deps.CountTargets(p)
+	if err != nil {
+		retryLater(ctx, deps, p, err, due, next, now)
+		return
+	}
+	active, err := hasActiveRun(ctx, p)
+	if err != nil {
+		retryLater(ctx, deps, p, err, due, next, now)
+		return
+	}
+	switch d := Decide(due, end, now, active, n); d {
+	case Fire:
+		if err := deps.StartPolicyRun(p, end); err != nil {
+			retryLater(ctx, deps, p, err, due, next, now)
+			return
+		}
+		_, _ = db.Col("patch_policies").UpdateOne(ctx, bson.M{"policy_id": p.PolicyID},
+			bson.M{"$set": bson.M{"last_run_at": now}, "$unset": bson.M{"last_skipped": ""}})
+	default:
+		recordSkip(ctx, deps, p, string(d), due, now)
+	}
+}
+
+func hasActiveRun(ctx context.Context, p models.PatchPolicy) (bool, error) {
+	err := db.Col("patch_runs").FindOne(ctx,
+		bson.M{"instance_id": p.InstanceID, "policy_id": p.PolicyID, "status": models.PatchRunRunning},
+		options.FindOne().SetProjection(bson.M{"_id": 1})).Err()
+	if err == nil {
+		return true, nil
+	}
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return false, nil
+	}
+	return false, err
+}
+
+// retryLater handles an error after the claim. The claim is put back, guarded
+// on the value just written so a concurrent edit to the policy is not undone,
+// and the next tick retries the same occurrence. Decide's missed rule bounds
+// the retries: once the occurrence is too late it is skipped as missed.
+func retryLater(ctx context.Context, deps Deps, p models.PatchPolicy, cause error, due, next, now time.Time) {
+	if _, err := db.Col("patch_policies").UpdateOne(ctx,
+		bson.M{"policy_id": p.PolicyID, "next_run_at": next},
+		bson.M{"$set": bson.M{"next_run_at": due}}); err != nil {
+		log.Printf("patchsched: policy %s: put back claim: %v", p.PolicyID, err)
+	}
+	recordSkip(ctx, deps, p, "error: "+cause.Error(), due, now)
+}
+
+func recordSkip(ctx context.Context, deps Deps, p models.PatchPolicy, reason string, due, at time.Time) {
+	_, _ = db.Col("patch_policies").UpdateOne(ctx, bson.M{"policy_id": p.PolicyID},
+		bson.M{"$set": bson.M{"last_skipped": models.Skip{Reason: reason, Due: due, At: at}}})
+	deps.LogEvent(p.InstanceID, "patch.skipped", "schedule", "", "",
+		"patch policy "+p.Name+" skipped "+due.Format(time.RFC3339)+": "+reason)
+}
+
+func disable(ctx context.Context, deps Deps, p models.PatchPolicy, reason string) {
+	_, _ = db.Col("patch_policies").UpdateOne(ctx, bson.M{"policy_id": p.PolicyID}, bson.M{
+		"$set":   bson.M{"enabled": false, "disabled_reason": reason},
+		"$unset": bson.M{"next_run_at": ""},
+	})
+	deps.LogEvent(p.InstanceID, "patch.policy_disabled", "schedule", "", "", "patch policy "+p.Name+" disabled: "+reason)
+}

server/internal/services/coreindexes.go

@@ -41,3 +41,28 @@ func EnsureAuthIndexes() error {
 	}
 	return nil
 }
+
+// EnsureMFAIndexes is fatal on failure like EnsureAuthIndexes, and for the same
+// reason: these unique indexes are a security property, not an optimisation. A
+// duplicate (instance_id, user_id) would make "this user's factors" ambiguous,
+// and a duplicate credential_id would let an assertion resolve to two users.
+func EnsureMFAIndexes() error {
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	if _, err := db.Col("user_mfa").Indexes().CreateOne(ctx, mongo.IndexModel{
+		Keys:    bson.D{{Key: "instance_id", Value: 1}, {Key: "user_id", Value: 1}},
+		Options: options.Index().SetUnique(true),
+	}); err != nil {
+		return err
+	}
+
+	_, err := db.Col("webauthn_credentials").Indexes().CreateMany(ctx, []mongo.IndexModel{
+		{
+			Keys:    bson.D{{Key: "instance_id", Value: 1}, {Key: "credential_id", Value: 1}},
+			Options: options.Index().SetUnique(true),
+		},
+		{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "user_id", Value: 1}}},
+	})
+	return err
+}

server/internal/services/heartbeats.go

@@ -0,0 +1,237 @@
+package services
+
+import (
+	"context"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"log"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+const (
+	HeartbeatPing  = "ping"
+	HeartbeatStart = "start"
+	HeartbeatFail  = "fail"
+)
+
+// MaxHeartbeatBody bounds what a /fail request can put into an incident cause
+// and a notification. A job's stderr can be megabytes; the first kilobyte is
+// what a human reads.
+const MaxHeartbeatBody = 1024
+
+const defaultHeartbeatGraceSec = 300
+
+var ErrHeartbeatNotFound = errors.New("heartbeat not found")
+
+// ErrInvalidMonitor marks a validation failure, which handlers answer with 400.
+var ErrInvalidMonitor = errors.New("invalid monitor")
+
+func NewHeartbeatToken() (string, string, error) {
+	raw := make([]byte, 24)
+	if _, err := rand.Read(raw); err != nil {
+		return "", "", err
+	}
+	tok := base64.RawURLEncoding.EncodeToString(raw)
+	return tok, HashHeartbeatToken(tok), nil
+}
+
+func HashHeartbeatToken(token string) string {
+	sum := sha256.Sum256([]byte(token))
+	return hex.EncodeToString(sum[:])
+}
+
+func validateHeartbeat(t *models.MonitorTarget) error {
+	if t.PeriodSec < 60 {
+		return fmt.Errorf("%w: period_sec must be at least 60", ErrInvalidMonitor)
+	}
+	if t.GraceSec < 0 {
+		return fmt.Errorf("%w: grace_sec must not be negative", ErrInvalidMonitor)
+	}
+	if t.GraceSec == 0 {
+		t.GraceSec = defaultHeartbeatGraceSec
+	}
+	return nil
+}
+
+func failMessage(body string) string {
+	body = strings.TrimSpace(body)
+	if len(body) > MaxHeartbeatBody {
+		body = body[:MaxHeartbeatBody]
+	}
+	if body == "" {
+		return "reported failure"
+	}
+	return "reported failure: " + body
+}
+
+// heartbeatVerdict decides whether a heartbeat is overdue. A heartbeat that has
+// never pinged is never down: the clock starts at the first ping, so creating
+// one before the job is deployed does not page anyone.
+func heartbeatVerdict(m models.Monitor, now time.Time) (bool, string) {
+	grace := time.Duration(m.Target.GraceSec) * time.Second
+	if m.State.StartedAt != nil && now.After(m.State.StartedAt.Add(grace)) {
+		return true, fmt.Sprintf("started %s, never finished", m.State.StartedAt.UTC().Format(time.RFC3339))
+	}
+	if m.State.LastPingAt == nil {
+		return false, ""
+	}
+	deadline := m.State.LastPingAt.Add(time.Duration(m.Target.PeriodSec)*time.Second + grace)
+	if now.After(deadline) {
+		return true, fmt.Sprintf("no ping since %s", m.State.LastPingAt.UTC().Format(time.RFC3339))
+	}
+	return false, ""
+}
+
+// RecordHeartbeat applies one ping. The state change is a single
+// FindOneAndUpdate on the token hash, so two pings racing each other cannot
+// both read the old state and lose one of the writes.
+func RecordHeartbeat(token, kind, body string, now time.Time) error {
+	ctx, cancel := monCtx()
+	defer cancel()
+
+	filter := bson.M{"heartbeat_token_hash": HashHeartbeatToken(token), "type": models.MonitorHeartbeat, "enabled": true}
+	set := bson.M{"state.last_check_at": now}
+	unset := bson.M{}
+	var next, message string
+	switch kind {
+	case HeartbeatStart:
+		set["state.started_at"] = now
+	case HeartbeatPing:
+		set["state.last_ping_at"] = now
+		set["state.status"] = models.StatusUp
+		set["state.message"] = ""
+		unset["state.started_at"] = ""
+		next = models.StatusUp
+	case HeartbeatFail:
+		message = failMessage(body)
+		set["state.last_ping_at"] = now
+		set["state.status"] = models.StatusDown
+		set["state.message"] = message
+		unset["state.started_at"] = ""
+		next = models.StatusDown
+	default:
+		return fmt.Errorf("unknown heartbeat kind %q", kind)
+	}
+	upd := bson.M{"$set": set}
+	if len(unset) > 0 {
+		upd["$unset"] = unset
+	}
+
+	// ReturnDocument Before: the previous status and started_at are what the
+	// transition and the duration need.
+	var before models.Monitor
+	err := db.Col("monitors").FindOneAndUpdate(ctx, filter, upd,
+		options.FindOneAndUpdate().SetReturnDocument(options.Before)).Decode(&before)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return ErrHeartbeatNotFound
+	}
+	if err != nil {
+		return err
+	}
+	if kind == HeartbeatStart {
+		return nil
+	}
+
+	latency := 0
+	if kind == HeartbeatPing && before.State.StartedAt != nil {
+		latency = int(now.Sub(*before.State.StartedAt).Milliseconds())
+	}
+	if latency > 0 {
+		db.Col("monitors").UpdateOne(ctx, bson.M{"monitor_id": before.MonitorID}, bson.M{"$set": bson.M{"state.latency_ms": latency}})
+	}
+	recordSample(ctx, &before, kind == HeartbeatPing, latency, now)
+	applyTransition(ctx, &before, "", "", before.State.Status, next, message, now)
+	return nil
+}
+
+// SweepHeartbeats marks overdue heartbeats down. Recovery only ever comes from
+// a ping, so the sweep never moves anything up.
+func SweepHeartbeats(now time.Time) {
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	cur, err := db.Col("monitors").Find(ctx, bson.M{
+		"type": models.MonitorHeartbeat, "enabled": true,
+		"state.status": bson.M{"$ne": models.StatusDown},
+	})
+	if err != nil {
+		log.Printf("heartbeats: list: %v", err)
+		return
+	}
+	var monitors []models.Monitor
+	if err := cur.All(ctx, &monitors); err != nil {
+		log.Printf("heartbeats: decode: %v", err)
+		return
+	}
+	for i := range monitors {
+		m := &monitors[i]
+		down, msg := heartbeatVerdict(*m, now)
+		if !down {
+			continue
+		}
+		// Guarded on the exact fields the verdict was computed from, so a
+		// ping landing between the read and this write (which changes
+		// last_ping_at or started_at without necessarily changing status)
+		// makes the filter match nothing rather than overwriting a monitor
+		// that is no longer overdue.
+		res, err := db.Col("monitors").UpdateOne(ctx,
+			sweepGuardFilter(*m),
+			bson.M{"$set": bson.M{"state.status": models.StatusDown, "state.message": msg, "state.last_check_at": now},
+				"$unset": bson.M{"state.started_at": ""}})
+		if err != nil || res.ModifiedCount == 0 {
+			continue
+		}
+		recordSample(ctx, m, false, 0, now)
+		applyTransition(ctx, m, "", "", m.State.Status, models.StatusDown, msg, now)
+	}
+}
+
+// sweepGuardFilter is the optimistic-concurrency filter for SweepHeartbeats'
+// update: it pins monitor_id, status, last_ping_at and started_at to the
+// values the verdict was computed from, so the update only applies when
+// nothing about the ping state changed underneath the sweep. last_ping_at is
+// absent before the first ping and started_at is $unset on every ping, so an
+// unset field is pinned with $exists:false rather than equality to nil.
+func sweepGuardFilter(m models.Monitor) bson.M {
+	filter := bson.M{"monitor_id": m.MonitorID, "state.status": m.State.Status}
+	if m.State.LastPingAt != nil {
+		filter["state.last_ping_at"] = *m.State.LastPingAt
+	} else {
+		filter["state.last_ping_at"] = bson.M{"$exists": false}
+	}
+	if m.State.StartedAt != nil {
+		filter["state.started_at"] = *m.State.StartedAt
+	} else {
+		filter["state.started_at"] = bson.M{"$exists": false}
+	}
+	return filter
+}
+
+func RotateHeartbeatToken(instanceID, monitorID string) (string, error) {
+	ctx, cancel := monCtx()
+	defer cancel()
+	tok, hash, err := NewHeartbeatToken()
+	if err != nil {
+		return "", err
+	}
+	res, err := db.Col("monitors").UpdateOne(ctx,
+		bson.M{"monitor_id": monitorID, "instance_id": instanceID, "type": models.MonitorHeartbeat},
+		bson.M{"$set": bson.M{"heartbeat_token_hash": hash}})
+	if err != nil {
+		return "", err
+	}
+	if res.MatchedCount == 0 {
+		return "", ErrHeartbeatNotFound
+	}
+	return tok, nil
+}

server/internal/services/heartbeats_test.go

@@ -0,0 +1,117 @@
+package services
+
+import (
+	"reflect"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+func hbMonitor(status string, lastPing, started *time.Time) models.Monitor {
+	return models.Monitor{
+		Type:   models.MonitorHeartbeat,
+		Target: models.MonitorTarget{PeriodSec: 3600, GraceSec: 300},
+		State:  models.MonitorState{Status: status, LastPingAt: lastPing, StartedAt: started},
+	}
+}
+
+func TestHeartbeatVerdict(t *testing.T) {
+	now := time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC)
+	at := func(d time.Duration) *time.Time { v := now.Add(-d); return &v }
+
+	cases := []struct {
+		name     string
+		m        models.Monitor
+		wantDown bool
+		wantMsg  string
+	}{
+		{"never pinged stays pending", hbMonitor(models.StatusPending, nil, nil), false, ""},
+		{"within period", hbMonitor(models.StatusUp, at(30*time.Minute), nil), false, ""},
+		{"inside grace", hbMonitor(models.StatusUp, at(62*time.Minute), nil), false, ""},
+		{"overdue", hbMonitor(models.StatusUp, at(66*time.Minute), nil), true, "no ping since"},
+		{"started inside grace", hbMonitor(models.StatusUp, at(10*time.Minute), at(4*time.Minute)), false, ""},
+		{"started never finished", hbMonitor(models.StatusUp, at(10*time.Minute), at(6*time.Minute)), true, "never finished"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			down, msg := heartbeatVerdict(c.m, now)
+			if down != c.wantDown || !strings.Contains(msg, c.wantMsg) {
+				t.Fatalf("got (%v,%q), want (%v, contains %q)", down, msg, c.wantDown, c.wantMsg)
+			}
+		})
+	}
+}
+
+func TestNewHeartbeatTokenHashes(t *testing.T) {
+	tok, hash, err := NewHeartbeatToken()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(tok) < 32 {
+		t.Fatalf("token too short: %d", len(tok))
+	}
+	if hash != HashHeartbeatToken(tok) || hash == tok {
+		t.Fatal("hash must be the SHA-256 of the token and differ from it")
+	}
+}
+
+func TestValidateHeartbeat(t *testing.T) {
+	tg := models.MonitorTarget{PeriodSec: 59}
+	if err := validateHeartbeat(&tg); err == nil {
+		t.Fatal("period under 60 must be rejected")
+	}
+	tg = models.MonitorTarget{PeriodSec: 60}
+	if err := validateHeartbeat(&tg); err != nil || tg.GraceSec != 300 {
+		t.Fatalf("grace should default to 300, got %d err %v", tg.GraceSec, err)
+	}
+	tg = models.MonitorTarget{PeriodSec: 60, GraceSec: -1}
+	if err := validateHeartbeat(&tg); err == nil {
+		t.Fatal("negative grace must be rejected")
+	}
+}
+
+func TestSweepGuardFilterPinsFields(t *testing.T) {
+	now := time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC)
+
+	// Both set: filter pins status, last_ping_at and started_at to their
+	// exact values.
+	m := hbMonitor(models.StatusUp, &now, &now)
+	m.MonitorID = "mon-1"
+	got := sweepGuardFilter(m)
+	want := bson.M{
+		"monitor_id":         "mon-1",
+		"state.status":       models.StatusUp,
+		"state.last_ping_at": now,
+		"state.started_at":   now,
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("got %+v, want %+v", got, want)
+	}
+
+	// Both unset: filter requires the fields to still be absent.
+	m2 := hbMonitor(models.StatusPending, nil, nil)
+	m2.MonitorID = "mon-2"
+	got2 := sweepGuardFilter(m2)
+	want2 := bson.M{
+		"monitor_id":         "mon-2",
+		"state.status":       models.StatusPending,
+		"state.last_ping_at": bson.M{"$exists": false},
+		"state.started_at":   bson.M{"$exists": false},
+	}
+	if !reflect.DeepEqual(got2, want2) {
+		t.Fatalf("got %+v, want %+v", got2, want2)
+	}
+}
+
+func TestTruncateHeartbeatBody(t *testing.T) {
+	long := strings.Repeat("x", MaxHeartbeatBody+50)
+	if got := failMessage(long); len(got) != len("reported failure: ")+MaxHeartbeatBody {
+		t.Fatalf("len = %d", len(got))
+	}
+	if got := failMessage("  "); got != "reported failure" {
+		t.Fatalf("empty body message = %q", got)
+	}
+}

server/internal/services/instancelock.go

@@ -0,0 +1,73 @@
+package services
+
+import (
+	"context"
+	"log"
+	"sync"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+// instanceLockTTL matches the host resolver's cache, so a lock and an unlock
+// both take effect within the same minute everywhere.
+const instanceLockTTL = 60 * time.Second
+
+type lockEntry struct {
+	locked bool
+	at     time.Time
+}
+
+var (
+	lockMu    sync.Mutex
+	lockCache = map[string]lockEntry{}
+
+	// lockLookup and lockNow are variables so tests can replace them.
+	lockLookup = func(ctx context.Context, instanceID string) (bool, error) {
+		n, err := db.Col("instances").CountDocuments(ctx, bson.M{
+			"instance_id": instanceID,
+			"locked_at":   bson.M{"$exists": true},
+		})
+		return n > 0, err
+	}
+	lockNow = time.Now
+)
+
+// InstanceLocked reports whether Vantage HQ has locked this instance under an
+// account dispute. HQ writes instances.locked_at through its cloudprov package;
+// nothing on this side ever sets or clears it.
+//
+// A read error answers false and is not cached. A database that cannot answer
+// this cannot serve the request that asked either, so failing open here costs
+// nothing that failing closed would save.
+func InstanceLocked(instanceID string) bool {
+	if instanceID == "" {
+		return false
+	}
+	lockMu.Lock()
+	if e, ok := lockCache[instanceID]; ok && lockNow().Sub(e.at) < instanceLockTTL {
+		lockMu.Unlock()
+		return e.locked
+	}
+	lockMu.Unlock()
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	locked, err := lockLookup(ctx, instanceID)
+	if err != nil {
+		log.Printf("instance lock: read %s: %v", instanceID, err)
+		return false
+	}
+	lockMu.Lock()
+	lockCache[instanceID] = lockEntry{locked: locked, at: lockNow()}
+	lockMu.Unlock()
+	return locked
+}
+
+// unlockedFilter narrows an instance query to instances HQ has not locked.
+// HQ clears the field with $unset, so absence is the whole test.
+func unlockedFilter(f bson.M) bson.M {
+	f["locked_at"] = bson.M{"$exists": false}
+	return f
+}

server/internal/services/instancelock_test.go

@@ -0,0 +1,86 @@
+package services
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+// stubLock replaces the Mongo lookup and the clock, and empties the cache, for
+// one test.
+func stubLock(t *testing.T, lookup func(context.Context, string) (bool, error), at *time.Time) {
+	t.Helper()
+	prevLookup, prevNow := lockLookup, lockNow
+	lockLookup = lookup
+	lockNow = func() time.Time { return *at }
+	lockMu.Lock()
+	lockCache = map[string]lockEntry{}
+	lockMu.Unlock()
+	t.Cleanup(func() {
+		lockLookup, lockNow = prevLookup, prevNow
+		lockMu.Lock()
+		lockCache = map[string]lockEntry{}
+		lockMu.Unlock()
+	})
+}
+
+func TestInstanceLockedCachesForTTL(t *testing.T) {
+	calls := 0
+	at := time.Date(2026, 9, 10, 10, 0, 0, 0, time.UTC)
+	stubLock(t, func(context.Context, string) (bool, error) { calls++; return true, nil }, &at)
+
+	if !InstanceLocked("i1") {
+		t.Fatal("want locked")
+	}
+	InstanceLocked("i1")
+	if calls != 1 {
+		t.Fatalf("lookups = %d within the TTL, want 1", calls)
+	}
+	at = at.Add(instanceLockTTL + time.Second)
+	InstanceLocked("i1")
+	if calls != 2 {
+		t.Fatalf("lookups = %d after the TTL, want 2", calls)
+	}
+}
+
+func TestInstanceLockedReadErrorIsNotCached(t *testing.T) {
+	calls := 0
+	at := time.Date(2026, 9, 10, 10, 0, 0, 0, time.UTC)
+	stubLock(t, func(context.Context, string) (bool, error) {
+		calls++
+		return false, errors.New("mongo down")
+	}, &at)
+
+	if InstanceLocked("i1") {
+		t.Fatal("a read error must answer false")
+	}
+	InstanceLocked("i1")
+	if calls != 2 {
+		t.Fatalf("lookups = %d, want 2: an error must not be cached", calls)
+	}
+}
+
+func TestInstanceLockedEmptyID(t *testing.T) {
+	at := time.Now()
+	stubLock(t, func(context.Context, string) (bool, error) {
+		t.Fatal("no lookup for an empty id")
+		return false, nil
+	}, &at)
+	if InstanceLocked("") {
+		t.Fatal("empty id is never locked")
+	}
+}
+
+func TestUnlockedFilter(t *testing.T) {
+	got := unlockedFilter(bson.M{"slug": "acme"})
+	want := bson.M{"slug": "acme", "locked_at": bson.M{"$exists": false}}
+	if len(got) != 2 || got["slug"] != "acme" {
+		t.Fatalf("got %v, want %v", got, want)
+	}
+	if cond, ok := got["locked_at"].(bson.M); !ok || cond["$exists"] != false {
+		t.Fatalf("got %v, want %v", got, want)
+	}
+}

server/internal/services/instances.go

@@ -25,6 +25,21 @@ func GetInstance(instanceID string) (*models.Instance, error) {
 }
 
 func GetInstanceBySlug(slug string) (*models.Instance, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	var o models.Instance
+	err := db.Col("instances").FindOne(ctx, unlockedFilter(bson.M{"slug": slug})).Decode(&o)
+	if err != nil {
+		return nil, err
+	}
+	return &o, nil
+}
+
+// GetInstanceBySlugIncludingLocked is GetInstanceBySlug without the lock
+// filter. Only the host resolver calls it, so it can tell "no such instance"
+// from "instance locked by Vantage HQ" and let the login page say which; it
+// still refuses a locked instance itself.
+func GetInstanceBySlugIncludingLocked(slug string) (*models.Instance, error) {
 	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	var o models.Instance

server/internal/services/inventory.go

@@ -2,11 +2,15 @@ package services
 
 import (
 	"context"
+	"errors"
+	"log"
 	"time"
 
 	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
 	"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
 	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
 )
 
 func StoreInventory(serverID string, r *pb.InventoryReport) error {
@@ -23,7 +27,13 @@ func StoreInventory(serverID string, r *pb.InventoryReport) error {
 		set["inventory.memory.used_bytes"] = r.Memory.UsedBytes
 	}
 	set["inventory.swap_used_bytes"] = r.SwapUsed
+	// Kept current so a patch reboot can be proven by a changed boot time,
+	// independent of clock skew between the host and the control plane.
+	if r.BootTimeUnix > 0 {
+		set["inventory.boot_time"] = time.Unix(r.BootTimeUnix, 0).UTC()
+	}
 
+	var unset bson.M
 	if r.IncludeStatic {
 		set["inventory.static_at"] = now
 		set["inventory.swap_total_bytes"] = r.SwapTotal
@@ -44,8 +54,44 @@ func StoreInventory(serverID string, r *pb.InventoryReport) error {
 			})
 		}
 		set["inventory.partitions"] = parts
+
+		var prev struct {
+			Inventory struct {
+				RebootRequired      bool       `bson:"reboot_required"`
+				RebootRequiredSince *time.Time `bson:"reboot_required_since"`
+			} `bson:"inventory"`
+		}
+		err := db.Col("servers").FindOne(ctx, bson.M{"server_id": serverID},
+			options.FindOne().SetProjection(bson.M{"inventory.reboot_required": 1, "inventory.reboot_required_since": 1})).Decode(&prev)
+		// On a transient read error the previous stamp is unknown. Leaving it
+		// alone beats resetting it to now, which would restart "pending for N
+		// days" and hide a long-overdue reboot.
+		if err != nil && !errors.Is(err, mongo.ErrNoDocuments) {
+			log.Printf("inventory: read reboot state for %s: %v", serverID, err)
+		} else if since, clear := rebootSinceUpdate(prev.Inventory.RebootRequired, prev.Inventory.RebootRequiredSince, r.RebootRequired, now); since != nil {
+			set["inventory.reboot_required_since"] = *since
+		} else if clear {
+			unset = bson.M{"inventory.reboot_required_since": ""}
+		}
 	}
 
-	_, err := db.Col("servers").UpdateOne(ctx, bson.M{"server_id": serverID}, bson.M{"$set": set})
+	upd := bson.M{"$set": set}
+	if unset != nil {
+		upd["$unset"] = unset
+	}
+	_, err := db.Col("servers").UpdateOne(ctx, bson.M{"server_id": serverID}, upd)
 	return err
 }
+
+// rebootSinceUpdate decides how reboot_required_since changes. The stamp is
+// kept from the first report that needed a reboot, so "pending for 7 days"
+// means seven days, not seven days since the last static report.
+func rebootSinceUpdate(prevRequired bool, prevSince *time.Time, nowRequired bool, now time.Time) (*time.Time, bool) {
+	if !nowRequired {
+		return nil, prevSince != nil || prevRequired
+	}
+	if prevSince == nil {
+		return &now, false
+	}
+	return nil, false
+}

server/internal/services/inventory_reboot_test.go

@@ -0,0 +1,24 @@
+package services
+
+import (
+	"testing"
+	"time"
+)
+
+func TestRebootSinceUpdate(t *testing.T) {
+	now := time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC)
+	earlier := now.Add(-72 * time.Hour)
+
+	if set, unset := rebootSinceUpdate(false, nil, true, now); set == nil || !set.Equal(now) || unset {
+		t.Fatal("turning on must stamp now")
+	}
+	if set, unset := rebootSinceUpdate(true, &earlier, true, now); set != nil || unset {
+		t.Fatal("staying on must keep the original stamp")
+	}
+	if set, _ := rebootSinceUpdate(true, nil, true, now); set == nil {
+		t.Fatal("on with no stamp (pre-upgrade data) must stamp now")
+	}
+	if set, unset := rebootSinceUpdate(true, &earlier, false, now); set != nil || !unset {
+		t.Fatal("turning off must clear the stamp")
+	}
+}

server/internal/services/metricfilter_test.go

@@ -0,0 +1,20 @@
+package services
+
+import (
+	"testing"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+func TestFilterByVisibleServer(t *testing.T) {
+	incs := []models.Incident{{IncidentID: "a"}, {IncidentID: "b", ServerID: "s1"}, {IncidentID: "c", ServerID: "s2"}}
+	id := func(i models.Incident) string { return i.ServerID }
+
+	if got := FilterByVisibleServer(incs, id, nil, false); len(got) != 3 {
+		t.Fatalf("unrestricted keeps all, got %d", len(got))
+	}
+	got := FilterByVisibleServer(incs, id, map[string]bool{"s1": true}, true)
+	if len(got) != 2 || got[0].IncidentID != "a" || got[1].IncidentID != "b" {
+		t.Fatalf("restricted keeps serverless and visible only, got %+v", got)
+	}
+}

server/internal/services/metricrules.go

@@ -0,0 +1,215 @@
+package services
+
+import (
+	"errors"
+	"fmt"
+	"path"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+// ErrMonitorOutOfScope marks a metric monitor whose selector reaches beyond a
+// restricted token's tag scope, on create, update or delete.
+var ErrMonitorOutOfScope = errors.New("selector is outside this credential's tag scope")
+
+// selectorWithinScope holds when every server the selector can match is also
+// inside the token's scope - true exactly when the selector pins every pair
+// the scope does.
+func selectorWithinScope(sel, tokenScope map[string]string) bool {
+	for k, v := range tokenScope {
+		if sel[k] != v {
+			return false
+		}
+	}
+	return true
+}
+
+const (
+	MetricDiskPct            = "disk_pct"
+	MetricDiskFreeGB         = "disk_free_gb"
+	MetricMemPct             = "mem_pct"
+	MetricLoadPerCore        = "load_per_core"
+	MetricUnitFailed         = "unit_failed"
+	MetricContainerUnhealthy = "container_unhealthy"
+	MetricRebootPendingDays  = "reboot_pending_days"
+	MetricAgentOfflineMin    = "agent_offline_min"
+)
+
+// metricStaleAfter is how old an agent's metrics can be before a rule stops
+// judging them. A dead agent's last report must not hold an alert open or
+// clear one; agent_offline_min is the rule for a dead agent.
+const metricStaleAfter = 5 * time.Minute
+
+func metricNeedsWorkloads(kind string) bool {
+	return kind == MetricUnitFailed || kind == MetricContainerUnhealthy
+}
+
+func metricUsesThreshold(kind string) bool {
+	return !metricNeedsWorkloads(kind)
+}
+
+func validateMetric(m *models.Monitor) error {
+	t := &m.Target
+	switch t.Metric {
+	case MetricDiskPct, MetricDiskFreeGB, MetricMemPct, MetricLoadPerCore,
+		MetricUnitFailed, MetricContainerUnhealthy, MetricRebootPendingDays, MetricAgentOfflineMin:
+	default:
+		return fmt.Errorf("%w: unknown metric %q", ErrInvalidMonitor, t.Metric)
+	}
+	if metricUsesThreshold(t.Metric) && t.Threshold <= 0 {
+		return fmt.Errorf("%w: threshold must be greater than 0", ErrInvalidMonitor)
+	}
+	if (t.Metric == MetricDiskPct || t.Metric == MetricMemPct) && t.Threshold > 100 {
+		return fmt.Errorf("%w: threshold must be 100 or less", ErrInvalidMonitor)
+	}
+	if t.Mount != "" && !path.IsAbs(t.Mount) {
+		return fmt.Errorf("%w: mount must be an absolute path", ErrInvalidMonitor)
+	}
+	if m.ForSec < 0 {
+		m.ForSec = 0
+	}
+	return nil
+}
+
+// EvaluateMetric judges one server against one rule. ok=false means there is
+// nothing trustworthy to judge - stale metrics, a mount that does not exist on
+// this host - and the caller keeps the previous state rather than guessing.
+func EvaluateMetric(t models.MonitorTarget, srv models.Server, wls []models.Workload, now time.Time) (bool, float64, string, bool) {
+	if t.Metric == MetricAgentOfflineMin {
+		if srv.LastSeen == nil {
+			return false, 0, "", false
+		}
+		mins := now.Sub(*srv.LastSeen).Minutes()
+		return mins >= t.Threshold, mins, fmt.Sprintf("agent not seen for %.0f minutes", mins), true
+	}
+	inv := srv.Inventory
+	if inv == nil || inv.MetricsAt == nil || now.Sub(*inv.MetricsAt) > metricStaleAfter {
+		return false, 0, "", false
+	}
+
+	switch t.Metric {
+	case MetricDiskPct, MetricDiskFreeGB:
+		return evalDisk(t, inv.Partitions)
+	case MetricMemPct:
+		if inv.Memory.TotalBytes == 0 {
+			return false, 0, "", false
+		}
+		pct := float64(inv.Memory.UsedBytes) / float64(inv.Memory.TotalBytes) * 100
+		return pct >= t.Threshold, pct, fmt.Sprintf("memory %.1f%% used", pct), true
+	case MetricLoadPerCore:
+		if inv.CPU.Cores == 0 {
+			return false, 0, "", false
+		}
+		v := inv.CPU.Load1 / float64(inv.CPU.Cores)
+		return v >= t.Threshold, v, fmt.Sprintf("load %.2f per core", v), true
+	case MetricRebootPendingDays:
+		if !inv.RebootRequired || inv.RebootRequiredSince == nil {
+			return false, 0, "", true
+		}
+		days := now.Sub(*inv.RebootRequiredSince).Hours() / 24
+		return days >= t.Threshold, days, fmt.Sprintf("reboot pending for %.0f days", days), true
+	case MetricUnitFailed:
+		return evalWorkloads(wls, "unit", func(w models.Workload) bool { return w.State == "failed" }, "failed")
+	case MetricContainerUnhealthy:
+		return evalWorkloads(wls, "container", func(w models.Workload) bool { return w.Health == "unhealthy" }, "unhealthy")
+	}
+	return false, 0, "", false
+}
+
+// evalDisk reports the worst matching partition, so "any mount" names the one
+// that is actually full.
+func evalDisk(t models.MonitorTarget, parts []models.Partition) (bool, float64, string, bool) {
+	found := false
+	var worstBreach bool
+	var worstVal float64
+	var worstMsg string
+	for _, p := range parts {
+		if p.TotalBytes == 0 || p.UsedBytes > p.TotalBytes || (t.Mount != "" && p.Mountpoint != t.Mount) {
+			continue
+		}
+		var breach bool
+		var val float64
+		var msg string
+		if t.Metric == MetricDiskPct {
+			val = float64(p.UsedBytes) / float64(p.TotalBytes) * 100
+			breach = val >= t.Threshold
+			msg = fmt.Sprintf("%s %.1f%% used", p.Mountpoint, val)
+		} else {
+			val = float64(p.TotalBytes-p.UsedBytes) / 1e9
+			breach = val <= t.Threshold
+			msg = fmt.Sprintf("%s %.1f GB free", p.Mountpoint, val)
+		}
+		worse := !found ||
+			(t.Metric == MetricDiskPct && val > worstVal) ||
+			(t.Metric == MetricDiskFreeGB && val < worstVal)
+		if worse {
+			worstBreach, worstVal, worstMsg = breach, val, msg
+		}
+		found = true
+	}
+	if !found {
+		return false, 0, "", false
+	}
+	return worstBreach, worstVal, worstMsg, true
+}
+
+func evalWorkloads(wls []models.Workload, kind string, bad func(models.Workload) bool, word string) (bool, float64, string, bool) {
+	var names []string
+	for _, w := range wls {
+		if w.Kind == kind && bad(w) {
+			names = append(names, w.Name)
+		}
+	}
+	if len(names) == 0 {
+		return false, 0, "", true
+	}
+	return true, float64(len(names)), fmt.Sprintf("%s %s: %s", kind, word, strings.Join(names, ", ")), true
+}
+
+// nextServerState applies the "for N seconds" gate. BreachSince is the first
+// sweep that saw the condition, so the gate measures continuous breach and a
+// single clear sweep restarts it.
+func nextServerState(prev *models.MonitorServerState, breach bool, forSec int, now time.Time) (string, *time.Time) {
+	if !breach {
+		return models.StatusUp, nil
+	}
+	since := now
+	if prev != nil && prev.BreachSince != nil {
+		since = *prev.BreachSince
+	}
+	// A server already down stays down while the breach lasts, even if for_sec
+	// was raised meanwhile. Dropping it to pending would never resolve the open
+	// incident, because only an up transition closes it.
+	if prev != nil && prev.Status == models.StatusDown {
+		return models.StatusDown, &since
+	}
+	if now.Sub(since) >= time.Duration(forSec)*time.Second {
+		return models.StatusDown, &since
+	}
+	return models.StatusPending, &since
+}
+
+func rollupParent(states []models.MonitorServerState) (string, string) {
+	if len(states) == 0 {
+		return models.StatusUp, "no matching servers"
+	}
+	down, pending := 0, 0
+	for _, s := range states {
+		switch s.Status {
+		case models.StatusDown:
+			down++
+		case models.StatusPending:
+			pending++
+		}
+	}
+	msg := fmt.Sprintf("%d of %d servers breaching", down, len(states))
+	switch {
+	case down > 0:
+		return models.StatusDown, msg
+	case pending > 0:
+		return models.StatusPending, msg
+	}
+	return models.StatusUp, msg
+}

server/internal/services/metricrules_test.go

@@ -0,0 +1,130 @@
+package services
+
+import (
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+var evalNow = time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC)
+
+func fresh() *time.Time { t := evalNow.Add(-time.Minute); return &t }
+
+func srvWith(inv models.Inventory) models.Server {
+	if inv.MetricsAt == nil {
+		inv.MetricsAt = fresh()
+	}
+	ls := evalNow.Add(-30 * time.Second)
+	return models.Server{ServerID: "s1", Hostname: "web-01", Inventory: &inv, LastSeen: &ls}
+}
+
+func TestEvaluateMetric(t *testing.T) {
+	disk := models.Inventory{Partitions: []models.Partition{
+		{Mountpoint: "/", TotalBytes: 100e9, UsedBytes: 50e9},
+		{Mountpoint: "/var", TotalBytes: 100e9, UsedBytes: 95e9},
+	}}
+	stale := evalNow.Add(-10 * time.Minute)
+	since := evalNow.Add(-8 * 24 * time.Hour)
+	oldSeen := evalNow.Add(-20 * time.Minute)
+
+	cases := []struct {
+		name       string
+		t          models.MonitorTarget
+		srv        models.Server
+		wls        []models.Workload
+		wantBreach bool
+		wantOK     bool
+		msgHas     string
+	}{
+		{"disk pct any mount breaches", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 90}, srvWith(disk), nil, true, true, "/var"},
+		{"disk pct named mount clear", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 90, Mount: "/"}, srvWith(disk), nil, false, true, "/"},
+		{"disk pct missing mount skips", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 90, Mount: "/data"}, srvWith(disk), nil, false, false, ""},
+		{"disk free gb breaches", models.MonitorTarget{Metric: MetricDiskFreeGB, Threshold: 10}, srvWith(disk), nil, true, true, "GB free"},
+		{"zero total partition ignored", models.MonitorTarget{Metric: MetricDiskPct, Threshold: 1}, srvWith(models.Inventory{Partitions: []models.Partition{{Mountpoint: "/proc"}}}), nil, false, false, ""},
+		{"used over total partition ignored", models.MonitorTarget{Metric: MetricDiskFreeGB, Threshold: 10}, srvWith(models.Inventory{Partitions: []models.Partition{{Mountpoint: "/", TotalBytes: 100e9, UsedBytes: 150e9}}}), nil, false, false, ""},
+		{"mem pct", models.MonitorTarget{Metric: MetricMemPct, Threshold: 80}, srvWith(models.Inventory{Memory: models.MemInfo{TotalBytes: 100, UsedBytes: 85}}), nil, true, true, "memory"},
+		{"load per core", models.MonitorTarget{Metric: MetricLoadPerCore, Threshold: 1.5}, srvWith(models.Inventory{CPU: models.CPUInfo{Cores: 4, Load1: 8}}), nil, true, true, "load"},
+		{"load no cores skips", models.MonitorTarget{Metric: MetricLoadPerCore, Threshold: 1.5}, srvWith(models.Inventory{CPU: models.CPUInfo{Load1: 8}}), nil, false, false, ""},
+		{"stale inventory skips", models.MonitorTarget{Metric: MetricMemPct, Threshold: 1}, srvWith(models.Inventory{MetricsAt: &stale, Memory: models.MemInfo{TotalBytes: 100, UsedBytes: 99}}), nil, false, false, ""},
+		{"unit failed", models.MonitorTarget{Metric: MetricUnitFailed}, srvWith(models.Inventory{}), []models.Workload{{Kind: "unit", Name: "backup.service", State: "failed"}}, true, true, "backup.service"},
+		{"container unhealthy clear", models.MonitorTarget{Metric: MetricContainerUnhealthy}, srvWith(models.Inventory{}), []models.Workload{{Kind: "container", Name: "db", Health: "healthy"}}, false, true, ""},
+		{"reboot pending days", models.MonitorTarget{Metric: MetricRebootPendingDays, Threshold: 7}, srvWith(models.Inventory{RebootRequired: true, RebootRequiredSince: &since}), nil, true, true, "reboot"},
+		{"reboot not required", models.MonitorTarget{Metric: MetricRebootPendingDays, Threshold: 7}, srvWith(models.Inventory{}), nil, false, true, ""},
+		{"agent offline ignores stale inventory", models.MonitorTarget{Metric: MetricAgentOfflineMin, Threshold: 10},
+			func() models.Server { s := srvWith(models.Inventory{MetricsAt: &stale}); s.LastSeen = &oldSeen; return s }(), nil, true, true, "not seen"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			breach, _, msg, ok := EvaluateMetric(c.t, c.srv, c.wls, evalNow)
+			if breach != c.wantBreach || ok != c.wantOK || !strings.Contains(msg, c.msgHas) {
+				t.Fatalf("got breach=%v ok=%v msg=%q", breach, ok, msg)
+			}
+		})
+	}
+}
+
+func TestNextServerState(t *testing.T) {
+	past := func(d time.Duration) *time.Time { v := evalNow.Add(-d); return &v }
+
+	st, since := nextServerState(nil, true, 300, evalNow)
+	if st != models.StatusPending || since == nil || !since.Equal(evalNow) {
+		t.Fatalf("new breach: %s %v", st, since)
+	}
+	st, _ = nextServerState(&models.MonitorServerState{Status: models.StatusPending, BreachSince: past(2 * time.Minute)}, true, 300, evalNow)
+	if st != models.StatusPending {
+		t.Fatalf("under for_sec should stay pending, got %s", st)
+	}
+	st, since = nextServerState(&models.MonitorServerState{Status: models.StatusPending, BreachSince: past(6 * time.Minute)}, true, 300, evalNow)
+	if st != models.StatusDown || !since.Equal(*past(6 * time.Minute)) {
+		t.Fatalf("over for_sec should be down keeping since, got %s %v", st, since)
+	}
+	st, _ = nextServerState(nil, true, 0, evalNow)
+	if st != models.StatusDown {
+		t.Fatalf("for_sec 0 is down immediately, got %s", st)
+	}
+	// Raising for_sec mid-breach must not demote a down server to pending:
+	// pending never resolves an incident, so it would stay open forever.
+	st, since = nextServerState(&models.MonitorServerState{Status: models.StatusDown, BreachSince: past(2 * time.Minute)}, true, 3600, evalNow)
+	if st != models.StatusDown || !since.Equal(*past(2 * time.Minute)) {
+		t.Fatalf("down with larger for_sec should stay down keeping since, got %s %v", st, since)
+	}
+	st, since = nextServerState(&models.MonitorServerState{Status: models.StatusDown, BreachSince: past(time.Hour)}, false, 300, evalNow)
+	if st != models.StatusUp || since != nil {
+		t.Fatalf("clear should be up with no since, got %s %v", st, since)
+	}
+}
+
+func TestRollupParent(t *testing.T) {
+	if s, m := rollupParent(nil); s != models.StatusUp || m != "no matching servers" {
+		t.Fatalf("empty: %s %q", s, m)
+	}
+	states := []models.MonitorServerState{{Status: models.StatusUp}, {Status: models.StatusPending}, {Status: models.StatusDown}}
+	if s, m := rollupParent(states); s != models.StatusDown || m != "1 of 3 servers breaching" {
+		t.Fatalf("got %s %q", s, m)
+	}
+	if s, _ := rollupParent(states[:2]); s != models.StatusPending {
+		t.Fatalf("pending wins over up, got %s", s)
+	}
+}
+
+func TestValidateMetric(t *testing.T) {
+	bad := []models.MonitorTarget{
+		{Metric: "cpu_magic", Threshold: 1},
+		{Metric: MetricDiskPct, Threshold: 0},
+		{Metric: MetricDiskPct, Threshold: 101},
+		{Metric: MetricDiskPct, Threshold: 90, Mount: "var"},
+	}
+	for _, tg := range bad {
+		m := models.Monitor{Type: models.MonitorMetric, Target: tg}
+		if err := validateMetric(&m); !errors.Is(err, ErrInvalidMonitor) {
+			t.Errorf("%+v: want ErrInvalidMonitor, got %v", tg, err)
+		}
+	}
+	ok := models.Monitor{Type: models.MonitorMetric, ForSec: -5, Target: models.MonitorTarget{Metric: MetricUnitFailed}}
+	if err := validateMetric(&ok); err != nil || ok.ForSec != 0 {
+		t.Fatalf("unit_failed needs no threshold and negative for_sec clamps to 0: %v %d", err, ok.ForSec)
+	}
+}

server/internal/services/metricscope_test.go

@@ -0,0 +1,27 @@
+package services
+
+import "testing"
+
+// A restricted token must not be able to create a rule that watches servers it
+// cannot see: the per-server table and incident messages would disclose them.
+func TestSelectorWithinScope(t *testing.T) {
+	scope := map[string]string{"env": "prod"}
+	cases := []struct {
+		sel  map[string]string
+		want bool
+	}{
+		{nil, false},
+		{map[string]string{"env": "dev"}, false},
+		{map[string]string{"role": "web"}, false},
+		{map[string]string{"env": "prod"}, true},
+		{map[string]string{"env": "prod", "role": "web"}, true},
+	}
+	for _, c := range cases {
+		if got := selectorWithinScope(c.sel, scope); got != c.want {
+			t.Errorf("sel %v: got %v want %v", c.sel, got, c.want)
+		}
+	}
+	if !selectorWithinScope(nil, nil) {
+		t.Error("an unrestricted credential may use any selector, including the whole fleet")
+	}
+}

server/internal/services/metricsweep.go

@@ -0,0 +1,153 @@
+package services
+
+import (
+	"context"
+	"log"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+// EnsureMonitorServerStateIndexes declares the one key every read and upsert
+// uses. Unique, so a double sweep across a leader handover cannot fork a
+// server's state into two documents.
+func EnsureMonitorServerStateIndexes() error {
+	_, err := db.Col("monitor_server_states").Indexes().CreateOne(context.Background(), mongo.IndexModel{
+		Keys:    bson.D{{Key: "monitor_id", Value: 1}, {Key: "server_id", Value: 1}},
+		Options: options.Index().SetUnique(true),
+	})
+	if err != nil {
+		log.Printf("warning: monitor_server_states indexes: %v", err)
+	}
+	// Every heartbeat ping looks its monitor up by token hash. Sparse because
+	// only heartbeat monitors carry one.
+	_, err = db.Col("monitors").Indexes().CreateOne(context.Background(), mongo.IndexModel{
+		Keys:    bson.D{{Key: "heartbeat_token_hash", Value: 1}},
+		Options: options.Index().SetSparse(true),
+	})
+	if err != nil {
+		log.Printf("warning: monitors heartbeat_token_hash index: %v", err)
+	}
+	return nil
+}
+
+func ListMonitorServerStates(instanceID, monitorID string) ([]models.MonitorServerState, error) {
+	ctx, cancel := monCtx()
+	defer cancel()
+	cur, err := db.Col("monitor_server_states").Find(ctx, bson.M{"instance_id": instanceID, "monitor_id": monitorID})
+	if err != nil {
+		return nil, err
+	}
+	out := []models.MonitorServerState{}
+	if err := cur.All(ctx, &out); err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// ponytail: every metric monitor re-lists its servers each sweep; cache the
+// fleet per instance per sweep if large fleets show up in the sweep log.
+func SweepMetricMonitors(now time.Time) {
+	ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
+	defer cancel()
+	cur, err := db.Col("monitors").Find(ctx, bson.M{"type": models.MonitorMetric, "enabled": true})
+	if err != nil {
+		log.Printf("metrics: list: %v", err)
+		return
+	}
+	var monitors []models.Monitor
+	if err := cur.All(ctx, &monitors); err != nil {
+		log.Printf("metrics: decode: %v", err)
+		return
+	}
+	for i := range monitors {
+		sweepOneMetric(ctx, &monitors[i], now)
+	}
+}
+
+func sweepOneMetric(ctx context.Context, m *models.Monitor, now time.Time) {
+	defer func() {
+		if r := recover(); r != nil {
+			log.Printf("metrics: monitor %s panic: %v", m.MonitorID, r)
+		}
+	}()
+
+	servers, err := ListServersFiltered(m.InstanceID, m.Target.Selector)
+	if err != nil {
+		log.Printf("metrics: servers for %s: %v", m.MonitorID, err)
+		return
+	}
+	prevStates, err := ListMonitorServerStates(m.InstanceID, m.MonitorID)
+	if err != nil {
+		log.Printf("metrics: states for %s: %v", m.MonitorID, err)
+		return
+	}
+	prevByServer := map[string]*models.MonitorServerState{}
+	for i := range prevStates {
+		prevByServer[prevStates[i].ServerID] = &prevStates[i]
+	}
+
+	col := db.Col("monitor_server_states")
+	current := make([]models.MonitorServerState, 0, len(servers))
+	matched := map[string]bool{}
+	for _, srv := range servers {
+		matched[srv.ServerID] = true
+		prev := prevByServer[srv.ServerID]
+
+		var wls []models.Workload
+		if metricNeedsWorkloads(m.Target.Metric) {
+			if sw, err := GetWorkloads(m.InstanceID, srv.ServerID); err == nil && sw != nil {
+				wls = sw.Workloads
+			}
+		}
+		breach, value, msg, ok := EvaluateMetric(m.Target, srv, wls, now)
+		if !ok {
+			if prev != nil {
+				current = append(current, *prev)
+			}
+			continue
+		}
+
+		status, since := nextServerState(prev, breach, m.ForSec, now)
+		st := models.MonitorServerState{
+			InstanceID: m.InstanceID, MonitorID: m.MonitorID, ServerID: srv.ServerID,
+			Status: status, BreachSince: since, Value: value, Message: msg, UpdatedAt: now,
+		}
+		if _, err := col.ReplaceOne(ctx, bson.M{"monitor_id": m.MonitorID, "server_id": srv.ServerID}, st,
+			options.Replace().SetUpsert(true)); err != nil {
+			log.Printf("metrics: save state %s/%s: %v", m.MonitorID, srv.ServerID, err)
+			continue
+		}
+		current = append(current, st)
+
+		// A server seen for the first time has no previous status to leave, so
+		// it can open an incident but never announce a recovery.
+		prevStatus := models.StatusPending
+		if prev != nil {
+			prevStatus = prev.Status
+		}
+		applyTransition(ctx, m, srv.ServerID, srv.Hostname, prevStatus, status, msg, now)
+	}
+
+	// Servers that left the selector, or the fleet, did not recover: their
+	// incidents close quietly and their state goes.
+	for _, prev := range prevStates {
+		if matched[prev.ServerID] {
+			continue
+		}
+		if prev.Status == models.StatusDown {
+			resolveIncident(ctx, m, prev.ServerID, now)
+		}
+		col.DeleteOne(ctx, bson.M{"monitor_id": m.MonitorID, "server_id": prev.ServerID})
+	}
+
+	status, msg := rollupParent(current)
+	db.Col("monitors").UpdateOne(ctx, bson.M{"monitor_id": m.MonitorID}, bson.M{"$set": bson.M{
+		"state.status": status, "state.message": msg, "state.last_check_at": now,
+	}})
+	recordSample(ctx, m, status != models.StatusDown, 0, now)
+}

server/internal/services/mfa.go

@@ -0,0 +1,489 @@
+package services
+
+import (
+	"context"
+	"crypto/rand"
+	"crypto/sha256"
+	"crypto/subtle"
+	"encoding/base32"
+	"encoding/hex"
+	"errors"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"github.com/pquerna/otp"
+	"github.com/pquerna/otp/totp"
+	"github.com/redis/go-redis/v9"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+var (
+	ErrNoMFA        = errors.New("no multi-factor authentication is enrolled")
+	ErrBadCode      = errors.New("that code is not valid")
+	ErrCodeReplayed = errors.New("that code has already been used")
+)
+
+// Factor names travel to the browser, which chooses which prompt to draw.
+const (
+	FactorTOTP     = "totp"
+	FactorWebAuthn = "webauthn"
+	FactorRecovery = "recovery"
+	FactorPassword = "password"
+)
+
+const recoveryCodeCount = 10
+
+// RedisClient is set by main.go after auth.InitRedis succeeds. services must
+// not import auth: auth already imports services, and Go has no import
+// cycles.
+var RedisClient *redis.Client
+
+func mfaCtx() (context.Context, context.CancelFunc) {
+	return context.WithTimeout(context.Background(), 5*time.Second)
+}
+
+// UsersWithMFA returns the set of user IDs in this instance holding a factor.
+// One query per collection, not one per member, so the member list stays
+// cheap however many users an instance has.
+func UsersWithMFA(instanceID string) (map[string]bool, error) {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	out := map[string]bool{}
+
+	cur, err := db.Col("user_mfa").Find(ctx,
+		bson.M{"instance_id": instanceID, "totp_confirmed_at": bson.M{"$exists": true}})
+	if err != nil {
+		return nil, err
+	}
+	var rows []models.UserMFA
+	if err := cur.All(ctx, &rows); err != nil {
+		return nil, err
+	}
+	for _, r := range rows {
+		out[r.UserID] = true
+	}
+
+	var userIDs []string
+	if err := db.Col("webauthn_credentials").Distinct(ctx, "user_id",
+		bson.M{"instance_id": instanceID}).Decode(&userIDs); err != nil {
+		return nil, err
+	}
+	for _, id := range userIDs {
+		out[id] = true
+	}
+	return out, nil
+}
+
+// GenerateRecoveryCodes returns the codes to show the user once, and the
+// hashed records to store. The plaintext is never persisted.
+func GenerateRecoveryCodes() ([]string, []models.RecoveryCode, error) {
+	plain := make([]string, 0, recoveryCodeCount)
+	stored := make([]models.RecoveryCode, 0, recoveryCodeCount)
+	for i := 0; i < recoveryCodeCount; i++ {
+		b := make([]byte, 5)
+		if _, err := rand.Read(b); err != nil {
+			return nil, nil, err
+		}
+		// Crockford-ish base32 without padding: 8 characters, no case to get
+		// wrong when read off paper.
+		code := strings.ToLower(base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(b))
+		plain = append(plain, code)
+		stored = append(stored, models.RecoveryCode{Hash: HashRecoveryCode(code)})
+	}
+	return plain, stored, nil
+}
+
+// NormaliseRecoveryCode makes a code typed off paper comparable: no case, no
+// spaces, no dashes.
+func NormaliseRecoveryCode(code string) string {
+	r := strings.NewReplacer(" ", "", "-", "", "\t", "")
+	return strings.ToLower(r.Replace(strings.TrimSpace(code)))
+}
+
+func HashRecoveryCode(code string) string {
+	sum := sha256.Sum256([]byte(NormaliseRecoveryCode(code)))
+	return hex.EncodeToString(sum[:])
+}
+
+// ConsumeRecoveryCode reports which unused code matches, if any. It does not
+// write: the caller marks the index used so the database update and the audit
+// event stay in one place.
+func ConsumeRecoveryCode(codes []models.RecoveryCode, input string, now time.Time) (int, bool) {
+	want := HashRecoveryCode(input)
+	for i, c := range codes {
+		if c.UsedAt != nil {
+			continue
+		}
+		if subtle.ConstantTimeCompare([]byte(c.Hash), []byte(want)) == 1 {
+			return i, true
+		}
+	}
+	return 0, false
+}
+
+func GetUserMFA(instanceID, userID string) (*models.UserMFA, error) {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	var m models.UserMFA
+	err := db.Col("user_mfa").FindOne(ctx, bson.M{"instance_id": instanceID, "user_id": userID}).Decode(&m)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	return &m, nil
+}
+
+func CountPasskeys(instanceID, userID string) (int64, error) {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	return db.Col("webauthn_credentials").CountDocuments(ctx, bson.M{"instance_id": instanceID, "user_id": userID})
+}
+
+// hasFactor is the single definition of "this user has MFA". An unconfirmed
+// TOTP secret does not count: a half-finished setup must not lock anyone out.
+func hasFactor(m *models.UserMFA, passkeys int) bool {
+	if passkeys > 0 {
+		return true
+	}
+	return m != nil && m.TOTPConfirmedAt != nil
+}
+
+// MFAMethods lists the factors a user can present, newest-friendly order. An
+// empty slice means they have no MFA at all.
+func MFAMethods(instanceID, userID string) ([]string, error) {
+	m, err := GetUserMFA(instanceID, userID)
+	if err != nil {
+		return nil, err
+	}
+	passkeys, err := CountPasskeys(instanceID, userID)
+	if err != nil {
+		return nil, err
+	}
+	methods := []string{}
+	if passkeys > 0 {
+		methods = append(methods, FactorWebAuthn)
+	}
+	if m != nil && m.TOTPConfirmedAt != nil {
+		methods = append(methods, FactorTOTP)
+	}
+	if len(methods) > 0 && m != nil {
+		for _, c := range m.RecoveryCodes {
+			if c.UsedAt == nil {
+				methods = append(methods, FactorRecovery)
+				break
+			}
+		}
+	}
+	return methods, nil
+}
+
+func HasMFA(instanceID, userID string) (bool, error) {
+	m, err := GetUserMFA(instanceID, userID)
+	if err != nil {
+		return false, err
+	}
+	passkeys, err := CountPasskeys(instanceID, userID)
+	if err != nil {
+		return false, err
+	}
+	return hasFactor(m, int(passkeys)), nil
+}
+
+// WebAuthnHandle returns the user's stable random handle, creating the user_mfa
+// document on first use.
+func WebAuthnHandle(instanceID, userID string) ([]byte, error) {
+	m, err := GetUserMFA(instanceID, userID)
+	if err != nil {
+		return nil, err
+	}
+	if m != nil && len(m.WebAuthnHandle) > 0 {
+		return m.WebAuthnHandle, nil
+	}
+	handle := make([]byte, 64)
+	if _, err := rand.Read(handle); err != nil {
+		return nil, err
+	}
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	_, err = db.Col("user_mfa").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID},
+		bson.M{
+			"$set":         bson.M{"updated_at": time.Now()},
+			"$setOnInsert": bson.M{"webauthn_handle": handle},
+		},
+		options.UpdateOne().SetUpsert(true))
+	if err != nil {
+		return nil, err
+	}
+	// Re-read: a concurrent caller may have won the upsert.
+	m, err = GetUserMFA(instanceID, userID)
+	if err != nil {
+		return nil, err
+	}
+	return m.WebAuthnHandle, nil
+}
+
+// StartTOTPSetup writes an unconfirmed secret and returns it with its otpauth
+// URI. An existing unconfirmed secret is replaced; a confirmed one is not
+// touched until ConfirmTOTP succeeds against the new secret.
+func StartTOTPSetup(instanceID, userID, issuer, account string) (string, string, error) {
+	key, err := totp.Generate(totp.GenerateOpts{Issuer: issuer, AccountName: account})
+	if err != nil {
+		return "", "", err
+	}
+	enc, err := encryptString(key.Secret())
+	if err != nil {
+		return "", "", err
+	}
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	_, err = db.Col("user_mfa").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID},
+		bson.M{"$set": bson.M{"totp_pending_enc": enc, "updated_at": time.Now()}},
+		options.UpdateOne().SetUpsert(true))
+	if err != nil {
+		return "", "", err
+	}
+	return key.Secret(), key.URL(), nil
+}
+
+// ConfirmTOTP promotes the pending secret to the live one.
+func ConfirmTOTP(instanceID, userID, code string) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	var raw struct {
+		Pending string `bson:"totp_pending_enc"`
+	}
+	if err := db.Col("user_mfa").FindOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID}).Decode(&raw); err != nil {
+		return ErrNoMFA
+	}
+	if raw.Pending == "" {
+		return ErrNoMFA
+	}
+	secret, err := decryptString(raw.Pending)
+	if err != nil {
+		return err
+	}
+	if !totp.Validate(code, secret) {
+		return ErrBadCode
+	}
+	now := time.Now()
+	_, err = db.Col("user_mfa").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID},
+		bson.M{
+			"$set":   bson.M{"totp_secret_enc": raw.Pending, "totp_confirmed_at": now, "updated_at": now},
+			"$unset": bson.M{"totp_pending_enc": ""},
+		})
+	return err
+}
+
+// VerifyTOTPCode checks a code against the confirmed secret and burns it, so
+// the same code cannot be replayed inside its 30-second window by an attacker
+// who shoulder-surfed it.
+func VerifyTOTPCode(instanceID, userID, code string) error {
+	m, err := GetUserMFA(instanceID, userID)
+	if err != nil {
+		return err
+	}
+	if m == nil || m.TOTPConfirmedAt == nil || m.TOTPSecretEnc == "" {
+		return ErrNoMFA
+	}
+	secret, err := decryptString(m.TOTPSecretEnc)
+	if err != nil {
+		return err
+	}
+	step, ok := matchTOTPStep(code, secret, time.Now())
+	if !ok {
+		return ErrBadCode
+	}
+	return burnTOTPStep(userID, step)
+}
+
+// matchTOTPStep returns the 30-second time step the code was generated for,
+// trying the current step and one either side (the accepted skew). The replay
+// guard must burn this step, not the current one: burning the current step
+// would let a code accepted as s-1 or s+1 be accepted again one step later.
+func matchTOTPStep(code, secret string, now time.Time) (int64, bool) {
+	cur := now.Unix() / 30
+	for _, step := range []int64{cur - 1, cur, cur + 1} {
+		want, err := totp.GenerateCodeCustom(secret, time.Unix(step*30, 0), totp.ValidateOpts{
+			Period: 30, Digits: otp.DigitsSix, Algorithm: otp.AlgorithmSHA1,
+		})
+		if err == nil && subtle.ConstantTimeCompare([]byte(want), []byte(code)) == 1 {
+			return step, true
+		}
+	}
+	return 0, false
+}
+
+// ClearMFA removes every factor. Used by an owner or admin reset.
+func ClearMFA(instanceID, userID string) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	if _, err := db.Col("user_mfa").DeleteOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID}); err != nil {
+		return err
+	}
+	_, err := db.Col("webauthn_credentials").DeleteMany(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID})
+	return err
+}
+
+// UseRecoveryCode consumes one unused code, marking it used by index so a
+// concurrent second attempt with the same code finds it spent.
+func UseRecoveryCode(instanceID, userID, input string) error {
+	m, err := GetUserMFA(instanceID, userID)
+	if err != nil {
+		return err
+	}
+	if m == nil || len(m.RecoveryCodes) == 0 {
+		return ErrNoMFA
+	}
+	idx, ok := ConsumeRecoveryCode(m.RecoveryCodes, input, time.Now())
+	if !ok {
+		return ErrBadCode
+	}
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	now := time.Now()
+	res, err := db.Col("user_mfa").UpdateOne(ctx,
+		bson.M{
+			"instance_id": instanceID, "user_id": userID,
+			"recovery_codes." + strconv.Itoa(idx) + ".used_at": bson.M{"$exists": false},
+		},
+		bson.M{"$set": bson.M{
+			"recovery_codes." + strconv.Itoa(idx) + ".used_at": now,
+			"updated_at": now,
+		}})
+	if err != nil {
+		return err
+	}
+	if res.MatchedCount == 0 {
+		return ErrBadCode
+	}
+	return nil
+}
+
+// RequireMFAForInstance reads the policy, defaulting to off on any error: a
+// database blip must not lock an entire instance out of its own control plane.
+func RequireMFAForInstance(instanceID string) bool {
+	s, err := GetSettings(instanceID)
+	if err != nil {
+		return false
+	}
+	return models.RequireMFA(s)
+}
+
+// burnTOTPStep makes a step single-use for 90 seconds - longer than the +-1
+// step window it could still validate in. Keyed on the time step the code
+// matched, never on the code itself: a raw code sitting in a Redis key name
+// would be a currently-valid credential readable by anything that can list
+// keys. Redis is already required for sessions.
+func burnTOTPStep(userID string, step int64) error {
+	if RedisClient == nil {
+		return nil
+	}
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	key := "km:totp:" + userID + ":" + strconv.FormatInt(step, 10)
+	ok, err := RedisClient.SetNX(ctx, key, 1, 90*time.Second).Result()
+	if err != nil {
+		return err
+	}
+	if !ok {
+		return ErrCodeReplayed
+	}
+	return nil
+}
+
+// IssueRecoveryCodes replaces the user's set and returns the plaintext once.
+// Callers must not persist or log the return value.
+func IssueRecoveryCodes(instanceID, userID string) ([]string, error) {
+	plain, stored, err := GenerateRecoveryCodes()
+	if err != nil {
+		return nil, err
+	}
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	_, err = db.Col("user_mfa").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID},
+		bson.M{"$set": bson.M{"recovery_codes": stored, "updated_at": time.Now()}},
+		options.UpdateOne().SetUpsert(true))
+	if err != nil {
+		return nil, err
+	}
+	return plain, nil
+}
+
+var ErrMFARequiredByPolicy = errors.New("this instance requires multi-factor authentication; add another factor before removing this one")
+
+// canRemoveFactor is pure so the rule is testable without a database.
+// removing is FactorTOTP or FactorWebAuthn, and for a passkey it means one of
+// the counted passkeys.
+func canRemoveFactor(requireMFA, hasTOTP bool, passkeys int, removing string) error {
+	if !requireMFA {
+		return nil
+	}
+	remaining := 0
+	if hasTOTP && removing != FactorTOTP {
+		remaining++
+	}
+	switch removing {
+	case FactorWebAuthn:
+		remaining += passkeys - 1
+	default:
+		remaining += passkeys
+	}
+	if remaining > 0 {
+		return nil
+	}
+	return ErrMFARequiredByPolicy
+}
+
+// CheckCanRemoveFactor reads the current state and applies the rule.
+func CheckCanRemoveFactor(instanceID, userID, removing string) error {
+	m, err := GetUserMFA(instanceID, userID)
+	if err != nil {
+		return err
+	}
+	passkeys, err := CountPasskeys(instanceID, userID)
+	if err != nil {
+		return err
+	}
+	return canRemoveFactor(RequireMFAForInstance(instanceID), m != nil && m.TOTPConfirmedAt != nil, int(passkeys), removing)
+}
+
+// RemoveTOTP clears only the TOTP factor, leaving passkeys and recovery codes.
+func RemoveTOTP(instanceID, userID string) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	_, err := db.Col("user_mfa").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID},
+		bson.M{
+			"$unset": bson.M{"totp_secret_enc": "", "totp_confirmed_at": "", "totp_pending_enc": ""},
+			"$set":   bson.M{"updated_at": time.Now()},
+		})
+	return err
+}
+
+// RecoveryCodesRemaining counts unused codes for the account page.
+func RecoveryCodesRemaining(m *models.UserMFA) int {
+	if m == nil {
+		return 0
+	}
+	n := 0
+	for _, c := range m.RecoveryCodes {
+		if c.UsedAt == nil {
+			n++
+		}
+	}
+	return n
+}

server/internal/services/mfa_policy_test.go

@@ -0,0 +1,31 @@
+package services
+
+import "testing"
+
+// Removing a factor under an MFA policy must leave at least one behind, or the
+// user locks themselves out of an instance that will then demand enrolment
+// they cannot complete without signing in.
+func TestCanRemoveFactor(t *testing.T) {
+	cases := []struct {
+		name        string
+		requireMFA  bool
+		totp        bool
+		passkeys    int
+		removing    string
+		wantRefusal bool
+	}{
+		{"policy off, last factor", false, true, 0, FactorTOTP, false},
+		{"policy on, totp plus passkey, drop totp", true, true, 1, FactorTOTP, false},
+		{"policy on, last totp", true, true, 0, FactorTOTP, true},
+		{"policy on, last passkey", true, false, 1, FactorWebAuthn, true},
+		{"policy on, two passkeys, drop one", true, false, 2, FactorWebAuthn, false},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			err := canRemoveFactor(tc.requireMFA, tc.totp, tc.passkeys, tc.removing)
+			if tc.wantRefusal != (err != nil) {
+				t.Fatalf("canRemoveFactor refusal = %v, want %v", err != nil, tc.wantRefusal)
+			}
+		})
+	}
+}

server/internal/services/mfa_scoped_test.go

@@ -0,0 +1,31 @@
+package services
+
+import "testing"
+
+// A tenant-scoped collection missing from ScopedCollections outlives its
+// instance when the instance is purged - here that means a former customer's
+// TOTP secrets and passkeys stay in the database forever.
+func TestMFACollectionsAreScoped(t *testing.T) {
+	for _, name := range []string{"user_mfa", "webauthn_credentials"} {
+		found := false
+		for _, got := range ScopedCollections {
+			if got == name {
+				found = true
+				break
+			}
+		}
+		if !found {
+			t.Errorf("%s is not in ScopedCollections", name)
+		}
+	}
+}
+
+// Per-server metric state is tenant data and must be purged with its instance.
+func TestMonitorServerStatesAreScoped(t *testing.T) {
+	for _, got := range ScopedCollections {
+		if got == "monitor_server_states" {
+			return
+		}
+	}
+	t.Error("monitor_server_states is not in ScopedCollections")
+}

server/internal/services/mfa_test.go

@@ -0,0 +1,115 @@
+package services
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"github.com/pquerna/otp"
+	"github.com/pquerna/otp/totp"
+)
+
+func TestGenerateRecoveryCodesReturnsTenUniqueHashedCodes(t *testing.T) {
+	plain, stored, err := GenerateRecoveryCodes()
+	if err != nil {
+		t.Fatalf("generate: %v", err)
+	}
+	if len(plain) != 10 || len(stored) != 10 {
+		t.Fatalf("want 10 codes, got %d plain and %d stored", len(plain), len(stored))
+	}
+	seen := map[string]bool{}
+	for i, p := range plain {
+		if seen[p] {
+			t.Fatalf("duplicate code %q", p)
+		}
+		seen[p] = true
+		if stored[i].Hash == p {
+			t.Fatal("code stored in plaintext")
+		}
+		if stored[i].Hash != HashRecoveryCode(p) {
+			t.Fatalf("stored hash does not match HashRecoveryCode for code %d", i)
+		}
+		if stored[i].UsedAt != nil {
+			t.Fatal("fresh code marked used")
+		}
+	}
+}
+
+func TestConsumeRecoveryCodeIsSingleUseAndFormatTolerant(t *testing.T) {
+	plain, stored, _ := GenerateRecoveryCodes()
+	now := time.Now()
+
+	idx, ok := ConsumeRecoveryCode(stored, plain[3], now)
+	if !ok || idx != 3 {
+		t.Fatalf("want index 3 consumed, got idx=%d ok=%v", idx, ok)
+	}
+	used := now
+	stored[3].UsedAt = &used
+
+	if _, ok := ConsumeRecoveryCode(stored, plain[3], now); ok {
+		t.Fatal("a used recovery code was accepted a second time")
+	}
+
+	// Users retype codes with different case and stray dashes or spaces.
+	messy := " " + strings.ToUpper(plain[4]) + " "
+	if _, ok := ConsumeRecoveryCode(stored, messy, now); !ok {
+		t.Fatal("normalisation rejected a valid code")
+	}
+}
+
+func TestConsumeRecoveryCodeRejectsUnknownCode(t *testing.T) {
+	_, stored, _ := GenerateRecoveryCodes()
+	if _, ok := ConsumeRecoveryCode(stored, "not-a-real-code", time.Now()); ok {
+		t.Fatal("unknown code accepted")
+	}
+}
+
+func TestHasFactorRequiresConfirmedTOTPOrAPasskey(t *testing.T) {
+	now := time.Now()
+	cases := []struct {
+		name     string
+		mfa      *models.UserMFA
+		passkeys int
+		want     bool
+	}{
+		{"nothing", nil, 0, false},
+		{"unconfirmed totp only", &models.UserMFA{TOTPSecretEnc: "ab"}, 0, false},
+		{"confirmed totp", &models.UserMFA{TOTPSecretEnc: "ab", TOTPConfirmedAt: &now}, 0, true},
+		{"passkey only", &models.UserMFA{}, 1, true},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := hasFactor(tc.mfa, tc.passkeys); got != tc.want {
+				t.Fatalf("hasFactor = %v, want %v", got, tc.want)
+			}
+		})
+	}
+}
+
+func TestMatchTOTPStepReturnsTheStepTheCodeWasGeneratedFor(t *testing.T) {
+	const secret = "JBSWY3DPEHPK3PXP"
+	now := time.Unix(1_800_000_015, 0) // mid-step, well clear of a boundary
+	cur := now.Unix() / 30
+	for _, off := range []int64{-1, 0, 1} {
+		code, err := totp.GenerateCodeCustom(secret, time.Unix((cur+off)*30, 0), totp.ValidateOpts{
+			Period: 30, Digits: otp.DigitsSix, Algorithm: otp.AlgorithmSHA1,
+		})
+		if err != nil {
+			t.Fatal(err)
+		}
+		step, ok := matchTOTPStep(code, secret, now)
+		if !ok || step != cur+off {
+			t.Fatalf("offset %d: got step %d ok=%v, want %d", off, step, ok, cur+off)
+		}
+	}
+	far, _ := totp.GenerateCodeCustom(secret, time.Unix((cur+5)*30, 0), totp.ValidateOpts{
+		Period: 30, Digits: otp.DigitsSix, Algorithm: otp.AlgorithmSHA1,
+	})
+	if _, ok := matchTOTPStep(far, secret, now); ok {
+		t.Fatal("a code from five steps ahead must not match")
+	}
+	if _, ok := matchTOTPStep("000000x", secret, now); ok {
+		t.Fatal("a malformed code must not match")
+	}
+}

server/internal/services/migrate_instance.go

@@ -37,6 +37,7 @@ var ScopedCollections = []string{
 	"incidents",
 	"monitor_rollups",
 	"monitor_samples",
+	"monitor_server_states",
 	"notification_channels",
 	"console_sessions",
 	"audit_logs",
@@ -48,6 +49,12 @@ var ScopedCollections = []string{
 	"server_workloads",
 	"status_pages",
 	"status_incidents",
+	"maintenance_windows",
+	"patch_policies",
+	"patch_runs",
+	"patch_run_outputs",
+	"user_mfa",
+	"webauthn_credentials",
 }
 
 // collectionRenames maps the two collections whose names change. Ordered so the

server/internal/services/monitors.go

@@ -103,7 +103,8 @@ func ListServerScheduledMonitors() ([]models.Monitor, error) {
 func listMonitorsForRunner(instanceID, runner string) ([]models.Monitor, error) {
 	ctx, cancel := monCtx()
 	defer cancel()
-	filter := bson.M{"runner": runner, "enabled": true}
+	filter := bson.M{"runner": runner, "enabled": true,
+		"type": bson.M{"$nin": []string{models.MonitorMetric, models.MonitorHeartbeat}}}
 	if instanceID != "" {
 		filter["instance_id"] = instanceID
 	}
@@ -187,6 +188,32 @@ func CreateMonitor(instanceID string, m *models.Monitor, tokenScope map[string]s
 	if m.Runner == "" {
 		m.Runner = models.RunnerServer
 	}
+	if m.Type == models.MonitorHeartbeat {
+		if err := validateHeartbeat(&m.Target); err != nil {
+			return nil, err
+		}
+		tok, hash, err := NewHeartbeatToken()
+		if err != nil {
+			return nil, err
+		}
+		m.HeartbeatTokenHash = hash
+		m.HeartbeatToken = tok
+		// A heartbeat is never run, so neither where nor how often applies.
+		m.Runner = models.RunnerServer
+		m.IntervalSec = 0
+	}
+	if m.Type == models.MonitorMetric {
+		if err := validateMetric(m); err != nil {
+			return nil, err
+		}
+		if !selectorWithinScope(m.Target.Selector, tokenScope) {
+			return nil, ErrMonitorOutOfScope
+		}
+		// A metric monitor is evaluated by sweeping stored server inventory,
+		// never run against a single named agent.
+		m.Runner = models.RunnerServer
+		m.IntervalSec = 0
+	}
 	m.State = models.MonitorState{Status: models.StatusPending}
 	if _, err := db.Col("monitors").InsertOne(ctx, m); err != nil {
 		return nil, err
@@ -198,6 +225,50 @@ func UpdateMonitor(instanceID, monitorID string, upd bson.M, tokenScope map[stri
 	ctx, cancel := monCtx()
 	defer cancel()
 
+	existing, err := GetMonitor(instanceID, monitorID)
+	if err != nil {
+		return err
+	}
+	if existing == nil {
+		return ErrMonitorNotFound
+	}
+	unset, err := prepareMonitorUpdate(existing, upd)
+	if err != nil {
+		return err
+	}
+	// Editing any field of a metric monitor the token could not have created is
+	// refused before any write, so a restricted token cannot rename or disable
+	// a fleet-wide rule it cannot fully see.
+	if existing.Type == models.MonitorMetric && !selectorWithinScope(existing.Target.Selector, tokenScope) {
+		return ErrMonitorOutOfScope
+	}
+
+	if raw, present := upd["target"]; present && models.IsPassiveMonitor(existing.Type) {
+		tg, ok := raw.(models.MonitorTarget)
+		if !ok {
+			return fmt.Errorf("%w: target must be an object", ErrInvalidMonitor)
+		}
+		switch existing.Type {
+		case models.MonitorHeartbeat:
+			if err := validateHeartbeat(&tg); err != nil {
+				return err
+			}
+		case models.MonitorMetric:
+			forSec := existing.ForSec
+			if v, ok := upd["for_sec"].(int); ok {
+				forSec = v
+			}
+			probe := models.Monitor{Type: existing.Type, Target: tg, ForSec: forSec}
+			if err := validateMetric(&probe); err != nil {
+				return err
+			}
+			if !selectorWithinScope(tg.Selector, tokenScope) {
+				return ErrMonitorOutOfScope
+			}
+		}
+		upd["target"] = tg
+	}
+
 	if raw, present := upd["channel_ids"]; present {
 		ids, ok := raw.([]string)
 		if !ok {
@@ -231,13 +302,68 @@ func UpdateMonitor(instanceID, monitorID string, upd bson.M, tokenScope map[stri
 			upd["runner"] = models.RunnerServer
 		}
 	}
-	_, err := db.Col("monitors").UpdateOne(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID}, bson.M{"$set": upd})
+	change := bson.M{"$set": upd}
+	if len(unset) > 0 {
+		change["$unset"] = unset
+	}
+	_, err = db.Col("monitors").UpdateOne(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID}, change)
 	return err
 }
 
-func DeleteMonitor(instanceID, monitorID string) error {
+// ErrMonitorNotFound is returned when an update names a monitor that does not
+// exist in the instance.
+var ErrMonitorNotFound = errors.New("monitor not found")
+
+// prepareMonitorUpdate applies the checks and rewrites that need no database:
+// refusing a type change to or from a passive type, clamping for_sec, and
+// resetting a heartbeat's ping state when it is re-enabled. It returns the
+// fields to $unset.
+func prepareMonitorUpdate(existing *models.Monitor, upd bson.M) (bson.M, error) {
+	if raw, present := upd["type"]; present {
+		if t, _ := raw.(string); t != existing.Type && (models.IsPassiveMonitor(t) || models.IsPassiveMonitor(existing.Type)) {
+			bad := existing.Type
+			if models.IsPassiveMonitor(t) {
+				bad = t
+			}
+			return nil, fmt.Errorf("%w: type cannot be changed to or from %s", ErrInvalidMonitor, bad)
+		}
+	}
+	// A negative gate means nothing; treat it as "alert immediately", as
+	// create does.
+	if v, ok := upd["for_sec"].(int); ok && v < 0 {
+		upd["for_sec"] = 0
+	}
+	unset := bson.M{}
+	// A heartbeat disabled for a while still holds its last ping, so the next
+	// sweep after re-enabling would page at once. Start it over as if new.
+	if on, _ := upd["enabled"].(bool); on && !existing.Enabled && existing.Type == models.MonitorHeartbeat {
+		unset["state.last_ping_at"] = ""
+		unset["state.started_at"] = ""
+		upd["state.status"] = models.StatusPending
+	}
+	return unset, nil
+}
+
+// DeleteMonitor deletes a monitor and its associated data. For a metric
+// monitor, deletion is refused when its selector reaches beyond tokenScope -
+// the same out-of-scope check applied on create and update - so a restricted
+// token cannot remove a fleet-wide rule it could not have made. Other monitor
+// types delete as before, unaffected by tokenScope.
+func DeleteMonitor(instanceID, monitorID string, tokenScope map[string]string) error {
 	ctx, cancel := monCtx()
 	defer cancel()
+
+	existing, err := GetMonitor(instanceID, monitorID)
+	if err != nil {
+		return err
+	}
+	if existing == nil {
+		return nil
+	}
+	if existing.Type == models.MonitorMetric && !selectorWithinScope(existing.Target.Selector, tokenScope) {
+		return ErrMonitorOutOfScope
+	}
+
 	res, err := db.Col("monitors").DeleteOne(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID})
 	if err != nil {
 		return err
@@ -249,6 +375,7 @@ func DeleteMonitor(instanceID, monitorID string) error {
 	db.Col("incidents").DeleteMany(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID})
 	db.Col("monitor_rollups").DeleteMany(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID})
 	db.Col("monitor_samples").DeleteMany(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID})
+	db.Col("monitor_server_states").DeleteMany(ctx, bson.M{"monitor_id": monitorID, "instance_id": instanceID})
 	return nil
 }
 
@@ -342,24 +469,7 @@ func ingestResult(instanceID, runner, monitorID string, res checker.Result) erro
 
 	now := time.Now()
 	prev := m.State.Status
-	retries := m.Retries
-	if retries < 1 {
-		retries = 1
-	}
-
-	newStatus := prev
-	fails := m.State.Fails
-	if res.Up {
-		fails = 0
-		newStatus = models.StatusUp
-	} else {
-		fails++
-		if fails >= retries {
-			newStatus = models.StatusDown
-		} else if prev == "" || prev == models.StatusPending {
-			newStatus = models.StatusPending
-		}
-	}
+	newStatus, fails := decideStatus(prev, m.State.Fails, m.Retries, res.Up)
 
 	state := bson.M{
 		"state.status":        newStatus,
@@ -375,56 +485,36 @@ func ingestResult(instanceID, runner, monitorID string, res checker.Result) erro
 		return err
 	}
 
-	bucket := now.Truncate(time.Hour)
-	up := 0
-	if res.Up {
-		up = 1
-	}
+	recordSample(ctx, m, res.Up, res.LatencyMs, now)
 
-	/* The sample is the same result at full resolution, expiring by TTL. It is
-	   written next to the rollup rather than instead of it: the rollup is what
-	   survives, the sample is what the sub-hour views read. */
-	db.Col("monitor_samples").InsertOne(ctx, models.MonitorSample{
-		InstanceID: m.InstanceID,
-		MonitorID:  monitorID,
-		At:         now,
-		Up:         res.Up,
-		LatencyMs:  res.LatencyMs,
-	})
-
-	db.Col("monitor_rollups").UpdateOne(ctx,
-		bson.M{"monitor_id": monitorID, "period_start": bucket},
-		bson.M{
-			"$inc":         bson.M{"checks": 1, "up_count": up, "sum_latency": int64(res.LatencyMs)},
-			"$setOnInsert": bson.M{"instance_id": m.InstanceID},
-		},
-		options.UpdateOne().SetUpsert(true))
-
-	if newStatus != prev {
-		switch newStatus {
-		case models.StatusDown:
-			inc := models.Incident{
-				InstanceID: m.InstanceID,
-				IncidentID: uuid.NewString(),
-				MonitorID:  monitorID,
-				StartedAt:  now,
-				Cause:      res.Message,
-			}
-			db.Col("incidents").InsertOne(ctx, inc)
-			notifyTransition(m, newStatus, res.Message)
-		case models.StatusUp:
-			if prev == models.StatusDown {
-				db.Col("incidents").UpdateOne(ctx,
-					bson.M{"monitor_id": monitorID, "instance_id": m.InstanceID, "resolved_at": nil},
-					bson.M{"$set": bson.M{"resolved_at": now}})
-				notifyTransition(m, newStatus, res.Message)
-			}
-		}
-	}
+	applyTransition(ctx, m, "", "", prev, newStatus, res.Message, now)
 	return nil
 }
 
-func notifyTransition(m *models.Monitor, newStatus, message string) {
+// recordSample writes one result at full resolution and folds it into the
+// hourly rollup, the two records every history view reads.
+func recordSample(ctx context.Context, m *models.Monitor, up bool, latencyMs int, now time.Time) {
+	u := 0
+	if up {
+		u = 1
+	}
+	db.Col("monitor_samples").InsertOne(ctx, models.MonitorSample{
+		InstanceID: m.InstanceID,
+		MonitorID:  m.MonitorID,
+		At:         now,
+		Up:         up,
+		LatencyMs:  latencyMs,
+	})
+	db.Col("monitor_rollups").UpdateOne(ctx,
+		bson.M{"monitor_id": m.MonitorID, "period_start": now.Truncate(time.Hour)},
+		bson.M{
+			"$inc":         bson.M{"checks": 1, "up_count": u, "sum_latency": int64(latencyMs)},
+			"$setOnInsert": bson.M{"instance_id": m.InstanceID},
+		},
+		options.UpdateOne().SetUpsert(true))
+}
+
+func notifyTransition(m *models.Monitor, serverName, oldStatus, newStatus, message string) {
 	if len(m.ChannelIDs) == 0 {
 		return
 	}
@@ -435,8 +525,9 @@ func notifyTransition(m *models.Monitor, newStatus, message string) {
 	}
 	ev := notify.Event{
 		MonitorName: m.Name,
+		ServerName:  serverName,
 		Type:        m.Type,
-		OldStatus:   m.State.Status,
+		OldStatus:   oldStatus,
 		NewStatus:   newStatus,
 		Message:     message,
 		Time:        time.Now(),

server/internal/services/monitors_test.go

@@ -0,0 +1,45 @@
+package services
+
+import (
+	"errors"
+	"testing"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+func TestPrepareMonitorUpdate(t *testing.T) {
+	hb := &models.Monitor{Type: models.MonitorHeartbeat, Enabled: false}
+
+	if _, err := prepareMonitorUpdate(hb, bson.M{"type": models.MonitorHTTP}); !errors.Is(err, ErrInvalidMonitor) {
+		t.Fatalf("type change should be ErrInvalidMonitor, got %v", err)
+	}
+
+	upd := bson.M{"enabled": true}
+	unset, err := prepareMonitorUpdate(hb, upd)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, ok := unset["state.last_ping_at"]; !ok {
+		t.Fatalf("re-enable should clear last_ping_at, got %v", unset)
+	}
+	if _, ok := unset["state.started_at"]; !ok {
+		t.Fatalf("re-enable should clear started_at, got %v", unset)
+	}
+	if upd["state.status"] != models.StatusPending {
+		t.Fatalf("re-enable should set pending, got %v", upd)
+	}
+
+	on := &models.Monitor{Type: models.MonitorHeartbeat, Enabled: true}
+	if unset, _ := prepareMonitorUpdate(on, bson.M{"enabled": true}); len(unset) != 0 {
+		t.Fatalf("already enabled should not reset state, got %v", unset)
+	}
+
+	upd = bson.M{"for_sec": -5}
+	if _, err := prepareMonitorUpdate(&models.Monitor{Type: models.MonitorMetric}, upd); err != nil {
+		t.Fatal(err)
+	}
+	if upd["for_sec"] != 0 {
+		t.Fatalf("negative for_sec should clamp to 0, got %v", upd["for_sec"])
+	}
+}

server/internal/services/monitortransition.go

@@ -0,0 +1,78 @@
+package services
+
+import (
+	"context"
+	"log"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"github.com/google/uuid"
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+// decideStatus is the pull-check retry state machine: a success is always up,
+// failures below the retry count leave the status alone (or pending when there
+// is none yet), and reaching the count is down.
+func decideStatus(prev string, fails, retries int, up bool) (string, int) {
+	if retries < 1 {
+		retries = 1
+	}
+	if up {
+		return models.StatusUp, 0
+	}
+	fails++
+	if fails >= retries {
+		return models.StatusDown, fails
+	}
+	if prev == "" || prev == models.StatusPending {
+		return models.StatusPending, fails
+	}
+	return prev, fails
+}
+
+// applyTransition is the one place a status change becomes an incident and a
+// notification. serverID is empty for every monitor except a metric monitor,
+// which keeps one incident per server so each breach opens and resolves on its
+// own.
+func applyTransition(ctx context.Context, m *models.Monitor, serverID, serverName, prev, next, message string, now time.Time) {
+	if next == prev {
+		return
+	}
+	switch next {
+	case models.StatusDown:
+		inc := models.Incident{
+			InstanceID: m.InstanceID,
+			IncidentID: uuid.NewString(),
+			MonitorID:  m.MonitorID,
+			ServerID:   serverID,
+			StartedAt:  now,
+			Cause:      message,
+		}
+		if _, err := db.Col("incidents").InsertOne(ctx, inc); err != nil {
+			log.Printf("monitors: open incident for %s: %v", m.MonitorID, err)
+		}
+		notifyTransition(m, serverName, prev, next, message)
+	case models.StatusUp:
+		if prev != models.StatusDown {
+			return
+		}
+		resolveIncident(ctx, m, serverID, now)
+		notifyTransition(m, serverName, prev, next, message)
+	}
+}
+
+// resolveIncident closes the open incident for a monitor (and server, for a
+// metric monitor) without notifying. It is also used when a server stops
+// matching a metric monitor's selector: nothing recovered, so nobody is told.
+func resolveIncident(ctx context.Context, m *models.Monitor, serverID string, now time.Time) {
+	filter := bson.M{"monitor_id": m.MonitorID, "instance_id": m.InstanceID, "resolved_at": nil}
+	if serverID != "" {
+		filter["server_id"] = serverID
+	} else {
+		filter["server_id"] = bson.M{"$exists": false}
+	}
+	if _, err := db.Col("incidents").UpdateMany(ctx, filter, bson.M{"$set": bson.M{"resolved_at": now}}); err != nil {
+		log.Printf("monitors: resolve incident for %s: %v", m.MonitorID, err)
+	}
+}

server/internal/services/monitortransition_test.go

@@ -0,0 +1,36 @@
+package services
+
+import (
+	"testing"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+// decideStatus is the retry state machine lifted out of ingestResult. These
+// cases pin its existing behaviour so the refactor cannot change it.
+func TestDecideStatus(t *testing.T) {
+	cases := []struct {
+		name      string
+		prev      string
+		fails     int
+		retries   int
+		up        bool
+		wantState string
+		wantFails int
+	}{
+		{"up resets fails", models.StatusDown, 3, 2, true, models.StatusUp, 0},
+		{"first failure below retries is pending", models.StatusPending, 0, 3, false, models.StatusPending, 1},
+		{"failure below retries keeps up", models.StatusUp, 0, 3, false, models.StatusUp, 1},
+		{"failure reaching retries is down", models.StatusUp, 2, 3, false, models.StatusDown, 3},
+		{"retries below one treated as one", models.StatusUp, 0, 0, false, models.StatusDown, 1},
+		{"empty prev failing below retries is pending", "", 0, 2, false, models.StatusPending, 1},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			got, fails := decideStatus(c.prev, c.fails, c.retries, c.up)
+			if got != c.wantState || fails != c.wantFails {
+				t.Fatalf("got (%s,%d), want (%s,%d)", got, fails, c.wantState, c.wantFails)
+			}
+		})
+	}
+}

server/internal/services/passkeys.go

@@ -0,0 +1,118 @@
+package services
+
+import (
+	"encoding/hex"
+	"errors"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+)
+
+var ErrNoPasskey = errors.New("no such passkey")
+
+func ListPasskeys(instanceID, userID string) ([]models.WebAuthnCredential, error) {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	cur, err := db.Col("webauthn_credentials").Find(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID})
+	if err != nil {
+		return nil, err
+	}
+	defer cur.Close(ctx)
+	out := []models.WebAuthnCredential{}
+	if err := cur.All(ctx, &out); err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// GetPasskeyByCredentialID resolves a credential inside one instance. The
+// instance scope is not optional: an unscoped lookup would let a credential
+// registered on one tenant assert on another.
+func GetPasskeyByCredentialID(instanceID string, credID []byte) (*models.WebAuthnCredential, error) {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	var c models.WebAuthnCredential
+	err := db.Col("webauthn_credentials").FindOne(ctx,
+		bson.M{"instance_id": instanceID, "credential_id": credID}).Decode(&c)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return nil, ErrNoPasskey
+	}
+	if err != nil {
+		return nil, err
+	}
+	return &c, nil
+}
+
+func SavePasskey(instanceID, userID, name string, credID, publicKey, aaguid []byte, signCount uint32, transports []string, backupEligible, backupState bool) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	if name == "" {
+		name = "Passkey"
+	}
+	_, err := db.Col("webauthn_credentials").InsertOne(ctx, models.WebAuthnCredential{
+		InstanceID:      instanceID,
+		UserID:          userID,
+		CredentialID:    credID,
+		CredentialIDHex: hex.EncodeToString(credID),
+		PublicKey:       publicKey,
+		AAGUID:          aaguid,
+		SignCount:       signCount,
+		Transports:      transports,
+		BackupEligible:  &backupEligible,
+		BackupState:     backupState,
+		Name:            name,
+		CreatedAt:       time.Now(),
+	})
+	return err
+}
+
+// TouchPasskey records use and the new signature counter. A counter that fails
+// to advance can mean a cloned authenticator, so the caller checks it before
+// calling this.
+func TouchPasskey(instanceID string, credID []byte, signCount uint32, backupEligible, backupState bool) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	now := time.Now()
+	_, err := db.Col("webauthn_credentials").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "credential_id": credID},
+		bson.M{"$set": bson.M{
+			"sign_count": signCount, "last_used_at": now,
+			// Records eligibility on rows that predate it; unchanged otherwise,
+			// because an assertion only succeeds when it matched.
+			"backup_eligible": backupEligible, "backup_state": backupState,
+		}})
+	return err
+}
+
+func RenamePasskey(instanceID, userID, credIDHex, name string) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	res, err := db.Col("webauthn_credentials").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID, "credential_id_hex": credIDHex},
+		bson.M{"$set": bson.M{"name": name}})
+	if err != nil {
+		return err
+	}
+	if res.MatchedCount == 0 {
+		return ErrNoPasskey
+	}
+	return nil
+}
+
+func DeletePasskey(instanceID, userID, credIDHex string) error {
+	ctx, cancel := mfaCtx()
+	defer cancel()
+	res, err := db.Col("webauthn_credentials").DeleteOne(ctx,
+		bson.M{"instance_id": instanceID, "user_id": userID, "credential_id_hex": credIDHex})
+	if err != nil {
+		return err
+	}
+	if res.DeletedCount == 0 {
+		return ErrNoPasskey
+	}
+	return nil
+}

server/internal/services/patch_indexes.go

@@ -0,0 +1,46 @@
+package services
+
+import (
+	"context"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+// EnsurePatchIndexes builds the patching indexes. The command_id index is the
+// one that matters: every PatchResult is matched to its server run through it.
+func EnsurePatchIndexes() error {
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	if _, err := db.Col("maintenance_windows").Indexes().CreateOne(ctx, mongo.IndexModel{
+		Keys:    bson.D{{Key: "instance_id", Value: 1}, {Key: "window_id", Value: 1}},
+		Options: options.Index().SetUnique(true),
+	}); err != nil {
+		return err
+	}
+	if _, err := db.Col("patch_policies").Indexes().CreateMany(ctx, []mongo.IndexModel{
+		{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "policy_id", Value: 1}}, Options: options.Index().SetUnique(true)},
+		{Keys: bson.D{{Key: "enabled", Value: 1}, {Key: "next_run_at", Value: 1}}},
+		{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "window_id", Value: 1}}},
+	}); err != nil {
+		return err
+	}
+	if _, err := db.Col("patch_run_outputs").Indexes().CreateOne(ctx, mongo.IndexModel{
+		Keys:    bson.D{{Key: "instance_id", Value: 1}, {Key: "run_id", Value: 1}, {Key: "server_id", Value: 1}},
+		Options: options.Index().SetUnique(true),
+	}); err != nil {
+		return err
+	}
+	_, err := db.Col("patch_runs").Indexes().CreateMany(ctx, []mongo.IndexModel{
+		{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "run_id", Value: 1}}, Options: options.Index().SetUnique(true)},
+		{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "policy_id", Value: 1}, {Key: "started_at", Value: -1}}},
+		{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "servers.server_id", Value: 1}, {Key: "started_at", Value: -1}}},
+		{Keys: bson.D{{Key: "status", Value: 1}}},
+		{Keys: bson.D{{Key: "servers.command_id", Value: 1}}},
+	})
+	return err
+}

server/internal/services/patch_policies.go

@@ -0,0 +1,254 @@
+package services
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchsched"
+	"github.com/google/uuid"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+var (
+	ErrPolicyInvalid  = errors.New("invalid patch policy")
+	ErrPolicyNotFound = errors.New("patch policy not found")
+)
+
+// ValidatePolicy checks a policy without touching the database. The window's
+// existence and the token's tag scope are checked by Create and Update.
+func ValidatePolicy(p models.PatchPolicy) error {
+	if n := strings.TrimSpace(p.Name); n == "" || len(n) > 100 {
+		return fmt.Errorf("%w: name must be 1 to 100 characters", ErrPolicyInvalid)
+	}
+	if p.WindowID == "" {
+		return fmt.Errorf("%w: a maintenance window is required", ErrPolicyInvalid)
+	}
+	if p.Scope != models.PatchScopeAll && p.Scope != models.PatchScopeSecurity {
+		return fmt.Errorf("%w: scope must be %q or %q", ErrPolicyInvalid, models.PatchScopeAll, models.PatchScopeSecurity)
+	}
+	if p.Reboot != models.PatchRebootNever && p.Reboot != models.PatchRebootIfRequired {
+		return fmt.Errorf("%w: reboot must be %q or %q", ErrPolicyInvalid, models.PatchRebootNever, models.PatchRebootIfRequired)
+	}
+	if p.MaxConcurrent < 0 || p.MaxConcurrent > 1000 {
+		return fmt.Errorf("%w: max concurrent must be between 0 and 1000", ErrPolicyInvalid)
+	}
+	// Same rule as workflows: an empty selector matches nothing, and saying so
+	// at save time beats a policy that silently patches nobody every Sunday.
+	if len(p.TargetServerIDs) == 0 && len(p.TargetTags) == 0 {
+		return ErrNoTargets
+	}
+	return ValidateTags(p.TargetTags)
+}
+
+func nextRunFor(p models.PatchPolicy, w models.MaintenanceWindow, now time.Time) *time.Time {
+	if !p.Enabled {
+		return nil
+	}
+	start, err := patchsched.NextStart(w.Cron, w.TZ, now)
+	if err != nil {
+		return nil
+	}
+	return &start
+}
+
+func ListPolicies(instanceID string) ([]models.PatchPolicy, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	cur, err := db.Col("patch_policies").Find(ctx, bson.M{"instance_id": instanceID}, options.Find().SetSort(bson.M{"name": 1}))
+	if err != nil {
+		return nil, err
+	}
+	out := []models.PatchPolicy{}
+	return out, cur.All(ctx, &out)
+}
+
+func GetPolicy(instanceID, policyID string) (*models.PatchPolicy, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	var p models.PatchPolicy
+	err := db.Col("patch_policies").FindOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID}).Decode(&p)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return nil, ErrPolicyNotFound
+	}
+	return &p, err
+}
+
+// preparePolicy runs every check a save needs and returns the policy's window.
+func preparePolicy(instanceID string, p *models.PatchPolicy, tokenScope map[string]string) (*models.MaintenanceWindow, error) {
+	p.Name = strings.TrimSpace(p.Name)
+	if p.TargetServerIDs == nil {
+		p.TargetServerIDs = []string{}
+	}
+	if err := ValidatePolicy(*p); err != nil {
+		return nil, err
+	}
+	w, err := GetWindow(instanceID, p.WindowID)
+	if errors.Is(err, ErrWindowNotFound) {
+		return nil, fmt.Errorf("%w: that maintenance window does not exist", ErrPolicyInvalid)
+	}
+	if err != nil {
+		return nil, err
+	}
+	// validateTargetServers returns a plain "target server X not found"; wrap
+	// it so the handler can answer 400 without matching on text.
+	if err := validateTargetServers(instanceID, p.TargetServerIDs, tokenScope); err != nil {
+		return nil, fmt.Errorf("%w: %v", ErrPolicyInvalid, err)
+	}
+	if err := validateWorkflowTargetScope(instanceID, p.TargetServerIDs, p.TargetTags, tokenScope); err != nil {
+		return nil, err
+	}
+	return w, nil
+}
+
+func CreatePolicy(instanceID string, p models.PatchPolicy, tokenScope map[string]string) (*models.PatchPolicy, error) {
+	w, err := preparePolicy(instanceID, &p, tokenScope)
+	if err != nil {
+		return nil, err
+	}
+	ctx, cancel := patchCtx()
+	defer cancel()
+	now := time.Now()
+	p.InstanceID, p.PolicyID = instanceID, uuid.New().String()
+	p.CreatedAt, p.UpdatedAt = now, now
+	p.NextRunAt = nextRunFor(p, *w, now)
+	p.LastRunAt, p.LastSkipped, p.DisabledReason = nil, nil, ""
+	if _, err := db.Col("patch_policies").InsertOne(ctx, p); err != nil {
+		return nil, err
+	}
+	return &p, nil
+}
+
+func UpdatePolicy(instanceID, policyID string, p models.PatchPolicy, tokenScope map[string]string) (*models.PatchPolicy, error) {
+	// Missing policy answers 404 before any validation error.
+	if _, err := GetPolicy(instanceID, policyID); err != nil {
+		return nil, err
+	}
+	w, err := preparePolicy(instanceID, &p, tokenScope)
+	if err != nil {
+		return nil, err
+	}
+	ctx, cancel := patchCtx()
+	defer cancel()
+	now := time.Now()
+	set := bson.M{
+		"name": p.Name, "enabled": p.Enabled, "window_id": p.WindowID,
+		"target_server_ids": p.TargetServerIDs, "target_tags": p.TargetTags,
+		"scope": p.Scope, "reboot": p.Reboot, "max_concurrent": p.MaxConcurrent,
+		"notify_channel_ids": p.NotifyChannelIDs, "updated_at": now,
+	}
+	update := bson.M{"$set": set}
+	if next := nextRunFor(p, *w, now); next != nil {
+		set["next_run_at"] = *next
+		// Re-enabling is the operator's answer to whatever disabled it.
+		update["$unset"] = bson.M{"disabled_reason": ""}
+	} else {
+		update["$unset"] = bson.M{"next_run_at": ""}
+	}
+	if _, err := db.Col("patch_policies").UpdateOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID}, update); err != nil {
+		return nil, err
+	}
+	return GetPolicy(instanceID, policyID)
+}
+
+func DeletePolicy(instanceID, policyID string) error {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	res, err := db.Col("patch_policies").DeleteOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID})
+	if err != nil {
+		return err
+	}
+	if res.DeletedCount == 0 {
+		return ErrPolicyNotFound
+	}
+	return nil
+}
+
+// recomputePolicySchedules moves every enabled policy on this window to the
+// window's next start.
+func recomputePolicySchedules(ctx context.Context, w models.MaintenanceWindow) error {
+	start, err := patchsched.NextStart(w.Cron, w.TZ, time.Now())
+	if err != nil {
+		return err
+	}
+	_, err = db.Col("patch_policies").UpdateMany(ctx,
+		bson.M{"instance_id": w.InstanceID, "window_id": w.WindowID, "enabled": true},
+		bson.M{"$set": bson.M{"next_run_at": start}})
+	return err
+}
+
+// CountPolicyTargets resolves the selector as a run would, at this moment.
+func CountPolicyTargets(p models.PatchPolicy) (int, error) {
+	servers, err := ResolveTargets(p.InstanceID, p.TargetServerIDs, p.TargetTags)
+	if errors.Is(err, ErrNoTargets) {
+		return 0, nil
+	}
+	return len(servers), err
+}
+
+var ErrPatchRunActive = errors.New("a run of this policy is already in progress")
+
+// PoliciesForWindow lists every policy that uses a window, enabled or not.
+func PoliciesForWindow(instanceID, windowID string) ([]models.PatchPolicy, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	cur, err := db.Col("patch_policies").Find(ctx, bson.M{"instance_id": instanceID, "window_id": windowID})
+	if err != nil {
+		return nil, err
+	}
+	out := []models.PatchPolicy{}
+	return out, cur.All(ctx, &out)
+}
+
+// CheckWindowScope refuses a tag-restricted token changing a window when any
+// policy using it targets servers outside its restriction: moving or
+// deleting the window moves or stops those servers' patching.
+func CheckWindowScope(instanceID, windowID string, tokenScope map[string]string) error {
+	if len(tokenScope) == 0 {
+		return nil
+	}
+	ps, err := PoliciesForWindow(instanceID, windowID)
+	if err != nil {
+		return err
+	}
+	for _, p := range ps {
+		if err := CheckPolicyScope(instanceID, p, tokenScope); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// CheckPolicyScope refuses a tag-restricted token acting on a policy whose
+// targets reach outside its restriction, with the workflow rule unchanged.
+func CheckPolicyScope(instanceID string, p models.PatchPolicy, tokenScope map[string]string) error {
+	return validateWorkflowTargetScope(instanceID, p.TargetServerIDs, p.TargetTags, tokenScope)
+}
+
+// StartRunNow opens a window of the policy's usual length starting now. It is
+// how an operator tests a policy on a Tuesday afternoon.
+func StartRunNow(instanceID, policyID, actor string, tokenScope map[string]string) (*models.PatchRun, error) {
+	p, err := GetPolicy(instanceID, policyID)
+	if err != nil {
+		return nil, err
+	}
+	if err := CheckPolicyScope(instanceID, *p, tokenScope); err != nil {
+		return nil, err
+	}
+	w, err := GetWindow(instanceID, p.WindowID)
+	if err != nil {
+		return nil, err
+	}
+	ctx, cancel := patchCtx()
+	defer cancel()
+	if db.Col("patch_runs").FindOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID, "status": models.PatchRunRunning}).Err() == nil {
+		return nil, ErrPatchRunActive
+	}
+	return StartPolicyRun(*p, patchsched.WindowEnd(time.Now(), w.DurationMinutes), models.PatchSourceRunNow, actor)
+}

server/internal/services/patch_runs.go

@@ -0,0 +1,623 @@
+package services
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"log"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/notify"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchrun"
+	"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
+	"github.com/google/uuid"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+// ErrAgentOffline is declared in consoleproxy.go and reused here: both mean
+// the same thing, the target's agent is not on the command stream.
+var (
+	ErrPatchRunNotFound = errors.New("patch run not found")
+	ErrPatchRunFinished = errors.New("patch run has already finished")
+)
+
+const patchRunsCol = "patch_runs"
+
+// patchRunOutputsCol holds each server's output tail, one document per
+// (run_id, server_id). Kept out of the run document because a large run with
+// up to 64KB of output per server would pass MongoDB's 16MB document limit.
+const patchRunOutputsCol = "patch_run_outputs"
+
+// storePatchOutput upserts one server's output tail for a run.
+func storePatchOutput(ctx context.Context, instanceID, runID, serverID, output string) error {
+	_, err := db.Col(patchRunOutputsCol).UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "run_id": runID, "server_id": serverID},
+		bson.M{"$set": bson.M{"output": output, "updated_at": time.Now()}},
+		options.UpdateOne().SetUpsert(true))
+	return err
+}
+
+// fillPatchOutputs puts each server's stored output back on the run, so the
+// API response carries it as before. A run written before outputs moved out
+// keeps the output it has inline.
+func fillPatchOutputs(ctx context.Context, run *models.PatchRun) error {
+	cur, err := db.Col(patchRunOutputsCol).Find(ctx, bson.M{"instance_id": run.InstanceID, "run_id": run.RunID})
+	if err != nil {
+		return err
+	}
+	var outs []struct {
+		ServerID string `bson:"server_id"`
+		Output   string `bson:"output"`
+	}
+	if err := cur.All(ctx, &outs); err != nil {
+		return err
+	}
+	byServer := make(map[string]string, len(outs))
+	for _, o := range outs {
+		byServer[o.ServerID] = o.Output
+	}
+	for i := range run.Servers {
+		if out, ok := byServer[run.Servers[i].ServerID]; ok {
+			run.Servers[i].Output = out
+		}
+	}
+	return nil
+}
+
+// serverBootTime is the boot time the server last reported, or nil.
+func serverBootTime(ctx context.Context, instanceID, serverID string) *time.Time {
+	var doc struct {
+		Inventory struct {
+			BootTime *time.Time `bson:"boot_time"`
+		} `bson:"inventory"`
+	}
+	if err := db.Col("servers").FindOne(ctx, bson.M{"instance_id": instanceID, "server_id": serverID},
+		options.FindOne().SetProjection(bson.M{"inventory.boot_time": 1})).Decode(&doc); err != nil {
+		return nil
+	}
+	return doc.Inventory.BootTime
+}
+
+func newServerRun(s models.Server, now time.Time) models.PatchServerRun {
+	r := models.PatchServerRun{ServerID: s.ServerID, Hostname: s.Hostname, Status: models.PatchSrvQueued, PendingBefore: InstallableUpdateCount(s.AvailableUpdates)}
+	if !patchrun.AgentSupportsPatchResults(s.AgentVersion) {
+		v := s.AgentVersion
+		if v == "" {
+			v = "unknown"
+		}
+		r.Status = models.PatchSrvAgentTooOld
+		r.Error = fmt.Sprintf("agent %s predates patch results; update it to %s or later", v, patchrun.MinAgentVersion)
+		r.FinishedAt = &now
+	}
+	return r
+}
+
+// StartPolicyRun records a run for a policy and dispatches its first batch
+// immediately. The run document is written before any command leaves, so a
+// fast agent's result always finds it.
+func StartPolicyRun(p models.PatchPolicy, windowEnd time.Time, source, actor string) (*models.PatchRun, error) {
+	targets, err := ResolveTargets(p.InstanceID, p.TargetServerIDs, p.TargetTags)
+	if err != nil {
+		return nil, err
+	}
+	now := time.Now()
+	run := models.PatchRun{
+		InstanceID: p.InstanceID, RunID: uuid.New().String(), PolicyID: p.PolicyID, PolicyName: p.Name,
+		TriggeredBy: actor, Source: source, Scope: p.Scope, Reboot: p.Reboot, MaxConcurrent: p.MaxConcurrent,
+		WindowEnd: &windowEnd, Status: models.PatchRunRunning, StartedAt: now,
+	}
+	for _, s := range targets {
+		run.Servers = append(run.Servers, newServerRun(s, now))
+	}
+	ctx, cancel := patchCtx()
+	defer cancel()
+	if _, err := db.Col(patchRunsCol).InsertOne(ctx, run); err != nil {
+		return nil, err
+	}
+	LogEvent(p.InstanceID, "patch.run_started", actor, "", "",
+		fmt.Sprintf("patch policy %s started on %d servers (%s, reboot %s)", p.Name, len(run.Servers), p.Scope, p.Reboot))
+	advanceRun(ctx, run.RunID)
+	return &run, nil
+}
+
+// StartManualRun is Apply updates on one server. An agent too old to answer
+// still gets the command it always got, and the run says it cannot know the
+// outcome rather than claiming one.
+func StartManualRun(instanceID string, srv *models.Server, actor, source string) (*models.PatchRun, error) {
+	now := time.Now()
+	sr := newServerRun(*srv, now)
+	legacy := sr.Status == models.PatchSrvAgentTooOld
+	sr.Status, sr.Error, sr.FinishedAt, sr.StartedAt = models.PatchSrvPatching, "", nil, &now
+	run := models.PatchRun{
+		InstanceID: instanceID, RunID: uuid.New().String(), TriggeredBy: actor, Source: source,
+		Scope: models.PatchScopeAll, Reboot: models.PatchRebootNever, Status: models.PatchRunRunning, StartedAt: now,
+	}
+
+	ctx, cancel := patchCtx()
+	defer cancel()
+
+	var dispatchErr error
+	if !Dispatcher.IsConnected(srv.ServerID) {
+		dispatchErr = ErrAgentOffline
+	}
+	if dispatchErr == nil && legacy {
+		dispatchErr = DispatchApplyUpdates(srv.ServerID)
+		if dispatchErr == nil {
+			sr.Status = models.PatchSrvSucceeded
+			sr.Error = "no result reported: agent predates patch results"
+			sr.FinishedAt = &now
+		}
+	}
+	if dispatchErr != nil {
+		sr.Status, sr.Error, sr.FinishedAt = models.PatchSrvFailed, dispatchErr.Error(), &now
+	} else if !legacy {
+		sr.CommandID = uuid.New().String()
+	}
+	run.Servers = []models.PatchServerRun{sr}
+	if _, err := db.Col(patchRunsCol).InsertOne(ctx, run); err != nil {
+		return nil, err
+	}
+	if dispatchErr == nil && !legacy {
+		if err := dispatchPatch(srv.ServerID, sr.CommandID, run); err != nil {
+			_, _ = setServer(ctx, run.RunID, srv.ServerID, models.PatchSrvPatching,
+				bson.M{"status": models.PatchSrvFailed, "error": err.Error(), "finished_at": time.Now()})
+			dispatchErr = ErrAgentOffline
+		}
+	}
+	finalizeRun(ctx, run.RunID)
+	if dispatchErr != nil {
+		return &run, ErrAgentOffline
+	}
+	return &run, nil
+}
+
+func dispatchPatch(serverID, commandID string, run models.PatchRun) error {
+	cmd := &pb.ApplyUpdatesCmd{Scope: run.Scope, RebootIfRequired: run.Reboot == models.PatchRebootIfRequired}
+	if run.WindowEnd != nil {
+		cmd.DeadlineUnix = run.WindowEnd.Unix()
+	}
+	return Dispatcher.dispatch(serverID, &pb.ServerCommand{CommandId: commandID, ApplyUpdates: cmd})
+}
+
+// setServer updates one server run, but only if it is still in status from.
+// That guard is what stops a tick from overwriting a result that landed while
+// the tick was working.
+func setServer(ctx context.Context, runID, serverID, from string, set bson.M) (bool, error) {
+	fields := bson.M{}
+	for k, v := range set {
+		fields["servers.$."+k] = v
+	}
+	res, err := db.Col(patchRunsCol).UpdateOne(ctx,
+		bson.M{"run_id": runID, "servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": from}}},
+		bson.M{"$set": fields})
+	if err != nil {
+		return false, err
+	}
+	return res.MatchedCount > 0, nil
+}
+
+// setServerForCommand is setServer scoped to one command: a late result from
+// an older, already-superseded command (say, a fresh dispatch to the same
+// server after a failed send) must not land on the new attempt.
+func setServerForCommand(ctx context.Context, runID, serverID, commandID, from string, set bson.M) (bool, error) {
+	fields := bson.M{}
+	for k, v := range set {
+		fields["servers.$."+k] = v
+	}
+	res, err := db.Col(patchRunsCol).UpdateOne(ctx,
+		bson.M{"run_id": runID, "servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": from, "command_id": commandID}}},
+		bson.M{"$set": fields})
+	if err != nil {
+		return false, err
+	}
+	return res.MatchedCount > 0, nil
+}
+
+// claimServerForDispatch atomically moves one server from queued/waiting_offline
+// to patching, guarded on the run still being running and not cancelled: a
+// tick that loaded the run before CancelPatchRun wrote cancelled_at must not
+// dispatch to it.
+func claimServerForDispatch(ctx context.Context, runID, serverID, from, commandID string, now time.Time) (bool, error) {
+	res, err := db.Col(patchRunsCol).UpdateOne(ctx,
+		bson.M{
+			"run_id":       runID,
+			"status":       models.PatchRunRunning,
+			"cancelled_at": bson.M{"$exists": false},
+			"servers":      bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": from}},
+		},
+		bson.M{"$set": bson.M{
+			"servers.$.status":     models.PatchSrvPatching,
+			"servers.$.command_id": commandID,
+			"servers.$.started_at": now,
+		}})
+	if err != nil {
+		return false, err
+	}
+	return res.MatchedCount > 0, nil
+}
+
+// loadRun reads a run for the tick and result paths, which never need output,
+// so any inline output left on an old document is not read either.
+func loadRun(ctx context.Context, filter bson.M) (*models.PatchRun, error) {
+	var run models.PatchRun
+	err := db.Col(patchRunsCol).FindOne(ctx, filter, options.FindOne().SetProjection(bson.M{"servers.output": 0})).Decode(&run)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return nil, ErrPatchRunNotFound
+	}
+	return &run, err
+}
+
+// AdvancePatchRuns is the scheduler's second job each tick.
+func AdvancePatchRuns(ctx context.Context) {
+	cur, err := db.Col(patchRunsCol).Find(ctx, bson.M{"status": models.PatchRunRunning}, options.Find().SetProjection(bson.M{"run_id": 1}))
+	if err != nil {
+		log.Printf("patch runs: find running: %v", err)
+		return
+	}
+	var ids []struct {
+		RunID string `bson:"run_id"`
+	}
+	if err := cur.All(ctx, &ids); err != nil {
+		log.Printf("patch runs: decode running: %v", err)
+		return
+	}
+	for _, r := range ids {
+		if ctx.Err() != nil {
+			return
+		}
+		advanceRun(ctx, r.RunID)
+	}
+}
+
+func advanceRun(ctx context.Context, runID string) {
+	run, err := loadRun(ctx, bson.M{"run_id": runID})
+	if err != nil {
+		log.Printf("patch run %s: load: %v", runID, err)
+		return
+	}
+	now := time.Now()
+	connected := map[string]bool{}
+	for _, s := range run.Servers {
+		if !patchrun.IsTerminal(s.Status) {
+			connected[s.ServerID] = Dispatcher.IsConnected(s.ServerID)
+		}
+	}
+	for _, tr := range patchrun.Advance(*run, now, connected) {
+		if tr.Dispatch {
+			// Each server's claim and any reset-after-failed-send get their
+			// own fresh, short-lived context rather than sharing the caller's:
+			// advanceRun can dispatch to many servers in one tick, each ack
+			// waiting up to Dispatcher's own timeout, and a caller-supplied
+			// context (StartPolicyRun's patchCtx, or a leader context that
+			// AdvancePatchRuns loses partway through) would otherwise expire
+			// mid-loop and silently drop a later server's writes.
+			cmdID := uuid.New().String()
+			claimCtx, claimCancel := patchCtx()
+			ok, err := claimServerForDispatch(claimCtx, runID, tr.ServerID, tr.From, cmdID, now)
+			claimCancel()
+			if err != nil {
+				log.Printf("patch run %s: server %s: claim: %v", runID, tr.ServerID, err)
+				continue
+			}
+			if !ok {
+				continue
+			}
+			if err := dispatchPatch(tr.ServerID, cmdID, *run); err != nil {
+				// The agent dropped between the connection check and the send.
+				// Back to waiting: the next tick tries again while the window
+				// is open.
+				resetCtx, resetCancel := patchCtx()
+				if _, rerr := setServerForCommand(resetCtx, runID, tr.ServerID, cmdID, models.PatchSrvPatching,
+					bson.M{"status": models.PatchSrvWaitingOffline, "command_id": ""}); rerr != nil {
+					log.Printf("patch run %s: server %s: reset after failed dispatch: %v", runID, tr.ServerID, rerr)
+				}
+				resetCancel()
+			}
+			continue
+		}
+		set := bson.M{"status": tr.To}
+		if tr.Error != "" {
+			set["error"] = tr.Error
+		}
+		if patchrun.IsTerminal(tr.To) {
+			set["finished_at"] = now
+		}
+		if _, err := setServer(ctx, runID, tr.ServerID, tr.From, set); err != nil {
+			log.Printf("patch run %s: server %s: %v", runID, tr.ServerID, err)
+		}
+	}
+	finalizeRun(ctx, runID)
+}
+
+func finalizeRun(ctx context.Context, runID string) {
+	run, err := loadRun(ctx, bson.M{"run_id": runID})
+	if err != nil || run.Status != models.PatchRunRunning {
+		return
+	}
+	status, done := patchrun.Finalize(*run)
+	if !done {
+		return
+	}
+	now := time.Now()
+	res, err := db.Col(patchRunsCol).UpdateOne(ctx,
+		bson.M{"run_id": runID, "status": models.PatchRunRunning},
+		bson.M{"$set": bson.M{"status": status, "finished_at": now}})
+	if err != nil || res.MatchedCount == 0 {
+		return // finalised by someone else
+	}
+	run.Status = status
+	name := run.PolicyName
+	if name == "" {
+		name = "manual update"
+	}
+	LogEvent(run.InstanceID, "patch.run_finished", run.TriggeredBy, "", "",
+		fmt.Sprintf("patch run %s (%s) %s: %s", runID, name, status, patchrun.Summary(*run)))
+	if status == models.PatchRunPartial || status == models.PatchRunFailed ||
+		(status == models.PatchRunCancelled && hasBadOutcome(*run)) {
+		notifyPatchRun(*run)
+	}
+}
+
+// hasBadOutcome reports whether any server failed or was not reached. A
+// cancelled run with such a result still alerts: cancelling does not make a
+// failure somebody else's problem.
+func hasBadOutcome(run models.PatchRun) bool {
+	for _, s := range run.Servers {
+		switch s.Status {
+		case models.PatchSrvFailed, models.PatchSrvMissedOffline, models.PatchSrvWindowClosed:
+			return true
+		}
+	}
+	return false
+}
+
+func notifyPatchRun(run models.PatchRun) {
+	if run.PolicyID == "" {
+		return // a manual run was watched by the person who clicked
+	}
+	p, err := GetPolicy(run.InstanceID, run.PolicyID)
+	if err != nil || len(p.NotifyChannelIDs) == 0 {
+		return
+	}
+	chs, err := GetChannels(run.InstanceID, p.NotifyChannelIDs)
+	if err != nil {
+		log.Printf("patch run %s: channels: %v", run.RunID, err)
+		return
+	}
+	ev := notify.Event{
+		MonitorName: fmt.Sprintf("Patch policy %q %s", run.PolicyName, run.Status),
+		Type:        notify.TypePatch,
+		NewStatus:   run.Status,
+		Message:     fmt.Sprintf("%s (run %s)", patchrun.Summary(run), run.RunID),
+		Time:        time.Now(),
+	}
+	for _, ch := range chs {
+		if err := notify.Dispatch(ch, ev); err != nil {
+			log.Printf("patch run %s: notify %s: %v", run.RunID, ch.Name, err)
+		}
+	}
+}
+
+// RecordPatchResult is called by whichever pod holds the agent's stream. The
+// filter names this agent's own server, so one agent cannot answer for
+// another's command.
+func RecordPatchResult(instanceID, serverID string, r *pb.PatchResult) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	run, err := loadRun(ctx, bson.M{"instance_id": instanceID,
+		"servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "command_id": r.CommandId}}})
+	if err != nil {
+		log.Printf("patch result %s from %s matches no run: %v", r.CommandId, serverID, err)
+		return
+	}
+	for _, s := range run.Servers {
+		if s.ServerID != serverID || s.CommandID != r.CommandId {
+			continue
+		}
+		updated, ok := patchrun.ApplyResult(s, r, time.Now())
+		if !ok {
+			return
+		}
+		// The output goes to patch_run_outputs, not the run document.
+		set := bson.M{"status": updated.Status, "error": updated.Error}
+		if updated.PendingAfter != nil {
+			set["pending_after"] = *updated.PendingAfter
+		}
+		if updated.RebootedAt != nil {
+			set["rebooted_at"] = *updated.RebootedAt
+		}
+		if updated.Status == models.PatchSrvRebooting {
+			// The boot time before the reboot: a later report with a
+			// different one proves the restart.
+			if bt := serverBootTime(ctx, instanceID, serverID); bt != nil {
+				set["boot_time_before"] = *bt
+			}
+		}
+		if updated.FinishedAt != nil {
+			set["finished_at"] = *updated.FinishedAt
+		}
+		ok, err := setServerForCommand(ctx, run.RunID, serverID, r.CommandId, models.PatchSrvPatching, set)
+		if err != nil {
+			log.Printf("patch run %s: server %s: record result: %v", run.RunID, serverID, err)
+		}
+		if !ok {
+			return
+		}
+		if err := storePatchOutput(ctx, instanceID, run.RunID, serverID, updated.Output); err != nil {
+			log.Printf("patch run %s: server %s: store output: %v", run.RunID, serverID, err)
+		}
+		if updated.Status == models.PatchSrvRebooting {
+			LogEvent(instanceID, "patch.reboot", run.TriggeredBy, serverID, "",
+				fmt.Sprintf("%s rebooting for patch policy %s", s.Hostname, run.PolicyName))
+		}
+	}
+	finalizeRun(ctx, run.RunID)
+}
+
+// VerifyPatchReboots settles any rebooting server run for this server from a
+// static inventory report.
+func VerifyPatchReboots(instanceID, serverID string, bootTime time.Time, rebootRequired bool) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	cur, err := db.Col(patchRunsCol).Find(ctx, bson.M{"instance_id": instanceID, "status": models.PatchRunRunning,
+		"servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": models.PatchSrvRebooting}}})
+	if err != nil {
+		return
+	}
+	var runs []models.PatchRun
+	if err := cur.All(ctx, &runs); err != nil {
+		return
+	}
+	now := time.Now()
+	for _, run := range runs {
+		for _, s := range run.Servers {
+			if s.ServerID != serverID {
+				continue
+			}
+			updated, ok := patchrun.VerifyReboot(s, bootTime, rebootRequired, now)
+			if !ok {
+				continue
+			}
+			set := bson.M{"status": updated.Status, "error": updated.Error, "finished_at": now}
+			if updated.VerifiedAt != nil {
+				set["verified_at"] = *updated.VerifiedAt
+			}
+			_, _ = setServer(ctx, run.RunID, serverID, models.PatchSrvRebooting, set)
+		}
+		finalizeRun(ctx, run.RunID)
+	}
+}
+
+// CancelPatchRun stops further dispatch. Servers already patching finish:
+// killing a package manager mid-transaction is worse than letting it end.
+func CancelPatchRun(instanceID, runID string) error {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	res, err := db.Col(patchRunsCol).UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "run_id": runID, "status": models.PatchRunRunning, "cancelled_at": bson.M{"$exists": false}},
+		bson.M{"$set": bson.M{"cancelled_at": time.Now()}})
+	if err != nil {
+		return err
+	}
+	if res.MatchedCount == 0 {
+		if _, err := GetPatchRun(instanceID, runID); err != nil {
+			return err
+		}
+		return ErrPatchRunFinished
+	}
+	advanceRun(ctx, runID)
+	return nil
+}
+
+func GetPatchRun(instanceID, runID string) (*models.PatchRun, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	var run models.PatchRun
+	err := db.Col(patchRunsCol).FindOne(ctx, bson.M{"instance_id": instanceID, "run_id": runID}).Decode(&run)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return nil, ErrPatchRunNotFound
+	}
+	if err != nil {
+		return nil, err
+	}
+	if err := fillPatchOutputs(ctx, &run); err != nil {
+		return nil, err
+	}
+	return &run, nil
+}
+
+// ListPatchRuns omits output: a list of fifty runs would otherwise carry up
+// to 64KB per server.
+func ListPatchRuns(instanceID, policyID, serverID string, limit int64) ([]models.PatchRun, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	filter := bson.M{"instance_id": instanceID}
+	if policyID != "" {
+		filter["policy_id"] = policyID
+	}
+	if serverID != "" {
+		filter["servers.server_id"] = serverID
+	}
+	if limit <= 0 || limit > 200 {
+		limit = 50
+	}
+	cur, err := db.Col(patchRunsCol).Find(ctx, filter, options.Find().
+		SetSort(bson.M{"started_at": -1}).SetLimit(limit).SetProjection(bson.M{"servers.output": 0}))
+	if err != nil {
+		return nil, err
+	}
+	out := []models.PatchRun{}
+	return out, cur.All(ctx, &out)
+}
+
+// ScopePatchRun removes servers outside a tag-restricted token's reach, so a
+// run record cannot name a host the caller could not otherwise see.
+func ScopePatchRun(instanceID string, run *models.PatchRun, tokenScope map[string]string) error {
+	if len(tokenScope) == 0 {
+		return nil
+	}
+	ids := make([]string, 0, len(run.Servers))
+	for _, s := range run.Servers {
+		ids = append(ids, s.ServerID)
+	}
+	visible, err := ResolveTargetsScoped(instanceID, ids, nil, tokenScope)
+	if err != nil && !errors.Is(err, ErrNoTargets) {
+		return err
+	}
+	keep := map[string]bool{}
+	for _, s := range visible {
+		keep[s.ServerID] = true
+	}
+	kept := run.Servers[:0]
+	for _, s := range run.Servers {
+		if keep[s.ServerID] {
+			kept = append(kept, s)
+		}
+	}
+	run.Servers = kept
+	return nil
+}
+
+// sweepPatchRuns deletes finished runs past their instance's workflow log
+// retention: they are the same kind of record, and one setting governs both.
+func sweepPatchRuns() {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+	defer cancel()
+	cur, err := db.Col(patchRunsCol).Find(ctx, bson.M{"finished_at": bson.M{"$ne": nil}},
+		options.Find().SetProjection(bson.M{"run_id": 1, "instance_id": 1, "finished_at": 1}))
+	if err != nil {
+		log.Printf("patch run sweep: %v", err)
+		return
+	}
+	defer cur.Close(ctx)
+	cache := map[string]int{}
+	now := time.Now()
+	for cur.Next(ctx) {
+		var r struct {
+			RunID      string     `bson:"run_id"`
+			InstanceID string     `bson:"instance_id"`
+			FinishedAt *time.Time `bson:"finished_at"`
+		}
+		if cur.Decode(&r) != nil || r.FinishedAt == nil {
+			continue
+		}
+		days, ok := cache[r.InstanceID]
+		if !ok {
+			days = defaultRetentionDays
+			if v, err := GetWorkflowLogRetentionDays(r.InstanceID); err == nil {
+				days = v
+			}
+			cache[r.InstanceID] = days
+		}
+		if days <= 0 || !r.FinishedAt.Before(now.AddDate(0, 0, -days)) {
+			continue
+		}
+		if _, err := db.Col(patchRunsCol).DeleteOne(ctx, bson.M{"run_id": r.RunID}); err == nil {
+			_, _ = db.Col(patchRunOutputsCol).DeleteMany(ctx, bson.M{"instance_id": r.InstanceID, "run_id": r.RunID})
+		}
+	}
+}

server/internal/services/patch_scoped_test.go

@@ -0,0 +1,20 @@
+package services
+
+import "testing"
+
+// A tenant-scoped collection missing from ScopedCollections outlives its
+// instance when the instance is purged.
+func TestPatchCollectionsAreScoped(t *testing.T) {
+	for _, name := range []string{"maintenance_windows", "patch_policies", "patch_runs", "patch_run_outputs"} {
+		found := false
+		for _, got := range ScopedCollections {
+			if got == name {
+				found = true
+				break
+			}
+		}
+		if !found {
+			t.Errorf("ScopedCollections is missing %q", name)
+		}
+	}
+}

server/internal/services/patch_validate_test.go

@@ -0,0 +1,90 @@
+package services
+
+import (
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+)
+
+func goodWindow() models.MaintenanceWindow {
+	return models.MaintenanceWindow{Name: "Sunday", Cron: "0 2 * * 0", TZ: "Europe/London", DurationMinutes: 120}
+}
+
+func TestValidateWindow(t *testing.T) {
+	if err := ValidateWindow(goodWindow()); err != nil {
+		t.Fatalf("good window: %v", err)
+	}
+	bad := map[string]func(*models.MaintenanceWindow){
+		"empty name":     func(w *models.MaintenanceWindow) { w.Name = "  " },
+		"long name":      func(w *models.MaintenanceWindow) { w.Name = strings.Repeat("a", 101) },
+		"too short":      func(w *models.MaintenanceWindow) { w.DurationMinutes = 14 },
+		"too long":       func(w *models.MaintenanceWindow) { w.DurationMinutes = 721 },
+		"bad cron":       func(w *models.MaintenanceWindow) { w.Cron = "every sunday" },
+		"six-field cron": func(w *models.MaintenanceWindow) { w.Cron = "0 0 2 * * 0" },
+		"bad tz":         func(w *models.MaintenanceWindow) { w.TZ = "Mars/Olympus" },
+		"no tz":          func(w *models.MaintenanceWindow) { w.TZ = "" },
+	}
+	for name, mut := range bad {
+		w := goodWindow()
+		mut(&w)
+		if err := ValidateWindow(w); !errors.Is(err, ErrWindowInvalid) {
+			t.Errorf("%s: err = %v, want ErrWindowInvalid", name, err)
+		}
+	}
+}
+
+func TestPreviewWindowSpansDoNotOverlap(t *testing.T) {
+	from := time.Date(2026, 9, 14, 12, 0, 0, 0, time.UTC)
+	spans, err := PreviewWindow("0 * * * *", "UTC", 90, from, 3)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(spans) != 3 {
+		t.Fatalf("got %d spans", len(spans))
+	}
+	for i := 1; i < len(spans); i++ {
+		if spans[i].Start.Before(spans[i-1].End) {
+			t.Fatalf("span %d starts %s before previous ends %s", i, spans[i].Start, spans[i-1].End)
+		}
+	}
+	if !spans[0].End.Equal(spans[0].Start.Add(90 * time.Minute)) {
+		t.Fatal("end must be start + duration")
+	}
+}
+
+func goodPolicy() models.PatchPolicy {
+	return models.PatchPolicy{
+		Name: "Sunday prod", WindowID: "w1", TargetTags: map[string]string{"env": "prod"},
+		Scope: models.PatchScopeSecurity, Reboot: models.PatchRebootNever,
+	}
+}
+
+func TestValidatePolicy(t *testing.T) {
+	if err := ValidatePolicy(goodPolicy()); err != nil {
+		t.Fatalf("good policy: %v", err)
+	}
+	bad := map[string]func(*models.PatchPolicy){
+		"empty name":    func(p *models.PatchPolicy) { p.Name = "" },
+		"no window":     func(p *models.PatchPolicy) { p.WindowID = "" },
+		"bad scope":     func(p *models.PatchPolicy) { p.Scope = "everything" },
+		"bad reboot":    func(p *models.PatchPolicy) { p.Reboot = "always" },
+		"negative cap":  func(p *models.PatchPolicy) { p.MaxConcurrent = -1 },
+		"huge cap":      func(p *models.PatchPolicy) { p.MaxConcurrent = 1001 },
+		"uppercase tag": func(p *models.PatchPolicy) { p.TargetTags = map[string]string{"Env": "prod"} },
+	}
+	for name, mut := range bad {
+		p := goodPolicy()
+		mut(&p)
+		if err := ValidatePolicy(p); err == nil {
+			t.Errorf("%s: want an error", name)
+		}
+	}
+	p := goodPolicy()
+	p.TargetTags = nil
+	if err := ValidatePolicy(p); !errors.Is(err, ErrNoTargets) {
+		t.Errorf("empty selector: err = %v, want ErrNoTargets", err)
+	}
+}

server/internal/services/patch_window.go

@@ -0,0 +1,166 @@
+package services
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchsched"
+	"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
+	"github.com/google/uuid"
+	"go.mongodb.org/mongo-driver/v2/bson"
+	"go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+var (
+	ErrWindowInvalid  = errors.New("invalid maintenance window")
+	ErrWindowNotFound = errors.New("maintenance window not found")
+	ErrWindowInUse    = errors.New("maintenance window is used by a patch policy")
+)
+
+const (
+	minWindowMinutes = 15
+	maxWindowMinutes = 720
+)
+
+func patchCtx() (context.Context, context.CancelFunc) {
+	return context.WithTimeout(context.Background(), 10*time.Second)
+}
+
+func ValidateWindow(w models.MaintenanceWindow) error {
+	if n := strings.TrimSpace(w.Name); n == "" || len(n) > 100 {
+		return fmt.Errorf("%w: name must be 1 to 100 characters", ErrWindowInvalid)
+	}
+	if w.DurationMinutes < minWindowMinutes || w.DurationMinutes > maxWindowMinutes {
+		return fmt.Errorf("%w: duration must be between %d and %d minutes", ErrWindowInvalid, minWindowMinutes, maxWindowMinutes)
+	}
+	if _, err := workflowsched.ParseSchedule(w.Cron, w.TZ); err != nil {
+		return fmt.Errorf("%w: %v", ErrWindowInvalid, err)
+	}
+	return nil
+}
+
+type WindowSpan struct {
+	Start time.Time `json:"start"`
+	End   time.Time `json:"end"`
+}
+
+// PreviewWindow returns the next n windows, computed exactly as the scheduler
+// computes them, so the editor cannot disagree with what will fire.
+func PreviewWindow(cron, tz string, durationMinutes int, from time.Time, n int) ([]WindowSpan, error) {
+	if err := ValidateWindow(models.MaintenanceWindow{Name: "preview", Cron: cron, TZ: tz, DurationMinutes: durationMinutes}); err != nil {
+		return nil, err
+	}
+	out := make([]WindowSpan, 0, n)
+	for len(out) < n {
+		start, err := patchsched.NextStart(cron, tz, from)
+		if err != nil {
+			return nil, fmt.Errorf("%w: %v", ErrWindowInvalid, err)
+		}
+		end := patchsched.WindowEnd(start, durationMinutes)
+		out = append(out, WindowSpan{Start: start, End: end})
+		from = patchsched.Later(start, end)
+	}
+	return out, nil
+}
+
+func ListWindows(instanceID string) ([]models.MaintenanceWindow, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	cur, err := db.Col("maintenance_windows").Find(ctx, bson.M{"instance_id": instanceID}, options.Find().SetSort(bson.M{"name": 1}))
+	if err != nil {
+		return nil, err
+	}
+	out := []models.MaintenanceWindow{}
+	return out, cur.All(ctx, &out)
+}
+
+func GetWindow(instanceID, windowID string) (*models.MaintenanceWindow, error) {
+	w, err := LookupWindow(instanceID, windowID)
+	if err == nil && w == nil {
+		return nil, ErrWindowNotFound
+	}
+	return w, err
+}
+
+// LookupWindow is GetWindow for the scheduler, which must tell "gone" (nil,
+// nil: disable the policy) from a database error (retry next tick).
+func LookupWindow(instanceID, windowID string) (*models.MaintenanceWindow, error) {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	var w models.MaintenanceWindow
+	err := db.Col("maintenance_windows").FindOne(ctx, bson.M{"instance_id": instanceID, "window_id": windowID}).Decode(&w)
+	if errors.Is(err, mongo.ErrNoDocuments) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	return &w, nil
+}
+
+func CreateWindow(instanceID string, w models.MaintenanceWindow) (*models.MaintenanceWindow, error) {
+	w.Name = strings.TrimSpace(w.Name)
+	if err := ValidateWindow(w); err != nil {
+		return nil, err
+	}
+	ctx, cancel := patchCtx()
+	defer cancel()
+	w.InstanceID, w.WindowID = instanceID, uuid.New().String()
+	w.CreatedAt = time.Now()
+	w.UpdatedAt = w.CreatedAt
+	if _, err := db.Col("maintenance_windows").InsertOne(ctx, w); err != nil {
+		return nil, err
+	}
+	return &w, nil
+}
+
+// UpdateWindow saves the window and moves next_run_at on every enabled policy
+// that uses it, so an edited Sunday becomes the next Sunday everywhere at once.
+func UpdateWindow(instanceID, windowID string, w models.MaintenanceWindow) (*models.MaintenanceWindow, error) {
+	w.Name = strings.TrimSpace(w.Name)
+	if err := ValidateWindow(w); err != nil {
+		return nil, err
+	}
+	ctx, cancel := patchCtx()
+	defer cancel()
+	res, err := db.Col("maintenance_windows").UpdateOne(ctx,
+		bson.M{"instance_id": instanceID, "window_id": windowID},
+		bson.M{"$set": bson.M{"name": w.Name, "cron": w.Cron, "tz": w.TZ, "duration_minutes": w.DurationMinutes, "updated_at": time.Now()}})
+	if err != nil {
+		return nil, err
+	}
+	if res.MatchedCount == 0 {
+		return nil, ErrWindowNotFound
+	}
+	saved, err := GetWindow(instanceID, windowID)
+	if err != nil {
+		return nil, err
+	}
+	return saved, recomputePolicySchedules(ctx, *saved)
+}
+
+func DeleteWindow(instanceID, windowID string) error {
+	ctx, cancel := patchCtx()
+	defer cancel()
+	n, err := db.Col("patch_policies").CountDocuments(ctx, bson.M{"instance_id": instanceID, "window_id": windowID})
+	if err != nil {
+		return err
+	}
+	if n > 0 {
+		return ErrWindowInUse
+	}
+	res, err := db.Col("maintenance_windows").DeleteOne(ctx, bson.M{"instance_id": instanceID, "window_id": windowID})
+	if err != nil {
+		return err
+	}
+	if res.DeletedCount == 0 {
+		return ErrWindowNotFound
+	}
+	return nil
+}

server/internal/services/reap.go

@@ -58,12 +58,35 @@ func reapAfter() time.Duration {
 	return d
 }
 
+// freeReapFilter selects Free instances past the reap window. A locked instance
+// is excluded: it is under an open dispute in Vantage HQ, and a restore must
+// find it intact. Its fate is the dispute's to decide.
+func freeReapFilter(cutoff time.Time) bson.M {
+	return bson.M{
+		"license_tier":   license.TierFree,
+		"license_expiry": bson.M{"$ne": nil, "$lt": cutoff},
+		"locked_at":      bson.M{"$exists": false},
+	}
+}
+
+// terminatedPurgeFilter selects instances whose dispute failed in Vantage HQ.
+// Both fields are positive assertions and only HQ writes them: locked_at alone
+// is an open dispute, which must never be purged, and a self-hosted control
+// plane never has either.
+func terminatedPurgeFilter(now time.Time) bson.M {
+	return bson.M{
+		"locked_at":   bson.M{"$exists": true},
+		"purge_after": bson.M{"$exists": true, "$lt": now},
+	}
+}
+
 // purgeInstance deletes an instance and every document scoped to it.
 //
 // Unexported and unguarded: it trusts its caller completely and performs an
 // irreversible delete on whatever instance ID it is handed. The tier and expiry
 // gate - Free tier, an expiry that exists, an expiry past the window - lives in
-// ReapFreeInstances, which is the only caller. Do not export this.
+// ReapFreeInstances and ReapTerminatedInstances, which are the only callers.
+// Do not export this.
 //
 // Idempotent: re-running over a half-deleted instance completes it. The instance
 // document goes last, so an interrupted purge is retried on the next sweep
@@ -107,10 +130,7 @@ func ReapFreeInstances(ctx context.Context) (checked, purged int, err error) {
 	}
 	cutoff := time.Now().UTC().Add(-window)
 
-	cur, err := db.Col("instances").Find(ctx, bson.M{
-		"license_tier":   license.TierFree,
-		"license_expiry": bson.M{"$ne": nil, "$lt": cutoff},
-	})
+	cur, err := db.Col("instances").Find(ctx, freeReapFilter(cutoff))
 	if err != nil {
 		return 0, 0, err
 	}
@@ -152,20 +172,54 @@ func ReapFreeInstances(ctx context.Context) (checked, purged int, err error) {
 	return checked, purged, nil
 }
 
-// StartReaper sweeps once at boot, then on a ticker until ctx is cancelled, and
-// logs loudly which mode it is in.
-//
-// The pass at boot follows inject.StartReconciler's precedent and earns its keep
-// the same way: it makes a restart a supported way to force a sweep, which is
-// the only way this code can be exercised on demand - the ticker is hourly and
-// deletion is measured in days.
-func StartReaper(ctx context.Context) {
-	window := reapAfter()
-	if window == 0 {
-		log.Printf("reaper: DISABLED (FREE_INSTANCE_REAP_AFTER is unset or zero)")
-		return
+// ReapTerminatedInstances deletes instances Vantage HQ has marked for purge
+// after a failed dispute. It does not depend on FREE_INSTANCE_REAP_AFTER:
+// this is an explicit staff decision, not a policy default.
+func ReapTerminatedInstances(ctx context.Context) (checked, purged int, err error) {
+	cur, err := db.Col("instances").Find(ctx, terminatedPurgeFilter(time.Now().UTC()))
+	if err != nil {
+		return 0, 0, err
 	}
-	log.Printf("reaper: ENABLED - Free instances are deleted %s after their licence expires", window)
+	var doomed []struct {
+		InstanceID string    `bson:"instance_id"`
+		Name       string    `bson:"name"`
+		Slug       string    `bson:"slug"`
+		PurgeAfter time.Time `bson:"purge_after"`
+	}
+	if err := cur.All(ctx, &doomed); err != nil {
+		return 0, 0, err
+	}
+
+	for _, d := range doomed {
+		checked++
+		// Logged before the delete, for the same reason as the Free reaper:
+		// afterwards there is nothing left to describe.
+		log.Printf("REAPING terminated instance %s (%s, slug=%s) - dispute failed in Vantage HQ, purge authorised from %s",
+			d.InstanceID, d.Name, d.Slug, d.PurgeAfter.Format(time.RFC3339))
+		LogEvent(d.InstanceID, "instance.purged_terminated", "system", "", "",
+			fmt.Sprintf("dispute failed in Vantage HQ, purge authorised from %s", d.PurgeAfter.Format(time.RFC3339)))
+
+		counts, err := purgeInstance(ctx, d.InstanceID)
+		if err != nil {
+			log.Printf("reaper: purge of terminated %s failed after %v: %v", d.InstanceID, counts, err)
+			continue
+		}
+		purged++
+		log.Printf("reaped terminated instance %s: %v", d.InstanceID, counts)
+	}
+	return checked, purged, nil
+}
+
+// StartReaper sweeps once at boot, then hourly until ctx is cancelled. The
+// terminated sweep always runs; the Free sweep keeps its own off switch, and
+// the boot log names both modes.
+func StartReaper(ctx context.Context) {
+	if window := reapAfter(); window == 0 {
+		log.Printf("reaper: Free reaping DISABLED (FREE_INSTANCE_REAP_AFTER is unset or zero)")
+	} else {
+		log.Printf("reaper: Free reaping ENABLED - Free instances are deleted %s after their licence expires", window)
+	}
+	log.Printf("reaper: terminated purge ENABLED - instances Vantage HQ marks for purge are deleted once purge_after passes")
 
 	go func() {
 		reapOnce(ctx)
@@ -187,12 +241,14 @@ func reapOnce(ctx context.Context) {
 	runCtx, cancel := context.WithTimeout(ctx, 10*time.Minute)
 	defer cancel()
 
-	checked, purged, err := ReapFreeInstances(runCtx)
-	if err != nil {
-		log.Printf("reaper: %v", err)
-		return
+	if checked, purged, err := ReapFreeInstances(runCtx); err != nil {
+		log.Printf("reaper: free: %v", err)
+	} else if purged > 0 {
+		log.Printf("reaper: free: checked %d, purged %d", checked, purged)
 	}
-	if purged > 0 {
-		log.Printf("reaper: checked %d, purged %d", checked, purged)
+	if checked, purged, err := ReapTerminatedInstances(runCtx); err != nil {
+		log.Printf("reaper: terminated: %v", err)
+	} else if purged > 0 {
+		log.Printf("reaper: terminated: checked %d, purged %d", checked, purged)
 	}
 }

server/internal/services/reap_test.go

@@ -0,0 +1,33 @@
+package services
+
+import (
+	"reflect"
+	"testing"
+	"time"
+
+	"gitea.hostxtra.co.uk/vantage/vantage-shared/license"
+	"go.mongodb.org/mongo-driver/v2/bson"
+)
+
+func TestFreeReapFilterSkipsLockedInstances(t *testing.T) {
+	cutoff := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+	want := bson.M{
+		"license_tier":   license.TierFree,
+		"license_expiry": bson.M{"$ne": nil, "$lt": cutoff},
+		"locked_at":      bson.M{"$exists": false},
+	}
+	if got := freeReapFilter(cutoff); !reflect.DeepEqual(got, want) {
+		t.Fatalf("got %v, want %v", got, want)
+	}
+}
+
+func TestTerminatedPurgeFilterNeedsBothFields(t *testing.T) {
+	now := time.Date(2026, 9, 17, 10, 0, 0, 0, time.UTC)
+	want := bson.M{
+		"locked_at":   bson.M{"$exists": true},
+		"purge_after": bson.M{"$exists": true, "$lt": now},
+	}
+	if got := terminatedPurgeFilter(now); !reflect.DeepEqual(got, want) {
+		t.Fatalf("got %v, want %v", got, want)
+	}
+}

server/internal/services/scopes.go

@@ -11,8 +11,8 @@ import (
 // the vocabulary below.
 var ErrInvalidScope = errors.New("invalid scope")
 
-// ScopeResources is the whole vocabulary. Ten resources, each with :read and
-// :write, and write implies read on the same resource.
+// ScopeResources is the whole vocabulary. Eleven resources, each with :read
+// and :write, and write implies read on the same resource.
 //
 // It is deliberately coarse. A scope per endpoint is a table nobody maintains,
 // and a route added without an entry either fails closed and breaks, or
@@ -27,6 +27,7 @@ var ScopeResources = []string{
 	"workloads",
 	"settings",
 	"status",
+	"patching",
 	// mcp:read is permission to reach the MCP endpoint at all; mcp:write is
 	// permission for its write tools, which are not merely refused without it
 	// but omitted from tools/list entirely.

server/internal/services/secret_mask_test.go

@@ -0,0 +1,27 @@
+package services
+
+import (
+	"encoding/base64"
+	"net/url"
+	"strings"
+	"testing"
+)
+
+func TestMaskSecretsEncodedAndMultiline(t *testing.T) {
+	pem := "-----BEGIN KEY-----\nAbCdEfGh12345678\n-----END KEY-----"
+	secrets := map[string]string{"TOKEN": "p@ss w0rd/x+y", "KEY": pem, "SHORT": "ab"}
+	for _, in := range []string{
+		"raw p@ss w0rd/x+y",
+		"b64 " + base64.StdEncoding.EncodeToString([]byte("p@ss w0rd/x+y")),
+		"url " + url.QueryEscape("p@ss w0rd/x+y"),
+		"AbCdEfGh12345678",
+	} {
+		out := maskSecrets(in, secrets)
+		if !strings.Contains(out, "***") || strings.Contains(out, "w0rd") || strings.Contains(out, "AbCd") {
+			t.Errorf("not masked: %q -> %q", in, out)
+		}
+	}
+	if out := maskSecrets("about", secrets); out != "about" {
+		t.Errorf("short secret masked ordinary text: %q", out)
+	}
+}

server/internal/services/servers.go

@@ -201,6 +201,12 @@ func ValidateAgentToken(serverID, agentToken string) (*models.Server, error) {
 	if s.InstanceID == "" {
 		return nil, fmt.Errorf("server %s has no org", serverID)
 	}
+
+	// An agent of a locked instance is refused exactly like a bad token, so it
+	// keeps retrying with backoff and reconnects on its own if HQ restores it.
+	if InstanceLocked(s.InstanceID) {
+		return nil, fmt.Errorf("invalid agent token")
+	}
 	return &s, nil
 }