Compare commits

...
18 Commits
Author SHA1 Message Date
mrhid6 3f2d20868e fix(patching): final review fixes
Chart Release / chart (push) Successful in 19s
Server Deploy / deploy (push) Successful in 6m12s
- no dispatch in the last 15 minutes of a window; no-result timeout from dispatch time
- per-server output moves to patch_run_outputs (16MB document limit)
- reboot proven by a changed boot time; RebootTimeout 45m, ResultGrace 20m
- window update and delete are server-scoped against the policies using them
- scheduler puts the claim back on an error after it, so the next tick retries
- cancelled runs with failures alert; MCP apply_updates audits per server
- apply-updates 503 body documented; openapi regenerated
- web: cleared numeric fields no longer save as 0; Run now asks for confirmation
2026-09-15 13:49:28 +00:00
mrhid6 3ecea7c39f feat(web): show patch policy coverage on servers and open the run after Apply updates 2026-09-15 13:23:19 +00:00
mrhid6 68c613fd40 feat(web): patch run detail page with per-server output 2026-09-15 12:24:17 +00:00
mrhid6 70c239021d feat(web): patching page with policies, windows and runs 2026-09-15 11:19:01 +00:00
mrhid6 3a7618f82f feat(web): maintenance window and patch policy editors 2026-09-15 10:59:06 +00:00
mrhid6 7809419202 fix(web): anchor patch agent version regex to mirror server parseVersion 2026-09-15 09:43:56 +00:00
mrhid6 59e7ef63fe feat(web): patching API client, status vocabulary and navigation 2026-09-15 09:41:07 +00:00
mrhid6 b5bbf28c63 feat: patching REST API, patching scope, run IDs from apply-updates and MCP 2026-09-15 09:31:51 +00:00
mrhid6 17c9f813fc fix: patchsched hasActiveRun must not treat a real DB error as no active run 2026-09-15 09:19:51 +00:00
mrhid6 139658864b feat: patch scheduler loop; record patch results and verify reboots from the agent stream 2026-09-15 09:17:27 +00:00
mrhid6 1bb2ba7f2b fix: patch run dispatch - per-server contexts, cancel race, result command guard
Fix round 1 review findings on the patch run service:
- advanceRun no longer runs every server's dispatch claim and failed-send
  reset on the caller's shared short context; each gets its own fresh
  patchCtx(), and a failed reset write is logged instead of discarded.
- The dispatch claim (queued/waiting_offline -> patching) now also requires
  the run to still be status running with no cancelled_at, closing a race
  where a tick that loaded the run just before CancelPatchRun wrote
  cancelled_at could still dispatch.
- RecordPatchResult's write is now guarded on command_id too, so a late
  result for a superseded command cannot land on a re-dispatched attempt.
2026-09-15 09:09:12 +00:00
mrhid6 fef886c93b feat: patch run service - dispatch, results, reboot verification, cancel, alerts, retention 2026-09-15 09:04:23 +00:00
mrhid6 3a1614066e feat: maintenance window and patch policy services
Named patch_window.go (not patch_windows.go) since the _windows.go
suffix is Go's implicit GOOS build constraint and would silently
exclude the file on non-Windows builds.
2026-09-15 08:50:36 +00:00
mrhid6 c0e26d0493 feat: patchsched fire/skip decision and window arithmetic 2026-09-15 08:44:52 +00:00
mrhid6 b60daf0461 feat: patchrun - pure state machine for patch runs
Implements the patchrun package with a pure functional state machine for managing
patch runs. Contains no database dependencies - the services layer loads a run,
asks this package what should change, and writes changes guarded by expected status.

All 14 test cases pass, covering:
- Agent version parsing and support detection
- Concurrency limits and queueing
- Window deadlines and offline handling
- Result timeouts (ResultGrace, ManualTimeout, RebootTimeout)
- Reboot verification with boot time proof
- Run finalization logic
- Summary generation for alerts
2026-09-15 08:39:21 +00:00
mrhid6 8f1ea6d5a0 feat: patch models, scoped collections and indexes; pin vantage-shared v0.5.0 2026-09-15 08:27:04 +00:00
mrhid6 e63e773cda docs: implementation plan for scheduled patching; align spec audit names, alert payload and run source 2026-09-14 15:32:47 +00:00
mrhid6 48116bf737 docs: design for maintenance windows and scheduled patching 2026-09-14 14:41:24 +00:00
48 changed files with 11646 additions and 47 deletions
+58 -6
View File
@@ -173,6 +173,51 @@ Skips are recorded and surfaced, not just logged: past the 1h grace window is
`missed`, an active run is `already_running`, and a schedule that no longer
parses is disabled rather than left spinning the loop every 30 seconds forever.
### Scheduled patching
Four collections: `maintenance_windows` (cron start, IANA zone, duration),
`patch_policies` (selector, window, `all|security`, `never|if_required`,
concurrency cap, channels), `patch_runs` (one per firing or manual Apply
updates, one `servers[]` entry per target) and `patch_run_outputs` (one per
run and server, holding the package manager's output tail). The output lives
apart from the run because a large run with up to 64KB per server would pass
MongoDB's 16MB document limit; `GetPatchRun` fills `servers[].output` back in
memory so the API shape is unchanged, and the tick paths never read it. All
four are in `ScopedCollections`.
**Runs are driven by database state, not goroutines.** A run can last hours; a
goroutine-driven run is stranded at `running` when its pod dies. `patchsched`
ticks every 30s inside the housekeeping leader: it claims due policies with the
workflowsched `next_run_at` pattern, then advances every running run. Every
decision is a pure function in `internal/patchrun` (`Advance`, `ApplyResult`,
`VerifyReboot`, `Finalize`) and every write is guarded by the server run's
current status, so a result landing mid-tick is never overwritten.
**The window end never kills a package manager.** No server is dispatched in
the last `patchrun.LatestStartBeforeEnd` (15 minutes) of a window; queued and
waiting servers close at the window end as before. A server already patching
may finish past the end: its no-result timeout is its own dispatch time plus
`ManualTimeout` (2h, the agent's backstop) plus `ResultGrace` (20 minutes), for
windowed and manual runs alike. `RebootTimeout` is 45 minutes.
**Results do not cross the bus.** The pod holding the agent's stream writes
`PatchResult` straight into the run, found by `servers.command_id` and the
agent's own server ID. A reboot is settled by the first static inventory report
whose boot time differs from (is later than) `boot_time_before`, the
`inventory.boot_time` recorded when the server moved to rebooting, so host and
server clock skew does not matter. Without `boot_time_before` the report's boot
time must be later than `rebooted_at`. A report during the one-minute grace
does not count either way.
**Old agents must never receive a scope.** An agent before
`patchrun.MinAgentVersion` ignores `scope` and installs everything, so policy
runs mark it `agent_too_old` and do not dispatch. A manual Apply updates still
sends such an agent the empty command and records "no result reported".
The next window starts after `max(now, windowEnd)`, so windows never overlap,
including across a daylight-saving fall-back. `patchsched` must not import
`services`; its dependencies are injected from `main.go`.
### Server tags and workflow targeting
A server carries `tags map[string]string` - lowercase `[a-z0-9_-]`, key ≤32,
@@ -337,10 +382,12 @@ need a PowerShell Gallery install on every host and fails on an air-gapped
fleet. `CurrentVersion` is empty on Windows and `NewVersion` carries the KB
article ID: a Windows update is not a version bump of a named package.
**The agent never reboots a host.** `ApplyUpdatesCmd` installs and stops there;
`inventory.reboot_required` reports that one is owed, set on the static snapshot
every 15 minutes. Linux fills it too, from `/var/run/reboot-required` or
`dnf needs-restarting -r`.
**The agent reboots a host only when a patch command asks and a reboot is
owed.** `ApplyUpdatesCmd` carries `scope`, `reboot_if_required` and
`deadline_unix`; an empty command still means "everything, no reboot". The
agent answers with `PatchResult` and, when rebooting, sends it first and then
restarts after a one-minute grace. `inventory.reboot_required` is still set on
the static snapshot every 15 minutes and at agent start.
### Package inventory and CVE findings
@@ -887,7 +934,7 @@ service Vantage {
}
```
`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.
`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, `StepOutputChunk`, or `PatchResult`. `AgentMessage` now also carries `PatchResult`, the answer to `ApplyUpdatesCmd`.
`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`, `OpenProxyCmd`, `PingCmd`, `RefreshWorkloadsCmd`, `ControlWorkloadCmd`,
`WorkloadLogsCmd`.
@@ -955,6 +1002,11 @@ status-pages GET,POST /status-pages · GET,PUT,DELETE /status-pages/:pageId (ow
GET,POST /status-pages/:pageId/incidents
PUT,DELETE /status-pages/:pageId/incidents/:incidentId
POST /status-pages/:pageId/incidents/:incidentId/updates
patching GET,POST /maintenance-windows · POST /maintenance-windows/preview
GET,PUT,DELETE /maintenance-windows/:id (writes: owner|admin)
GET,POST /patch-policies · GET,PUT,DELETE /patch-policies/:id
POST /patch-policies/:id/run-now (writes: owner|admin)
GET /patch-runs · GET /patch-runs/:runId · POST /patch-runs/:runId/cancel
audit GET /audit
agent GET /agent/latest-version
settings GET,PUT /settings · POST /settings/secrets-token (owner|admin)
@@ -1015,7 +1067,7 @@ plane, each of which this codebase enforces:
## MongoDB Collections
`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `server_workloads` · `api_tokens` · `status_pages` · `status_incidents` · `migrations`
`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `server_workloads` · `api_tokens` · `status_pages` · `status_incidents` · `maintenance_windows` · `patch_policies` · `patch_runs` · `patch_run_outputs` · `migrations`
Every document except `migrations` carries `org_id`. Struct definitions are the source of truth - see `server/internal/models/`.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,539 @@
# Maintenance windows and scheduled patching
Date: 2026-09-14
## Goal
Let an operator say "install security updates on every `env:prod` server,
Sundays 02:00 to 04:00 Europe/London, and reboot them if the OS says a reboot
is owed", and have Vantage do it, report exactly what happened per server, and
say so loudly when it did not.
Today the only patching path is **Apply updates**, which runs immediately on one
server, installs everything, never reboots, and reports nothing back: the agent
logs a failure locally and the control plane never hears of it.
Out of scope, deliberately:
- **Update rings / staged rollout.** A policy runs all its targets within one
window, bounded only by `max_concurrent`. Rings are their own feature and
build on this one.
- **Package holds and per-package allow-lists.** Scope is `all` or `security`.
- **Live-streamed patch output.** A capped output tail is stored per server;
`workflow_log_lines` is not reused.
- **Alert muting during windows.** The window is a standalone object so this can
reference it later, but v1 does not mute anything.
- **Licence gating.** Patching is free on every tier, consistent with
`applyUpdates` already being exempt from the licence gate.
## Current state
- `POST /api/servers/:id/apply-updates` calls `services.DispatchApplyUpdates`,
which sends an empty `ApplyUpdatesCmd` and returns. No result is awaited or
recorded.
- Agent `handleApplyUpdates` runs `updates.ApplyAll()` and, on success only,
sends an empty `ReportUpdates`. Output is discarded (`exec.Cmd.Run()`).
- The apt path wraps `apt-get update` **and** `apt-get upgrade` in one 5-minute
context, so a large upgrade can be killed partway. dnf, yum, zypper, pacman
and apk run with no timeout at all.
- The agent never reboots. `inventory.reboot_required` is set on the 15-minute
static snapshot, and at agent start.
- `workflowsched` already solves scheduling: 5-field cron, IANA timezone,
`next_run_at` persisted and claimed atomically, recorded skips, run inside
`bus.RunAsLeader("housekeeping", ...)`.
- `default_steps/` ships `apply_package_updates` and `reboot_server` bash steps.
They stay as they are; this feature does not use them.
## Approach
A native patch path, not a workflow convention. The agent gains a result-bearing
update command with a scope and an opt-in reboot. Three new collections hold
windows, policies and runs. A new `patchsched` loop fires policies and advances
runs from state held in MongoDB, so a leader handover mid-window loses nothing.
The manual button and the vulnerability page move onto the same run model, so
every patch Vantage performs has a record.
Rejected alternatives:
- **Build on the workflow engine** (script steps per package manager). The patch
logic would exist twice, as scripts and as the agent's `updates` package; the
engine sends one step list to every target, so a mixed Linux/Windows policy
needs per-OS branching it does not have; and the manual button would stay
fire-and-forget.
- **Native command, recorded as a `WorkflowRun`.** Saves one page, but binds
patching to `steps_snapshot` and log sequencing that do not describe it, and
breaks the rule that a run always shows the script that ran.
## Data model
All three collections carry `instance_id` and are added to
`services.ScopedCollections`, so instance purge covers them.
### `maintenance_windows`
```go
type MaintenanceWindow struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"instance_id"`
WindowID string `bson:"window_id" json:"window_id"`
Name string `bson:"name" json:"name"`
Cron string `bson:"cron" json:"cron"` // 5-field, window start
TZ string `bson:"tz" json:"tz"` // IANA name
DurationMinutes int `bson:"duration_minutes" json:"duration_minutes"` // 15..720
CreatedAt time.Time `bson:"created_at" json:"created_at"`
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
}
```
A window answers "when" and nothing else: no targets, no behaviour. Cron and TZ
are validated through `workflowsched.NextOccurrence`, exactly as workflow
schedules are. Deleting a window referenced by any policy is refused with 409
`window_in_use`. Editing a window recomputes `next_run_at` on every policy that
references it, in the same service call.
### `patch_policies`
```go
type PatchPolicy struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"instance_id"`
PolicyID string `bson:"policy_id" json:"policy_id"`
Name string `bson:"name" json:"name"`
Enabled bool `bson:"enabled" json:"enabled"`
WindowID string `bson:"window_id" json:"window_id"`
TargetServerIDs []string `bson:"target_server_ids" json:"target_server_ids"`
TargetTags map[string]string `bson:"target_tags,omitempty" json:"target_tags,omitempty"`
Scope string `bson:"scope" json:"scope"` // "all" | "security"
Reboot string `bson:"reboot" json:"reboot"` // "never" | "if_required"
MaxConcurrent int `bson:"max_concurrent" json:"max_concurrent"` // 0 = no cap
NotifyChannelIDs []string `bson:"notify_channel_ids,omitempty" json:"notify_channel_ids,omitempty"`
NextRunAt *time.Time `bson:"next_run_at,omitempty" json:"next_run_at,omitempty"`
LastRunAt *time.Time `bson:"last_run_at,omitempty" json:"last_run_at,omitempty"`
LastSkipped *Skip `bson:"last_skipped,omitempty" json:"last_skipped,omitempty"`
DisabledReason string `bson:"disabled_reason,omitempty" json:"disabled_reason,omitempty"`
CreatedAt time.Time `bson:"created_at" json:"created_at"`
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
}
```
Targets use the workflow selector semantics unchanged: the distinct union of
named servers and tag matches, resolved through `services.ResolveTargets` at
fire time, and an empty selector matches nothing (`ErrNoTargets` on save).
`Skip` is the existing `models.Skip`.
Index: `{instance_id: 1, enabled: 1, next_run_at: 1}`.
### `patch_runs`
```go
type PatchRun struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"instance_id"`
RunID string `bson:"run_id" json:"run_id"`
PolicyID string `bson:"policy_id,omitempty" json:"policy_id,omitempty"` // empty for manual
PolicyName string `bson:"policy_name,omitempty" json:"policy_name,omitempty"`
TriggeredBy string `bson:"triggered_by" json:"triggered_by"` // "schedule" | actor
Source string `bson:"source" json:"source"` // schedule | server | vulnerabilities | mcp | run_now
CancelledAt *time.Time `bson:"cancelled_at,omitempty" json:"cancelled_at,omitempty"`
Scope string `bson:"scope" json:"scope"` // snapshot
Reboot string `bson:"reboot" json:"reboot"` // snapshot
MaxConcurrent int `bson:"max_concurrent" json:"max_concurrent"` // snapshot
WindowEnd *time.Time `bson:"window_end,omitempty" json:"window_end,omitempty"` // nil for manual
Status string `bson:"status" json:"status"` // running | succeeded | partial | failed | cancelled
StartedAt time.Time `bson:"started_at" json:"started_at"`
FinishedAt *time.Time `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
Servers []PatchServerRun `bson:"servers" json:"servers"`
}
type PatchServerRun struct {
ServerID string `bson:"server_id" json:"server_id"`
Hostname string `bson:"hostname" json:"hostname"`
Status string `bson:"status" json:"status"`
CommandID string `bson:"command_id,omitempty" json:"-"`
PendingBefore int `bson:"pending_before" json:"pending_before"`
PendingAfter *int `bson:"pending_after,omitempty" json:"pending_after,omitempty"`
RebootedAt *time.Time `bson:"rebooted_at,omitempty" json:"rebooted_at,omitempty"`
VerifiedAt *time.Time `bson:"verified_at,omitempty" json:"verified_at,omitempty"`
Output string `bson:"output,omitempty" json:"output,omitempty"` // tail, max 64KB
Error string `bson:"error,omitempty" json:"error,omitempty"`
StartedAt *time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
FinishedAt *time.Time `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
}
```
Server run statuses:
| Status | Terminal | Meaning |
| ----------------- | -------- | -------------------------------------------------------------------- |
| `queued` | no | Waiting for a concurrency slot |
| `waiting_offline` | no | Agent not connected; retried each tick while the window is open |
| `patching` | no | Command dispatched, no result yet |
| `rebooting` | no | Agent announced a reboot; awaiting a post-boot inventory report |
| `succeeded` | yes | Patched, and rebooted and verified if a reboot was owed and allowed |
| `failed` | yes | Package manager failed, no result, or reboot not verified |
| `unsupported` | yes | Security-only requested on a host with no security metadata |
| `agent_too_old` | yes | Agent predates patch results; not dispatched |
| `missed_offline` | yes | Offline for the whole window |
| `window_closed` | yes | Still queued when the window ended |
| `cancelled` | yes | Run cancelled before this server was dispatched |
Scope, reboot and concurrency are snapshotted onto the run so editing a policy
never changes what a past run shows. `PendingBefore` is the server's
`available_updates` count at dispatch; `PendingAfter` comes from the result.
Indexes: `{instance_id: 1, policy_id: 1, started_at: -1}`,
`{instance_id: 1, "servers.server_id": 1, started_at: -1}`,
`{status: 1}` (the tick's scan), and `{"servers.command_id": 1}` (result lookup).
Runs are swept under `workflow_log_retention_days` by the existing log sweeper.
## Wire contract
Changed in `vantage-shared` (`grpc/pb` and `proto/vantage/v1/vantage.proto` in
the same commit):
```proto
message ApplyUpdatesCmd {
string scope = 1; // "" or "all" | "security"
bool reboot_if_required = 2;
int64 deadline_unix = 3; // 0 = none; the agent caps at 2h
}
message PatchResult { // new AgentMessage oneof variant: PatchResult patch_result = 8;
string command_id = 1;
string status = 2; // ok | failed | unsupported | busy
string message = 3;
string output_tail = 4; // at most 64KB, newest bytes kept
int32 pending_after = 5;
bool reboot_required = 6;
bool rebooting = 7;
}
message InventoryReport {
// existing fields ...
int64 boot_time_unix = 11; // every report
}
```
The existing comment on `InventoryReport.reboot_required` ("The agent never
reboots") is reworded in the same commit to match the new reboot rule.
An empty `ApplyUpdatesCmd` means `scope: all`, no reboot, no deadline: exactly
today's behaviour, so a new agent under an old server is unaffected.
### Old agents
An old agent ignores the new fields. Under `scope: security` it would install
**everything**, and it sends no `PatchResult`. The server therefore gates on
`servers.agent_version`:
- `agentSupportsPatchResults(version)` is a semver comparison against the first
agent release carrying this feature. Empty, unparseable and dev versions are
treated as too old.
- A policy run marks an older server `agent_too_old` without dispatching.
- A manual Apply updates on an older server still dispatches the empty command,
records the server run as `succeeded` immediately with
`error: "no result reported: agent predates patch results"`, and the UI shows
it as unverified. The button keeps working through the transition.
## Agent
`internal/updates`:
- `Apply(opts ApplyOptions) (Result, error)` replaces `ApplyAll()`.
`ApplyOptions{Scope string; Deadline time.Time}`. `Result{Output []byte;
Unsupported bool}`.
- A package-level mutex: a second `Apply` while one runs returns `ErrBusy`.
- Index refresh (`apt-get update`, and nothing else) keeps its own 5-minute
timeout. The upgrade runs under a context ending at `Deadline`, or 2h from
start when no deadline is given. The existing single 5-minute context is
removed.
- Combined stdout and stderr go to a 64KB ring buffer that keeps the newest
bytes.
- Non-interactive everywhere: `DEBIAN_FRONTEND=noninteractive` and
`-o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold` for
apt, `--non-interactive` for zypper, `--noconfirm` stays for pacman.
Security-only, per package manager:
| Manager | Command | No security metadata |
| --------- | ---------------------------------------------------------------------------------------------------- | -------------------- |
| apt | temp sources list of entries whose suite ends in `-security`, from `/etc/apt/sources.list`, `*.list` and deb822 `*.sources`; `apt-get update` and `apt-get upgrade -y` with `-o Dir::Etc::SourceList=<tmp> -o Dir::Etc::SourceParts=-` | no such entries: `unsupported` |
| dnf / yum | `<pm> upgrade --security -y` | n/a |
| zypper | `zypper --non-interactive patch --category security` | n/a |
| pacman | none | always `unsupported` |
| apk | none | always `unsupported` |
| Windows | existing COM search, keeping updates whose `Categories` include Security Updates (`0FA1201D-4330-4FA8-8AE9-B877473B6441`) or Critical Updates (`E6CF1350-C01B-414D-A61F-263D14D133B4`) | n/a |
`unsupported` installs nothing. It never falls back to `all`.
apt security source filtering is a pure function over file contents
(`securitySources(files map[string]string) (list string, ok bool)`) so it is
testable on any platform.
`internal/sync` `handleApplyUpdates`:
1. `updates.Apply(...)` with the command's scope and deadline.
2. Re-run `CheckAvailable`, `ReportUpdates` the result (on success **and**
failure, so counts are fresh either way), read `RebootRequired()`.
3. If `reboot_if_required`, a reboot is owed, and at least 5 minutes remain
before the deadline: send `PatchResult{status: ok, rebooting: true, ...}` on
the command stream, then run `shutdown -r +1` (Linux) or
`shutdown /r /t 60 /c "Vantage patch policy"` (Windows). The one-minute
grace lets the result leave before the host goes down.
4. Otherwise send `PatchResult` with the outcome.
`internal/inventory` reports `boot_time_unix` on every report (Linux
`/proc/stat` `btime`, Windows via the existing collection).
The agent's documented promise changes from "never reboots a host" to "never
reboots a host unless the command explicitly asks and the OS reports a reboot
is owed". `vantage-agent/CLAUDE.md` and this repository's CLAUDE.md are updated
in the same change.
## Scheduler and run lifecycle
### `patchsched`
Runs inside the existing `RunAsLeader("housekeeping", ...)`, ticking every 30s.
It must not import `services` (the same cycle `workflowsched` avoids); its
dependencies are injected from `main.go` as `patchsched.Deps`.
Each tick does two things.
**Fire due policies.** For each enabled policy with `next_run_at <= now`, claim
it with the `workflowsched` pattern: `UpdateOne` matching `policy_id` **and**
the current `next_run_at`, setting the next occurrence. A zero match means
another process claimed it. Then `Decide(due, windowEnd, now, running, nTargets)`:
| Condition | Result |
| ----------------------------------------------------- | -------------------------------------------------- |
| window missing or no longer parses | disable policy, set `disabled_reason`, audit |
| `now >= windowEnd` or `now > due + 1h` | skip `missed` |
| a run of this policy has `status: running` | skip `already_running` |
| targets resolve to zero servers | skip `no_targets` |
| otherwise | fire |
Skips set `last_skipped` and write `patch.skipped`, as workflow skips do. Firing
creates a `patch_run` with every target `queued` (or `agent_too_old`),
`WindowEnd = due + duration`, sets `last_run_at`, and writes `patch.run_started`
with actor `schedule`.
**Advance running runs.** For each `patch_run` with `status: running`, load
connection state for its servers and apply `Advance(run, now, connected)`, a
pure function returning the transitions:
- While `now < WindowEnd` (or always, for a manual run): move `queued` and
`waiting_offline` servers to `patching` up to `MaxConcurrent` in flight
(`patching` plus `rebooting`), dispatching `ApplyUpdatesCmd` with
`deadline_unix = WindowEnd`. A dispatch that fails (503, agent offline) moves
the server to `waiting_offline`.
- When `now >= WindowEnd`: `queued` to `window_closed`, `waiting_offline` to
`missed_offline`.
- `patching` with no result past `deadline + 10m` (manual: start + 2h + 10m):
`failed`, `"no result from agent"`.
- `rebooting` with no verifying report past `RebootedAt + 20m`: `failed`,
`"did not come back within 20 minutes"`.
- When every server is terminal, `Finalize` sets the run status: `succeeded` if
all servers succeeded, `failed` if none did, otherwise `partial`.
`unsupported`, `agent_too_old`, `missed_offline` and `window_closed` are not
successes. Then notify (below) and write `patch.run_finished`.
Every transition is written with a filter on the server's current status, so a
result arriving concurrently is never overwritten by a stale tick.
### Results
`CommandStream`, on whichever pod holds the agent's stream, handles
`PatchResult` by updating the server run found by `servers.command_id`, via
`ApplyResult(serverRun, result, now)`:
- `ok` without `rebooting`: `succeeded`, `PendingAfter`, output.
- `ok` with `rebooting`: `rebooting`, `RebootedAt = now`, write `patch.reboot`.
- `failed` or `busy`: `failed` with the message.
- `unsupported`: `unsupported`.
Nothing awaits the result on the bus. The run document is the only state, so a
pod or leader change mid-window loses nothing. A result for an unknown command
ID is dropped and logged.
### Reboot verification
`ReportInventory` checks, for this server, any server run in `rebooting`, and
only on reports with `include_static` set: `reboot_required` is only computed
on static snapshots, so a metrics-only report would read as "no reboot owed".
The agent sends a static snapshot at start, so the first report after a reboot
qualifies.
`VerifyReboot(serverRun, bootTime, rebootRequired, now)`:
- `bootTime <= RebootedAt`: not yet rebooted, no change. This is why boot time
is used rather than "a report arrived": a static snapshot sent during the
one-minute grace must not count.
- `bootTime > RebootedAt` and `reboot_required` false: `succeeded`,
`VerifiedAt = now`.
- `bootTime > RebootedAt` and `reboot_required` still true: `failed`,
`"still requires a reboot after restarting"`.
### Manual runs
`POST /servers/:id/apply-updates` and the vulnerability page's Apply updates
create a one-server `patch_run` (`scope: all`, `reboot: never`, no window,
`TriggeredBy` the actor, `Source` one of `server`, `vulnerabilities` or `mcp`)
and dispatch immediately in
the handler rather than waiting for the next tick. A dispatch failure still
answers 503, and the run is recorded as `failed` so the attempt is not lost.
Timeouts and finalisation go through the same tick.
### Cancel
`POST /patch-runs/:runId/cancel` moves `queued` and `waiting_offline` servers to
`cancelled`. Servers already `patching` or `rebooting` continue: interrupting a
package manager mid-transaction is worse than letting it finish. The run
finalises as `cancelled` once the in-flight servers settle, unless none were in
flight, in which case immediately.
### Notifications
When a run finalises as `partial` or `failed`, one summary goes to each channel
in `NotifyChannelIDs` through the existing `notify` dispatch, as a new event
type `patch`: `[Vantage] Patch policy "Sunday prod" partial: 38 succeeded,
2 failed, 1 missed offline (run 3f2a...)`. The webhook payload is the existing
event shape with that summary as its message; no new payload fields. A clean
run sends nothing.
## REST API
All under `/api`, each route registered in `routeScopes` (so
`AssertScopeMapComplete` passes) and annotated for `swag`; `openapi.json` is
regenerated and committed.
```
maintenance-windows GET,POST /maintenance-windows · GET,PUT,DELETE /maintenance-windows/:id
POST /maintenance-windows/preview {cron,tz,duration_minutes} -> next 3 {start,end}
patch-policies GET,POST /patch-policies · GET,PUT,DELETE /patch-policies/:id
POST /patch-policies/:id/run-now window = now .. now + window duration
patch-runs GET /patch-runs?policy_id=&server_id=&limit= · GET /patch-runs/:runId
POST /patch-runs/:runId/cancel
servers POST /servers/:id/apply-updates 202 {run_id, message}
```
- New scope resource **`patching`** (`:read`, `:write`). Windows, policies and
runs use it. `apply-updates` stays on `servers:write`.
- Creating, editing and deleting windows and policies, and run-now, are
owner or admin (`RequireRole`). Viewing and cancelling runs are open to every
role. Apply updates keeps its current access.
- Saving a policy through a tag-restricted token is refused if its selector
reaches outside the restriction, reusing `validateWorkflowTargetScope`
unchanged. Scheduled firing passes a nil token scope, as workflow schedules do.
- `apply-updates` keeps `message` in its response for existing scripts and adds
`run_id`.
- MCP `apply_updates` returns the `run_id`. No new MCP tools.
Errors: `400` validation (`invalid_cron`, `invalid_tz`, `invalid_duration`,
`invalid_scope`, `invalid_reboot`, `no_targets`), `404` unknown window, policy
or run, `409 window_in_use`, `503` agent offline on apply-updates.
## Audit
Every event uses the `patch.` prefix so the audit page groups them under one
category: `patch.window_created|window_updated|window_deleted`,
`patch.policy_created|policy_updated|policy_deleted|policy_disabled`,
`patch.run_started`, `patch.run_finished`, `patch.skipped`, `patch.cancelled`,
`patch.reboot` (one per server Vantage reboots, naming the policy),
and the existing `updates.applied` for manual runs, now carrying the run ID.
## Frontend
`web/`, dark tokens only, no hex, pills carry shape and label.
- Sidebar: **Patching** in the Fleet group.
- `/patching`, three tabs:
- **Policies**: name, next window in its own timezone, resolved target count
via `web/lib/targets.ts`, scope and reboot chips, last-run pill, enabled
toggle. A disabled policy shows its `disabled_reason`.
- **Windows**: name, schedule in words, duration, policies using it.
- **Runs**: newest first, filter by policy.
- **Policy editor** (modal): name; window picker with inline "New window";
targets with `DualListBox` plus tag rows; scope radio, noting apk and pacman
hosts report unsupported for security-only; reboot radio, where "If required"
shows "Up to N servers may reboot during this window"; max concurrent;
notification channels. Shows "N of M targets need an agent update" linking to
the servers when any target is too old.
- **Window editor** (modal): name, presets writing cron (Nightly 02:00,
Sunday 02:00, Saturday 22:00, Monthly 1st 02:00), cron field, timezone,
duration; next 3 occurrences from `/maintenance-windows/preview`.
- `/patching/runs/[runId]`: header with status, trigger, window end and counts
per status; table of servers with status pill, updates installed
(`PendingBefore - PendingAfter`), rebooted and verified times, error; each row
expands to the output tail on the `--well` surface. Cancel while running. The
page polls while the run is `running`.
- **Server detail, OS updates panel**: "Covered by *Sunday prod*, next window
Sun 21 Sep 02:00 BST" or "Not covered by any patch policy"; last patch run
with link. Apply updates navigates to the new run.
- **Vulnerabilities**: Apply updates navigates to the new run.
## Documentation
`vantage-docs`:
- New `docs/vantage/patching.md`: windows, policies, scope per package manager
(including unsupported), reboot rule and verification, statuses table,
offline and window-close behaviour, agent version requirement.
- `vantage/servers.md`: remove the "Applying updates is not scheduled or staged"
warning, describe the run record, update the reboot sentence.
- `vantage/vulnerabilities.md`: Apply updates creates a run; link to patching.
- `hq/licensing-and-entitlements.md`: patching is available on every tier.
- `reference/api-tokens.md`: the `patching` scope.
`vantage-app/CLAUDE.md`: new "Scheduled patching" subsystem section, updated
"Inventory and OS updates", collection list, REST list, `ServerCommand` and
`AgentMessage` variants. `vantage-agent/CLAUDE.md`: the reboot promise.
## Testing
The repository tests pure functions without a database, so the logic is shaped
for that and the Mongo layer is a thin shell.
Server:
- `patchsched.Decide`: on time, late within grace, past grace, past window
end, already running, zero targets, and a DST case (`Europe/London`, last
Sunday of October, 01:30 start occurring twice; `NextOccurrence` behaviour is
asserted, not assumed).
- `patchrun.Advance`: concurrency cap never exceeded counting `rebooting`;
offline then online inside the window dispatches; offline to window end is
`missed_offline`; queued at window end is `window_closed`; nothing dispatched
after window end; no-result timeout; reboot timeout; cancelled runs dispatch
nothing; manual runs ignore window rules.
- `patchrun.ApplyResult` for each result status.
- `patchrun.VerifyReboot`: boot time before, equal to and after `RebootedAt`,
with reboot still owed and cleared.
- `patchrun.Finalize`: every mix of terminal statuses.
- `agentSupportsPatchResults`: older, equal, newer, pre-release, empty,
unparseable.
- Validation of windows and policies.
- `scopes_test` covers the new routes; a token tag-restriction case in the
style of `workflow_target_scope_test`.
Agent:
- `securitySources`: Debian and Ubuntu `.list`, Ubuntu 24.04 deb822 `.sources`,
commented lines, a file set with no security suites (`ok == false`).
- Ring buffer keeps the newest 64KB.
- Windows security category filtering, as a parser test beside `winparse_test`.
- Manual verification on Debian 12, Ubuntu 24.04, Rocky 9 and Windows Server
2022: security-only and all, reboot and verification, window close
mid-queue, agent busy.
## Rollout
1. `vantage-shared`: `ApplyUpdatesCmd` fields, `PatchResult`,
`InventoryReport.boot_time_unix`, proto and `pb` together. Release a tag.
2. `vantage-app`: bump the pin; ship collections, indexes, `patchsched`,
results handling, API, UI, MCP change and docs. With no new agents yet,
policies show every target as `agent_too_old` and the manual button works as
before, now with a run record.
3. `vantage-agent`: bump the pin, ship the agent changes, tag `agent/v*`. Set
that version as the gate constant in the server in step 2 (the gate names a
version that does not exist yet until step 3 ships, which is harmless: every
agent reads as too old until it does).
4. `vantage-docs`: publish the patching page.
+17
View File
@@ -23,7 +23,9 @@ import (
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
grpcserver "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/mcp"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/monitorsched"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchsched"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulnsched"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
@@ -179,6 +181,10 @@ func runSchemaSetup() {
log.Printf("warning: failed to ensure status page indexes: %v", err)
}
if err := services.EnsurePatchIndexes(); err != nil {
log.Printf("warning: patch indexes: %v", err)
}
if err := services.EnsureAuditIndexes(); err != nil {
log.Printf("warning: failed to ensure audit indexes: %v", err)
}
@@ -253,6 +259,17 @@ func serve() {
LogEvent: services.LogEvent,
})
patchsched.Start(jobCtx, patchsched.Deps{
LookupWindow: services.LookupWindow,
CountTargets: services.CountPolicyTargets,
StartPolicyRun: func(p models.PatchPolicy, windowEnd time.Time) error {
_, err := services.StartPolicyRun(p, windowEnd, models.PatchSourceSchedule, "schedule")
return err
},
AdvanceRuns: services.AdvancePatchRuns,
LogEvent: services.LogEvent,
})
vulnsched.Start(jobCtx, vulnsched.Deps{
LogEvent: services.LogEvent,
SendDigest: services.SendVulnDigest,
+2 -1
View File
@@ -38,6 +38,7 @@ require (
github.com/stretchr/objx v0.5.3 // indirect
github.com/stretchr/testify v1.12.1 // indirect
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
github.com/yuin/goldmark v1.8.6 // indirect
go.etcd.io/bbolt v1.5.0 // indirect
go.opentelemetry.io/otel v1.46.0 // indirect
go.opentelemetry.io/otel/trace v1.46.0 // indirect
@@ -46,7 +47,7 @@ require (
)
require (
gitea.hostxtra.co.uk/vantage/vantage-shared v0.3.3
gitea.hostxtra.co.uk/vantage/vantage-shared v0.5.0
github.com/bytedance/sonic v1.15.3 // indirect
github.com/bytedance/sonic/loader v0.5.2 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
+4 -2
View File
@@ -1,5 +1,5 @@
gitea.hostxtra.co.uk/vantage/vantage-shared v0.3.3 h1:+ak67Hj1C92hNfTE63nlF1c2wx4L8mML+1aI6VJnukc=
gitea.hostxtra.co.uk/vantage/vantage-shared v0.3.3/go.mod h1:dWjeOFLltQ8sv9Pnn1xRxGfWGgqa2fkG0esuaJLoPXQ=
gitea.hostxtra.co.uk/vantage/vantage-shared v0.5.0 h1:xwSIEkQKTd4Qk+BYHvoGN+h84Isr2h5qqnitUWF1m2w=
gitea.hostxtra.co.uk/vantage/vantage-shared v0.5.0/go.mod h1:Zo66XhqF8No3dveIowLCepvMxVg8KnhsNMz0k0Xpuck=
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986 h1:2a30xLN2sUZcMXl50hg+PJCIDdJgIvIbVcKqLJ/ZrtM=
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986/go.mod h1:NT+jyeCzXk6vXR5MTkdn4z64TgGfE5HMLC8qfj5unl8=
github.com/aquasecurity/trivy-db v0.0.0-20260813095258-0e0340a01b57 h1:A3Lz/9ip/qigafSxqBWcu7S8i+tJbQS7DB2V0XibOKs=
@@ -152,6 +152,8 @@ github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT0
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfSfmXjznFBSZNN13rSJjlIOI1fUNAtF7rmI=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
File diff suppressed because it is too large Load Diff
+24 -12
View File
@@ -187,6 +187,7 @@ func RegisterRoutes(r *gin.Engine) {
apiGroup.GET("/servers/:id/workloads/:wid/logs", auth.RequireRole("owner", "admin"), getWorkloadLogs)
registerStatusPageRoutes(apiGroup)
registerPatchingRoutes(apiGroup)
}
}
@@ -748,30 +749,41 @@ func updateAgent(c *gin.Context) {
// applyUpdates godoc
//
// @Summary Apply pending OS updates on a server
// @Description Dispatches ApplyUpdatesCmd. Exempt from the licence gate: security patching is never paywalled.
// @Description Starts a manual patch run (all updates, no reboot) and returns its ID. Exempt from the licence gate: security patching is never paywalled.
// @Tags servers
// @Produce json
// @Param id path string true "Server ID"
// @Success 202 {object} MessageResponse
// @Failure 404 {object} ErrorResponse
// @Failure 503 {object} ErrorResponse
// @Param id path string true "Server ID"
// @Param source query string false "vulnerabilities when started from the vulnerabilities page"
// @Success 202 {object} ApplyUpdatesResponse
// @Failure 404 {object} ErrorResponse
// @Failure 503 {object} ApplyUpdatesErrorResponse "agent offline; the attempt is recorded as run_id"
// @Security cookieAuth
// @Security bearerAuth
// @Router /servers/{id}/apply-updates [post]
func applyUpdates(c *gin.Context) {
id := c.Param("id")
s, err := services.GetServerScoped(auth.InstanceID(c), id, auth.ServerScope(c))
instanceID := auth.InstanceID(c)
s, err := services.GetServerScoped(instanceID, c.Param("id"), auth.ServerScope(c))
if err != nil {
c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
return
}
if err := services.DispatchApplyUpdates(s.ServerID); err != nil {
c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error()})
source := models.PatchSourceServer
if c.Query("source") == models.PatchSourceVulnerabilities {
source = models.PatchSourceVulnerabilities
}
run, err := services.StartManualRun(instanceID, s, actorFromCtx(c), source)
if errors.Is(err, services.ErrAgentOffline) {
// The attempt is still recorded, so it has a run ID to show.
c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error(), "run_id": run.RunID})
return
}
services.LogEvent(auth.InstanceID(c), "updates.applied", actorFromCtx(c), s.ServerID, "", fmt.Sprintf("package update command dispatched to %s", s.Hostname))
c.JSON(http.StatusAccepted, MessageResponse{Message: "apply updates command sent to agent"})
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
services.LogEvent(instanceID, "updates.applied", actorFromCtx(c), s.ServerID, "",
fmt.Sprintf("package update run %s started on %s", run.RunID, s.Hostname))
c.JSON(http.StatusAccepted, ApplyUpdatesResponse{Message: "apply updates command sent to agent", RunID: run.RunID})
}
// handleUpdateScript serves a dynamically generated shell script that
+455
View File
@@ -0,0 +1,455 @@
package api
import (
"errors"
"fmt"
"net/http"
"strconv"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
"github.com/gin-gonic/gin"
)
// registerPatchingRoutes mounts maintenance windows, patch policies and patch
// runs. Free on every tier: security patching is never paywalled, so no
// RequireFeature here. Writes that decide what reboots and when are owner or
// admin; watching and cancelling a run is open to every role.
func registerPatchingRoutes(g *gin.RouterGroup) {
admin := auth.RequireRole("owner", "admin")
g.GET("/maintenance-windows", listWindows)
g.POST("/maintenance-windows", admin, createWindow)
g.POST("/maintenance-windows/preview", previewWindow)
g.GET("/maintenance-windows/:id", getWindow)
g.PUT("/maintenance-windows/:id", admin, updateWindow)
g.DELETE("/maintenance-windows/:id", admin, deleteWindow)
g.GET("/patch-policies", listPolicies)
g.POST("/patch-policies", admin, createPolicy)
g.GET("/patch-policies/:id", getPolicy)
g.PUT("/patch-policies/:id", admin, updatePolicy)
g.DELETE("/patch-policies/:id", admin, deletePolicy)
g.POST("/patch-policies/:id/run-now", admin, runPolicyNow)
g.GET("/patch-runs", listPatchRuns)
g.GET("/patch-runs/:runId", getPatchRun)
g.POST("/patch-runs/:runId/cancel", cancelPatchRun)
}
// patchError maps every patching service error once.
func patchError(c *gin.Context, err error) {
switch {
case errors.Is(err, services.ErrWindowNotFound), errors.Is(err, services.ErrPolicyNotFound), errors.Is(err, services.ErrPatchRunNotFound):
c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
case errors.Is(err, services.ErrWindowInUse), errors.Is(err, services.ErrPatchRunActive), errors.Is(err, services.ErrPatchRunFinished):
c.JSON(http.StatusConflict, gin.H{"error": err.Error()})
case errors.Is(err, services.ErrWindowInvalid), errors.Is(err, services.ErrPolicyInvalid),
errors.Is(err, services.ErrNoTargets), errors.Is(err, services.ErrInvalidTag),
errors.Is(err, services.ErrWorkflowTargetOutOfScope):
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
default:
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
}
}
// listWindows godoc
//
// @Summary List maintenance windows
// @Tags patching
// @Produce json
// @Success 200 {array} models.MaintenanceWindow
// @Failure 500 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /maintenance-windows [get]
func listWindows(c *gin.Context) {
ws, err := services.ListWindows(auth.InstanceID(c))
if err != nil {
patchError(c, err)
return
}
c.JSON(http.StatusOK, ws)
}
// createWindow godoc
//
// @Summary Create a maintenance window
// @Tags patching
// @Accept json
// @Produce json
// @Param body body models.MaintenanceWindow true "Window"
// @Success 201 {object} models.MaintenanceWindow
// @Failure 400 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /maintenance-windows [post]
func createWindow(c *gin.Context) {
var body models.MaintenanceWindow
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
w, err := services.CreateWindow(auth.InstanceID(c), body)
if err != nil {
patchError(c, err)
return
}
services.LogEvent(auth.InstanceID(c), "patch.window_created", actorFromCtx(c), "", "",
fmt.Sprintf("maintenance window %s: %s %s for %d minutes", w.Name, w.Cron, w.TZ, w.DurationMinutes))
c.JSON(http.StatusCreated, w)
}
// previewWindow godoc
//
// @Summary Preview the next three maintenance windows
// @Description Computed by the scheduler's own code, so the editor and the scheduler agree.
// @Tags patching
// @Accept json
// @Produce json
// @Param body body WindowPreviewRequest true "Schedule"
// @Success 200 {array} services.WindowSpan
// @Failure 400 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /maintenance-windows/preview [post]
func previewWindow(c *gin.Context) {
var body WindowPreviewRequest
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
spans, err := services.PreviewWindow(body.Cron, body.TZ, body.DurationMinutes, time.Now(), 3)
if err != nil {
patchError(c, err)
return
}
c.JSON(http.StatusOK, spans)
}
// getWindow godoc
//
// @Summary Get a maintenance window
// @Tags patching
// @Produce json
// @Param id path string true "Window ID"
// @Success 200 {object} models.MaintenanceWindow
// @Failure 404 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /maintenance-windows/{id} [get]
func getWindow(c *gin.Context) {
w, err := services.GetWindow(auth.InstanceID(c), c.Param("id"))
if err != nil {
patchError(c, err)
return
}
c.JSON(http.StatusOK, w)
}
// updateWindow godoc
//
// @Summary Update a maintenance window
// @Description Moves the next run of every enabled policy using it.
// @Tags patching
// @Accept json
// @Produce json
// @Param id path string true "Window ID"
// @Param body body models.MaintenanceWindow true "Window"
// @Success 200 {object} models.MaintenanceWindow
// @Failure 400 {object} ErrorResponse
// @Failure 404 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /maintenance-windows/{id} [put]
func updateWindow(c *gin.Context) {
var body models.MaintenanceWindow
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
if err := services.CheckWindowScope(auth.InstanceID(c), c.Param("id"), auth.ServerScope(c)); err != nil {
patchError(c, err)
return
}
w, err := services.UpdateWindow(auth.InstanceID(c), c.Param("id"), body)
if err != nil {
patchError(c, err)
return
}
services.LogEvent(auth.InstanceID(c), "patch.window_updated", actorFromCtx(c), "", "",
fmt.Sprintf("maintenance window %s: %s %s for %d minutes", w.Name, w.Cron, w.TZ, w.DurationMinutes))
c.JSON(http.StatusOK, w)
}
// deleteWindow godoc
//
// @Summary Delete a maintenance window
// @Tags patching
// @Param id path string true "Window ID"
// @Success 204
// @Failure 404 {object} ErrorResponse
// @Failure 409 {object} ErrorResponse "window_in_use"
// @Security cookieAuth
// @Security bearerAuth
// @Router /maintenance-windows/{id} [delete]
func deleteWindow(c *gin.Context) {
if err := services.CheckWindowScope(auth.InstanceID(c), c.Param("id"), auth.ServerScope(c)); err != nil {
patchError(c, err)
return
}
if err := services.DeleteWindow(auth.InstanceID(c), c.Param("id")); err != nil {
patchError(c, err)
return
}
services.LogEvent(auth.InstanceID(c), "patch.window_deleted", actorFromCtx(c), "", "", "maintenance window "+c.Param("id")+" deleted")
c.Status(http.StatusNoContent)
}
// listPolicies godoc
//
// @Summary List patch policies
// @Tags patching
// @Produce json
// @Success 200 {array} models.PatchPolicy
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-policies [get]
func listPolicies(c *gin.Context) {
ps, err := services.ListPolicies(auth.InstanceID(c))
if err != nil {
patchError(c, err)
return
}
c.JSON(http.StatusOK, ps)
}
// createPolicy godoc
//
// @Summary Create a patch policy
// @Tags patching
// @Accept json
// @Produce json
// @Param body body models.PatchPolicy true "Policy"
// @Success 201 {object} models.PatchPolicy
// @Failure 400 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-policies [post]
func createPolicy(c *gin.Context) {
var body models.PatchPolicy
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
p, err := services.CreatePolicy(auth.InstanceID(c), body, auth.ServerScope(c))
if err != nil {
patchError(c, err)
return
}
services.LogEvent(auth.InstanceID(c), "patch.policy_created", actorFromCtx(c), "", "",
fmt.Sprintf("patch policy %s: scope %s, reboot %s, enabled %v", p.Name, p.Scope, p.Reboot, p.Enabled))
c.JSON(http.StatusCreated, p)
}
// getPolicy godoc
//
// @Summary Get a patch policy
// @Tags patching
// @Produce json
// @Param id path string true "Policy ID"
// @Success 200 {object} models.PatchPolicy
// @Failure 404 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-policies/{id} [get]
func getPolicy(c *gin.Context) {
p, err := services.GetPolicy(auth.InstanceID(c), c.Param("id"))
if err != nil {
patchError(c, err)
return
}
c.JSON(http.StatusOK, p)
}
// updatePolicy godoc
//
// @Summary Update a patch policy
// @Tags patching
// @Accept json
// @Produce json
// @Param id path string true "Policy ID"
// @Param body body models.PatchPolicy true "Policy"
// @Success 200 {object} models.PatchPolicy
// @Failure 400 {object} ErrorResponse
// @Failure 404 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-policies/{id} [put]
func updatePolicy(c *gin.Context) {
var body models.PatchPolicy
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
p, err := services.UpdatePolicy(auth.InstanceID(c), c.Param("id"), body, auth.ServerScope(c))
if err != nil {
patchError(c, err)
return
}
services.LogEvent(auth.InstanceID(c), "patch.policy_updated", actorFromCtx(c), "", "",
fmt.Sprintf("patch policy %s: scope %s, reboot %s, enabled %v", p.Name, p.Scope, p.Reboot, p.Enabled))
c.JSON(http.StatusOK, p)
}
// deletePolicy godoc
//
// @Summary Delete a patch policy
// @Tags patching
// @Param id path string true "Policy ID"
// @Success 204
// @Failure 404 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-policies/{id} [delete]
func deletePolicy(c *gin.Context) {
instanceID := auth.InstanceID(c)
p, err := services.GetPolicy(instanceID, c.Param("id"))
if err != nil {
patchError(c, err)
return
}
if err := services.CheckPolicyScope(instanceID, *p, auth.ServerScope(c)); err != nil {
patchError(c, err)
return
}
if err := services.DeletePolicy(instanceID, p.PolicyID); err != nil {
patchError(c, err)
return
}
services.LogEvent(instanceID, "patch.policy_deleted", actorFromCtx(c), "", "", "patch policy "+p.Name+" deleted")
c.Status(http.StatusNoContent)
}
// runPolicyNow godoc
//
// @Summary Run a patch policy now
// @Description Opens a window of the policy's usual length starting now.
// @Tags patching
// @Produce json
// @Param id path string true "Policy ID"
// @Success 202 {object} models.PatchRun
// @Failure 404 {object} ErrorResponse
// @Failure 409 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-policies/{id}/run-now [post]
func runPolicyNow(c *gin.Context) {
run, err := services.StartRunNow(auth.InstanceID(c), c.Param("id"), actorFromCtx(c), auth.ServerScope(c))
if err != nil {
patchError(c, err)
return
}
c.JSON(http.StatusAccepted, run)
}
// listPatchRuns godoc
//
// @Summary List patch runs
// @Tags patching
// @Produce json
// @Param policy_id query string false "Filter by policy"
// @Param server_id query string false "Filter by server"
// @Param limit query int false "At most 200, default 50"
// @Success 200 {array} models.PatchRun
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-runs [get]
func listPatchRuns(c *gin.Context) {
instanceID := auth.InstanceID(c)
limit, _ := strconv.ParseInt(c.Query("limit"), 10, 64)
runs, err := services.ListPatchRuns(instanceID, c.Query("policy_id"), c.Query("server_id"), limit)
if err != nil {
patchError(c, err)
return
}
out := runs[:0]
for i := range runs {
before := len(runs[i].Servers)
if err := services.ScopePatchRun(instanceID, &runs[i], auth.ServerScope(c)); err != nil {
patchError(c, err)
return
}
if before == 0 || len(runs[i].Servers) > 0 {
out = append(out, runs[i])
}
}
c.JSON(http.StatusOK, out)
}
// getPatchRun godoc
//
// @Summary Get a patch run, with per-server output
// @Tags patching
// @Produce json
// @Param runId path string true "Run ID"
// @Success 200 {object} models.PatchRun
// @Failure 404 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-runs/{runId} [get]
func getPatchRun(c *gin.Context) {
run, ok := scopedRun(c)
if ok {
c.JSON(http.StatusOK, run)
}
}
// cancelPatchRun godoc
//
// @Summary Cancel a patch run
// @Description Stops further dispatch. Servers already patching finish.
// @Tags patching
// @Param runId path string true "Run ID"
// @Success 204
// @Failure 404 {object} ErrorResponse
// @Failure 409 {object} ErrorResponse
// @Security cookieAuth
// @Security bearerAuth
// @Router /patch-runs/{runId}/cancel [post]
func cancelPatchRun(c *gin.Context) {
run, ok := scopedRun(c)
if !ok {
return
}
if err := services.CancelPatchRun(auth.InstanceID(c), run.RunID); err != nil {
patchError(c, err)
return
}
services.LogEvent(auth.InstanceID(c), "patch.cancelled", actorFromCtx(c), "", "", "patch run "+run.RunID+" cancelled")
c.Status(http.StatusNoContent)
}
// scopedRun loads a run for a caller. A tag-restricted token that cannot see
// every server in the run gets a 404, the same answer as a run that does not
// exist, rather than a partial record it could act on.
func scopedRun(c *gin.Context) (*models.PatchRun, bool) {
instanceID := auth.InstanceID(c)
run, err := services.GetPatchRun(instanceID, c.Param("runId"))
if err != nil {
patchError(c, err)
return nil, false
}
before := len(run.Servers)
if err := services.ScopePatchRun(instanceID, run, auth.ServerScope(c)); err != nil {
patchError(c, err)
return nil, false
}
if len(run.Servers) != before {
patchError(c, services.ErrPatchRunNotFound)
return nil, false
}
return run, true
}
@@ -0,0 +1,34 @@
package api
import "testing"
// Every patching route must carry a scope and a server-scope declaration, or
// boot fails. Asserting the exact scope here keeps a copy-paste of
// "patching:read" onto a write route from slipping through.
func TestPatchingRouteScopes(t *testing.T) {
want := map[string]string{
"GET /api/maintenance-windows": "patching:read",
"POST /api/maintenance-windows": "patching:write",
"POST /api/maintenance-windows/preview": "patching:read",
"GET /api/maintenance-windows/:id": "patching:read",
"PUT /api/maintenance-windows/:id": "patching:write",
"DELETE /api/maintenance-windows/:id": "patching:write",
"GET /api/patch-policies": "patching:read",
"POST /api/patch-policies": "patching:write",
"GET /api/patch-policies/:id": "patching:read",
"PUT /api/patch-policies/:id": "patching:write",
"DELETE /api/patch-policies/:id": "patching:write",
"POST /api/patch-policies/:id/run-now": "patching:write",
"GET /api/patch-runs": "patching:read",
"GET /api/patch-runs/:runId": "patching:read",
"POST /api/patch-runs/:runId/cancel": "patching:write",
}
for route, scope := range want {
if got := routeScopes[route]; got != scope {
t.Errorf("%s: scope %q, want %q", route, got, scope)
}
if _, ok := serverScopedRoutes[route]; !ok {
t.Errorf("%s: missing from serverScopedRoutes", route)
}
}
}
+17
View File
@@ -175,6 +175,23 @@ var routeScopes = map[string]string{
"PUT /api/status-pages/:pageId/incidents/:incidentId": "status:write",
"DELETE /api/status-pages/:pageId/incidents/:incidentId": "status:write",
"POST /api/status-pages/:pageId/incidents/:incidentId/updates": "status:write",
// Scheduled patching: maintenance windows, patch policies and patch runs.
"GET /api/maintenance-windows": "patching:read",
"POST /api/maintenance-windows": "patching:write",
"POST /api/maintenance-windows/preview": "patching:read",
"GET /api/maintenance-windows/:id": "patching:read",
"PUT /api/maintenance-windows/:id": "patching:write",
"DELETE /api/maintenance-windows/:id": "patching:write",
"GET /api/patch-policies": "patching:read",
"POST /api/patch-policies": "patching:write",
"GET /api/patch-policies/:id": "patching:read",
"PUT /api/patch-policies/:id": "patching:write",
"DELETE /api/patch-policies/:id": "patching:write",
"POST /api/patch-policies/:id/run-now": "patching:write",
"GET /api/patch-runs": "patching:read",
"GET /api/patch-runs/:runId": "patching:read",
"POST /api/patch-runs/:runId/cancel": "patching:write",
}
// RequireScopes enforces routeScopes for token-authenticated requests and does
+26
View File
@@ -398,6 +398,32 @@ var serverScopedRoutes = map[string]scopeDecl{
// Each tool touching server data applies the caller's selector itself.
"POST /api/mcp": exempt,
"GET /api/mcp": exempt,
// Maintenance windows are a cron expression, a zone and a duration. They
// name no server and return no server data, so reading and creating one
// is exempt. Changing or deleting one moves or stops the patching of
// every policy using it, so each is refused when any of those policies
// targets servers outside the token's tag restriction.
"GET /api/maintenance-windows": exempt,
"POST /api/maintenance-windows": exempt,
"POST /api/maintenance-windows/preview": exempt,
"GET /api/maintenance-windows/:id": exempt,
"PUT /api/maintenance-windows/:id": scoped,
"DELETE /api/maintenance-windows/:id": scoped,
// Reading a policy returns its selector (server IDs and tag pairs) and no
// hostname, inventory or state, the same data a workflow's targets carry.
"GET /api/patch-policies": exempt,
"GET /api/patch-policies/:id": exempt,
// Writes and run-now act on the policy's targets, so each is refused when
// those targets reach outside the token's tag restriction.
"POST /api/patch-policies": scoped,
"PUT /api/patch-policies/:id": scoped,
"DELETE /api/patch-policies/:id": scoped,
"POST /api/patch-policies/:id/run-now": scoped,
// Runs name hostnames; ScopePatchRun removes servers the token cannot see.
"GET /api/patch-runs": scoped,
"GET /api/patch-runs/:runId": scoped,
"POST /api/patch-runs/:runId/cancel": scoped,
}
// AssertServerScopeMapComplete refuses to boot when any registered /api route
+23
View File
@@ -297,3 +297,26 @@ type StatusIncidentUpdateRequest struct {
Status string `json:"status" binding:"required"`
Body string `json:"body" binding:"required"`
}
// --- patching ---
// ApplyUpdatesResponse keeps the message existing scripts read and adds the
// run that records what happened.
type ApplyUpdatesResponse struct {
Message string `json:"message"`
RunID string `json:"run_id,omitempty"`
}
// ApplyUpdatesErrorResponse is the 503 body of apply-updates: the agent is
// offline, and the attempt is still recorded as a run.
type ApplyUpdatesErrorResponse struct {
Error string `json:"error"`
RunID string `json:"run_id,omitempty"`
}
// WindowPreviewRequest is the body of POST /maintenance-windows/preview.
type WindowPreviewRequest struct {
Cron string `json:"cron"`
TZ string `json:"tz"`
DurationMinutes int `json:"duration_minutes"`
}
+9
View File
@@ -236,6 +236,12 @@ func (s *vantageServer) ReportInventory(ctx context.Context, req *pb.InventoryRe
if err := services.StoreInventory(srv.ServerID, req); err != nil {
log.Printf("store inventory for %s: %v", srv.ServerID, err)
}
// Only static snapshots compute reboot_required, so only they can settle a
// reboot. The agent sends one at start, so the first report after a
// reboot qualifies.
if req.IncludeStatic && req.BootTimeUnix > 0 {
services.VerifyPatchReboots(srv.InstanceID, srv.ServerID, time.Unix(req.BootTimeUnix, 0), req.RebootRequired)
}
return &pb.InventoryReportResponse{}, nil
}
@@ -330,6 +336,9 @@ func (s *vantageServer) CommandStream(stream pb.Vantage_CommandStreamServer) err
if m.WorkloadLogsResult != nil {
services.WorkloadResults.Deliver(m.WorkloadLogsResult)
}
if m.PatchResult != nil {
services.RecordPatchResult(srv.InstanceID, srv.ServerID, m.PatchResult)
}
if m.StepResult != nil {
services.StepResults.Deliver(m.StepResult)
}
+20 -11
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
)
@@ -156,16 +157,16 @@ type updateBatchResult struct {
Servers int `json:"servers"`
Succeeded []string `json:"succeeded"`
Failed map[string]string `json:"failed,omitempty"`
RunIDs map[string]string `json:"run_ids,omitempty"` // server ID -> patch run ID
}
// apply_updates. The REST route (internal/api/handlers.go's applyUpdates) is
// per-server: POST /servers/:id/apply-updates resolves one server with
// services.GetServerScoped and calls services.DispatchApplyUpdates(serverID).
// There is no fleet-wide variant of that service call to invoke once, so this
// tool resolves the requested targets through ResolveTargetsScoped exactly as
// the brief describes, then calls the same DispatchApplyUpdates the REST route
// calls, once per resolved server - the identical dispatch, just looped
// instead of hardcoded to one server_id from the URL.
// services.GetServerScoped and calls services.StartManualRun. There is no
// fleet-wide variant of that service call to invoke once, so this tool
// resolves the requested targets through ResolveTargetsScoped, then calls the
// same StartManualRun once per resolved server and writes the same
// updates.applied audit event per server with its run ID.
func init() {
All().Register(Tool{
Name: "apply_updates",
@@ -178,9 +179,9 @@ func init() {
Write: true,
Scope: "servers:write",
Description: "Apply pending OS package updates on real servers, selected by " +
"server_ids and/or tags. This installs packages on real machines right now and " +
"cannot be undone from here. A server may need a reboot afterward, which this " +
"tool does not do.",
"server_ids and/or tags. Starts one manual patch run per server. This installs " +
"packages on real machines right now and cannot be undone from here. A server " +
"may need a reboot afterward, which this tool does not do.",
Handler: func(ctx context.Context, c Caller, args map[string]any) (any, error) {
ids := stringSliceArg(args, "server_ids")
targets, err := services.ResolveTargetsScoped(c.InstanceID, ids, tagArg(args), c.TokenScope)
@@ -192,12 +193,20 @@ func init() {
}
result := updateBatchResult{Servers: len(targets), Failed: map[string]string{}}
for _, srv := range targets {
if err := services.DispatchApplyUpdates(srv.ServerID); err != nil {
for i := range targets {
srv := targets[i]
run, err := services.StartManualRun(c.InstanceID, &srv, "mcp:"+c.TokenName, models.PatchSourceMCP)
if err != nil {
result.Failed[srv.ServerID] = err.Error()
continue
}
services.LogEvent(c.InstanceID, "updates.applied", "mcp:"+c.TokenName, srv.ServerID, "",
fmt.Sprintf("package update run %s started on %s", run.RunID, srv.Hostname))
result.Succeeded = append(result.Succeeded, srv.ServerID)
if result.RunIDs == nil {
result.RunIDs = map[string]string{}
}
result.RunIDs[srv.ServerID] = run.RunID
}
if len(result.Failed) == 0 {
result.Failed = nil
+116
View File
@@ -0,0 +1,116 @@
package models
import (
"time"
"go.mongodb.org/mongo-driver/v2/bson"
)
const (
PatchScopeAll = "all"
PatchScopeSecurity = "security"
PatchRebootNever = "never"
PatchRebootIfRequired = "if_required"
PatchRunRunning = "running"
PatchRunSucceeded = "succeeded"
PatchRunPartial = "partial"
PatchRunFailed = "failed"
PatchRunCancelled = "cancelled"
PatchSrvQueued = "queued"
PatchSrvWaitingOffline = "waiting_offline"
PatchSrvPatching = "patching"
PatchSrvRebooting = "rebooting"
PatchSrvSucceeded = "succeeded"
PatchSrvFailed = "failed"
PatchSrvUnsupported = "unsupported"
PatchSrvAgentTooOld = "agent_too_old"
PatchSrvMissedOffline = "missed_offline"
PatchSrvWindowClosed = "window_closed"
PatchSrvCancelled = "cancelled"
PatchSourceSchedule = "schedule"
PatchSourceRunNow = "run_now"
PatchSourceServer = "server"
PatchSourceVulnerabilities = "vulnerabilities"
PatchSourceMCP = "mcp"
)
// MaintenanceWindow answers "when" and nothing else. Policies reference it by
// ID, so one window can later serve alert muting and status page maintenance
// without a second definition of the same Sunday morning.
type MaintenanceWindow struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"instance_id"`
WindowID string `bson:"window_id" json:"window_id"`
Name string `bson:"name" json:"name"`
Cron string `bson:"cron" json:"cron"` // 5-field, window start
TZ string `bson:"tz" json:"tz"` // IANA name
DurationMinutes int `bson:"duration_minutes" json:"duration_minutes"`
CreatedAt time.Time `bson:"created_at" json:"created_at"`
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
}
type PatchPolicy struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"instance_id"`
PolicyID string `bson:"policy_id" json:"policy_id"`
Name string `bson:"name" json:"name"`
Enabled bool `bson:"enabled" json:"enabled"`
WindowID string `bson:"window_id" json:"window_id"`
TargetServerIDs []string `bson:"target_server_ids" json:"target_server_ids"`
TargetTags map[string]string `bson:"target_tags,omitempty" json:"target_tags,omitempty"`
Scope string `bson:"scope" json:"scope"`
Reboot string `bson:"reboot" json:"reboot"`
MaxConcurrent int `bson:"max_concurrent" json:"max_concurrent"` // 0 = no cap
NotifyChannelIDs []string `bson:"notify_channel_ids,omitempty" json:"notify_channel_ids,omitempty"`
NextRunAt *time.Time `bson:"next_run_at,omitempty" json:"next_run_at,omitempty"`
LastRunAt *time.Time `bson:"last_run_at,omitempty" json:"last_run_at,omitempty"`
LastSkipped *Skip `bson:"last_skipped,omitempty" json:"last_skipped,omitempty"`
DisabledReason string `bson:"disabled_reason,omitempty" json:"disabled_reason,omitempty"`
CreatedAt time.Time `bson:"created_at" json:"created_at"`
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
}
// PatchRun is one firing of a policy, or one manual Apply updates. Scope,
// reboot and concurrency are copied from the policy at fire time so editing
// the policy never rewrites what a past run shows.
type PatchRun struct {
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
InstanceID string `bson:"instance_id" json:"instance_id"`
RunID string `bson:"run_id" json:"run_id"`
PolicyID string `bson:"policy_id,omitempty" json:"policy_id,omitempty"`
PolicyName string `bson:"policy_name,omitempty" json:"policy_name,omitempty"`
TriggeredBy string `bson:"triggered_by" json:"triggered_by"`
Source string `bson:"source" json:"source"`
Scope string `bson:"scope" json:"scope"`
Reboot string `bson:"reboot" json:"reboot"`
MaxConcurrent int `bson:"max_concurrent" json:"max_concurrent"`
WindowEnd *time.Time `bson:"window_end,omitempty" json:"window_end,omitempty"`
Status string `bson:"status" json:"status"`
CancelledAt *time.Time `bson:"cancelled_at,omitempty" json:"cancelled_at,omitempty"`
StartedAt time.Time `bson:"started_at" json:"started_at"`
FinishedAt *time.Time `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
Servers []PatchServerRun `bson:"servers" json:"servers"`
}
type PatchServerRun struct {
ServerID string `bson:"server_id" json:"server_id"`
Hostname string `bson:"hostname" json:"hostname"`
Status string `bson:"status" json:"status"`
CommandID string `bson:"command_id,omitempty" json:"-"`
PendingBefore int `bson:"pending_before" json:"pending_before"`
PendingAfter *int `bson:"pending_after,omitempty" json:"pending_after,omitempty"`
RebootedAt *time.Time `bson:"rebooted_at,omitempty" json:"rebooted_at,omitempty"`
VerifiedAt *time.Time `bson:"verified_at,omitempty" json:"verified_at,omitempty"`
// BootTimeBefore is the host's reported boot time when the reboot was
// announced. A later report with a different boot time proves the
// restart without comparing the host clock to the server clock.
BootTimeBefore *time.Time `bson:"boot_time_before,omitempty" json:"-"`
Output string `bson:"output,omitempty" json:"output,omitempty"`
Error string `bson:"error,omitempty" json:"error,omitempty"`
StartedAt *time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
FinishedAt *time.Time `bson:"finished_at,omitempty" json:"finished_at,omitempty"`
}
+3
View File
@@ -42,6 +42,9 @@ type Inventory struct {
RebootRequired bool `bson:"reboot_required,omitempty" json:"reboot_required,omitempty"`
MetricsAt *time.Time `bson:"metrics_at,omitempty" json:"metrics_at,omitempty"`
StaticAt *time.Time `bson:"static_at,omitempty" json:"static_at,omitempty"`
// BootTime is the host's last reported boot time, stored on every report
// that carries one so a patch reboot can be proven by a changed boot.
BootTime *time.Time `bson:"boot_time,omitempty" json:"boot_time,omitempty"`
}
type Server struct {
+5 -1
View File
@@ -11,6 +11,10 @@ import (
// monitor check. MonitorName carries the hostname in that case.
const TypeServer = "server"
// TypePatch marks a patch run summary. Like a vulnerability digest it is a
// headline, not a transition, so title() adds no verb.
const TypePatch = "patch"
type Event struct {
MonitorName string
Type string
@@ -26,7 +30,7 @@ func (e Event) title() string {
verb = "is DOWN"
}
var s string
if e.Type == TypeVuln {
if e.Type == TypeVuln || e.Type == TypePatch {
// A digest is not a transition. MonitorName already carries the whole
// headline ("12 new critical across 4 servers"), so no verb applies.
s = fmt.Sprintf("[Vantage] %s", e.MonitorName)
+17
View File
@@ -0,0 +1,17 @@
package notify
import (
"testing"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
)
// A patch summary is not a transition: it must not read "is DOWN" or
// "recovered".
func TestPatchEventTitle(t *testing.T) {
ev := Event{MonitorName: `Patch policy "Sunday prod" partial`, Type: TypePatch, NewStatus: models.PatchRunPartial, Message: "38 succeeded, 2 failed"}
want := `[Vantage] Patch policy "Sunday prod" partial: 38 succeeded, 2 failed`
if got := ev.title(); got != want {
t.Fatalf("got %q, want %q", got, want)
}
}
+270
View File
@@ -0,0 +1,270 @@
// Package patchrun is the patch run state machine as pure functions. Nothing
// here touches the database: the services layer loads a run, asks this
// package what should change, and writes that change guarded by the status it
// expected. That split is what makes the rules testable without MongoDB.
package patchrun
import (
"fmt"
"strconv"
"strings"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
)
// MinAgentVersion is the first agent release that honours ApplyUpdatesCmd's
// scope and answers with a PatchResult. An older agent ignores the scope and
// installs everything, so a policy must never dispatch to one.
const MinAgentVersion = "1.4.0"
const (
// ResultGrace covers what the agent does after the upgrade and before it
// answers: a pending-update re-check of up to 10 minutes (Windows Update
// search) plus a 2 minute reboot check, with room to spare.
ResultGrace = 20 * time.Minute
// ManualTimeout is the agent's backstop for one started upgrade, counted
// from its own start. The window end never stops a running upgrade.
ManualTimeout = 2 * time.Hour
// RebootTimeout is how long a rebooting server has to send a post-boot
// inventory report. Windows cumulative updates routinely take over 20.
RebootTimeout = 45 * time.Minute
// LatestStartBeforeEnd is the tail of a window in which no server starts
// patching: a late start would run long past the window end.
LatestStartBeforeEnd = 15 * time.Minute
)
// AgentSupportsPatchResults compares major.minor.patch. Empty, "dev" and
// anything unparseable count as too old, and a pre-release of exactly the
// minimum version counts as older than it, as semver orders them.
func AgentSupportsPatchResults(version string) bool {
have, pre, ok := parseVersion(version)
if !ok {
return false
}
want, _, _ := parseVersion(MinAgentVersion)
for i := 0; i < 3; i++ {
if have[i] != want[i] {
return have[i] > want[i]
}
}
return !pre
}
func parseVersion(v string) ([3]int, bool, bool) {
var out [3]int
v = strings.TrimPrefix(strings.TrimSpace(v), "v")
v, _, _ = strings.Cut(v, "+")
core, pre, hasPre := strings.Cut(v, "-")
parts := strings.Split(core, ".")
if len(parts) != 3 {
return out, false, false
}
for i, p := range parts {
n, err := strconv.Atoi(p)
if err != nil || n < 0 {
return out, false, false
}
out[i] = n
}
return out, hasPre && pre != "", true
}
// IsTerminal reports whether a server run has finished.
func IsTerminal(status string) bool {
switch status {
case models.PatchSrvQueued, models.PatchSrvWaitingOffline, models.PatchSrvPatching, models.PatchSrvRebooting:
return false
}
return true
}
// Transition is one change Advance wants made. The caller writes it guarded
// by From, so a result that arrived meanwhile is never overwritten. Dispatch
// means: send ApplyUpdatesCmd, and set To only once the command is on its way.
type Transition struct {
ServerID string
From string
To string
Error string
Dispatch bool
}
// Advance returns what should change on this tick. connected says which
// agents hold a command stream right now.
func Advance(run models.PatchRun, now time.Time, connected map[string]bool) []Transition {
if run.Status != models.PatchRunRunning {
return nil
}
windowOpen := run.WindowEnd == nil || now.Before(*run.WindowEnd)
// In the last LatestStartBeforeEnd of a window nothing new starts: queued
// and waiting servers simply wait, and close at WindowEnd as usual.
mayStart := run.WindowEnd == nil || now.Before(run.WindowEnd.Add(-LatestStartBeforeEnd))
inFlight := 0
for _, s := range run.Servers {
if s.Status == models.PatchSrvPatching || s.Status == models.PatchSrvRebooting {
inFlight++
}
}
var out []Transition
for _, s := range run.Servers {
switch s.Status {
case models.PatchSrvQueued, models.PatchSrvWaitingOffline:
switch {
case run.CancelledAt != nil:
out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvCancelled})
case !windowOpen:
to := models.PatchSrvWindowClosed
if s.Status == models.PatchSrvWaitingOffline {
to = models.PatchSrvMissedOffline
}
out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: to})
case !mayStart:
// The window tail: no dispatch, no transition.
case run.MaxConcurrent > 0 && inFlight >= run.MaxConcurrent:
// No slot this tick.
case !connected[s.ServerID]:
if s.Status == models.PatchSrvQueued {
out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvWaitingOffline})
}
default:
out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvPatching, Dispatch: true})
inFlight++
}
case models.PatchSrvPatching:
if now.After(resultDeadline(run, s)) {
out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvFailed, Error: "no result from agent"})
}
case models.PatchSrvRebooting:
if s.RebootedAt != nil && now.After(s.RebootedAt.Add(RebootTimeout)) {
out = append(out, Transition{ServerID: s.ServerID, From: s.Status, To: models.PatchSrvFailed, Error: fmt.Sprintf("did not come back within %d minutes", int(RebootTimeout.Minutes()))})
}
}
}
return out
}
// resultDeadline is the same for windowed and manual runs: the agent lets a
// started upgrade finish past the window end, so the window end says nothing
// about when a result is due. The base is the server's dispatch time.
func resultDeadline(run models.PatchRun, s models.PatchServerRun) time.Time {
start := run.StartedAt
if s.StartedAt != nil {
start = *s.StartedAt
}
return start.Add(ManualTimeout + ResultGrace)
}
// ApplyResult folds an agent's PatchResult into the server run. It only acts
// on a server that is patching; anything else is a late or duplicate result.
func ApplyResult(s models.PatchServerRun, r *pb.PatchResult, now time.Time) (models.PatchServerRun, bool) {
if s.Status != models.PatchSrvPatching {
return s, false
}
s.Output = r.OutputTail
if r.PendingAfter >= 0 {
v := int(r.PendingAfter)
s.PendingAfter = &v
}
switch r.Status {
case pb.PatchStatusOK:
if r.Rebooting {
s.Status = models.PatchSrvRebooting
s.RebootedAt = &now
return s, true
}
s.Status = models.PatchSrvSucceeded
case pb.PatchStatusUnsupported:
s.Status = models.PatchSrvUnsupported
s.Error = r.Message
default:
s.Status = models.PatchSrvFailed
s.Error = r.Message
if s.Error == "" {
s.Error = "agent reported a failure"
}
}
s.FinishedAt = &now
return s, true
}
// VerifyReboot settles a rebooting server from a static inventory report.
// When the boot time reported before the reboot is known, a later boot time
// is the proof: both come from the host clock, so skew against the server
// clock does not matter. Otherwise only a boot time later than the reboot
// command counts. Either way a snapshot sent during the one-minute grace
// period, before the host went down, does not.
func VerifyReboot(s models.PatchServerRun, bootTime time.Time, rebootRequired bool, now time.Time) (models.PatchServerRun, bool) {
if s.Status != models.PatchSrvRebooting || s.RebootedAt == nil {
return s, false
}
proven := bootTime.After(*s.RebootedAt)
if s.BootTimeBefore != nil {
proven = bootTime.After(*s.BootTimeBefore)
}
if !proven {
return s, false
}
if rebootRequired {
s.Status = models.PatchSrvFailed
s.Error = "still requires a reboot after restarting"
} else {
s.Status = models.PatchSrvSucceeded
s.VerifiedAt = &now
}
s.FinishedAt = &now
return s, true
}
// Finalize says whether the run is over and how it ended. Only succeeded
// counts as success: unsupported, agent_too_old and the window outcomes did
// not patch anything.
func Finalize(run models.PatchRun) (string, bool) {
ok := 0
for _, s := range run.Servers {
if !IsTerminal(s.Status) {
return "", false
}
if s.Status == models.PatchSrvSucceeded {
ok++
}
}
switch {
case run.CancelledAt != nil:
return models.PatchRunCancelled, true
case len(run.Servers) > 0 && ok == len(run.Servers):
return models.PatchRunSucceeded, true
case ok == 0:
return models.PatchRunFailed, true
default:
return models.PatchRunPartial, true
}
}
var summaryOrder = []struct{ status, label string }{
{models.PatchSrvSucceeded, "succeeded"},
{models.PatchSrvFailed, "failed"},
{models.PatchSrvUnsupported, "unsupported"},
{models.PatchSrvAgentTooOld, "need an agent update"},
{models.PatchSrvMissedOffline, "missed offline"},
{models.PatchSrvWindowClosed, "window closed"},
{models.PatchSrvCancelled, "cancelled"},
}
// Summary is the one-line count used in alerts, e.g. "38 succeeded, 2 failed".
func Summary(run models.PatchRun) string {
counts := map[string]int{}
for _, s := range run.Servers {
counts[s.Status]++
}
var parts []string
for _, o := range summaryOrder {
if n := counts[o.status]; n > 0 {
parts = append(parts, fmt.Sprintf("%d %s", n, o.label))
}
}
return strings.Join(parts, ", ")
}
+322
View File
@@ -0,0 +1,322 @@
package patchrun
import (
"testing"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
)
var t0 = time.Date(2026, 9, 20, 2, 0, 0, 0, time.UTC)
func tp(t time.Time) *time.Time { return &t }
func srv(id, status string) models.PatchServerRun {
return models.PatchServerRun{ServerID: id, Hostname: id, Status: status}
}
func windowRun(max int, servers ...models.PatchServerRun) models.PatchRun {
return models.PatchRun{Status: models.PatchRunRunning, StartedAt: t0, WindowEnd: tp(t0.Add(2 * time.Hour)), MaxConcurrent: max, Servers: servers}
}
func find(ts []Transition, id string) *Transition {
for i := range ts {
if ts[i].ServerID == id {
return &ts[i]
}
}
return nil
}
func TestAgentSupportsPatchResults(t *testing.T) {
cases := map[string]bool{
"1.4.0": true, "v1.4.0": true, "1.4.1": true, "1.10.0": true, "2.0.0": true,
"1.3.5": false, "1.4.0-rc1": false, "1.5.0-rc1": true,
"": false, "dev": false, "1.4": false, "x.y.z": false,
}
for v, want := range cases {
if got := AgentSupportsPatchResults(v); got != want {
t.Errorf("%q: got %v, want %v", v, got, want)
}
}
}
func TestAdvanceRespectsConcurrencyCountingRebooting(t *testing.T) {
run := windowRun(2,
srv("a", models.PatchSrvRebooting),
srv("b", models.PatchSrvQueued),
srv("c", models.PatchSrvQueued),
)
run.Servers[0].RebootedAt = tp(t0)
ts := Advance(run, t0.Add(time.Minute), map[string]bool{"b": true, "c": true})
if tr := find(ts, "b"); tr == nil || !tr.Dispatch || tr.To != models.PatchSrvPatching {
t.Fatalf("b should dispatch, got %+v", tr)
}
if tr := find(ts, "c"); tr != nil {
t.Fatalf("c must wait for a slot, got %+v", tr)
}
}
func TestAdvanceUnlimitedWhenZero(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvQueued), srv("b", models.PatchSrvQueued))
ts := Advance(run, t0, map[string]bool{"a": true, "b": true})
if len(ts) != 2 || !ts[0].Dispatch || !ts[1].Dispatch {
t.Fatalf("both should dispatch: %+v", ts)
}
}
func TestAdvanceOfflineWaitsThenDispatches(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvQueued))
ts := Advance(run, t0, map[string]bool{})
if len(ts) != 1 || ts[0].To != models.PatchSrvWaitingOffline || ts[0].Dispatch {
t.Fatalf("offline should wait: %+v", ts)
}
run.Servers[0].Status = models.PatchSrvWaitingOffline
if ts := Advance(run, t0.Add(time.Minute), map[string]bool{}); len(ts) != 0 {
t.Fatalf("still offline must be a no-op, got %+v", ts)
}
ts = Advance(run, t0.Add(2*time.Minute), map[string]bool{"a": true})
if len(ts) != 1 || !ts[0].Dispatch {
t.Fatalf("back online should dispatch: %+v", ts)
}
}
func TestAdvanceWindowCloses(t *testing.T) {
run := windowRun(1, srv("q", models.PatchSrvQueued), srv("w", models.PatchSrvWaitingOffline))
ts := Advance(run, t0.Add(2*time.Hour), map[string]bool{"q": true, "w": true})
if tr := find(ts, "q"); tr == nil || tr.To != models.PatchSrvWindowClosed || tr.Dispatch {
t.Fatalf("queued at window end: %+v", tr)
}
if tr := find(ts, "w"); tr == nil || tr.To != models.PatchSrvMissedOffline {
t.Fatalf("waiting at window end: %+v", tr)
}
}
// A windowed run times out from the server's own dispatch time, not from the
// window end: a server dispatched late in the window may finish past it.
func TestAdvanceNoResultTimeoutFromDispatch(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvPatching))
run.Servers[0].StartedAt = tp(t0.Add(90 * time.Minute))
deadline := t0.Add(90*time.Minute + ManualTimeout + ResultGrace)
if ts := Advance(run, run.WindowEnd.Add(ResultGrace+time.Minute), nil); len(ts) != 0 {
t.Fatalf("past WindowEnd+grace but inside the dispatch timeout: %+v", ts)
}
if ts := Advance(run, deadline.Add(-time.Minute), nil); len(ts) != 0 {
t.Fatalf("inside the dispatch timeout: %+v", ts)
}
ts := Advance(run, deadline.Add(time.Minute), nil)
if len(ts) != 1 || ts[0].To != models.PatchSrvFailed || ts[0].Error == "" {
t.Fatalf("past the dispatch timeout: %+v", ts)
}
}
// Without a server StartedAt the run's own start is the base.
func TestAdvanceNoResultTimeoutFallsBackToRunStart(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvPatching))
if ts := Advance(run, t0.Add(2*time.Hour+19*time.Minute), nil); len(ts) != 0 {
t.Fatalf("inside timeout: %+v", ts)
}
if ts := Advance(run, t0.Add(2*time.Hour+21*time.Minute), nil); len(ts) != 1 || ts[0].To != models.PatchSrvFailed {
t.Fatalf("past timeout: %+v", ts)
}
}
func TestAdvanceManualRunTimeout(t *testing.T) {
run := models.PatchRun{Status: models.PatchRunRunning, StartedAt: t0, Servers: []models.PatchServerRun{srv("a", models.PatchSrvPatching)}}
run.Servers[0].StartedAt = tp(t0)
if ts := Advance(run, t0.Add(2*time.Hour+19*time.Minute), nil); len(ts) != 0 {
t.Fatalf("manual inside timeout: %+v", ts)
}
if ts := Advance(run, t0.Add(2*time.Hour+21*time.Minute), nil); len(ts) != 1 || ts[0].To != models.PatchSrvFailed {
t.Fatalf("manual past timeout: %+v", ts)
}
}
func TestTimingConstants(t *testing.T) {
if ResultGrace != 20*time.Minute || RebootTimeout != 45*time.Minute || LatestStartBeforeEnd != 15*time.Minute {
t.Fatalf("ResultGrace=%v RebootTimeout=%v LatestStartBeforeEnd=%v", ResultGrace, RebootTimeout, LatestStartBeforeEnd)
}
}
// No server starts patching in the last 15 minutes of a window: it would
// either be cut short or run long past the window end.
func TestAdvanceNoDispatchInWindowTail(t *testing.T) {
run := windowRun(0, srv("q", models.PatchSrvQueued), srv("w", models.PatchSrvWaitingOffline), srv("o", models.PatchSrvQueued))
cutoff := run.WindowEnd.Add(-LatestStartBeforeEnd)
online := map[string]bool{"q": true, "w": true}
for _, at := range []time.Time{cutoff, cutoff.Add(time.Minute), run.WindowEnd.Add(-time.Second)} {
if ts := Advance(run, at, online); len(ts) != 0 {
t.Fatalf("at %v: nothing may change in the window tail, got %+v", at.Sub(t0), ts)
}
}
}
func TestAdvanceDispatchJustBeforeWindowTail(t *testing.T) {
run := windowRun(0, srv("q", models.PatchSrvQueued))
at := run.WindowEnd.Add(-LatestStartBeforeEnd - time.Second)
ts := Advance(run, at, map[string]bool{"q": true})
if len(ts) != 1 || !ts[0].Dispatch {
t.Fatalf("dispatch must be allowed just before the tail: %+v", ts)
}
}
// A manual run has no window and no tail.
func TestAdvanceManualRunHasNoTail(t *testing.T) {
run := models.PatchRun{Status: models.PatchRunRunning, StartedAt: t0, Servers: []models.PatchServerRun{srv("a", models.PatchSrvQueued)}}
if ts := Advance(run, t0.Add(10*time.Hour), map[string]bool{"a": true}); len(ts) != 1 || !ts[0].Dispatch {
t.Fatalf("manual run must dispatch: %+v", ts)
}
}
func TestAdvanceRebootTimeout(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvRebooting))
run.Servers[0].RebootedAt = tp(t0)
if ts := Advance(run, t0.Add(44*time.Minute), nil); len(ts) != 0 {
t.Fatalf("inside reboot timeout: %+v", ts)
}
ts := Advance(run, t0.Add(46*time.Minute), nil)
if len(ts) != 1 || ts[0].To != models.PatchSrvFailed || ts[0].Error != "did not come back within 45 minutes" {
t.Fatalf("past reboot timeout: %+v", ts)
}
}
func TestAdvanceCancelledDispatchesNothing(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvQueued), srv("b", models.PatchSrvWaitingOffline), srv("c", models.PatchSrvPatching))
run.CancelledAt = tp(t0)
ts := Advance(run, t0.Add(time.Minute), map[string]bool{"a": true, "b": true})
for _, id := range []string{"a", "b"} {
if tr := find(ts, id); tr == nil || tr.To != models.PatchSrvCancelled || tr.Dispatch {
t.Errorf("%s: %+v", id, tr)
}
}
if find(ts, "c") != nil {
t.Error("an in-flight server must be left to finish")
}
}
func TestAdvanceIgnoresFinishedRun(t *testing.T) {
run := windowRun(0, srv("a", models.PatchSrvQueued))
run.Status = models.PatchRunSucceeded
if ts := Advance(run, t0, map[string]bool{"a": true}); ts != nil {
t.Fatalf("got %+v", ts)
}
}
func TestApplyResult(t *testing.T) {
now := t0.Add(10 * time.Minute)
cases := []struct {
r pb.PatchResult
want string
}{
{pb.PatchResult{Status: pb.PatchStatusOK, PendingAfter: 0}, models.PatchSrvSucceeded},
{pb.PatchResult{Status: pb.PatchStatusOK, Rebooting: true}, models.PatchSrvRebooting},
{pb.PatchResult{Status: pb.PatchStatusFailed, Message: "apt broke"}, models.PatchSrvFailed},
{pb.PatchResult{Status: pb.PatchStatusBusy, Message: "busy"}, models.PatchSrvFailed},
{pb.PatchResult{Status: pb.PatchStatusUnsupported, Message: "no metadata"}, models.PatchSrvUnsupported},
}
for _, c := range cases {
got, ok := ApplyResult(srv("a", models.PatchSrvPatching), &c.r, now)
if !ok || got.Status != c.want {
t.Errorf("%s: got %s ok=%v, want %s", c.r.Status, got.Status, ok, c.want)
}
if c.want == models.PatchSrvRebooting && (got.RebootedAt == nil || got.FinishedAt != nil) {
t.Errorf("rebooting must set RebootedAt and leave FinishedAt nil: %+v", got)
}
}
if _, ok := ApplyResult(srv("a", models.PatchSrvSucceeded), &pb.PatchResult{Status: pb.PatchStatusOK}, now); ok {
t.Error("a result for a server not patching must be ignored")
}
got, _ := ApplyResult(srv("a", models.PatchSrvPatching), &pb.PatchResult{Status: pb.PatchStatusOK, PendingAfter: -1}, now)
if got.PendingAfter != nil {
t.Error("PendingAfter -1 means unknown and must stay nil")
}
}
func TestVerifyReboot(t *testing.T) {
s := srv("a", models.PatchSrvRebooting)
s.RebootedAt = tp(t0)
now := t0.Add(5 * time.Minute)
if _, ok := VerifyReboot(s, t0.Add(-time.Hour), false, now); ok {
t.Error("boot before the reboot command is not proof")
}
if _, ok := VerifyReboot(s, t0, false, now); ok {
t.Error("boot equal to the reboot command is not proof")
}
got, ok := VerifyReboot(s, t0.Add(2*time.Minute), false, now)
if !ok || got.Status != models.PatchSrvSucceeded || got.VerifiedAt == nil {
t.Errorf("clean reboot: %+v", got)
}
got, ok = VerifyReboot(s, t0.Add(2*time.Minute), true, now)
if !ok || got.Status != models.PatchSrvFailed || got.Error == "" {
t.Errorf("still owed: %+v", got)
}
if _, ok := VerifyReboot(srv("a", models.PatchSrvPatching), t0.Add(time.Hour), false, now); ok {
t.Error("only rebooting servers verify")
}
}
// With the boot time recorded before the reboot, a changed boot time is the
// proof, whatever the skew between the host clock and the server clock.
func TestVerifyRebootChangedBoot(t *testing.T) {
s := srv("a", models.PatchSrvRebooting)
s.RebootedAt = tp(t0)
s.BootTimeBefore = tp(t0.Add(-10 * 24 * time.Hour))
now := t0.Add(5 * time.Minute)
// The host clock runs 10 minutes slow: its new boot time reads earlier
// than the server's RebootedAt, yet the boot did change.
got, ok := VerifyReboot(s, t0.Add(-8*time.Minute), false, now)
if !ok || got.Status != models.PatchSrvSucceeded {
t.Fatalf("changed boot behind a slow clock must be proven: %+v ok=%v", got, ok)
}
// A fast host clock with an unchanged boot is not proof.
if _, ok := VerifyReboot(s, *s.BootTimeBefore, false, now); ok {
t.Error("an unchanged boot time is not proof")
}
if _, ok := VerifyReboot(s, s.BootTimeBefore.Add(-time.Minute), false, now); ok {
t.Error("an earlier boot time is not proof")
}
}
func TestFinalize(t *testing.T) {
mk := func(statuses ...string) models.PatchRun {
r := windowRun(0)
for i, s := range statuses {
r.Servers = append(r.Servers, srv(string(rune('a'+i)), s))
}
return r
}
cases := []struct {
run models.PatchRun
want string
done bool
}{
{mk(models.PatchSrvSucceeded, models.PatchSrvSucceeded), models.PatchRunSucceeded, true},
{mk(models.PatchSrvSucceeded, models.PatchSrvFailed), models.PatchRunPartial, true},
{mk(models.PatchSrvSucceeded, models.PatchSrvUnsupported), models.PatchRunPartial, true},
{mk(models.PatchSrvSucceeded, models.PatchSrvAgentTooOld), models.PatchRunPartial, true},
{mk(models.PatchSrvMissedOffline, models.PatchSrvWindowClosed), models.PatchRunFailed, true},
{mk(models.PatchSrvSucceeded, models.PatchSrvPatching), "", false},
{mk(), models.PatchRunFailed, true},
}
for i, c := range cases {
got, done := Finalize(c.run)
if got != c.want || done != c.done {
t.Errorf("case %d: got %q/%v, want %q/%v", i, got, done, c.want, c.done)
}
}
cancelled := mk(models.PatchSrvSucceeded, models.PatchSrvCancelled)
cancelled.CancelledAt = tp(t0)
if got, done := Finalize(cancelled); got != models.PatchRunCancelled || !done {
t.Errorf("cancelled: %q/%v", got, done)
}
}
func TestSummary(t *testing.T) {
r := windowRun(0,
srv("a", models.PatchSrvSucceeded), srv("b", models.PatchSrvSucceeded),
srv("c", models.PatchSrvFailed), srv("d", models.PatchSrvMissedOffline))
if got := Summary(r); got != "2 succeeded, 1 failed, 1 missed offline" {
t.Fatalf("got %q", got)
}
}
+55
View File
@@ -0,0 +1,55 @@
// Package patchsched fires patch policies at the start of their maintenance
// window and advances running patch runs. Like workflowsched it runs under the
// housekeeping leader lock and must not import services: services imports
// this package for NextStart and WindowEnd.
package patchsched
import (
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
)
type Decision string
const (
Fire Decision = "fire"
SkipMissed Decision = "missed"
SkipRunning Decision = "already_running"
SkipNoTargets Decision = "no_targets"
)
// Decide is the whole fire or skip rule for one due policy. Missed comes
// first, as in workflowsched: a stale occurrence is recorded as missed
// whatever else is true. A window that has already closed is missed even
// inside the hour of grace, because there is no time left to patch in.
func Decide(due, windowEnd, now time.Time, runActive bool, targets int) Decision {
if !now.Before(windowEnd) || now.Sub(due) > workflowsched.GraceWindow {
return SkipMissed
}
if runActive {
return SkipRunning
}
if targets == 0 {
return SkipNoTargets
}
return Fire
}
func WindowEnd(start time.Time, durationMinutes int) time.Time {
return start.Add(time.Duration(durationMinutes) * time.Minute)
}
// NextStart is the first window start strictly after from. Callers pass
// Later(now, currentWindowEnd) so windows never overlap, including across a
// daylight-saving fall-back where the same wall-clock time occurs twice.
func NextStart(cron, tz string, from time.Time) (time.Time, error) {
return workflowsched.NextOccurrence(cron, tz, from)
}
func Later(a, b time.Time) time.Time {
if a.After(b) {
return a
}
return b
}
+66
View File
@@ -0,0 +1,66 @@
package patchsched
import (
"testing"
"time"
)
var due = time.Date(2026, 9, 20, 2, 0, 0, 0, time.UTC)
func TestDecide(t *testing.T) {
end := due.Add(2 * time.Hour)
cases := []struct {
name string
now time.Time
end time.Time
running bool
targets int
want Decision
}{
{"on time", due, end, false, 3, Fire},
{"late within grace", due.Add(59 * time.Minute), end, false, 3, Fire},
{"past grace", due.Add(61 * time.Minute), end, false, 3, SkipMissed},
{"window already over", due.Add(20 * time.Minute), due.Add(15 * time.Minute), false, 3, SkipMissed},
{"missed wins over running", due.Add(2 * time.Hour), end, true, 3, SkipMissed},
{"previous run active", due, end, true, 3, SkipRunning},
{"no targets", due, end, false, 0, SkipNoTargets},
}
for _, c := range cases {
if got := Decide(due, c.end, c.now, c.running, c.targets); got != c.want {
t.Errorf("%s: got %s, want %s", c.name, got, c.want)
}
}
}
// Europe/London falls back on 25 October 2026, so 01:30 happens twice. The
// next window is computed from the end of the current one, so a two-hour
// window starting at the first 01:30 cannot fire again at the second.
func TestNextStartAcrossFallBack(t *testing.T) {
loc, _ := time.LoadLocation("Europe/London")
from := time.Date(2026, 10, 24, 12, 0, 0, 0, time.UTC)
first, err := NextStart("30 1 * * 0", "Europe/London", from)
if err != nil {
t.Fatal(err)
}
if d := first.In(loc); d.Day() != 25 || d.Month() != time.October || d.Hour() != 1 || d.Minute() != 30 {
t.Fatalf("first = %s", d)
}
end := WindowEnd(first, 120)
next, err := NextStart("30 1 * * 0", "Europe/London", Later(first.Add(time.Minute), end))
if err != nil {
t.Fatal(err)
}
if d := next.In(loc); d.Day() != 1 || d.Month() != time.November || d.Hour() != 1 || d.Minute() != 30 {
t.Fatalf("next = %s, want 2026-11-01 01:30 London", d)
}
}
func TestWindowEndAndLater(t *testing.T) {
if got := WindowEnd(due, 90); !got.Equal(due.Add(90 * time.Minute)) {
t.Fatalf("WindowEnd = %s", got)
}
a, b := due, due.Add(time.Second)
if !Later(a, b).Equal(b) || !Later(b, a).Equal(b) {
t.Fatal("Later must return the later time")
}
}
+159
View File
@@ -0,0 +1,159 @@
package patchsched
import (
"context"
"errors"
"log"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
"go.mongodb.org/mongo-driver/v2/mongo/options"
)
const tickInterval = 30 * time.Second
// Deps are injected from main.go: services imports this package, so this
// package cannot import services.
type Deps struct {
// LookupWindow returns nil, nil when the window no longer exists.
LookupWindow func(instanceID, windowID string) (*models.MaintenanceWindow, error)
CountTargets func(p models.PatchPolicy) (int, error)
StartPolicyRun func(p models.PatchPolicy, windowEnd time.Time) error
AdvanceRuns func(ctx context.Context)
LogEvent func(instanceID, eventType, actor, serverID, keyID, details string)
}
// Start runs until ctx is cancelled, inside bus.RunAsLeader("housekeeping").
// Each tick fires due policies, then advances every running run.
func Start(ctx context.Context, deps Deps) {
go func() {
t := time.NewTicker(tickInterval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
fireDue(ctx, deps, time.Now())
deps.AdvanceRuns(ctx)
}
}
}()
}
func fireDue(ctx context.Context, deps Deps, now time.Time) {
cur, err := db.Col("patch_policies").Find(ctx, bson.M{"enabled": true, "next_run_at": bson.M{"$lte": now}})
if err != nil {
log.Printf("patchsched: find due: %v", err)
return
}
var due []models.PatchPolicy
if err := cur.All(ctx, &due); err != nil {
log.Printf("patchsched: decode due: %v", err)
return
}
for _, p := range due {
if ctx.Err() != nil {
return
}
process(ctx, deps, p, now)
}
}
func process(ctx context.Context, deps Deps, p models.PatchPolicy, now time.Time) {
if p.NextRunAt == nil {
return
}
due := *p.NextRunAt
w, err := deps.LookupWindow(p.InstanceID, p.WindowID)
if err != nil {
log.Printf("patchsched: policy %s: load window: %v", p.PolicyID, err)
return // a database error is retried next tick, not treated as "gone"
}
if w == nil {
disable(ctx, deps, p, "its maintenance window no longer exists")
return
}
end := WindowEnd(due, w.DurationMinutes)
next, err := NextStart(w.Cron, w.TZ, Later(now, end))
if err != nil {
disable(ctx, deps, p, "its maintenance window schedule is no longer valid: "+err.Error())
return
}
// The claim, as in workflowsched: matching the current next_run_at means a
// second process reaching this policy matches nothing.
res, err := db.Col("patch_policies").UpdateOne(ctx,
bson.M{"policy_id": p.PolicyID, "next_run_at": due},
bson.M{"$set": bson.M{"next_run_at": next}})
if err != nil || res.MatchedCount == 0 {
return
}
n, err := deps.CountTargets(p)
if err != nil {
retryLater(ctx, deps, p, err, due, next, now)
return
}
active, err := hasActiveRun(ctx, p)
if err != nil {
retryLater(ctx, deps, p, err, due, next, now)
return
}
switch d := Decide(due, end, now, active, n); d {
case Fire:
if err := deps.StartPolicyRun(p, end); err != nil {
retryLater(ctx, deps, p, err, due, next, now)
return
}
_, _ = db.Col("patch_policies").UpdateOne(ctx, bson.M{"policy_id": p.PolicyID},
bson.M{"$set": bson.M{"last_run_at": now}, "$unset": bson.M{"last_skipped": ""}})
default:
recordSkip(ctx, deps, p, string(d), due, now)
}
}
func hasActiveRun(ctx context.Context, p models.PatchPolicy) (bool, error) {
err := db.Col("patch_runs").FindOne(ctx,
bson.M{"instance_id": p.InstanceID, "policy_id": p.PolicyID, "status": models.PatchRunRunning},
options.FindOne().SetProjection(bson.M{"_id": 1})).Err()
if err == nil {
return true, nil
}
if errors.Is(err, mongo.ErrNoDocuments) {
return false, nil
}
return false, err
}
// retryLater handles an error after the claim. The claim is put back, guarded
// on the value just written so a concurrent edit to the policy is not undone,
// and the next tick retries the same occurrence. Decide's missed rule bounds
// the retries: once the occurrence is too late it is skipped as missed.
func retryLater(ctx context.Context, deps Deps, p models.PatchPolicy, cause error, due, next, now time.Time) {
if _, err := db.Col("patch_policies").UpdateOne(ctx,
bson.M{"policy_id": p.PolicyID, "next_run_at": next},
bson.M{"$set": bson.M{"next_run_at": due}}); err != nil {
log.Printf("patchsched: policy %s: put back claim: %v", p.PolicyID, err)
}
recordSkip(ctx, deps, p, "error: "+cause.Error(), due, now)
}
func recordSkip(ctx context.Context, deps Deps, p models.PatchPolicy, reason string, due, at time.Time) {
_, _ = db.Col("patch_policies").UpdateOne(ctx, bson.M{"policy_id": p.PolicyID},
bson.M{"$set": bson.M{"last_skipped": models.Skip{Reason: reason, Due: due, At: at}}})
deps.LogEvent(p.InstanceID, "patch.skipped", "schedule", "", "",
"patch policy "+p.Name+" skipped "+due.Format(time.RFC3339)+": "+reason)
}
func disable(ctx context.Context, deps Deps, p models.PatchPolicy, reason string) {
_, _ = db.Col("patch_policies").UpdateOne(ctx, bson.M{"policy_id": p.PolicyID}, bson.M{
"$set": bson.M{"enabled": false, "disabled_reason": reason},
"$unset": bson.M{"next_run_at": ""},
})
deps.LogEvent(p.InstanceID, "patch.policy_disabled", "schedule", "", "", "patch policy "+p.Name+" disabled: "+reason)
}
+5
View File
@@ -23,6 +23,11 @@ func StoreInventory(serverID string, r *pb.InventoryReport) error {
set["inventory.memory.used_bytes"] = r.Memory.UsedBytes
}
set["inventory.swap_used_bytes"] = r.SwapUsed
// Kept current so a patch reboot can be proven by a changed boot time,
// independent of clock skew between the host and the control plane.
if r.BootTimeUnix > 0 {
set["inventory.boot_time"] = time.Unix(r.BootTimeUnix, 0).UTC()
}
if r.IncludeStatic {
set["inventory.static_at"] = now
@@ -48,6 +48,10 @@ var ScopedCollections = []string{
"server_workloads",
"status_pages",
"status_incidents",
"maintenance_windows",
"patch_policies",
"patch_runs",
"patch_run_outputs",
}
// collectionRenames maps the two collections whose names change. Ordered so the
+46
View File
@@ -0,0 +1,46 @@
package services
import (
"context"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
"go.mongodb.org/mongo-driver/v2/mongo/options"
)
// EnsurePatchIndexes builds the patching indexes. The command_id index is the
// one that matters: every PatchResult is matched to its server run through it.
func EnsurePatchIndexes() error {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if _, err := db.Col("maintenance_windows").Indexes().CreateOne(ctx, mongo.IndexModel{
Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "window_id", Value: 1}},
Options: options.Index().SetUnique(true),
}); err != nil {
return err
}
if _, err := db.Col("patch_policies").Indexes().CreateMany(ctx, []mongo.IndexModel{
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "policy_id", Value: 1}}, Options: options.Index().SetUnique(true)},
{Keys: bson.D{{Key: "enabled", Value: 1}, {Key: "next_run_at", Value: 1}}},
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "window_id", Value: 1}}},
}); err != nil {
return err
}
if _, err := db.Col("patch_run_outputs").Indexes().CreateOne(ctx, mongo.IndexModel{
Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "run_id", Value: 1}, {Key: "server_id", Value: 1}},
Options: options.Index().SetUnique(true),
}); err != nil {
return err
}
_, err := db.Col("patch_runs").Indexes().CreateMany(ctx, []mongo.IndexModel{
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "run_id", Value: 1}}, Options: options.Index().SetUnique(true)},
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "policy_id", Value: 1}, {Key: "started_at", Value: -1}}},
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "servers.server_id", Value: 1}, {Key: "started_at", Value: -1}}},
{Keys: bson.D{{Key: "status", Value: 1}}},
{Keys: bson.D{{Key: "servers.command_id", Value: 1}}},
})
return err
}
+254
View File
@@ -0,0 +1,254 @@
package services
import (
"context"
"errors"
"fmt"
"strings"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchsched"
"github.com/google/uuid"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
"go.mongodb.org/mongo-driver/v2/mongo/options"
)
var (
ErrPolicyInvalid = errors.New("invalid patch policy")
ErrPolicyNotFound = errors.New("patch policy not found")
)
// ValidatePolicy checks a policy without touching the database. The window's
// existence and the token's tag scope are checked by Create and Update.
func ValidatePolicy(p models.PatchPolicy) error {
if n := strings.TrimSpace(p.Name); n == "" || len(n) > 100 {
return fmt.Errorf("%w: name must be 1 to 100 characters", ErrPolicyInvalid)
}
if p.WindowID == "" {
return fmt.Errorf("%w: a maintenance window is required", ErrPolicyInvalid)
}
if p.Scope != models.PatchScopeAll && p.Scope != models.PatchScopeSecurity {
return fmt.Errorf("%w: scope must be %q or %q", ErrPolicyInvalid, models.PatchScopeAll, models.PatchScopeSecurity)
}
if p.Reboot != models.PatchRebootNever && p.Reboot != models.PatchRebootIfRequired {
return fmt.Errorf("%w: reboot must be %q or %q", ErrPolicyInvalid, models.PatchRebootNever, models.PatchRebootIfRequired)
}
if p.MaxConcurrent < 0 || p.MaxConcurrent > 1000 {
return fmt.Errorf("%w: max concurrent must be between 0 and 1000", ErrPolicyInvalid)
}
// Same rule as workflows: an empty selector matches nothing, and saying so
// at save time beats a policy that silently patches nobody every Sunday.
if len(p.TargetServerIDs) == 0 && len(p.TargetTags) == 0 {
return ErrNoTargets
}
return ValidateTags(p.TargetTags)
}
func nextRunFor(p models.PatchPolicy, w models.MaintenanceWindow, now time.Time) *time.Time {
if !p.Enabled {
return nil
}
start, err := patchsched.NextStart(w.Cron, w.TZ, now)
if err != nil {
return nil
}
return &start
}
func ListPolicies(instanceID string) ([]models.PatchPolicy, error) {
ctx, cancel := patchCtx()
defer cancel()
cur, err := db.Col("patch_policies").Find(ctx, bson.M{"instance_id": instanceID}, options.Find().SetSort(bson.M{"name": 1}))
if err != nil {
return nil, err
}
out := []models.PatchPolicy{}
return out, cur.All(ctx, &out)
}
func GetPolicy(instanceID, policyID string) (*models.PatchPolicy, error) {
ctx, cancel := patchCtx()
defer cancel()
var p models.PatchPolicy
err := db.Col("patch_policies").FindOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID}).Decode(&p)
if errors.Is(err, mongo.ErrNoDocuments) {
return nil, ErrPolicyNotFound
}
return &p, err
}
// preparePolicy runs every check a save needs and returns the policy's window.
func preparePolicy(instanceID string, p *models.PatchPolicy, tokenScope map[string]string) (*models.MaintenanceWindow, error) {
p.Name = strings.TrimSpace(p.Name)
if p.TargetServerIDs == nil {
p.TargetServerIDs = []string{}
}
if err := ValidatePolicy(*p); err != nil {
return nil, err
}
w, err := GetWindow(instanceID, p.WindowID)
if errors.Is(err, ErrWindowNotFound) {
return nil, fmt.Errorf("%w: that maintenance window does not exist", ErrPolicyInvalid)
}
if err != nil {
return nil, err
}
// validateTargetServers returns a plain "target server X not found"; wrap
// it so the handler can answer 400 without matching on text.
if err := validateTargetServers(instanceID, p.TargetServerIDs, tokenScope); err != nil {
return nil, fmt.Errorf("%w: %v", ErrPolicyInvalid, err)
}
if err := validateWorkflowTargetScope(instanceID, p.TargetServerIDs, p.TargetTags, tokenScope); err != nil {
return nil, err
}
return w, nil
}
func CreatePolicy(instanceID string, p models.PatchPolicy, tokenScope map[string]string) (*models.PatchPolicy, error) {
w, err := preparePolicy(instanceID, &p, tokenScope)
if err != nil {
return nil, err
}
ctx, cancel := patchCtx()
defer cancel()
now := time.Now()
p.InstanceID, p.PolicyID = instanceID, uuid.New().String()
p.CreatedAt, p.UpdatedAt = now, now
p.NextRunAt = nextRunFor(p, *w, now)
p.LastRunAt, p.LastSkipped, p.DisabledReason = nil, nil, ""
if _, err := db.Col("patch_policies").InsertOne(ctx, p); err != nil {
return nil, err
}
return &p, nil
}
func UpdatePolicy(instanceID, policyID string, p models.PatchPolicy, tokenScope map[string]string) (*models.PatchPolicy, error) {
// Missing policy answers 404 before any validation error.
if _, err := GetPolicy(instanceID, policyID); err != nil {
return nil, err
}
w, err := preparePolicy(instanceID, &p, tokenScope)
if err != nil {
return nil, err
}
ctx, cancel := patchCtx()
defer cancel()
now := time.Now()
set := bson.M{
"name": p.Name, "enabled": p.Enabled, "window_id": p.WindowID,
"target_server_ids": p.TargetServerIDs, "target_tags": p.TargetTags,
"scope": p.Scope, "reboot": p.Reboot, "max_concurrent": p.MaxConcurrent,
"notify_channel_ids": p.NotifyChannelIDs, "updated_at": now,
}
update := bson.M{"$set": set}
if next := nextRunFor(p, *w, now); next != nil {
set["next_run_at"] = *next
// Re-enabling is the operator's answer to whatever disabled it.
update["$unset"] = bson.M{"disabled_reason": ""}
} else {
update["$unset"] = bson.M{"next_run_at": ""}
}
if _, err := db.Col("patch_policies").UpdateOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID}, update); err != nil {
return nil, err
}
return GetPolicy(instanceID, policyID)
}
func DeletePolicy(instanceID, policyID string) error {
ctx, cancel := patchCtx()
defer cancel()
res, err := db.Col("patch_policies").DeleteOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID})
if err != nil {
return err
}
if res.DeletedCount == 0 {
return ErrPolicyNotFound
}
return nil
}
// recomputePolicySchedules moves every enabled policy on this window to the
// window's next start.
func recomputePolicySchedules(ctx context.Context, w models.MaintenanceWindow) error {
start, err := patchsched.NextStart(w.Cron, w.TZ, time.Now())
if err != nil {
return err
}
_, err = db.Col("patch_policies").UpdateMany(ctx,
bson.M{"instance_id": w.InstanceID, "window_id": w.WindowID, "enabled": true},
bson.M{"$set": bson.M{"next_run_at": start}})
return err
}
// CountPolicyTargets resolves the selector as a run would, at this moment.
func CountPolicyTargets(p models.PatchPolicy) (int, error) {
servers, err := ResolveTargets(p.InstanceID, p.TargetServerIDs, p.TargetTags)
if errors.Is(err, ErrNoTargets) {
return 0, nil
}
return len(servers), err
}
var ErrPatchRunActive = errors.New("a run of this policy is already in progress")
// PoliciesForWindow lists every policy that uses a window, enabled or not.
func PoliciesForWindow(instanceID, windowID string) ([]models.PatchPolicy, error) {
ctx, cancel := patchCtx()
defer cancel()
cur, err := db.Col("patch_policies").Find(ctx, bson.M{"instance_id": instanceID, "window_id": windowID})
if err != nil {
return nil, err
}
out := []models.PatchPolicy{}
return out, cur.All(ctx, &out)
}
// CheckWindowScope refuses a tag-restricted token changing a window when any
// policy using it targets servers outside its restriction: moving or
// deleting the window moves or stops those servers' patching.
func CheckWindowScope(instanceID, windowID string, tokenScope map[string]string) error {
if len(tokenScope) == 0 {
return nil
}
ps, err := PoliciesForWindow(instanceID, windowID)
if err != nil {
return err
}
for _, p := range ps {
if err := CheckPolicyScope(instanceID, p, tokenScope); err != nil {
return err
}
}
return nil
}
// CheckPolicyScope refuses a tag-restricted token acting on a policy whose
// targets reach outside its restriction, with the workflow rule unchanged.
func CheckPolicyScope(instanceID string, p models.PatchPolicy, tokenScope map[string]string) error {
return validateWorkflowTargetScope(instanceID, p.TargetServerIDs, p.TargetTags, tokenScope)
}
// StartRunNow opens a window of the policy's usual length starting now. It is
// how an operator tests a policy on a Tuesday afternoon.
func StartRunNow(instanceID, policyID, actor string, tokenScope map[string]string) (*models.PatchRun, error) {
p, err := GetPolicy(instanceID, policyID)
if err != nil {
return nil, err
}
if err := CheckPolicyScope(instanceID, *p, tokenScope); err != nil {
return nil, err
}
w, err := GetWindow(instanceID, p.WindowID)
if err != nil {
return nil, err
}
ctx, cancel := patchCtx()
defer cancel()
if db.Col("patch_runs").FindOne(ctx, bson.M{"instance_id": instanceID, "policy_id": policyID, "status": models.PatchRunRunning}).Err() == nil {
return nil, ErrPatchRunActive
}
return StartPolicyRun(*p, patchsched.WindowEnd(time.Now(), w.DurationMinutes), models.PatchSourceRunNow, actor)
}
+623
View File
@@ -0,0 +1,623 @@
package services
import (
"context"
"errors"
"fmt"
"log"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/notify"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchrun"
"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
"github.com/google/uuid"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
"go.mongodb.org/mongo-driver/v2/mongo/options"
)
// ErrAgentOffline is declared in consoleproxy.go and reused here: both mean
// the same thing, the target's agent is not on the command stream.
var (
ErrPatchRunNotFound = errors.New("patch run not found")
ErrPatchRunFinished = errors.New("patch run has already finished")
)
const patchRunsCol = "patch_runs"
// patchRunOutputsCol holds each server's output tail, one document per
// (run_id, server_id). Kept out of the run document because a large run with
// up to 64KB of output per server would pass MongoDB's 16MB document limit.
const patchRunOutputsCol = "patch_run_outputs"
// storePatchOutput upserts one server's output tail for a run.
func storePatchOutput(ctx context.Context, instanceID, runID, serverID, output string) error {
_, err := db.Col(patchRunOutputsCol).UpdateOne(ctx,
bson.M{"instance_id": instanceID, "run_id": runID, "server_id": serverID},
bson.M{"$set": bson.M{"output": output, "updated_at": time.Now()}},
options.UpdateOne().SetUpsert(true))
return err
}
// fillPatchOutputs puts each server's stored output back on the run, so the
// API response carries it as before. A run written before outputs moved out
// keeps the output it has inline.
func fillPatchOutputs(ctx context.Context, run *models.PatchRun) error {
cur, err := db.Col(patchRunOutputsCol).Find(ctx, bson.M{"instance_id": run.InstanceID, "run_id": run.RunID})
if err != nil {
return err
}
var outs []struct {
ServerID string `bson:"server_id"`
Output string `bson:"output"`
}
if err := cur.All(ctx, &outs); err != nil {
return err
}
byServer := make(map[string]string, len(outs))
for _, o := range outs {
byServer[o.ServerID] = o.Output
}
for i := range run.Servers {
if out, ok := byServer[run.Servers[i].ServerID]; ok {
run.Servers[i].Output = out
}
}
return nil
}
// serverBootTime is the boot time the server last reported, or nil.
func serverBootTime(ctx context.Context, instanceID, serverID string) *time.Time {
var doc struct {
Inventory struct {
BootTime *time.Time `bson:"boot_time"`
} `bson:"inventory"`
}
if err := db.Col("servers").FindOne(ctx, bson.M{"instance_id": instanceID, "server_id": serverID},
options.FindOne().SetProjection(bson.M{"inventory.boot_time": 1})).Decode(&doc); err != nil {
return nil
}
return doc.Inventory.BootTime
}
func newServerRun(s models.Server, now time.Time) models.PatchServerRun {
r := models.PatchServerRun{ServerID: s.ServerID, Hostname: s.Hostname, Status: models.PatchSrvQueued, PendingBefore: len(s.AvailableUpdates)}
if !patchrun.AgentSupportsPatchResults(s.AgentVersion) {
v := s.AgentVersion
if v == "" {
v = "unknown"
}
r.Status = models.PatchSrvAgentTooOld
r.Error = fmt.Sprintf("agent %s predates patch results; update it to %s or later", v, patchrun.MinAgentVersion)
r.FinishedAt = &now
}
return r
}
// StartPolicyRun records a run for a policy and dispatches its first batch
// immediately. The run document is written before any command leaves, so a
// fast agent's result always finds it.
func StartPolicyRun(p models.PatchPolicy, windowEnd time.Time, source, actor string) (*models.PatchRun, error) {
targets, err := ResolveTargets(p.InstanceID, p.TargetServerIDs, p.TargetTags)
if err != nil {
return nil, err
}
now := time.Now()
run := models.PatchRun{
InstanceID: p.InstanceID, RunID: uuid.New().String(), PolicyID: p.PolicyID, PolicyName: p.Name,
TriggeredBy: actor, Source: source, Scope: p.Scope, Reboot: p.Reboot, MaxConcurrent: p.MaxConcurrent,
WindowEnd: &windowEnd, Status: models.PatchRunRunning, StartedAt: now,
}
for _, s := range targets {
run.Servers = append(run.Servers, newServerRun(s, now))
}
ctx, cancel := patchCtx()
defer cancel()
if _, err := db.Col(patchRunsCol).InsertOne(ctx, run); err != nil {
return nil, err
}
LogEvent(p.InstanceID, "patch.run_started", actor, "", "",
fmt.Sprintf("patch policy %s started on %d servers (%s, reboot %s)", p.Name, len(run.Servers), p.Scope, p.Reboot))
advanceRun(ctx, run.RunID)
return &run, nil
}
// StartManualRun is Apply updates on one server. An agent too old to answer
// still gets the command it always got, and the run says it cannot know the
// outcome rather than claiming one.
func StartManualRun(instanceID string, srv *models.Server, actor, source string) (*models.PatchRun, error) {
now := time.Now()
sr := newServerRun(*srv, now)
legacy := sr.Status == models.PatchSrvAgentTooOld
sr.Status, sr.Error, sr.FinishedAt, sr.StartedAt = models.PatchSrvPatching, "", nil, &now
run := models.PatchRun{
InstanceID: instanceID, RunID: uuid.New().String(), TriggeredBy: actor, Source: source,
Scope: models.PatchScopeAll, Reboot: models.PatchRebootNever, Status: models.PatchRunRunning, StartedAt: now,
}
ctx, cancel := patchCtx()
defer cancel()
var dispatchErr error
if !Dispatcher.IsConnected(srv.ServerID) {
dispatchErr = ErrAgentOffline
}
if dispatchErr == nil && legacy {
dispatchErr = DispatchApplyUpdates(srv.ServerID)
if dispatchErr == nil {
sr.Status = models.PatchSrvSucceeded
sr.Error = "no result reported: agent predates patch results"
sr.FinishedAt = &now
}
}
if dispatchErr != nil {
sr.Status, sr.Error, sr.FinishedAt = models.PatchSrvFailed, dispatchErr.Error(), &now
} else if !legacy {
sr.CommandID = uuid.New().String()
}
run.Servers = []models.PatchServerRun{sr}
if _, err := db.Col(patchRunsCol).InsertOne(ctx, run); err != nil {
return nil, err
}
if dispatchErr == nil && !legacy {
if err := dispatchPatch(srv.ServerID, sr.CommandID, run); err != nil {
_, _ = setServer(ctx, run.RunID, srv.ServerID, models.PatchSrvPatching,
bson.M{"status": models.PatchSrvFailed, "error": err.Error(), "finished_at": time.Now()})
dispatchErr = ErrAgentOffline
}
}
finalizeRun(ctx, run.RunID)
if dispatchErr != nil {
return &run, ErrAgentOffline
}
return &run, nil
}
func dispatchPatch(serverID, commandID string, run models.PatchRun) error {
cmd := &pb.ApplyUpdatesCmd{Scope: run.Scope, RebootIfRequired: run.Reboot == models.PatchRebootIfRequired}
if run.WindowEnd != nil {
cmd.DeadlineUnix = run.WindowEnd.Unix()
}
return Dispatcher.dispatch(serverID, &pb.ServerCommand{CommandId: commandID, ApplyUpdates: cmd})
}
// setServer updates one server run, but only if it is still in status from.
// That guard is what stops a tick from overwriting a result that landed while
// the tick was working.
func setServer(ctx context.Context, runID, serverID, from string, set bson.M) (bool, error) {
fields := bson.M{}
for k, v := range set {
fields["servers.$."+k] = v
}
res, err := db.Col(patchRunsCol).UpdateOne(ctx,
bson.M{"run_id": runID, "servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": from}}},
bson.M{"$set": fields})
if err != nil {
return false, err
}
return res.MatchedCount > 0, nil
}
// setServerForCommand is setServer scoped to one command: a late result from
// an older, already-superseded command (say, a fresh dispatch to the same
// server after a failed send) must not land on the new attempt.
func setServerForCommand(ctx context.Context, runID, serverID, commandID, from string, set bson.M) (bool, error) {
fields := bson.M{}
for k, v := range set {
fields["servers.$."+k] = v
}
res, err := db.Col(patchRunsCol).UpdateOne(ctx,
bson.M{"run_id": runID, "servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": from, "command_id": commandID}}},
bson.M{"$set": fields})
if err != nil {
return false, err
}
return res.MatchedCount > 0, nil
}
// claimServerForDispatch atomically moves one server from queued/waiting_offline
// to patching, guarded on the run still being running and not cancelled: a
// tick that loaded the run before CancelPatchRun wrote cancelled_at must not
// dispatch to it.
func claimServerForDispatch(ctx context.Context, runID, serverID, from, commandID string, now time.Time) (bool, error) {
res, err := db.Col(patchRunsCol).UpdateOne(ctx,
bson.M{
"run_id": runID,
"status": models.PatchRunRunning,
"cancelled_at": bson.M{"$exists": false},
"servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": from}},
},
bson.M{"$set": bson.M{
"servers.$.status": models.PatchSrvPatching,
"servers.$.command_id": commandID,
"servers.$.started_at": now,
}})
if err != nil {
return false, err
}
return res.MatchedCount > 0, nil
}
// loadRun reads a run for the tick and result paths, which never need output,
// so any inline output left on an old document is not read either.
func loadRun(ctx context.Context, filter bson.M) (*models.PatchRun, error) {
var run models.PatchRun
err := db.Col(patchRunsCol).FindOne(ctx, filter, options.FindOne().SetProjection(bson.M{"servers.output": 0})).Decode(&run)
if errors.Is(err, mongo.ErrNoDocuments) {
return nil, ErrPatchRunNotFound
}
return &run, err
}
// AdvancePatchRuns is the scheduler's second job each tick.
func AdvancePatchRuns(ctx context.Context) {
cur, err := db.Col(patchRunsCol).Find(ctx, bson.M{"status": models.PatchRunRunning}, options.Find().SetProjection(bson.M{"run_id": 1}))
if err != nil {
log.Printf("patch runs: find running: %v", err)
return
}
var ids []struct {
RunID string `bson:"run_id"`
}
if err := cur.All(ctx, &ids); err != nil {
log.Printf("patch runs: decode running: %v", err)
return
}
for _, r := range ids {
if ctx.Err() != nil {
return
}
advanceRun(ctx, r.RunID)
}
}
func advanceRun(ctx context.Context, runID string) {
run, err := loadRun(ctx, bson.M{"run_id": runID})
if err != nil {
log.Printf("patch run %s: load: %v", runID, err)
return
}
now := time.Now()
connected := map[string]bool{}
for _, s := range run.Servers {
if !patchrun.IsTerminal(s.Status) {
connected[s.ServerID] = Dispatcher.IsConnected(s.ServerID)
}
}
for _, tr := range patchrun.Advance(*run, now, connected) {
if tr.Dispatch {
// Each server's claim and any reset-after-failed-send get their
// own fresh, short-lived context rather than sharing the caller's:
// advanceRun can dispatch to many servers in one tick, each ack
// waiting up to Dispatcher's own timeout, and a caller-supplied
// context (StartPolicyRun's patchCtx, or a leader context that
// AdvancePatchRuns loses partway through) would otherwise expire
// mid-loop and silently drop a later server's writes.
cmdID := uuid.New().String()
claimCtx, claimCancel := patchCtx()
ok, err := claimServerForDispatch(claimCtx, runID, tr.ServerID, tr.From, cmdID, now)
claimCancel()
if err != nil {
log.Printf("patch run %s: server %s: claim: %v", runID, tr.ServerID, err)
continue
}
if !ok {
continue
}
if err := dispatchPatch(tr.ServerID, cmdID, *run); err != nil {
// The agent dropped between the connection check and the send.
// Back to waiting: the next tick tries again while the window
// is open.
resetCtx, resetCancel := patchCtx()
if _, rerr := setServerForCommand(resetCtx, runID, tr.ServerID, cmdID, models.PatchSrvPatching,
bson.M{"status": models.PatchSrvWaitingOffline, "command_id": ""}); rerr != nil {
log.Printf("patch run %s: server %s: reset after failed dispatch: %v", runID, tr.ServerID, rerr)
}
resetCancel()
}
continue
}
set := bson.M{"status": tr.To}
if tr.Error != "" {
set["error"] = tr.Error
}
if patchrun.IsTerminal(tr.To) {
set["finished_at"] = now
}
if _, err := setServer(ctx, runID, tr.ServerID, tr.From, set); err != nil {
log.Printf("patch run %s: server %s: %v", runID, tr.ServerID, err)
}
}
finalizeRun(ctx, runID)
}
func finalizeRun(ctx context.Context, runID string) {
run, err := loadRun(ctx, bson.M{"run_id": runID})
if err != nil || run.Status != models.PatchRunRunning {
return
}
status, done := patchrun.Finalize(*run)
if !done {
return
}
now := time.Now()
res, err := db.Col(patchRunsCol).UpdateOne(ctx,
bson.M{"run_id": runID, "status": models.PatchRunRunning},
bson.M{"$set": bson.M{"status": status, "finished_at": now}})
if err != nil || res.MatchedCount == 0 {
return // finalised by someone else
}
run.Status = status
name := run.PolicyName
if name == "" {
name = "manual update"
}
LogEvent(run.InstanceID, "patch.run_finished", run.TriggeredBy, "", "",
fmt.Sprintf("patch run %s (%s) %s: %s", runID, name, status, patchrun.Summary(*run)))
if status == models.PatchRunPartial || status == models.PatchRunFailed ||
(status == models.PatchRunCancelled && hasBadOutcome(*run)) {
notifyPatchRun(*run)
}
}
// hasBadOutcome reports whether any server failed or was not reached. A
// cancelled run with such a result still alerts: cancelling does not make a
// failure somebody else's problem.
func hasBadOutcome(run models.PatchRun) bool {
for _, s := range run.Servers {
switch s.Status {
case models.PatchSrvFailed, models.PatchSrvMissedOffline, models.PatchSrvWindowClosed:
return true
}
}
return false
}
func notifyPatchRun(run models.PatchRun) {
if run.PolicyID == "" {
return // a manual run was watched by the person who clicked
}
p, err := GetPolicy(run.InstanceID, run.PolicyID)
if err != nil || len(p.NotifyChannelIDs) == 0 {
return
}
chs, err := GetChannels(run.InstanceID, p.NotifyChannelIDs)
if err != nil {
log.Printf("patch run %s: channels: %v", run.RunID, err)
return
}
ev := notify.Event{
MonitorName: fmt.Sprintf("Patch policy %q %s", run.PolicyName, run.Status),
Type: notify.TypePatch,
NewStatus: run.Status,
Message: fmt.Sprintf("%s (run %s)", patchrun.Summary(run), run.RunID),
Time: time.Now(),
}
for _, ch := range chs {
if err := notify.Dispatch(ch, ev); err != nil {
log.Printf("patch run %s: notify %s: %v", run.RunID, ch.Name, err)
}
}
}
// RecordPatchResult is called by whichever pod holds the agent's stream. The
// filter names this agent's own server, so one agent cannot answer for
// another's command.
func RecordPatchResult(instanceID, serverID string, r *pb.PatchResult) {
ctx, cancel := patchCtx()
defer cancel()
run, err := loadRun(ctx, bson.M{"instance_id": instanceID,
"servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "command_id": r.CommandId}}})
if err != nil {
log.Printf("patch result %s from %s matches no run: %v", r.CommandId, serverID, err)
return
}
for _, s := range run.Servers {
if s.ServerID != serverID || s.CommandID != r.CommandId {
continue
}
updated, ok := patchrun.ApplyResult(s, r, time.Now())
if !ok {
return
}
// The output goes to patch_run_outputs, not the run document.
set := bson.M{"status": updated.Status, "error": updated.Error}
if updated.PendingAfter != nil {
set["pending_after"] = *updated.PendingAfter
}
if updated.RebootedAt != nil {
set["rebooted_at"] = *updated.RebootedAt
}
if updated.Status == models.PatchSrvRebooting {
// The boot time before the reboot: a later report with a
// different one proves the restart.
if bt := serverBootTime(ctx, instanceID, serverID); bt != nil {
set["boot_time_before"] = *bt
}
}
if updated.FinishedAt != nil {
set["finished_at"] = *updated.FinishedAt
}
ok, err := setServerForCommand(ctx, run.RunID, serverID, r.CommandId, models.PatchSrvPatching, set)
if err != nil {
log.Printf("patch run %s: server %s: record result: %v", run.RunID, serverID, err)
}
if !ok {
return
}
if err := storePatchOutput(ctx, instanceID, run.RunID, serverID, updated.Output); err != nil {
log.Printf("patch run %s: server %s: store output: %v", run.RunID, serverID, err)
}
if updated.Status == models.PatchSrvRebooting {
LogEvent(instanceID, "patch.reboot", run.TriggeredBy, serverID, "",
fmt.Sprintf("%s rebooting for patch policy %s", s.Hostname, run.PolicyName))
}
}
finalizeRun(ctx, run.RunID)
}
// VerifyPatchReboots settles any rebooting server run for this server from a
// static inventory report.
func VerifyPatchReboots(instanceID, serverID string, bootTime time.Time, rebootRequired bool) {
ctx, cancel := patchCtx()
defer cancel()
cur, err := db.Col(patchRunsCol).Find(ctx, bson.M{"instance_id": instanceID, "status": models.PatchRunRunning,
"servers": bson.M{"$elemMatch": bson.M{"server_id": serverID, "status": models.PatchSrvRebooting}}})
if err != nil {
return
}
var runs []models.PatchRun
if err := cur.All(ctx, &runs); err != nil {
return
}
now := time.Now()
for _, run := range runs {
for _, s := range run.Servers {
if s.ServerID != serverID {
continue
}
updated, ok := patchrun.VerifyReboot(s, bootTime, rebootRequired, now)
if !ok {
continue
}
set := bson.M{"status": updated.Status, "error": updated.Error, "finished_at": now}
if updated.VerifiedAt != nil {
set["verified_at"] = *updated.VerifiedAt
}
_, _ = setServer(ctx, run.RunID, serverID, models.PatchSrvRebooting, set)
}
finalizeRun(ctx, run.RunID)
}
}
// CancelPatchRun stops further dispatch. Servers already patching finish:
// killing a package manager mid-transaction is worse than letting it end.
func CancelPatchRun(instanceID, runID string) error {
ctx, cancel := patchCtx()
defer cancel()
res, err := db.Col(patchRunsCol).UpdateOne(ctx,
bson.M{"instance_id": instanceID, "run_id": runID, "status": models.PatchRunRunning, "cancelled_at": bson.M{"$exists": false}},
bson.M{"$set": bson.M{"cancelled_at": time.Now()}})
if err != nil {
return err
}
if res.MatchedCount == 0 {
if _, err := GetPatchRun(instanceID, runID); err != nil {
return err
}
return ErrPatchRunFinished
}
advanceRun(ctx, runID)
return nil
}
func GetPatchRun(instanceID, runID string) (*models.PatchRun, error) {
ctx, cancel := patchCtx()
defer cancel()
var run models.PatchRun
err := db.Col(patchRunsCol).FindOne(ctx, bson.M{"instance_id": instanceID, "run_id": runID}).Decode(&run)
if errors.Is(err, mongo.ErrNoDocuments) {
return nil, ErrPatchRunNotFound
}
if err != nil {
return nil, err
}
if err := fillPatchOutputs(ctx, &run); err != nil {
return nil, err
}
return &run, nil
}
// ListPatchRuns omits output: a list of fifty runs would otherwise carry up
// to 64KB per server.
func ListPatchRuns(instanceID, policyID, serverID string, limit int64) ([]models.PatchRun, error) {
ctx, cancel := patchCtx()
defer cancel()
filter := bson.M{"instance_id": instanceID}
if policyID != "" {
filter["policy_id"] = policyID
}
if serverID != "" {
filter["servers.server_id"] = serverID
}
if limit <= 0 || limit > 200 {
limit = 50
}
cur, err := db.Col(patchRunsCol).Find(ctx, filter, options.Find().
SetSort(bson.M{"started_at": -1}).SetLimit(limit).SetProjection(bson.M{"servers.output": 0}))
if err != nil {
return nil, err
}
out := []models.PatchRun{}
return out, cur.All(ctx, &out)
}
// ScopePatchRun removes servers outside a tag-restricted token's reach, so a
// run record cannot name a host the caller could not otherwise see.
func ScopePatchRun(instanceID string, run *models.PatchRun, tokenScope map[string]string) error {
if len(tokenScope) == 0 {
return nil
}
ids := make([]string, 0, len(run.Servers))
for _, s := range run.Servers {
ids = append(ids, s.ServerID)
}
visible, err := ResolveTargetsScoped(instanceID, ids, nil, tokenScope)
if err != nil && !errors.Is(err, ErrNoTargets) {
return err
}
keep := map[string]bool{}
for _, s := range visible {
keep[s.ServerID] = true
}
kept := run.Servers[:0]
for _, s := range run.Servers {
if keep[s.ServerID] {
kept = append(kept, s)
}
}
run.Servers = kept
return nil
}
// sweepPatchRuns deletes finished runs past their instance's workflow log
// retention: they are the same kind of record, and one setting governs both.
func sweepPatchRuns() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
cur, err := db.Col(patchRunsCol).Find(ctx, bson.M{"finished_at": bson.M{"$ne": nil}},
options.Find().SetProjection(bson.M{"run_id": 1, "instance_id": 1, "finished_at": 1}))
if err != nil {
log.Printf("patch run sweep: %v", err)
return
}
defer cur.Close(ctx)
cache := map[string]int{}
now := time.Now()
for cur.Next(ctx) {
var r struct {
RunID string `bson:"run_id"`
InstanceID string `bson:"instance_id"`
FinishedAt *time.Time `bson:"finished_at"`
}
if cur.Decode(&r) != nil || r.FinishedAt == nil {
continue
}
days, ok := cache[r.InstanceID]
if !ok {
days = defaultRetentionDays
if v, err := GetWorkflowLogRetentionDays(r.InstanceID); err == nil {
days = v
}
cache[r.InstanceID] = days
}
if days <= 0 || !r.FinishedAt.Before(now.AddDate(0, 0, -days)) {
continue
}
if _, err := db.Col(patchRunsCol).DeleteOne(ctx, bson.M{"run_id": r.RunID}); err == nil {
_, _ = db.Col(patchRunOutputsCol).DeleteMany(ctx, bson.M{"instance_id": r.InstanceID, "run_id": r.RunID})
}
}
}
@@ -0,0 +1,20 @@
package services
import "testing"
// A tenant-scoped collection missing from ScopedCollections outlives its
// instance when the instance is purged.
func TestPatchCollectionsAreScoped(t *testing.T) {
for _, name := range []string{"maintenance_windows", "patch_policies", "patch_runs", "patch_run_outputs"} {
found := false
for _, got := range ScopedCollections {
if got == name {
found = true
break
}
}
if !found {
t.Errorf("ScopedCollections is missing %q", name)
}
}
}
@@ -0,0 +1,90 @@
package services
import (
"errors"
"strings"
"testing"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
)
func goodWindow() models.MaintenanceWindow {
return models.MaintenanceWindow{Name: "Sunday", Cron: "0 2 * * 0", TZ: "Europe/London", DurationMinutes: 120}
}
func TestValidateWindow(t *testing.T) {
if err := ValidateWindow(goodWindow()); err != nil {
t.Fatalf("good window: %v", err)
}
bad := map[string]func(*models.MaintenanceWindow){
"empty name": func(w *models.MaintenanceWindow) { w.Name = " " },
"long name": func(w *models.MaintenanceWindow) { w.Name = strings.Repeat("a", 101) },
"too short": func(w *models.MaintenanceWindow) { w.DurationMinutes = 14 },
"too long": func(w *models.MaintenanceWindow) { w.DurationMinutes = 721 },
"bad cron": func(w *models.MaintenanceWindow) { w.Cron = "every sunday" },
"six-field cron": func(w *models.MaintenanceWindow) { w.Cron = "0 0 2 * * 0" },
"bad tz": func(w *models.MaintenanceWindow) { w.TZ = "Mars/Olympus" },
"no tz": func(w *models.MaintenanceWindow) { w.TZ = "" },
}
for name, mut := range bad {
w := goodWindow()
mut(&w)
if err := ValidateWindow(w); !errors.Is(err, ErrWindowInvalid) {
t.Errorf("%s: err = %v, want ErrWindowInvalid", name, err)
}
}
}
func TestPreviewWindowSpansDoNotOverlap(t *testing.T) {
from := time.Date(2026, 9, 14, 12, 0, 0, 0, time.UTC)
spans, err := PreviewWindow("0 * * * *", "UTC", 90, from, 3)
if err != nil {
t.Fatal(err)
}
if len(spans) != 3 {
t.Fatalf("got %d spans", len(spans))
}
for i := 1; i < len(spans); i++ {
if spans[i].Start.Before(spans[i-1].End) {
t.Fatalf("span %d starts %s before previous ends %s", i, spans[i].Start, spans[i-1].End)
}
}
if !spans[0].End.Equal(spans[0].Start.Add(90 * time.Minute)) {
t.Fatal("end must be start + duration")
}
}
func goodPolicy() models.PatchPolicy {
return models.PatchPolicy{
Name: "Sunday prod", WindowID: "w1", TargetTags: map[string]string{"env": "prod"},
Scope: models.PatchScopeSecurity, Reboot: models.PatchRebootNever,
}
}
func TestValidatePolicy(t *testing.T) {
if err := ValidatePolicy(goodPolicy()); err != nil {
t.Fatalf("good policy: %v", err)
}
bad := map[string]func(*models.PatchPolicy){
"empty name": func(p *models.PatchPolicy) { p.Name = "" },
"no window": func(p *models.PatchPolicy) { p.WindowID = "" },
"bad scope": func(p *models.PatchPolicy) { p.Scope = "everything" },
"bad reboot": func(p *models.PatchPolicy) { p.Reboot = "always" },
"negative cap": func(p *models.PatchPolicy) { p.MaxConcurrent = -1 },
"huge cap": func(p *models.PatchPolicy) { p.MaxConcurrent = 1001 },
"uppercase tag": func(p *models.PatchPolicy) { p.TargetTags = map[string]string{"Env": "prod"} },
}
for name, mut := range bad {
p := goodPolicy()
mut(&p)
if err := ValidatePolicy(p); err == nil {
t.Errorf("%s: want an error", name)
}
}
p := goodPolicy()
p.TargetTags = nil
if err := ValidatePolicy(p); !errors.Is(err, ErrNoTargets) {
t.Errorf("empty selector: err = %v, want ErrNoTargets", err)
}
}
+166
View File
@@ -0,0 +1,166 @@
package services
import (
"context"
"errors"
"fmt"
"strings"
"time"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/patchsched"
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
"github.com/google/uuid"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
"go.mongodb.org/mongo-driver/v2/mongo/options"
)
var (
ErrWindowInvalid = errors.New("invalid maintenance window")
ErrWindowNotFound = errors.New("maintenance window not found")
ErrWindowInUse = errors.New("maintenance window is used by a patch policy")
)
const (
minWindowMinutes = 15
maxWindowMinutes = 720
)
func patchCtx() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), 10*time.Second)
}
func ValidateWindow(w models.MaintenanceWindow) error {
if n := strings.TrimSpace(w.Name); n == "" || len(n) > 100 {
return fmt.Errorf("%w: name must be 1 to 100 characters", ErrWindowInvalid)
}
if w.DurationMinutes < minWindowMinutes || w.DurationMinutes > maxWindowMinutes {
return fmt.Errorf("%w: duration must be between %d and %d minutes", ErrWindowInvalid, minWindowMinutes, maxWindowMinutes)
}
if _, err := workflowsched.ParseSchedule(w.Cron, w.TZ); err != nil {
return fmt.Errorf("%w: %v", ErrWindowInvalid, err)
}
return nil
}
type WindowSpan struct {
Start time.Time `json:"start"`
End time.Time `json:"end"`
}
// PreviewWindow returns the next n windows, computed exactly as the scheduler
// computes them, so the editor cannot disagree with what will fire.
func PreviewWindow(cron, tz string, durationMinutes int, from time.Time, n int) ([]WindowSpan, error) {
if err := ValidateWindow(models.MaintenanceWindow{Name: "preview", Cron: cron, TZ: tz, DurationMinutes: durationMinutes}); err != nil {
return nil, err
}
out := make([]WindowSpan, 0, n)
for len(out) < n {
start, err := patchsched.NextStart(cron, tz, from)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrWindowInvalid, err)
}
end := patchsched.WindowEnd(start, durationMinutes)
out = append(out, WindowSpan{Start: start, End: end})
from = patchsched.Later(start, end)
}
return out, nil
}
func ListWindows(instanceID string) ([]models.MaintenanceWindow, error) {
ctx, cancel := patchCtx()
defer cancel()
cur, err := db.Col("maintenance_windows").Find(ctx, bson.M{"instance_id": instanceID}, options.Find().SetSort(bson.M{"name": 1}))
if err != nil {
return nil, err
}
out := []models.MaintenanceWindow{}
return out, cur.All(ctx, &out)
}
func GetWindow(instanceID, windowID string) (*models.MaintenanceWindow, error) {
w, err := LookupWindow(instanceID, windowID)
if err == nil && w == nil {
return nil, ErrWindowNotFound
}
return w, err
}
// LookupWindow is GetWindow for the scheduler, which must tell "gone" (nil,
// nil: disable the policy) from a database error (retry next tick).
func LookupWindow(instanceID, windowID string) (*models.MaintenanceWindow, error) {
ctx, cancel := patchCtx()
defer cancel()
var w models.MaintenanceWindow
err := db.Col("maintenance_windows").FindOne(ctx, bson.M{"instance_id": instanceID, "window_id": windowID}).Decode(&w)
if errors.Is(err, mongo.ErrNoDocuments) {
return nil, nil
}
if err != nil {
return nil, err
}
return &w, nil
}
func CreateWindow(instanceID string, w models.MaintenanceWindow) (*models.MaintenanceWindow, error) {
w.Name = strings.TrimSpace(w.Name)
if err := ValidateWindow(w); err != nil {
return nil, err
}
ctx, cancel := patchCtx()
defer cancel()
w.InstanceID, w.WindowID = instanceID, uuid.New().String()
w.CreatedAt = time.Now()
w.UpdatedAt = w.CreatedAt
if _, err := db.Col("maintenance_windows").InsertOne(ctx, w); err != nil {
return nil, err
}
return &w, nil
}
// UpdateWindow saves the window and moves next_run_at on every enabled policy
// that uses it, so an edited Sunday becomes the next Sunday everywhere at once.
func UpdateWindow(instanceID, windowID string, w models.MaintenanceWindow) (*models.MaintenanceWindow, error) {
w.Name = strings.TrimSpace(w.Name)
if err := ValidateWindow(w); err != nil {
return nil, err
}
ctx, cancel := patchCtx()
defer cancel()
res, err := db.Col("maintenance_windows").UpdateOne(ctx,
bson.M{"instance_id": instanceID, "window_id": windowID},
bson.M{"$set": bson.M{"name": w.Name, "cron": w.Cron, "tz": w.TZ, "duration_minutes": w.DurationMinutes, "updated_at": time.Now()}})
if err != nil {
return nil, err
}
if res.MatchedCount == 0 {
return nil, ErrWindowNotFound
}
saved, err := GetWindow(instanceID, windowID)
if err != nil {
return nil, err
}
return saved, recomputePolicySchedules(ctx, *saved)
}
func DeleteWindow(instanceID, windowID string) error {
ctx, cancel := patchCtx()
defer cancel()
n, err := db.Col("patch_policies").CountDocuments(ctx, bson.M{"instance_id": instanceID, "window_id": windowID})
if err != nil {
return err
}
if n > 0 {
return ErrWindowInUse
}
res, err := db.Col("maintenance_windows").DeleteOne(ctx, bson.M{"instance_id": instanceID, "window_id": windowID})
if err != nil {
return err
}
if res.DeletedCount == 0 {
return ErrWindowNotFound
}
return nil
}
+3 -2
View File
@@ -11,8 +11,8 @@ import (
// the vocabulary below.
var ErrInvalidScope = errors.New("invalid scope")
// ScopeResources is the whole vocabulary. Ten resources, each with :read and
// :write, and write implies read on the same resource.
// ScopeResources is the whole vocabulary. Eleven resources, each with :read
// and :write, and write implies read on the same resource.
//
// It is deliberately coarse. A scope per endpoint is a table nobody maintains,
// and a route added without an entry either fails closed and breaks, or
@@ -27,6 +27,7 @@ var ScopeResources = []string{
"workloads",
"settings",
"status",
"patching",
// mcp:read is permission to reach the MCP endpoint at all; mcp:write is
// permission for its write tools, which are not merely refused without it
// but omitted from tools/list entirely.
+2
View File
@@ -332,6 +332,7 @@ func HasServerRunLog(runID, serverID string) bool {
func StartLogSweeper(ctx context.Context) {
go func() {
sweepLogs()
sweepPatchRuns()
t := time.NewTicker(time.Hour)
defer t.Stop()
for {
@@ -340,6 +341,7 @@ func StartLogSweeper(ctx context.Context) {
return
case <-t.C:
sweepLogs()
sweepPatchRuns()
}
}
}()
+68
View File
@@ -0,0 +1,68 @@
"use client";
import { Suspense } from "react";
import { useRouter, useSearchParams } from "next/navigation";
import { useAuth } from "@/components/AuthProvider";
import { Card, CenteredSpinner } from "@/components/ui";
import { PolicyList } from "@/components/patching/PolicyList";
import { WindowList } from "@/components/patching/WindowList";
import { RunList } from "@/components/patching/RunList";
const TABS = [
{ id: "policies", label: "Policies" },
{ id: "windows", label: "Windows" },
{ id: "runs", label: "Runs" },
] as const;
type TabId = (typeof TABS)[number]["id"];
// useSearchParams is only allowed inside a Suspense boundary, as on the
// servers page.
export default function PatchingPage() {
return (
<Suspense fallback={<CenteredSpinner label="Loading patching" />}>
<PatchingInner />
</Suspense>
);
}
function PatchingInner() {
const router = useRouter();
const params = useSearchParams();
const { isAdmin } = useAuth();
const tab: TabId = (TABS.find((t) => t.id === params.get("tab"))?.id ?? "policies") as TabId;
return (
<div className="p-4 sm:p-6 lg:p-8">
<div className="mb-6">
<h1 className="text-2xl font-semibold text-text-primary">Patching</h1>
<p className="mt-1 max-w-2xl text-sm text-text-secondary">
Patch servers inside maintenance windows. Every patch Vantage performs, scheduled or clicked, is recorded as a run with a result for each server.
</p>
</div>
<div role="tablist" aria-label="Patching" className="mb-4 flex gap-1 border-b border-border">
{TABS.map((t) => (
<button
key={t.id}
role="tab"
id={`patching-tab-${t.id}`}
aria-selected={tab === t.id}
onClick={() => router.replace(`/patching?tab=${t.id}`)}
className={`-mb-px border-b-2 px-4 py-2 text-sm transition-colors focus:outline-none focus-visible:ring-2 focus-visible:ring-accent ${
tab === t.id ? "border-accent text-text-primary" : "border-transparent text-text-secondary hover:text-text-primary"
}`}
>
{t.label}
</button>
))}
</div>
<Card padding={false} role="tabpanel" aria-labelledby={`patching-tab-${tab}`}>
{tab === "policies" && <PolicyList canEdit={isAdmin} />}
{tab === "windows" && <WindowList canEdit={isAdmin} />}
{tab === "runs" && <RunList />}
</Card>
</div>
);
}
@@ -0,0 +1,142 @@
"use client";
import Link from "next/link";
import { useState } from "react";
import { useParams } from "next/navigation";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { api, PatchServerRun } from "@/lib/api";
import { AsyncBoundary, Badge, Button, Card, CenteredSpinner, Table, Tbody, Td, Th, Thead, Tr, friendlyMessage, useToast } from "@/components/ui";
import { RUN_STATUS, SERVER_STATUS } from "@/components/patching/status";
function installed(s: PatchServerRun): string {
if (s.pending_after === undefined || s.pending_after === null) return "n/a";
return String(Math.max(0, s.pending_before - s.pending_after));
}
function ServerRow({ s }: { s: PatchServerRun }) {
const [open, setOpen] = useState(false);
const meta = SERVER_STATUS[s.status];
return (
<>
<Tr>
<Td label="Server">
<Link href={`/servers/${s.server_id}`} className="font-mono text-sm text-accent hover:underline">
{s.hostname}
</Link>
</Td>
<Td label="Status">
<Badge variant={meta.variant}>{meta.label}</Badge>
</Td>
<Td label="Installed">
<span className="font-mono text-sm tabular-nums">{installed(s)}</span>
</Td>
<Td label="Reboot">
<span className="text-xs text-text-secondary">
{s.rebooted_at ? `rebooted ${new Date(s.rebooted_at).toLocaleTimeString()}` : "none"}
{s.verified_at && `, back ${new Date(s.verified_at).toLocaleTimeString()}`}
</span>
</Td>
<Td label="Detail">
<div className="flex items-center justify-end gap-3">
{s.error && <span className="max-w-xs truncate text-xs text-danger" title={s.error}>{s.error}</span>}
{s.output && (
<Button variant="ghost" size="sm" onClick={() => setOpen(!open)} aria-expanded={open}>
{open ? "Hide output" : "Output"}
</Button>
)}
</div>
</Td>
</Tr>
{open && s.output && (
<Tr>
<Td colSpan={5}>
<pre className="max-h-96 overflow-auto rounded bg-well px-4 py-3 font-mono text-[11.5px] leading-relaxed text-text-secondary">{s.output}</pre>
</Td>
</Tr>
)}
</>
);
}
export default function PatchRunPage() {
const { runId } = useParams<{ runId: string }>();
const queryClient = useQueryClient();
const toast = useToast();
const { data: run, isLoading, error, refetch } = useQuery({
queryKey: ["patch-run", runId],
queryFn: () => api.getPatchRun(runId),
refetchInterval: (q) => (q.state.data?.status === "running" ? 5_000 : false),
});
const cancel = useMutation({
mutationFn: () => api.cancelPatchRun(runId),
onSuccess: () => {
toast.success("Cancelled. Servers already patching will finish; nothing further starts.");
queryClient.invalidateQueries({ queryKey: ["patch-run", runId] });
},
onError: (e) => toast.error(friendlyMessage(e)),
});
if (isLoading) return <CenteredSpinner />;
const counts = new Map<string, number>();
run?.servers.forEach((s) => counts.set(s.status, (counts.get(s.status) ?? 0) + 1));
return (
<div className="p-4 sm:p-6 lg:p-8">
<AsyncBoundary isLoading={false} error={error} onRetry={refetch}>
{run && (
<>
<Link href="/patching?tab=runs" className="text-sm text-text-secondary hover:text-accent">
Back to patch runs
</Link>
<div className="mt-3 flex flex-wrap items-start justify-between gap-4">
<div>
<h1 className="text-2xl font-semibold text-text-primary">{run.policy_name ?? "Manual update"}</h1>
<p className="mt-1 text-sm text-text-secondary">
Started {new Date(run.started_at).toLocaleString()} by {run.triggered_by}.{" "}
{run.scope === "security" ? "Security updates only" : "All pending updates"}, {run.reboot === "if_required" ? "reboot if required" : "no reboot"}.
{run.window_end && ` Window ends ${new Date(run.window_end).toLocaleString()}.`}
</p>
</div>
<div className="flex items-center gap-3">
<Badge variant={RUN_STATUS[run.status].variant}>{RUN_STATUS[run.status].label}</Badge>
{run.status === "running" && !run.cancelled_at && (
<Button variant="secondary" size="sm" loading={cancel.isPending} onClick={() => cancel.mutate()}>
Cancel run
</Button>
)}
</div>
</div>
<div className="mt-4 flex flex-wrap gap-x-6 gap-y-2">
{[...counts.entries()].map(([status, n]) => (
<span key={status} className="text-sm text-text-secondary">
<span className="font-mono tabular-nums text-text-primary">{n}</span> {SERVER_STATUS[status as PatchServerRun["status"]].label}
</span>
))}
</div>
<Card padding={false} className="mt-6">
<Table>
<Thead>
<Tr>
<Th>Server</Th>
<Th>Status</Th>
<Th>Installed</Th>
<Th>Reboot</Th>
<Th />
</Tr>
</Thead>
<Tbody>
{run.servers.map((s) => (
<ServerRow key={s.server_id} s={s} />
))}
</Tbody>
</Table>
</Card>
</>
)}
</AsyncBoundary>
</div>
);
}
+7 -1
View File
@@ -130,7 +130,13 @@ export default function ServerDetailPage() {
const { mutate: applyUpdates, isPending: isApplying } = useMutation({
mutationFn: () => api.applyUpdates(serverId),
onSuccess: () => toast.success("Update command sent. Patching runs in the background and may take several minutes."),
onSuccess: (res) => {
if (res.run_id) {
router.push(`/patching/runs/${res.run_id}`);
} else {
toast.success("Update command sent.");
}
},
onError: toast.error,
});
+6 -4
View File
@@ -2,6 +2,7 @@
import { useMemo, useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { useRouter } from "next/navigation";
import { api, vulnerabilities, type FindingState, type Severity, type VulnFinding } from "@/lib/api";
import { useAuth } from "@/components/AuthProvider";
import { Button, Card, Pagination, usePagination, useToast } from "@/components/ui";
@@ -42,6 +43,7 @@ const FIX_FILTERS: { key: string; label: string; hasFix: boolean | undefined }[]
export default function VulnerabilitiesPage() {
const { isAdmin } = useAuth();
const qc = useQueryClient();
const router = useRouter();
const [state, setState] = useState<FindingState>("open");
const [severity, setSeverity] = useState<Severity | "">("");
@@ -112,10 +114,10 @@ export default function VulnerabilitiesPage() {
});
const applyUpdates = useMutation({
mutationFn: (serverId: string) => api.applyUpdates(serverId),
// This one had no feedback of any kind: the button dispatched a patch
// run to a whole server and the page did not change in any way.
onSuccess: (_data, serverId) => toast.success(`Update command sent to ${serverName(serverId)}.`),
mutationFn: (serverId: string) => api.applyUpdates(serverId, "vulnerabilities"),
onSuccess: (res) => {
if (res.run_id) router.push(`/patching/runs/${res.run_id}`);
},
onError: toast.error,
});
+9
View File
@@ -176,6 +176,14 @@ function TokenIcon() {
);
}
function PatchIcon() {
return (
<svg className="h-5 w-5" fill="none" viewBox="0 0 24 24" stroke="currentColor" strokeWidth={1.5}>
<path strokeLinecap="round" strokeLinejoin="round" d="M6.75 3v2.25M17.25 3v2.25M3 18.75V7.5a2.25 2.25 0 012.25-2.25h13.5A2.25 2.25 0 0121 7.5v11.25m-18 0A2.25 2.25 0 005.25 21h13.5A2.25 2.25 0 0021 18.75m-18 0v-7.5A2.25 2.25 0 015.25 9h13.5A2.25 2.25 0 0121 11.25v7.5m-9-6l-1.5 3h3l-1.5 3" />
</svg>
);
}
const navGroups: NavGroup[] = [
{
label: "Fleet",
@@ -184,6 +192,7 @@ const navGroups: NavGroup[] = [
{ href: "/workloads", label: "Workloads", icon: <WorkloadIcon /> },
{ href: "/monitors", label: "Monitors", icon: <MonitorIcon /> },
{ href: "/vulnerabilities", label: "Vulnerabilities", icon: <ShieldIcon /> },
{ href: "/patching", label: "Patching", icon: <PatchIcon /> },
],
},
{
+188
View File
@@ -0,0 +1,188 @@
"use client";
import Link from "next/link";
import { useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { useRouter } from "next/navigation";
import { api, PatchPolicy } from "@/lib/api";
import { resolveTargets } from "@/lib/targets";
import { AsyncBoundary, Badge, Button, ConfirmDialog, EmptyState, Table, TableSkeleton, Tbody, Td, Th, Thead, Tr, friendlyMessage, useToast } from "@/components/ui";
import { PolicyModal } from "./PolicyModal";
import { RUN_STATUS } from "./status";
const SKIP_REASON: Record<string, string> = {
missed: "the control plane was not running when the window opened",
already_running: "the previous run was still going",
no_targets: "no servers matched",
};
// runNowBody says what clicking Run now does to real machines, before it does.
function runNowBody(p: PatchPolicy, count: number): string {
const servers = `${count} server${count === 1 ? "" : "s"}`;
const installs = p.scope === "security" ? "security updates" : "all pending updates";
const reboot = p.reboot === "if_required" ? "Servers that need a reboot will restart." : "No server will be rebooted.";
return `This starts a window of the usual length now and installs ${installs} on ${servers}. ${reboot}`;
}
export function PolicyList({ canEdit }: { canEdit: boolean }) {
const router = useRouter();
const queryClient = useQueryClient();
const toast = useToast();
const [editing, setEditing] = useState<PatchPolicy | "new" | null>(null);
const [deleting, setDeleting] = useState<PatchPolicy | null>(null);
const [running, setRunning] = useState<PatchPolicy | null>(null);
const { data: policies, isLoading, error, refetch } = useQuery({ queryKey: ["patch-policies"], queryFn: () => api.listPatchPolicies() });
const { data: windows } = useQuery({ queryKey: ["maintenance-windows"], queryFn: () => api.listMaintenanceWindows() });
const { data: servers } = useQuery({ queryKey: ["servers"], queryFn: () => api.listServers() });
const { data: runs } = useQuery({ queryKey: ["patch-runs", "all"], queryFn: () => api.listPatchRuns({ limit: 50 }) });
const runNow = useMutation({
mutationFn: (p: PatchPolicy) => api.runPatchPolicyNow(p.policy_id),
onSuccess: (run) => {
setRunning(null);
router.push(`/patching/runs/${run.run_id}`);
},
onError: (e) => {
setRunning(null);
toast.error(friendlyMessage(e));
},
});
const remove = useMutation({
mutationFn: (p: PatchPolicy) => api.deletePatchPolicy(p.policy_id),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["patch-policies"] });
toast.success("Policy deleted.");
setDeleting(null);
},
onError: (e) => toast.error(friendlyMessage(e)),
});
const windowName = (id: string) => windows?.find((w) => w.window_id === id);
const lastRun = (id: string) => runs?.find((r) => r.policy_id === id);
return (
<>
{canEdit && (
<div className="flex justify-end border-b border-border px-6 py-3">
<Button variant="primary" size="sm" onClick={() => setEditing("new")} disabled={(windows ?? []).length === 0} title={(windows ?? []).length === 0 ? "Create a maintenance window first" : undefined}>
New policy
</Button>
</div>
)}
<AsyncBoundary
isLoading={isLoading}
error={error}
onRetry={refetch}
skeleton={<TableSkeleton columns={5} />}
isEmpty={!policies || policies.length === 0}
empty={<EmptyState title="No patch policies yet." description="A policy says which servers to patch, what to install and whether to reboot, inside a maintenance window." />}
>
<Table>
<Thead>
<Tr>
<Th>Policy</Th>
<Th>Next window</Th>
<Th>Targets</Th>
<Th>Installs</Th>
<Th>Last run</Th>
{canEdit && <Th />}
</Tr>
</Thead>
<Tbody>
{(policies ?? []).map((p) => {
const w = windowName(p.window_id);
const count = resolveTargets(servers ?? [], p.target_server_ids, p.target_tags ?? {}).length;
const last = lastRun(p.policy_id);
return (
<Tr key={p.policy_id}>
<Td label="Policy">
<div className="flex flex-col gap-1">
<span className="font-medium text-text-primary">{p.name}</span>
{!p.enabled && (
<span className="text-xs text-warning">
Disabled{p.disabled_reason ? `: ${p.disabled_reason}` : ""}
</span>
)}
{p.last_skipped && (
<span className="text-xs text-warning">
Skipped {new Date(p.last_skipped.due).toLocaleString()}: {SKIP_REASON[p.last_skipped.reason] ?? p.last_skipped.reason}
</span>
)}
</div>
</Td>
<Td label="Next window">
{p.enabled && p.next_run_at ? (
<span className="text-sm">
{new Date(p.next_run_at).toLocaleString(undefined, { timeZone: w?.tz, dateStyle: "medium", timeStyle: "short" })}
<span className="block text-xs text-text-tertiary">{w?.name}</span>
</span>
) : (
<span className="text-text-tertiary">none</span>
)}
</Td>
<Td label="Targets">
<span className="tabular-nums">{count}</span>
</Td>
<Td label="Installs">
<div className="flex flex-wrap gap-1.5">
<Badge variant="neutral">{p.scope === "security" ? "security only" : "all updates"}</Badge>
{p.reboot === "if_required" && <Badge variant="accent">reboots</Badge>}
</div>
</Td>
<Td label="Last run">
{last ? (
<Link href={`/patching/runs/${last.run_id}`}>
<Badge variant={RUN_STATUS[last.status].variant}>{RUN_STATUS[last.status].label}</Badge>
</Link>
) : (
<span className="text-text-tertiary">never</span>
)}
</Td>
{canEdit && (
<Td>
<div className="flex justify-end gap-2">
<Button variant="ghost" size="sm" loading={runNow.isPending && runNow.variables?.policy_id === p.policy_id} onClick={() => setRunning(p)}>
Run now
</Button>
<Button variant="ghost" size="sm" onClick={() => setEditing(p)}>
Edit
</Button>
<Button variant="ghost" size="sm" onClick={() => setDeleting(p)}>
Delete
</Button>
</div>
</Td>
)}
</Tr>
);
})}
</Tbody>
</Table>
</AsyncBoundary>
{editing && <PolicyModal initial={editing === "new" ? undefined : editing} onClose={() => setEditing(null)} />}
{running && (
<ConfirmDialog
open
title={`Run ${running.name} now?`}
body={runNowBody(running, resolveTargets(servers ?? [], running.target_server_ids, running.target_tags ?? {}).length)}
confirmLabel="Run now"
destructive={false}
loading={runNow.isPending}
onConfirm={() => runNow.mutate(running)}
onClose={() => setRunning(null)}
/>
)}
{deleting && (
<ConfirmDialog
open
title={`Delete ${deleting.name}?`}
body="The policy stops running. Its past runs stay on the Runs tab."
confirmLabel="Delete policy"
loading={remove.isPending}
onConfirm={() => remove.mutate(deleting)}
onClose={() => setDeleting(null)}
/>
)}
</>
);
}
+211
View File
@@ -0,0 +1,211 @@
"use client";
import { useMemo, useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { api, PatchPolicy, PatchReboot, PatchScope } from "@/lib/api";
import { resolveTargets } from "@/lib/targets";
import { Button, Modal, friendlyMessage, useToast } from "@/components/ui";
import { DualListBox } from "@/components/workflows/DualListBox";
import { WindowModal } from "./WindowModal";
import { agentSupportsPatchResults, describeCron, formatDuration, MIN_AGENT_VERSION, parseIntInRange } from "./status";
const inputClass = "w-full rounded-lg border border-border bg-surface-2 px-3 py-2 text-sm text-text-primary focus:border-accent/50 focus:outline-none focus:ring-1 focus:ring-accent/30";
function Radio<T extends string>({ name, value, current, onChange, title, hint }: { name: string; value: T; current: T; onChange: (v: T) => void; title: string; hint: string }) {
return (
<label className="flex items-start gap-3">
<input type="radio" id={`${name}-${value}`} name={name} checked={current === value} onChange={() => onChange(value)} className="mt-0.5 h-4 w-4 accent-accent" />
<span>
<span className="block text-sm text-text-primary">{title}</span>
<span className="mt-0.5 block text-xs text-text-tertiary">{hint}</span>
</span>
</label>
);
}
export function PolicyModal({ initial, onClose }: { initial?: PatchPolicy; onClose: () => void }) {
const queryClient = useQueryClient();
const toast = useToast();
const [name, setName] = useState(initial?.name ?? "");
const [enabled, setEnabled] = useState(initial?.enabled ?? true);
const [windowId, setWindowId] = useState(initial?.window_id ?? "");
const [targets, setTargets] = useState<string[]>(initial?.target_server_ids ?? []);
const [tagRows, setTagRows] = useState<[string, string][]>(Object.entries(initial?.target_tags ?? {}));
const [scope, setScope] = useState<PatchScope>(initial?.scope ?? "security");
const [reboot, setReboot] = useState<PatchReboot>(initial?.reboot ?? "never");
// Kept as the raw input so a cleared field is not silently read as 0 (no cap).
const [maxConcurrentRaw, setMaxConcurrentRaw] = useState(String(initial?.max_concurrent ?? 0));
const maxConcurrent = parseIntInRange(maxConcurrentRaw, 0, 1000);
const [channels, setChannels] = useState<string[]>(initial?.notify_channel_ids ?? []);
const [newWindow, setNewWindow] = useState(false);
const [error, setError] = useState<string | null>(null);
const { data: windows } = useQuery({ queryKey: ["maintenance-windows"], queryFn: () => api.listMaintenanceWindows() });
const { data: servers } = useQuery({ queryKey: ["servers"], queryFn: () => api.listServers() });
const { data: knownTags } = useQuery({ queryKey: ["server-tags"], queryFn: () => api.listKnownTags(), staleTime: 60_000 });
const { data: allChannels } = useQuery({ queryKey: ["channels"], queryFn: () => api.listChannels() });
const tags = useMemo(() => Object.fromEntries(tagRows.filter(([k, v]) => k && v)), [tagRows]);
const resolved = useMemo(() => resolveTargets(servers ?? [], targets, tags), [servers, targets, tags]);
const tooOld = resolved.filter((s) => !agentSupportsPatchResults(s.agent_version));
const { mutate: save, isPending } = useMutation({
mutationFn: () => {
const input = { name, enabled, window_id: windowId, target_server_ids: targets, target_tags: tags, scope, reboot, max_concurrent: maxConcurrent ?? 0, notify_channel_ids: channels };
return initial ? api.updatePatchPolicy(initial.policy_id, input) : api.createPatchPolicy(input);
},
onSuccess: (p) => {
queryClient.invalidateQueries({ queryKey: ["patch-policies"] });
toast.success(initial ? `Saved ${p.name}.` : `Created ${p.name}.`);
onClose();
},
onError: (e) => setError(friendlyMessage(e)),
});
return (
<>
<Modal open title={initial ? "Edit patch policy" : "New patch policy"} onClose={onClose} wide>
<div className="space-y-5">
{error && (
<div className="rounded border border-danger/30 bg-danger/10 px-3 py-2 text-sm text-danger" role="alert">
{error}
</div>
)}
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2">
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Name</span>
<input id="policy-name" className={inputClass} value={name} onChange={(e) => setName(e.target.value)} placeholder="Sunday prod security" />
</label>
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Maintenance window</span>
<div className="flex gap-2">
<select id="policy-window" className={inputClass} value={windowId} onChange={(e) => setWindowId(e.target.value)}>
<option value="">Choose a window</option>
{(windows ?? []).map((w) => (
<option key={w.window_id} value={w.window_id}>
{w.name}: {describeCron(w.cron)}, {formatDuration(w.duration_minutes)} ({w.tz})
</option>
))}
</select>
<Button variant="secondary" size="sm" onClick={() => setNewWindow(true)}>
New
</Button>
</div>
</label>
</div>
<div>
<span className="mb-1 block text-xs uppercase text-text-secondary">Target servers</span>
<DualListBox
items={(servers ?? []).map((s) => ({ id: s.server_id, label: s.hostname, hint: s.status === "active" ? undefined : s.status }))}
selected={targets}
onChange={setTargets}
selectedLabel="Targets"
emptyAvailable="Every server is a target."
emptySelected="No servers named."
/>
</div>
<div>
<span className="mb-1 block text-xs uppercase text-text-secondary">Target tags</span>
<p className="mb-2 text-[11px] text-text-tertiary">Servers carrying every tag below are patched too. Tags are read when the window opens, so a server tagged later is included.</p>
<datalist id="policy-tag-keys">
{Object.keys(knownTags ?? {}).map((k) => (
<option key={k} value={k} />
))}
</datalist>
<div className="flex flex-col gap-2">
{tagRows.map(([k, v], i) => (
<div key={i} className="flex gap-2">
<input id={`policy-tag-key-${i}`} list="policy-tag-keys" className={inputClass} value={k} placeholder="key" onChange={(e) => setTagRows(tagRows.map((r, j): [string, string] => (j === i ? [e.target.value, r[1]] : r)))} />
<input id={`policy-tag-value-${i}`} className={inputClass} value={v} placeholder="value" onChange={(e) => setTagRows(tagRows.map((r, j): [string, string] => (j === i ? [r[0], e.target.value] : r)))} />
<Button variant="ghost" size="sm" onClick={() => setTagRows(tagRows.filter((_, j) => j !== i))} aria-label={`Remove ${k || "tag"}`}>
Remove
</Button>
</div>
))}
<div>
<Button variant="secondary" size="sm" onClick={() => setTagRows([...tagRows, ["", ""]])}>
Add tag
</Button>
</div>
</div>
<p className="mt-2 text-xs text-text-secondary">
{resolved.length} server{resolved.length === 1 ? "" : "s"} targeted now.
{tooOld.length > 0 && (
<span className="text-warning">
{" "}
{tooOld.length} of {resolved.length} need an agent update to {MIN_AGENT_VERSION} or later and will be skipped until then.
</span>
)}
</p>
</div>
<div className="grid grid-cols-1 gap-5 sm:grid-cols-2">
<fieldset className="space-y-3">
<legend className="mb-2 text-xs uppercase text-text-secondary">What to install</legend>
<Radio name="scope" value="security" current={scope} onChange={setScope} title="Security updates only" hint="Servers using apk or pacman have no security metadata and report unsupported." />
<Radio name="scope" value="all" current={scope} onChange={setScope} title="All pending updates" hint="Everything the package manager would upgrade." />
</fieldset>
<fieldset className="space-y-3">
<legend className="mb-2 text-xs uppercase text-text-secondary">Reboots</legend>
<Radio name="reboot" value="never" current={reboot} onChange={setReboot} title="Never reboot" hint="Servers that need one show reboot required." />
<Radio name="reboot" value="if_required" current={reboot} onChange={setReboot} title="Reboot if required" hint="Only when the OS says so, and only with 5 minutes or more left in the window." />
{reboot === "if_required" && resolved.length > 0 && (
<p className="rounded-lg border border-warning/30 bg-warning/10 px-3 py-2 text-xs text-warning">
Up to {maxConcurrent ? Math.min(maxConcurrent, resolved.length) : resolved.length} server{resolved.length === 1 ? "" : "s"} may be rebooting at the same time during this window.
</p>
)}
</fieldset>
</div>
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2">
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">At most this many at once</span>
<input id="policy-max-concurrent" type="number" min={0} max={1000} className={inputClass} value={maxConcurrentRaw} aria-invalid={maxConcurrent === null} onChange={(e) => setMaxConcurrentRaw(e.target.value)} />
{maxConcurrent === null ? (
<span className="mt-1.5 block text-[11px] text-danger">Enter a whole number from 0 to 1000.</span>
) : (
<span className="mt-1.5 block text-[11px] text-text-tertiary">0 means no limit</span>
)}
</label>
<div>
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Alert when a run is not clean</span>
<div className="flex flex-col gap-1.5">
{(allChannels ?? []).length === 0 && <p className="text-xs text-text-tertiary">No notification channels yet.</p>}
{(allChannels ?? []).map((ch) => (
<label key={ch.channel_id} className="flex items-center gap-2 text-sm text-text-primary">
<input
type="checkbox"
id={`policy-channel-${ch.channel_id}`}
checked={channels.includes(ch.channel_id)}
onChange={(e) => setChannels(e.target.checked ? [...channels, ch.channel_id] : channels.filter((c) => c !== ch.channel_id))}
className="h-4 w-4 accent-accent"
/>
{ch.name}
</label>
))}
</div>
</div>
</div>
<label className="flex items-center gap-3">
<input type="checkbox" id="policy-enabled" checked={enabled} onChange={(e) => setEnabled(e.target.checked)} className="h-4 w-4 accent-accent" />
<span className="text-sm text-text-primary">Enabled</span>
</label>
<div className="flex justify-end gap-2">
<Button variant="secondary" onClick={onClose}>
Cancel
</Button>
<Button variant="primary" loading={isPending} disabled={!name.trim() || !windowId || maxConcurrent === null} onClick={() => save()}>
{initial ? "Save policy" : "Create policy"}
</Button>
</div>
</div>
</Modal>
{newWindow && <WindowModal onClose={() => setNewWindow(false)} onSaved={(w) => setWindowId(w.window_id)} />}
</>
);
}
+75
View File
@@ -0,0 +1,75 @@
"use client";
import Link from "next/link";
import { useQuery } from "@tanstack/react-query";
import { api } from "@/lib/api";
import { AsyncBoundary, Badge, EmptyState, Table, TableSkeleton, Tbody, Td, Th, Thead, Tr } from "@/components/ui";
import { RUN_STATUS } from "./status";
const SOURCE_LABEL: Record<string, string> = {
schedule: "window",
run_now: "run now",
server: "server page",
vulnerabilities: "vulnerabilities",
mcp: "agent (MCP)",
};
export function RunList({ policyId }: { policyId?: string }) {
const { data: runs, isLoading, error, refetch } = useQuery({
queryKey: ["patch-runs", policyId ?? "all"],
queryFn: () => api.listPatchRuns({ policy_id: policyId, limit: 50 }),
// A run in progress changes on every scheduler tick.
refetchInterval: (q) => (q.state.data?.some((r) => r.status === "running") ? 10_000 : false),
});
return (
<AsyncBoundary
isLoading={isLoading}
error={error}
onRetry={refetch}
skeleton={<TableSkeleton columns={5} />}
isEmpty={!runs || runs.length === 0}
empty={<EmptyState title="No patch runs yet." description="A run is recorded every time a policy's window opens and every time someone clicks Apply updates." />}
>
<Table>
<Thead>
<Tr>
<Th>Started</Th>
<Th>Policy</Th>
<Th>Status</Th>
<Th>Servers</Th>
<Th>Started by</Th>
</Tr>
</Thead>
<Tbody>
{(runs ?? []).map((r) => {
const ok = r.servers.filter((s) => s.status === "succeeded").length;
return (
<Tr key={r.run_id}>
<Td label="Started">
<Link href={`/patching/runs/${r.run_id}`} className="font-mono text-sm text-accent hover:underline">
{new Date(r.started_at).toLocaleString()}
</Link>
</Td>
<Td label="Policy">{r.policy_name ?? <span className="text-text-tertiary">manual</span>}</Td>
<Td label="Status">
<Badge variant={RUN_STATUS[r.status].variant}>{RUN_STATUS[r.status].label}</Badge>
</Td>
<Td label="Servers">
<span className="font-mono text-sm tabular-nums">
{ok}/{r.servers.length}
</span>
</Td>
<Td label="Started by">
<span className="text-sm text-text-secondary">
{r.triggered_by} <span className="text-text-tertiary">via {SOURCE_LABEL[r.source] ?? r.source}</span>
</span>
</Td>
</Tr>
);
})}
</Tbody>
</Table>
</AsyncBoundary>
);
}
+103
View File
@@ -0,0 +1,103 @@
"use client";
import { useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { api, MaintenanceWindow } from "@/lib/api";
import { AsyncBoundary, Button, ConfirmDialog, EmptyState, Table, TableSkeleton, Tbody, Td, Th, Thead, Tr, friendlyMessage, useToast } from "@/components/ui";
import { WindowModal } from "./WindowModal";
import { describeCron, formatDuration } from "./status";
export function WindowList({ canEdit }: { canEdit: boolean }) {
const queryClient = useQueryClient();
const toast = useToast();
const [editing, setEditing] = useState<MaintenanceWindow | "new" | null>(null);
const [deleting, setDeleting] = useState<MaintenanceWindow | null>(null);
const { data: windows, isLoading, error, refetch } = useQuery({ queryKey: ["maintenance-windows"], queryFn: () => api.listMaintenanceWindows() });
const { data: policies } = useQuery({ queryKey: ["patch-policies"], queryFn: () => api.listPatchPolicies() });
const { mutate: remove, isPending } = useMutation({
mutationFn: (w: MaintenanceWindow) => api.deleteMaintenanceWindow(w.window_id),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["maintenance-windows"] });
toast.success("Window deleted.");
setDeleting(null);
},
// 409 window_in_use arrives here with the server's own sentence.
onError: (e) => {
toast.error(friendlyMessage(e));
setDeleting(null);
},
});
const usedBy = (id: string) => (policies ?? []).filter((p) => p.window_id === id).length;
return (
<>
{canEdit && (
<div className="flex justify-end border-b border-border px-6 py-3">
<Button variant="primary" size="sm" onClick={() => setEditing("new")}>
New window
</Button>
</div>
)}
<AsyncBoundary
isLoading={isLoading}
error={error}
onRetry={refetch}
skeleton={<TableSkeleton columns={4} />}
isEmpty={!windows || windows.length === 0}
empty={<EmptyState title="No maintenance windows yet." description="A window is a recurring time slot, such as Sundays 02:00 to 04:00. Patch policies run inside one." />}
>
<Table>
<Thead>
<Tr>
<Th>Name</Th>
<Th>When</Th>
<Th>Length</Th>
<Th>Used by</Th>
{canEdit && <Th />}
</Tr>
</Thead>
<Tbody>
{(windows ?? []).map((w) => (
<Tr key={w.window_id}>
<Td label="Name">{w.name}</Td>
<Td label="When">
{describeCron(w.cron)} <span className="text-text-tertiary">({w.tz})</span>
</Td>
<Td label="Length">{formatDuration(w.duration_minutes)}</Td>
<Td label="Used by">
<span className="tabular-nums">{usedBy(w.window_id)}</span> {usedBy(w.window_id) === 1 ? "policy" : "policies"}
</Td>
{canEdit && (
<Td>
<div className="flex justify-end gap-2">
<Button variant="ghost" size="sm" onClick={() => setEditing(w)}>
Edit
</Button>
<Button variant="ghost" size="sm" onClick={() => setDeleting(w)} disabled={usedBy(w.window_id) > 0} title={usedBy(w.window_id) > 0 ? "Move its policies to another window first" : undefined}>
Delete
</Button>
</div>
</Td>
)}
</Tr>
))}
</Tbody>
</Table>
</AsyncBoundary>
{editing && <WindowModal initial={editing === "new" ? undefined : editing} onClose={() => setEditing(null)} />}
{deleting && (
<ConfirmDialog
open
title={`Delete ${deleting.name}?`}
body="The window is removed. No policy uses it, so nothing else changes."
confirmLabel="Delete window"
loading={isPending}
onConfirm={() => remove(deleting)}
onClose={() => setDeleting(null)}
/>
)}
</>
);
}
+141
View File
@@ -0,0 +1,141 @@
"use client";
import { useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { api, MaintenanceWindow } from "@/lib/api";
import { Button, Modal, friendlyMessage, useToast } from "@/components/ui";
import { parseIntInRange } from "./status";
/*
* Presets write cron underneath, as the workflow schedule card does, and the
* next three windows come from the server, so the editor cannot disagree with
* the scheduler about when a window opens.
*/
const PRESETS: { label: string; cron: string }[] = [
{ label: "Nightly, 02:00", cron: "0 2 * * *" },
{ label: "Sunday, 02:00", cron: "0 2 * * 0" },
{ label: "Saturday, 22:00", cron: "0 22 * * 6" },
{ label: "Monthly, 1st 02:00", cron: "0 2 1 * *" },
];
const ZONES = ["UTC", "Europe/London", "Europe/Berlin", "America/New_York", "America/Los_Angeles", "Asia/Singapore", "Australia/Sydney"];
const inputClass = "w-full rounded-lg border border-border bg-surface-2 px-3 py-2 text-sm text-text-primary focus:border-accent/50 focus:outline-none focus:ring-1 focus:ring-accent/30";
export function WindowModal({ initial, onClose, onSaved }: { initial?: MaintenanceWindow; onClose: () => void; onSaved?: (w: MaintenanceWindow) => void }) {
const queryClient = useQueryClient();
const toast = useToast();
const [name, setName] = useState(initial?.name ?? "");
const [cron, setCron] = useState(initial?.cron ?? "0 2 * * 0");
const [tz, setTz] = useState(initial?.tz ?? Intl.DateTimeFormat().resolvedOptions().timeZone ?? "UTC");
// Kept as the raw input so a cleared field never reaches the API as NaN.
const [durationRaw, setDurationRaw] = useState(String(initial?.duration_minutes ?? 120));
const duration = parseIntInRange(durationRaw, 15, 720);
const [error, setError] = useState<string | null>(null);
const { data: preview, isError: previewFailed, error: previewError } = useQuery({
queryKey: ["window-preview", cron, tz, duration],
queryFn: () => api.previewMaintenanceWindow({ cron, tz, duration_minutes: duration ?? 0 }),
enabled: duration !== null,
retry: false,
});
const { mutate: save, isPending } = useMutation({
mutationFn: () => {
const input = { name, cron, tz, duration_minutes: duration ?? 0 };
return initial ? api.updateMaintenanceWindow(initial.window_id, input) : api.createMaintenanceWindow(input);
},
onSuccess: (w) => {
queryClient.invalidateQueries({ queryKey: ["maintenance-windows"] });
queryClient.invalidateQueries({ queryKey: ["patch-policies"] });
toast.success(initial ? `Saved ${w.name}. Policies using it now follow the new times.` : `Created ${w.name}.`);
onSaved?.(w);
onClose();
},
onError: (e) => setError(friendlyMessage(e)),
});
return (
<Modal open title={initial ? "Edit maintenance window" : "New maintenance window"} onClose={onClose}>
<div className="space-y-4">
{error && (
<div className="rounded border border-danger/30 bg-danger/10 px-3 py-2 text-sm text-danger" role="alert">
{error}
</div>
)}
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Name</span>
<input id="window-name" className={inputClass} value={name} onChange={(e) => setName(e.target.value)} placeholder="Sunday early morning" />
</label>
<div className="flex flex-wrap gap-2">
{PRESETS.map((p) => (
<button
key={p.cron}
type="button"
onClick={() => setCron(p.cron)}
className={`rounded-lg border px-3 py-1.5 text-xs transition-colors focus:outline-none focus-visible:ring-2 focus-visible:ring-accent ${
cron === p.cron ? "border-accent bg-accent/10 text-accent" : "border-border bg-surface-2 text-text-secondary hover:border-accent/40"
}`}
>
{p.label}
</button>
))}
</div>
<div className="grid grid-cols-1 gap-4 sm:grid-cols-3">
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Starts (cron)</span>
<input id="window-cron" className={`${inputClass} font-mono`} value={cron} onChange={(e) => setCron(e.target.value)} />
<span className="mt-1.5 block font-mono text-[11px] text-text-tertiary">minute hour day month weekday</span>
</label>
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Timezone</span>
<select id="window-tz" className={inputClass} value={tz} onChange={(e) => setTz(e.target.value)}>
{[...new Set([tz, ...ZONES])].map((z) => (
<option key={z} value={z}>
{z}
</option>
))}
</select>
</label>
<label className="block">
<span className="mb-1.5 block text-sm font-medium text-text-secondary">Length (minutes)</span>
<input id="window-duration" type="number" min={15} max={720} step={15} className={inputClass} value={durationRaw} aria-invalid={duration === null} onChange={(e) => setDurationRaw(e.target.value)} />
{duration === null ? (
<span className="mt-1.5 block text-[11px] text-danger">Enter a whole number from 15 to 720.</span>
) : (
<span className="mt-1.5 block text-[11px] text-text-tertiary">15 to 720</span>
)}
</label>
</div>
<div className="rounded-lg bg-well px-4 py-3">
<p className="font-mono text-[10px] uppercase tracking-[0.16em] text-text-tertiary">Next three windows</p>
{previewFailed ? (
<p className="mt-1.5 font-mono text-[11.5px] text-danger">{friendlyMessage(previewError)}</p>
) : preview ? (
<ul className="mt-1.5 flex flex-col gap-0.5 font-mono text-[11.5px] text-text-secondary">
{preview.map((s) => (
<li key={s.start}>
{new Date(s.start).toLocaleString()} to {new Date(s.end).toLocaleTimeString()}
</li>
))}
</ul>
) : (
<p className="mt-1.5 font-mono text-[11.5px] text-text-tertiary">Working it out...</p>
)}
</div>
<div className="flex justify-end gap-2">
<Button variant="secondary" onClick={onClose}>
Cancel
</Button>
<Button variant="primary" loading={isPending} disabled={previewFailed || !name.trim() || duration === null} onClick={() => save()}>
{initial ? "Save window" : "Create window"}
</Button>
</div>
</div>
</Modal>
);
}
+80
View File
@@ -0,0 +1,80 @@
import type { PatchRunStatus, PatchServerStatus } from "@/lib/api";
type Variant = "success" | "warning" | "danger" | "neutral" | "accent";
/*
* One place that says how every patch state reads. Badge adds a dot to the
* three state variants, so each label here is also readable without colour.
*/
export const RUN_STATUS: Record<PatchRunStatus, { label: string; variant: Variant }> = {
running: { label: "running", variant: "accent" },
succeeded: { label: "succeeded", variant: "success" },
partial: { label: "partial", variant: "warning" },
failed: { label: "failed", variant: "danger" },
cancelled: { label: "cancelled", variant: "neutral" },
};
export const SERVER_STATUS: Record<PatchServerStatus, { label: string; variant: Variant }> = {
queued: { label: "queued", variant: "neutral" },
waiting_offline: { label: "waiting for agent", variant: "warning" },
patching: { label: "patching", variant: "accent" },
rebooting: { label: "rebooting", variant: "accent" },
succeeded: { label: "succeeded", variant: "success" },
failed: { label: "failed", variant: "danger" },
unsupported: { label: "unsupported", variant: "warning" },
agent_too_old: { label: "agent too old", variant: "warning" },
missed_offline: { label: "missed, offline", variant: "danger" },
window_closed: { label: "window closed", variant: "danger" },
cancelled: { label: "cancelled", variant: "neutral" },
};
/*
* Mirrors patchrun.MinAgentVersion and AgentSupportsPatchResults in the
* server. The server is the authority; this only lets the policy editor warn
* before saving. Change both together.
*/
export const MIN_AGENT_VERSION = "1.4.0";
/** Mirrors patchrun.AgentSupportsPatchResults (and its parseVersion) in the server. */
export function agentSupportsPatchResults(version?: string): boolean {
const m = /^v?(\d+)\.(\d+)\.(\d+)(?:-([^+]*))?(?:\+.*)?$/.exec((version ?? "").trim());
if (!m) return false;
const have = [Number(m[1]), Number(m[2]), Number(m[3])];
const want = MIN_AGENT_VERSION.split(".").map(Number);
for (let i = 0; i < 3; i++) {
if (have[i] !== want[i]) return have[i] > want[i];
}
const hasPrerelease = !!m[4];
return !hasPrerelease;
}
const DAYS = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"];
/** Plain words for the common shapes; anything else shows the expression itself. */
export function describeCron(cron: string): string {
const f = cron.trim().split(/\s+/);
if (f.length !== 5 || !/^\d+$/.test(f[0]) || !/^\d+$/.test(f[1])) return cron;
const at = `${f[1].padStart(2, "0")}:${f[0].padStart(2, "0")}`;
const [, , dom, mon, dow] = f;
if (dom === "*" && mon === "*" && dow === "*") return `Daily at ${at}`;
if (dom === "*" && mon === "*" && /^[0-6]$/.test(dow)) return `${DAYS[Number(dow)]}s at ${at}`;
if (/^\d+$/.test(dom) && mon === "*" && dow === "*") return `Day ${dom} of each month at ${at}`;
return cron;
}
export function formatDuration(minutes: number): string {
const h = Math.floor(minutes / 60);
const m = minutes % 60;
if (h === 0) return `${m} min`;
return m === 0 ? `${h} h` : `${h} h ${m} min`;
}
// parseIntInRange reads a form field kept as its raw string. It returns null
// for an empty, fractional or out-of-range value, so a cleared field is never
// sent as 0 or NaN.
export function parseIntInRange(raw: string, min: number, max: number): number | null {
const t = raw.trim();
if (!/^\d+$/.test(t)) return null;
const n = Number(t);
return n >= min && n <= max ? n : null;
}
+34 -1
View File
@@ -1,8 +1,12 @@
"use client";
import { useState } from "react";
import Link from "next/link";
import { useQuery } from "@tanstack/react-query";
import { api, ServerWithKeys } from "@/lib/api";
import { Badge, Button, Card, ConfirmDialog, Table, Tbody, Td, Th, Thead, Tr } from "@/components/ui";
import { matchesTags } from "@/lib/targets";
import { RUN_STATUS } from "@/components/patching/status";
/*
* Everything that changes what is installed on the machine: its OS packages,
@@ -42,6 +46,17 @@ export function MaintenanceTab({
const isWindows = server.os_info?.toLowerCase().includes("windows");
const agentCurrent = !!latestVersion && !!server.agent_version && server.agent_version === latestVersion;
const { data: policies } = useQuery({ queryKey: ["patch-policies"], queryFn: () => api.listPatchPolicies() });
const { data: windows } = useQuery({ queryKey: ["maintenance-windows"], queryFn: () => api.listMaintenanceWindows() });
const { data: lastRuns } = useQuery({ queryKey: ["patch-runs", "server", server.server_id], queryFn: () => api.listPatchRuns({ server_id: server.server_id, limit: 1 }) });
// Same selector rule as the scheduler: named, or carrying every tag.
const covering = (policies ?? []).filter((p) => p.enabled && (p.target_server_ids.includes(server.server_id) || matchesTags(server, p.target_tags ?? {})));
const next = covering
.filter((p) => p.next_run_at)
.sort((a, b) => (a.next_run_at! < b.next_run_at! ? -1 : 1))[0];
const nextWindow = next && windows?.find((w) => w.window_id === next.window_id);
const lastRun = lastRuns?.[0];
return (
<div className="grid grid-cols-1 gap-6 xl:grid-cols-2">
<Card padding={false}>
@@ -55,6 +70,24 @@ export function MaintenanceTab({
</div>
</div>
<div className="flex flex-wrap items-center justify-between gap-2 border-b border-border px-6 py-3 text-xs text-text-secondary">
{next && nextWindow ? (
<span>
Covered by <span className="text-text-primary">{next.name}</span>, next window{" "}
{new Date(next.next_run_at!).toLocaleString(undefined, { timeZone: nextWindow.tz, dateStyle: "medium", timeStyle: "short" })} ({nextWindow.tz})
</span>
) : (
<span>
Not covered by any patch policy. <Link href="/patching" className="text-accent hover:underline">Set one up</Link>
</span>
)}
{lastRun && (
<Link href={`/patching/runs/${lastRun.run_id}`} className="flex items-center gap-2 hover:text-accent">
Last run <Badge variant={RUN_STATUS[lastRun.status].variant}>{RUN_STATUS[lastRun.status].label}</Badge>
</Link>
)}
</div>
{updates.length === 0 ? (
<p className="px-6 py-10 text-center text-sm text-text-secondary">
{isWindows ? "No pending Windows updates. The agent checks hourly." : "No pending package updates. The agent checks hourly."}
@@ -94,7 +127,7 @@ export function MaintenanceTab({
<Button variant="primary" loading={isApplying} onClick={onApplyUpdates} disabled={server.status !== "active"} title={server.status !== "active" ? "Agent must be online to apply updates" : undefined}>
Apply updates
</Button>
<p className="text-xs text-text-tertiary">Upgrade runs in the background and may take several minutes.</p>
<p className="text-xs text-text-tertiary">Installs all pending updates now, without rebooting, and records a run.</p>
</div>
</>
)}
+153 -2
View File
@@ -417,6 +417,99 @@ export interface WorkflowRun {
server_runs: ServerRun[];
}
export type PatchScope = "all" | "security";
export type PatchReboot = "never" | "if_required";
export type PatchRunStatus = "running" | "succeeded" | "partial" | "failed" | "cancelled";
export type PatchServerStatus =
| "queued"
| "waiting_offline"
| "patching"
| "rebooting"
| "succeeded"
| "failed"
| "unsupported"
| "agent_too_old"
| "missed_offline"
| "window_closed"
| "cancelled";
export interface MaintenanceWindow {
window_id: string;
name: string;
cron: string;
tz: string;
duration_minutes: number;
created_at: string;
updated_at: string;
}
export interface MaintenanceWindowInput {
name: string;
cron: string;
tz: string;
duration_minutes: number;
}
export interface WindowSpan {
start: string;
end: string;
}
export interface PatchPolicy {
policy_id: string;
name: string;
enabled: boolean;
window_id: string;
target_server_ids: string[];
target_tags?: Record<string, string>;
scope: PatchScope;
reboot: PatchReboot;
max_concurrent: number;
notify_channel_ids?: string[];
next_run_at?: string;
last_run_at?: string;
last_skipped?: Skip;
disabled_reason?: string;
created_at: string;
updated_at: string;
}
export type PatchPolicyInput = Pick<
PatchPolicy,
"name" | "enabled" | "window_id" | "target_server_ids" | "target_tags" | "scope" | "reboot" | "max_concurrent" | "notify_channel_ids"
>;
export interface PatchServerRun {
server_id: string;
hostname: string;
status: PatchServerStatus;
pending_before: number;
pending_after?: number;
rebooted_at?: string;
verified_at?: string;
output?: string;
error?: string;
started_at?: string;
finished_at?: string;
}
export interface PatchRun {
run_id: string;
policy_id?: string;
policy_name?: string;
triggered_by: string;
source: "schedule" | "run_now" | "server" | "vulnerabilities" | "mcp";
scope: PatchScope;
reboot: PatchReboot;
max_concurrent: number;
window_end?: string;
status: PatchRunStatus;
cancelled_at?: string;
started_at: string;
finished_at?: string;
servers: PatchServerRun[];
}
export type Role = "owner" | "admin" | "member";
/** The session as returned by GET /auth/me mirrors auth.Session on the server. */
@@ -842,12 +935,70 @@ export const api = {
});
},
applyUpdates(serverId: string): Promise<{ message: string }> {
return request<{ message: string }>(`/servers/${serverId}/apply-updates`, {
applyUpdates(serverId: string, source?: "vulnerabilities"): Promise<{ message: string; run_id?: string }> {
const qs = source ? `?source=${source}` : "";
return request<{ message: string; run_id?: string }>(`/servers/${serverId}/apply-updates${qs}`, {
method: "POST",
});
},
listMaintenanceWindows(): Promise<MaintenanceWindow[]> {
return request<MaintenanceWindow[]>("/maintenance-windows");
},
createMaintenanceWindow(input: MaintenanceWindowInput): Promise<MaintenanceWindow> {
return request<MaintenanceWindow>("/maintenance-windows", { method: "POST", body: JSON.stringify(input) });
},
updateMaintenanceWindow(windowId: string, input: MaintenanceWindowInput): Promise<MaintenanceWindow> {
return request<MaintenanceWindow>(`/maintenance-windows/${windowId}`, { method: "PUT", body: JSON.stringify(input) });
},
deleteMaintenanceWindow(windowId: string): Promise<void> {
return request<void>(`/maintenance-windows/${windowId}`, { method: "DELETE" });
},
previewMaintenanceWindow(input: Omit<MaintenanceWindowInput, "name">): Promise<WindowSpan[]> {
return request<WindowSpan[]>("/maintenance-windows/preview", { method: "POST", body: JSON.stringify(input) });
},
listPatchPolicies(): Promise<PatchPolicy[]> {
return request<PatchPolicy[]>("/patch-policies");
},
createPatchPolicy(input: PatchPolicyInput): Promise<PatchPolicy> {
return request<PatchPolicy>("/patch-policies", { method: "POST", body: JSON.stringify(input) });
},
updatePatchPolicy(policyId: string, input: PatchPolicyInput): Promise<PatchPolicy> {
return request<PatchPolicy>(`/patch-policies/${policyId}`, { method: "PUT", body: JSON.stringify(input) });
},
deletePatchPolicy(policyId: string): Promise<void> {
return request<void>(`/patch-policies/${policyId}`, { method: "DELETE" });
},
runPatchPolicyNow(policyId: string): Promise<PatchRun> {
return request<PatchRun>(`/patch-policies/${policyId}/run-now`, { method: "POST" });
},
listPatchRuns(params: { policy_id?: string; server_id?: string; limit?: number } = {}): Promise<PatchRun[]> {
const qs = new URLSearchParams();
if (params.policy_id) qs.set("policy_id", params.policy_id);
if (params.server_id) qs.set("server_id", params.server_id);
if (params.limit) qs.set("limit", String(params.limit));
const suffix = qs.toString();
return request<PatchRun[]>(`/patch-runs${suffix ? `?${suffix}` : ""}`);
},
getPatchRun(runId: string): Promise<PatchRun> {
return request<PatchRun>(`/patch-runs/${runId}`);
},
cancelPatchRun(runId: string): Promise<void> {
return request<void>(`/patch-runs/${runId}/cancel`, { method: "POST" });
},
listAuditEvents(params: AuditQuery = {}): Promise<AuditPage> {
const qs = new URLSearchParams();
if (params.q) qs.set("q", params.q);
+1
View File
@@ -36,6 +36,7 @@ export const AUDIT_CATEGORIES: { value: string; label: string }[] = [
{ value: "console", label: "Console" },
{ value: "agent", label: "Agents" },
{ value: "updates", label: "OS updates" },
{ value: "patch", label: "Patching" },
{ value: "auth_provider", label: "Single sign-on" },
{ value: "settings", label: "Settings" },
{ value: "token", label: "API tokens" },