From 00c03c365d02ed43e83a7848a2b9617867f82ea6 Mon Sep 17 00:00:00 2001 From: mrhid6 Date: Thu, 6 Aug 2026 10:33:54 +0100 Subject: [PATCH] docs: design for package inventory and CVE findings Agents report installed packages; the control plane matches them against trivy-db and raises findings that link to the existing ApplyUpdatesCmd patching path. Scoped to sub-project A, Linux only. Container registry, image scanning and compliance baselines are separate specs. --- ...ckage-inventory-and-cve-findings-design.md | 524 ++++++++++++++++++ 1 file changed, 524 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md diff --git a/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md b/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md new file mode 100644 index 0000000..1b5024a --- /dev/null +++ b/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md @@ -0,0 +1,524 @@ +# Package inventory and CVE findings + +Date: 2026-08-06 + +Agents report the packages installed on each server. The control plane matches +them against distro security feeds and raises findings that link straight to +the patching path that already exists. A finding nobody can fix today can be +accepted with a reason and an expiry date rather than sitting red forever. + +This is one of four sub-projects sketched together and deliberately separated: + +| # | Sub-project | Depends on | +| - | ----------- | ---------- | +| A | **Package inventory + CVE findings** — this spec | nothing | +| B | Container/service registry | nothing | +| C | Container image scanning | A and B | +| D | Compliance profiles (baseline assertions) | shares A's findings UI only | + +A and B are independent of one another. C is the joiner and must not be +designed before both exist. D shares a page with A and nothing else — a +different collector, a different evaluation model and a different remediation +story — so folding it in here would double the size for no shared machinery. + +Scope of this spec is **A, Linux only.** Windows needs a separate source +(MSRC CVRF), a separate collector (`Get-HotFix` plus registry) and a KB +supersedence matcher that shares no code with the Linux path. That matches the +existing position that Windows agents are second-class by design, and the six +package managers `updates.go` already detects cover the whole Linux surface. + +--- + +## The trap this design is built around + +Distributions **backport** security fixes without changing the upstream +version. Ubuntu ships `openssl 3.0.2-0ubuntu1.15` patched against +CVE-2023-0286; NVD says version 3.0.2 is vulnerable. Matching installed +versions against NVD or CPE ranges therefore reports a fleet full of criticals +that are all already fixed. + +That is not merely noisy. It is fatal to the feature: once the first report is +mostly wrong, nobody reads the second one, and a genuine finding is lost in the +noise it created. Everything below follows from refusing to make that mistake. + +The correct source is the **distribution's own security feed**, keyed on the +distribution's own version string — Debian and Ubuntu OVAL/USN, Red Hat OVAL +v2, Alpine secdb. `trivy-db` is those feeds pre-merged into one BoltDB +artifact, rebuilt every six hours and published as an OCI artifact. + +--- + +## Where the vulnerability data comes from + +`trivy-db`, pulled server-side from `ghcr.io/aquasecurity/trivy-db:2`. + +The alternative considered was querying OSV.dev per scan, which needs no +storage and no puller. It was rejected on two counts: it requires outbound +internet on every scan, which breaks air-gapped installs; and it sends the +package list of a customer's entire fleet to a third party. The audience most +likely to buy vulnerability scanning is the audience least willing to do that. + +The blob is roughly 50MB, read-only, reproducible, and identified by a version +number. **It is not stored in Mongo and not written to `/data`** — +`server.persistence` defaults to off and nothing writes to `/data` any more. +It does not need durable storage: whichever pod needs it pulls it to its own +ephemeral temp directory. Nothing shared, nothing to back up, nothing to +migrate. + +`VANTAGE_TRIVY_DB_REF` overrides the default reference so a customer can mirror +the artifact into their own registry. It also covers the anonymous ghcr rate +limit, which the six-hourly pull cadence already makes unlikely to bite. + +--- + +## Only the leader matches + +This is the crux, and it falls out of the replica model already in the +codebase. + +Two things trigger matching, and they happen on different pods: + +1. a fleet-wide rescan when `trivy-db` updates — naturally the leader's job +2. a server's package list changing — handled by whichever pod holds *that + agent's* command stream + +If (2) matched inline, **every replica would need the 50MB database resident**, +and a database refresh would have N pods racing to rescan the same fleet and N +digests reaching the customer. That is the exact failure `RunAsLeader` exists +to prevent, and it is the same argument that put `monitorsched` behind the +lock. + +So `ReportPackages` does not match. It upserts the package list and sets +`scan_pending: true`. That is all it does. + +`server/internal/vulnsched` then runs inside the **existing** +`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, +`workflowsched` and the sweepers — one role, one lock. Every 60 seconds it: + +1. pulls `trivy-db` if the local copy is older than six hours +2. if the pulled version differs from `vulndb_meta.db_version`, marks **every** + server `scan_pending` +3. matches all `scan_pending` servers, clears the flag, diffs against existing + findings +4. emits **one** digest per tick covering everything newly opened + +Step 4 is why batching is structural rather than bolted on. A `trivy-db` +refresh can open several hundred findings across a fleet at once; one message +per finding would rate-limit the webhook or get the channel muted, and either +way the customer stops receiving the alerts they are paying for. The tick is +already the natural batch boundary, so **the failure cannot occur by +construction** rather than by a debounce someone has to maintain. + +`scan_pending` lives on the document rather than in memory, for the same reason +`next_run_at` and `workflow_log_seq` do: a leader handover between marking and +scanning would otherwise lose it. A handover costs the new leader one re-pull +of the database. + +The cost of this indirection is up to 60 seconds between an agent reporting a +changed package set and its findings updating. For vulnerability data that is +nothing, and it buys a single matching path instead of two. + +--- + +## Components + +``` +agent/internal/packages/ collect installed packages + /etc/os-release +proto/ ReportPackages RPC +server/internal/vulndb/ puller, BoltDB access, matcher +server/internal/vulnsched/ leader-owned tick: pull, scan, digest +server/internal/services/ findings, acceptance, alert rules +web/app/(app)/vulnerabilities/ fleet board; plus two server-detail tabs +``` + +`vulnsched` takes the dependencies it needs — `LogEvent` and the notification +dispatch — as a `vulnsched.Deps` injected from `main.go`, following +`workflowsched`. The manual rescan endpoint does not call into `vulnsched` at +all: it sets `scan_pending` on every server and lets the next tick find them, +so there is no path by which `services` imports the scheduler and no cycle to +avoid later. + +--- + +## The wire path + +A new `ReportPackages` RPC on the agent's existing hourly loop — the same +`runUpdateCheck` cadence, reusing `updates.go`'s `detectPM()`. + +```protobuf +rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse); + +message ReportPackagesRequest { + string server_id = 1; + string agent_token = 2; + string hash = 3; // sha256 of the sorted list + OSRelease os = 4; + repeated InstalledPackage packages = 5; // omitted when only offering a hash +} + +message ReportPackagesResponse { + bool need_full = 1; // hash differs; resend with packages populated +} +``` + +The agent calls once with `packages` empty. `need_full` true means the hash +differs from what the server holds, and the agent immediately calls again with +the list populated. + +The agent sends a SHA-256 of its sorted package list first. If it matches what +the server already holds, the server answers `unchanged` and the ~150KB body is +never sent. A machine's package set changes rarely, so almost every hour costs +one small message, and the rare changed hour costs one extra round trip. + +Folding the list into the existing 15-minute `InventoryReport` static snapshot +was rejected: it would re-send ~150KB per server every 15 minutes regardless of +change, roughly 40MB/hour of gRPC traffic on a 100-server fleet to transmit +data that is almost always identical. + +--- + +## Data model + +Four new collections. Every one carries `instance_id` except `vulndb_meta`, +which is explained below. + +### `server_packages` — one document per server, not per package + +```go +type ServerPackages struct { + ID primitive.ObjectID `bson:"_id"` + InstanceID primitive.ObjectID `bson:"instance_id"` + ServerID string `bson:"server_id"` + OS OSRelease `bson:"os"` // family, version_id, arch + Hash string `bson:"hash"` // sha256 of the sorted list + Packages []InstalledPackage `bson:"packages"` + CollectedAt time.Time `bson:"collected_at"` + ScanPending bool `bson:"scan_pending"` + ScannedAt time.Time `bson:"scanned_at"` + Status string `bson:"status"` // ok | unsupported + DBVersion int `bson:"db_version"` // last matched against +} + +type InstalledPackage struct { + Name string `bson:"name"` + Version string `bson:"version"` // distro version string, verbatim + Epoch int `bson:"epoch,omitempty"` + Arch string `bson:"arch"` + SourceName string `bson:"source_name,omitempty"` +} +``` + +One document rather than two thousand is what makes a report a **single atomic +upsert with no delta logic** — the hash already established that something +changed, so there is nothing to reconcile field by field. A typical Linux host +lands near 150KB, comfortably inside the 16MB document limit. + +Indexes: `{instance_id, server_id}` unique, and a multikey +`{instance_id, "packages.name"}` for fleet-wide package search. + +`SourceName` is not decoration. **Debian and Ubuntu advisories are keyed on the +source package**: a CVE against `openssl` covers the binaries `libssl3`, +`openssl` and `libssl-dev`, so matching on binary name alone misses two of the +three. + +`OS.VersionID` selects the feed. Ubuntu 22.04 and 24.04 publish different fixed +versions for the same CVE, so a scan without it is guesswork. + +### `vuln_findings` — one document per (server, CVE, package) + +```go +type VulnFinding struct { + ID primitive.ObjectID `bson:"_id"` + InstanceID primitive.ObjectID `bson:"instance_id"` + ServerID string `bson:"server_id"` + + CVEID string `bson:"cve_id"` + PackageName string `bson:"package_name"` + Installed string `bson:"installed_version"` + FixedIn string `bson:"fixed_in,omitempty"` + Severity string `bson:"severity"` + CVSSScore float64 `bson:"cvss_score,omitempty"` + Title string `bson:"title,omitempty"` + References []string `bson:"references,omitempty"` + + State string `bson:"state"` // open | fixed | accepted + FirstSeen time.Time `bson:"first_seen"` + LastSeen time.Time `bson:"last_seen"` + FixedAt *time.Time `bson:"fixed_at,omitempty"` + Accepted *Acceptance `bson:"accepted,omitempty"` +} + +type Acceptance struct { + By primitive.ObjectID `bson:"by"` + Reason string `bson:"reason"` + Until time.Time `bson:"until"` + At time.Time `bson:"at"` +} +``` + +Unique on `{instance_id, server_id, cve_id, package_name}`. That key is what +makes a rescan an idempotent upsert rather than a duplicate factory, and it is +what lets `first_seen` survive across scans. Query index +`{instance_id, state, severity}`. + +**An empty `FixedIn` is a real and common state** and must never be conflated +with "not vulnerable". A CVE with no vendor fix published yet is exactly the +finding people most need to see, and also the one that most needs acceptance, +because there is nothing to patch. + +Findings are **not deleted when a package is patched**. State moves to `fixed` +with `fixed_at` set, so "what did we remediate last quarter" remains +answerable — which is the question an auditor asks. + +### `vulndb_meta` — singleton, deliberately unscoped + +`db_version`, `pulled_at`, `last_full_scan_at`, `last_error`. It carries no +`instance_id` because the vulnerability database is a property of the +deployment, not of a tenant. Same reasoning as `migrations`. + +### `vuln_alert_rules` + +`instance_id`, `name`, `enabled`, `min_severity`, `tags map[string]string`, +`channel_ids []`, timestamps. + +The tag filter resolves through **`services.ResolveTargets`**, not a second +matcher. That function is already the single answer to which servers a +selector touches, and an alert rule that disagreed with a workflow about what +`env:prod` means would be worse than having no filter at all. + +--- + +## The matching engine + +``` +server/internal/vulndb/ + pull.go OCI fetch → temp dir, version compare against vulndb_meta + db.go BoltDB open, advisory lookup by (ecosystem, source, version) + match.go per-family matching, severity resolution + version.go dispatch to deb/rpm/apk comparator by OS family +``` + +Dependencies: `github.com/aquasecurity/trivy-db` for the BoltDB schema, plus +`go-deb-version`, `go-rpm-version` and `go-apk-version` — each a small +standalone module doing one job. The roughly 200 lines of per-distro advisory +lookup are ours. + +Importing `trivy` itself was rejected: it would pull a very large transitive +dependency tree into the server binary for one feature, and its Go API carries +no stability guarantee across minor versions. Shelling out to the `trivy` +binary against a generated SBOM was rejected for shipping a second binary in +the image and turning a library call into subprocess lifecycle, timeouts and +output-format drift. + +### Why the comparators are bought rather than written + +Version ordering is where this feature lives or dies, and its failure mode is +silent. `dpkg` ordering has epochs, and `~` sorts *before* the empty string, so +`3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own +segment rules and treats `~` and `^` differently again. A `strings.Compare` or +a semver parse orders `1.9` above `1.10` and reports a vulnerable fleet as +clean — a false negative, which nobody notices until it matters. + +### Scanning one server + +1. Load `server_packages`; resolve OS family and version to a `trivy-db` + ecosystem. +2. **Unsupported ecosystem → record `status: unsupported`, clear the flag, + write no findings.** +3. For each package: resolve source name, look up advisories, compare versions. +4. Upsert vulnerable results as `open`, preserving `first_seen`. +5. Any currently-`open` finding absent from this result set → `fixed`, stamp + `fixed_at`. +6. Any `accepted` finding past its `until` → back to `open`. +7. Clear `scan_pending`, stamp `scanned_at` and `db_version`. + +Steps 5 and 6 must run in that order, so a finding that is both absent and +expired settles as `fixed` rather than reopening on a package that no longer +carries it. + +Step 2 matters as much as any of the matching. Arch has no feed in `trivy-db`, +so an Arch host must report **unsupported**, never "0 findings". Reporting +clean when the truth is unknown is the same class of lie as a silently stale +database, and it is the reason `vulndb_meta.pulled_at` appears on screen rather +than only in a log. + +### Severity + +Resolved **vendor → NVD → unknown**, in that order, never invented. + +This will surface as "why is this critical CVE marked low", and the answer is +that Debian and Red Hat routinely downgrade an NVD score because the vulnerable +code path is not reachable in their build. Their rating is the accurate one for +that package, and showing NVD's above it would manufacture work that does not +need doing. + +--- + +## Findings lifecycle + +`open | fixed | accepted`. + +An accepted finding is suppressed from counts and alerts until its `until` +date, then reopens automatically. A reason is required. + +Acceptance with a mandatory expiry, rather than permanent dismissal, is what +keeps the feature usable in both directions. Without any acceptance mechanism, +a kernel CVE awaiting a reboot window sits red indefinitely and trains people +to ignore the page. With permanent dismissal, accepted findings accumulate +silently and nobody revisits them — the dismissal list becomes where risk goes +to be forgotten, which is precisely what an auditor asks to see. + +Retention: `settings.vuln_finding_retention_days`, a `*int` on the same pattern +as `workflow_log_retention_days` — nil means 90 days, 0 means forever. Only +`fixed` findings are swept, by a `StartVulnSweeper` inside the same +`RunAsLeader("housekeeping", …)` as the existing sweepers. `open` and +`accepted` findings are never swept at any setting. + +--- + +## Alerting + +Per-org rules over the existing `notification_channels`: severity threshold, +optional tag filter, target channels. + +A rescan emits one message summarising what newly opened — "12 new critical +across 4 servers" — never one message per finding. See the leader section for +why the tick boundary makes this structural. + +Modelling findings as a monitor type was rejected. It would reuse monitors' +state machine and channel wiring for free, but monitors are up/down for one +endpoint with retries and hourly rollups, none of which means anything for a +CVE; most fields would be disabled in the UI and the uptime graphs would be +polluted with a signal that is not uptime. + +This adds one `notify` payload type and a `vuln_digest.html.tmpl` / +`vuln_digest.txt.tmpl` pair in `shared/mail`, since `render_test.go` fails on +any template no case covers. + +--- + +## Entitlement + +`entitlement.features.vuln_scanning`, a boolean alongside the existing +per-instance feature toggles, so it can later be priced as a catalogue +`feature` component without a second migration. Off on Free. + +**The gate is checked at `ReportPackages`, not at display.** Gating only the UI +would still pay every write cost, and storage is the expensive half. + +The agent learns of it through the existing 30-second `SyncKeys` poll: +`SyncResponse` gains a `collect_packages` bool, and the hourly loop skips +collection entirely when it is false. So an ungated instance produces no +collection, no gRPC body, no document and no storage. `ReportPackages` still +re-checks the entitlement server-side and refuses — the agent flag is an +optimisation, the server check is the boundary. + +Turning the feature off does not delete existing findings; they stop being +served and stop updating. Deletion is the instance-deletion path's job. + +--- + +## REST API + +``` +GET /api/vulnerabilities # filter: severity, state, server, tags +GET /api/vulnerabilities/summary # severity counts + database freshness +POST /api/vulnerabilities/rescan # marks all scan_pending (owner|admin) +POST /api/vulnerabilities/:id/accept # reason + until (owner|admin) +DELETE /api/vulnerabilities/:id/accept # (owner|admin) +GET /api/servers/:id/vulnerabilities +GET /api/servers/:id/packages +GET /api/packages/search?name= # fleet-wide +GET,POST /api/vuln-rules · PUT,DELETE /api/vuln-rules/:id +``` + +Every mutating path writes an audit event, as all of them do. Acceptance is the +one decision people will be asked to justify, so `by`, `reason`, `until` and +`at` land in the audit record and not only on the document. + +--- + +## UI + +`/vulnerabilities` is a fleet board **grouped by CVE** — one row per CVE with +an affected-server count, expandable to the individual servers. The same CVE +across 40 servers is one decision, and a flat list of findings makes it look +like forty. + +Server detail gains **Vulnerabilities** and **Packages** tabs. Alert rules go +on `/settings/notifications`, beside the channels they consume. + +Remediation introduces no new mechanism: a finding carrying `fixed_in` renders +an **Apply updates** action calling the existing +`POST /api/servers/:id/apply-updates`, which is already `ApplyUpdatesCmd`. See +it, patch it, one place — and no second patching path to keep consistent with +the first. + +Database freshness is shown wherever findings are, not tucked into settings. A +fleet scanning against a three-week-old database must say so rather than +quietly report all-clear. + +--- + +## Testing + +**Version comparison, table-driven, per family.** The highest-value test in the +feature, because its failure mode is silent. The table must include the +backport case explicitly — installed `1:3.0.2-0ubuntu1.15` against advisory +fixed-in `1:3.0.2-0ubuntu1.15` resolving to *not vulnerable* — alongside tilde +ordering (`1.0~rc1` < `1.0`), epoch dominance (`1:1.0` > `2.0`) and +`1.9` < `1.10`. + +**Matcher golden tests.** A fixture package list and a small hand-built BoltDB +fixture asserting an exact finding set. No network: a test that pulls the real +database fails on a bad day and changes its expectations every six hours. + +**State transitions.** `open → fixed → reopened`, acceptance expiry, and +`first_seen` surviving a rescan. The last is easy to break with an upsert that +overwrites, and nothing notices until a report claims everything was discovered +yesterday. + +**Collector parsers**, one per package manager, from captured command output. +`updates.go` currently has no tests and its parsers are the same shape, so +these fixtures should cover both. + +**Scoping and gating.** Findings queries scoped by `instance_id`, and the +entitlement gate proven to write nothing rather than merely hide. + +--- + +## Failure modes + +| Failure | Behaviour | +| ------- | --------- | +| Database pull fails | Keep the last good copy and serve stale. Record `last_error`, surface `pulled_at` age. **Never clear findings** — a network blip must not read as "all fixed" | +| Unsupported distribution | `status: unsupported`, not zero findings | +| Agent stops reporting | Findings persist and `collected_at` age is shown. No auto-expiry: a silent agent is not a patched server | +| `trivy-db` schema version bumps | The puller refuses an unknown schema rather than mis-parsing it | +| ghcr anonymous rate limit | Backoff; `VANTAGE_TRIVY_DB_REF` mirrors to a private registry | +| Leadership lost mid-scan | The context is cancelled and the scan returns; `scan_pending` is still set, so the next leader picks it up | +| Instance deleted | **`server_packages` and `vuln_findings` must be added to the control plane's instance-deletion collection list.** Easy to miss, and missing it orphans a tenant's package data indefinitely | + +--- + +## Environment variables + +| Name | Required | Notes | +| ---- | -------- | ----- | +| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for air-gapped installs or to avoid the anonymous ghcr rate limit | +| `VANTAGE_VULNDB_DISABLED` | no | disables the puller and the scheduler entirely. Findings already written are still served and still marked stale | + +--- + +## Deliberately out of scope + +- **Windows.** Separate source, collector and matcher; its own spec. +- **Container image scanning.** Sub-project C; needs the container registry. +- **Compliance baseline assertions.** Sub-project D; shares this findings UI + and nothing else. +- **Language-level dependency scanning** (npm, pip, Go modules). `trivy-db` + covers these ecosystems, but finding the manifests on a host is a different + collection problem from asking the package manager what is installed. +- **Automatic patching on a finding.** Remediation is one click, not zero. An + unattended upgrade triggered by a CVE feed is a fleet-wide change driven by a + third party's data, which is not a decision to take away from an operator.