docs: design for package inventory and CVE findings

Agents report installed packages; the control plane matches them against
trivy-db and raises findings that link to the existing ApplyUpdatesCmd
patching path.

Scoped to sub-project A, Linux only. Container registry, image scanning
and compliance baselines are separate specs.
This commit is contained in:
2026-08-06 10:33:54 +01:00
parent dc8dd3dd58
commit 00c03c365d
@@ -0,0 +1,524 @@
# Package inventory and CVE findings
Date: 2026-08-06
Agents report the packages installed on each server. The control plane matches
them against distro security feeds and raises findings that link straight to
the patching path that already exists. A finding nobody can fix today can be
accepted with a reason and an expiry date rather than sitting red forever.
This is one of four sub-projects sketched together and deliberately separated:
| # | Sub-project | Depends on |
| - | ----------- | ---------- |
| A | **Package inventory + CVE findings** — this spec | nothing |
| B | Container/service registry | nothing |
| C | Container image scanning | A and B |
| D | Compliance profiles (baseline assertions) | shares A's findings UI only |
A and B are independent of one another. C is the joiner and must not be
designed before both exist. D shares a page with A and nothing else — a
different collector, a different evaluation model and a different remediation
story — so folding it in here would double the size for no shared machinery.
Scope of this spec is **A, Linux only.** Windows needs a separate source
(MSRC CVRF), a separate collector (`Get-HotFix` plus registry) and a KB
supersedence matcher that shares no code with the Linux path. That matches the
existing position that Windows agents are second-class by design, and the six
package managers `updates.go` already detects cover the whole Linux surface.
---
## The trap this design is built around
Distributions **backport** security fixes without changing the upstream
version. Ubuntu ships `openssl 3.0.2-0ubuntu1.15` patched against
CVE-2023-0286; NVD says version 3.0.2 is vulnerable. Matching installed
versions against NVD or CPE ranges therefore reports a fleet full of criticals
that are all already fixed.
That is not merely noisy. It is fatal to the feature: once the first report is
mostly wrong, nobody reads the second one, and a genuine finding is lost in the
noise it created. Everything below follows from refusing to make that mistake.
The correct source is the **distribution's own security feed**, keyed on the
distribution's own version string — Debian and Ubuntu OVAL/USN, Red Hat OVAL
v2, Alpine secdb. `trivy-db` is those feeds pre-merged into one BoltDB
artifact, rebuilt every six hours and published as an OCI artifact.
---
## Where the vulnerability data comes from
`trivy-db`, pulled server-side from `ghcr.io/aquasecurity/trivy-db:2`.
The alternative considered was querying OSV.dev per scan, which needs no
storage and no puller. It was rejected on two counts: it requires outbound
internet on every scan, which breaks air-gapped installs; and it sends the
package list of a customer's entire fleet to a third party. The audience most
likely to buy vulnerability scanning is the audience least willing to do that.
The blob is roughly 50MB, read-only, reproducible, and identified by a version
number. **It is not stored in Mongo and not written to `/data`**
`server.persistence` defaults to off and nothing writes to `/data` any more.
It does not need durable storage: whichever pod needs it pulls it to its own
ephemeral temp directory. Nothing shared, nothing to back up, nothing to
migrate.
`VANTAGE_TRIVY_DB_REF` overrides the default reference so a customer can mirror
the artifact into their own registry. It also covers the anonymous ghcr rate
limit, which the six-hourly pull cadence already makes unlikely to bite.
---
## Only the leader matches
This is the crux, and it falls out of the replica model already in the
codebase.
Two things trigger matching, and they happen on different pods:
1. a fleet-wide rescan when `trivy-db` updates — naturally the leader's job
2. a server's package list changing — handled by whichever pod holds *that
agent's* command stream
If (2) matched inline, **every replica would need the 50MB database resident**,
and a database refresh would have N pods racing to rescan the same fleet and N
digests reaching the customer. That is the exact failure `RunAsLeader` exists
to prevent, and it is the same argument that put `monitorsched` behind the
lock.
So `ReportPackages` does not match. It upserts the package list and sets
`scan_pending: true`. That is all it does.
`server/internal/vulnsched` then runs inside the **existing**
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`,
`workflowsched` and the sweepers — one role, one lock. Every 60 seconds it:
1. pulls `trivy-db` if the local copy is older than six hours
2. if the pulled version differs from `vulndb_meta.db_version`, marks **every**
server `scan_pending`
3. matches all `scan_pending` servers, clears the flag, diffs against existing
findings
4. emits **one** digest per tick covering everything newly opened
Step 4 is why batching is structural rather than bolted on. A `trivy-db`
refresh can open several hundred findings across a fleet at once; one message
per finding would rate-limit the webhook or get the channel muted, and either
way the customer stops receiving the alerts they are paying for. The tick is
already the natural batch boundary, so **the failure cannot occur by
construction** rather than by a debounce someone has to maintain.
`scan_pending` lives on the document rather than in memory, for the same reason
`next_run_at` and `workflow_log_seq` do: a leader handover between marking and
scanning would otherwise lose it. A handover costs the new leader one re-pull
of the database.
The cost of this indirection is up to 60 seconds between an agent reporting a
changed package set and its findings updating. For vulnerability data that is
nothing, and it buys a single matching path instead of two.
---
## Components
```
agent/internal/packages/ collect installed packages + /etc/os-release
proto/ ReportPackages RPC
server/internal/vulndb/ puller, BoltDB access, matcher
server/internal/vulnsched/ leader-owned tick: pull, scan, digest
server/internal/services/ findings, acceptance, alert rules
web/app/(app)/vulnerabilities/ fleet board; plus two server-detail tabs
```
`vulnsched` takes the dependencies it needs — `LogEvent` and the notification
dispatch — as a `vulnsched.Deps` injected from `main.go`, following
`workflowsched`. The manual rescan endpoint does not call into `vulnsched` at
all: it sets `scan_pending` on every server and lets the next tick find them,
so there is no path by which `services` imports the scheduler and no cycle to
avoid later.
---
## The wire path
A new `ReportPackages` RPC on the agent's existing hourly loop — the same
`runUpdateCheck` cadence, reusing `updates.go`'s `detectPM()`.
```protobuf
rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse);
message ReportPackagesRequest {
string server_id = 1;
string agent_token = 2;
string hash = 3; // sha256 of the sorted list
OSRelease os = 4;
repeated InstalledPackage packages = 5; // omitted when only offering a hash
}
message ReportPackagesResponse {
bool need_full = 1; // hash differs; resend with packages populated
}
```
The agent calls once with `packages` empty. `need_full` true means the hash
differs from what the server holds, and the agent immediately calls again with
the list populated.
The agent sends a SHA-256 of its sorted package list first. If it matches what
the server already holds, the server answers `unchanged` and the ~150KB body is
never sent. A machine's package set changes rarely, so almost every hour costs
one small message, and the rare changed hour costs one extra round trip.
Folding the list into the existing 15-minute `InventoryReport` static snapshot
was rejected: it would re-send ~150KB per server every 15 minutes regardless of
change, roughly 40MB/hour of gRPC traffic on a 100-server fleet to transmit
data that is almost always identical.
---
## Data model
Four new collections. Every one carries `instance_id` except `vulndb_meta`,
which is explained below.
### `server_packages` — one document per server, not per package
```go
type ServerPackages struct {
ID primitive.ObjectID `bson:"_id"`
InstanceID primitive.ObjectID `bson:"instance_id"`
ServerID string `bson:"server_id"`
OS OSRelease `bson:"os"` // family, version_id, arch
Hash string `bson:"hash"` // sha256 of the sorted list
Packages []InstalledPackage `bson:"packages"`
CollectedAt time.Time `bson:"collected_at"`
ScanPending bool `bson:"scan_pending"`
ScannedAt time.Time `bson:"scanned_at"`
Status string `bson:"status"` // ok | unsupported
DBVersion int `bson:"db_version"` // last matched against
}
type InstalledPackage struct {
Name string `bson:"name"`
Version string `bson:"version"` // distro version string, verbatim
Epoch int `bson:"epoch,omitempty"`
Arch string `bson:"arch"`
SourceName string `bson:"source_name,omitempty"`
}
```
One document rather than two thousand is what makes a report a **single atomic
upsert with no delta logic** — the hash already established that something
changed, so there is nothing to reconcile field by field. A typical Linux host
lands near 150KB, comfortably inside the 16MB document limit.
Indexes: `{instance_id, server_id}` unique, and a multikey
`{instance_id, "packages.name"}` for fleet-wide package search.
`SourceName` is not decoration. **Debian and Ubuntu advisories are keyed on the
source package**: a CVE against `openssl` covers the binaries `libssl3`,
`openssl` and `libssl-dev`, so matching on binary name alone misses two of the
three.
`OS.VersionID` selects the feed. Ubuntu 22.04 and 24.04 publish different fixed
versions for the same CVE, so a scan without it is guesswork.
### `vuln_findings` — one document per (server, CVE, package)
```go
type VulnFinding struct {
ID primitive.ObjectID `bson:"_id"`
InstanceID primitive.ObjectID `bson:"instance_id"`
ServerID string `bson:"server_id"`
CVEID string `bson:"cve_id"`
PackageName string `bson:"package_name"`
Installed string `bson:"installed_version"`
FixedIn string `bson:"fixed_in,omitempty"`
Severity string `bson:"severity"`
CVSSScore float64 `bson:"cvss_score,omitempty"`
Title string `bson:"title,omitempty"`
References []string `bson:"references,omitempty"`
State string `bson:"state"` // open | fixed | accepted
FirstSeen time.Time `bson:"first_seen"`
LastSeen time.Time `bson:"last_seen"`
FixedAt *time.Time `bson:"fixed_at,omitempty"`
Accepted *Acceptance `bson:"accepted,omitempty"`
}
type Acceptance struct {
By primitive.ObjectID `bson:"by"`
Reason string `bson:"reason"`
Until time.Time `bson:"until"`
At time.Time `bson:"at"`
}
```
Unique on `{instance_id, server_id, cve_id, package_name}`. That key is what
makes a rescan an idempotent upsert rather than a duplicate factory, and it is
what lets `first_seen` survive across scans. Query index
`{instance_id, state, severity}`.
**An empty `FixedIn` is a real and common state** and must never be conflated
with "not vulnerable". A CVE with no vendor fix published yet is exactly the
finding people most need to see, and also the one that most needs acceptance,
because there is nothing to patch.
Findings are **not deleted when a package is patched**. State moves to `fixed`
with `fixed_at` set, so "what did we remediate last quarter" remains
answerable — which is the question an auditor asks.
### `vulndb_meta` — singleton, deliberately unscoped
`db_version`, `pulled_at`, `last_full_scan_at`, `last_error`. It carries no
`instance_id` because the vulnerability database is a property of the
deployment, not of a tenant. Same reasoning as `migrations`.
### `vuln_alert_rules`
`instance_id`, `name`, `enabled`, `min_severity`, `tags map[string]string`,
`channel_ids []`, timestamps.
The tag filter resolves through **`services.ResolveTargets`**, not a second
matcher. That function is already the single answer to which servers a
selector touches, and an alert rule that disagreed with a workflow about what
`env:prod` means would be worse than having no filter at all.
---
## The matching engine
```
server/internal/vulndb/
pull.go OCI fetch → temp dir, version compare against vulndb_meta
db.go BoltDB open, advisory lookup by (ecosystem, source, version)
match.go per-family matching, severity resolution
version.go dispatch to deb/rpm/apk comparator by OS family
```
Dependencies: `github.com/aquasecurity/trivy-db` for the BoltDB schema, plus
`go-deb-version`, `go-rpm-version` and `go-apk-version` — each a small
standalone module doing one job. The roughly 200 lines of per-distro advisory
lookup are ours.
Importing `trivy` itself was rejected: it would pull a very large transitive
dependency tree into the server binary for one feature, and its Go API carries
no stability guarantee across minor versions. Shelling out to the `trivy`
binary against a generated SBOM was rejected for shipping a second binary in
the image and turning a library call into subprocess lifecycle, timeouts and
output-format drift.
### Why the comparators are bought rather than written
Version ordering is where this feature lives or dies, and its failure mode is
silent. `dpkg` ordering has epochs, and `~` sorts *before* the empty string, so
`3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own
segment rules and treats `~` and `^` differently again. A `strings.Compare` or
a semver parse orders `1.9` above `1.10` and reports a vulnerable fleet as
clean — a false negative, which nobody notices until it matters.
### Scanning one server
1. Load `server_packages`; resolve OS family and version to a `trivy-db`
ecosystem.
2. **Unsupported ecosystem → record `status: unsupported`, clear the flag,
write no findings.**
3. For each package: resolve source name, look up advisories, compare versions.
4. Upsert vulnerable results as `open`, preserving `first_seen`.
5. Any currently-`open` finding absent from this result set → `fixed`, stamp
`fixed_at`.
6. Any `accepted` finding past its `until` → back to `open`.
7. Clear `scan_pending`, stamp `scanned_at` and `db_version`.
Steps 5 and 6 must run in that order, so a finding that is both absent and
expired settles as `fixed` rather than reopening on a package that no longer
carries it.
Step 2 matters as much as any of the matching. Arch has no feed in `trivy-db`,
so an Arch host must report **unsupported**, never "0 findings". Reporting
clean when the truth is unknown is the same class of lie as a silently stale
database, and it is the reason `vulndb_meta.pulled_at` appears on screen rather
than only in a log.
### Severity
Resolved **vendor → NVD → unknown**, in that order, never invented.
This will surface as "why is this critical CVE marked low", and the answer is
that Debian and Red Hat routinely downgrade an NVD score because the vulnerable
code path is not reachable in their build. Their rating is the accurate one for
that package, and showing NVD's above it would manufacture work that does not
need doing.
---
## Findings lifecycle
`open | fixed | accepted`.
An accepted finding is suppressed from counts and alerts until its `until`
date, then reopens automatically. A reason is required.
Acceptance with a mandatory expiry, rather than permanent dismissal, is what
keeps the feature usable in both directions. Without any acceptance mechanism,
a kernel CVE awaiting a reboot window sits red indefinitely and trains people
to ignore the page. With permanent dismissal, accepted findings accumulate
silently and nobody revisits them — the dismissal list becomes where risk goes
to be forgotten, which is precisely what an auditor asks to see.
Retention: `settings.vuln_finding_retention_days`, a `*int` on the same pattern
as `workflow_log_retention_days` — nil means 90 days, 0 means forever. Only
`fixed` findings are swept, by a `StartVulnSweeper` inside the same
`RunAsLeader("housekeeping", …)` as the existing sweepers. `open` and
`accepted` findings are never swept at any setting.
---
## Alerting
Per-org rules over the existing `notification_channels`: severity threshold,
optional tag filter, target channels.
A rescan emits one message summarising what newly opened — "12 new critical
across 4 servers" — never one message per finding. See the leader section for
why the tick boundary makes this structural.
Modelling findings as a monitor type was rejected. It would reuse monitors'
state machine and channel wiring for free, but monitors are up/down for one
endpoint with retries and hourly rollups, none of which means anything for a
CVE; most fields would be disabled in the UI and the uptime graphs would be
polluted with a signal that is not uptime.
This adds one `notify` payload type and a `vuln_digest.html.tmpl` /
`vuln_digest.txt.tmpl` pair in `shared/mail`, since `render_test.go` fails on
any template no case covers.
---
## Entitlement
`entitlement.features.vuln_scanning`, a boolean alongside the existing
per-instance feature toggles, so it can later be priced as a catalogue
`feature` component without a second migration. Off on Free.
**The gate is checked at `ReportPackages`, not at display.** Gating only the UI
would still pay every write cost, and storage is the expensive half.
The agent learns of it through the existing 30-second `SyncKeys` poll:
`SyncResponse` gains a `collect_packages` bool, and the hourly loop skips
collection entirely when it is false. So an ungated instance produces no
collection, no gRPC body, no document and no storage. `ReportPackages` still
re-checks the entitlement server-side and refuses — the agent flag is an
optimisation, the server check is the boundary.
Turning the feature off does not delete existing findings; they stop being
served and stop updating. Deletion is the instance-deletion path's job.
---
## REST API
```
GET /api/vulnerabilities # filter: severity, state, server, tags
GET /api/vulnerabilities/summary # severity counts + database freshness
POST /api/vulnerabilities/rescan # marks all scan_pending (owner|admin)
POST /api/vulnerabilities/:id/accept # reason + until (owner|admin)
DELETE /api/vulnerabilities/:id/accept # (owner|admin)
GET /api/servers/:id/vulnerabilities
GET /api/servers/:id/packages
GET /api/packages/search?name= # fleet-wide
GET,POST /api/vuln-rules · PUT,DELETE /api/vuln-rules/:id
```
Every mutating path writes an audit event, as all of them do. Acceptance is the
one decision people will be asked to justify, so `by`, `reason`, `until` and
`at` land in the audit record and not only on the document.
---
## UI
`/vulnerabilities` is a fleet board **grouped by CVE** — one row per CVE with
an affected-server count, expandable to the individual servers. The same CVE
across 40 servers is one decision, and a flat list of findings makes it look
like forty.
Server detail gains **Vulnerabilities** and **Packages** tabs. Alert rules go
on `/settings/notifications`, beside the channels they consume.
Remediation introduces no new mechanism: a finding carrying `fixed_in` renders
an **Apply updates** action calling the existing
`POST /api/servers/:id/apply-updates`, which is already `ApplyUpdatesCmd`. See
it, patch it, one place — and no second patching path to keep consistent with
the first.
Database freshness is shown wherever findings are, not tucked into settings. A
fleet scanning against a three-week-old database must say so rather than
quietly report all-clear.
---
## Testing
**Version comparison, table-driven, per family.** The highest-value test in the
feature, because its failure mode is silent. The table must include the
backport case explicitly — installed `1:3.0.2-0ubuntu1.15` against advisory
fixed-in `1:3.0.2-0ubuntu1.15` resolving to *not vulnerable* — alongside tilde
ordering (`1.0~rc1` < `1.0`), epoch dominance (`1:1.0` > `2.0`) and
`1.9` < `1.10`.
**Matcher golden tests.** A fixture package list and a small hand-built BoltDB
fixture asserting an exact finding set. No network: a test that pulls the real
database fails on a bad day and changes its expectations every six hours.
**State transitions.** `open → fixed → reopened`, acceptance expiry, and
`first_seen` surviving a rescan. The last is easy to break with an upsert that
overwrites, and nothing notices until a report claims everything was discovered
yesterday.
**Collector parsers**, one per package manager, from captured command output.
`updates.go` currently has no tests and its parsers are the same shape, so
these fixtures should cover both.
**Scoping and gating.** Findings queries scoped by `instance_id`, and the
entitlement gate proven to write nothing rather than merely hide.
---
## Failure modes
| Failure | Behaviour |
| ------- | --------- |
| Database pull fails | Keep the last good copy and serve stale. Record `last_error`, surface `pulled_at` age. **Never clear findings** — a network blip must not read as "all fixed" |
| Unsupported distribution | `status: unsupported`, not zero findings |
| Agent stops reporting | Findings persist and `collected_at` age is shown. No auto-expiry: a silent agent is not a patched server |
| `trivy-db` schema version bumps | The puller refuses an unknown schema rather than mis-parsing it |
| ghcr anonymous rate limit | Backoff; `VANTAGE_TRIVY_DB_REF` mirrors to a private registry |
| Leadership lost mid-scan | The context is cancelled and the scan returns; `scan_pending` is still set, so the next leader picks it up |
| Instance deleted | **`server_packages` and `vuln_findings` must be added to the control plane's instance-deletion collection list.** Easy to miss, and missing it orphans a tenant's package data indefinitely |
---
## Environment variables
| Name | Required | Notes |
| ---- | -------- | ----- |
| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for air-gapped installs or to avoid the anonymous ghcr rate limit |
| `VANTAGE_VULNDB_DISABLED` | no | disables the puller and the scheduler entirely. Findings already written are still served and still marked stale |
---
## Deliberately out of scope
- **Windows.** Separate source, collector and matcher; its own spec.
- **Container image scanning.** Sub-project C; needs the container registry.
- **Compliance baseline assertions.** Sub-project D; shares this findings UI
and nothing else.
- **Language-level dependency scanning** (npm, pip, Go modules). `trivy-db`
covers these ecosystems, but finding the manifests on a host is a different
collection problem from asking the package manager what is installed.
- **Automatic patching on a finding.** Remediation is one click, not zero. An
unattended upgrade triggered by a CVE feed is a fleet-wide change driven by a
third party's data, which is not a decision to take away from an operator.