# Package inventory and CVE findings

Date: 2026-08-06

Agents report the packages installed on each server. The control plane matches
them against distro security feeds and raises findings that link straight to
the patching path that already exists. A finding nobody can fix today can be
accepted with a reason and an expiry date rather than sitting red forever.

This is one of four sub-projects sketched together and deliberately separated:

| # | Sub-project | Depends on |
| - | ----------- | ---------- |
| A | **Package inventory + CVE findings** — this spec | nothing |
| B | Container/service registry | nothing |
| C | Container image scanning | A and B |
| D | Compliance profiles (baseline assertions) | shares A's findings UI only |

A and B are independent of one another. C is the joiner and must not be
designed before both exist. D shares a page with A and nothing else — a
different collector, a different evaluation model and a different remediation
story — so folding it in here would double the size for no shared machinery.

Scope of this spec is **A, Linux only.** Windows needs a separate source
(MSRC CVRF), a separate collector (`Get-HotFix` plus registry) and a KB
supersedence matcher that shares no code with the Linux path. That matches the
existing position that Windows agents are second-class by design, and the six
package managers `updates.go` already detects cover the whole Linux surface.

---

## The trap this design is built around

Distributions **backport** security fixes without changing the upstream
version. Ubuntu ships `openssl 3.0.2-0ubuntu1.15` patched against
CVE-2023-0286; NVD says version 3.0.2 is vulnerable. Matching installed
versions against NVD or CPE ranges therefore reports a fleet full of criticals
that are all already fixed.

That is not merely noisy. It is fatal to the feature: once the first report is
mostly wrong, nobody reads the second one, and a genuine finding is lost in the
noise it created. Everything below follows from refusing to make that mistake.

The correct source is the **distribution's own security feed**, keyed on the
distribution's own version string — Debian and Ubuntu OVAL/USN, Red Hat OVAL
v2, Alpine secdb. `trivy-db` is those feeds pre-merged into one BoltDB
artifact, rebuilt every six hours and published as an OCI artifact.

---

## Where the vulnerability data comes from

`trivy-db`, pulled server-side from `ghcr.io/aquasecurity/trivy-db:2`.

The alternative considered was querying OSV.dev per scan, which needs no
storage and no puller. It was rejected on two counts: it requires outbound
internet on every scan, which breaks air-gapped installs; and it sends the
package list of a customer's entire fleet to a third party. The audience most
likely to buy vulnerability scanning is the audience least willing to do that.

The blob is roughly 50MB, read-only, reproducible, and identified by a version
number. **It is not stored in Mongo and not written to `/data`** —
`server.persistence` defaults to off and nothing writes to `/data` any more.
It does not need durable storage: whichever pod needs it pulls it to its own
ephemeral temp directory. Nothing shared, nothing to back up, nothing to
migrate.

`VANTAGE_TRIVY_DB_REF` overrides the default reference so a customer can mirror
the artifact into their own registry. It also covers the anonymous ghcr rate
limit, which the six-hourly pull cadence already makes unlikely to bite.

---

## Only the leader matches

This is the crux, and it falls out of the replica model already in the
codebase.

Two things trigger matching, and they happen on different pods:

1. a fleet-wide rescan when `trivy-db` updates — naturally the leader's job
2. a server's package list changing — handled by whichever pod holds *that
   agent's* command stream

If (2) matched inline, **every replica would need the 50MB database resident**,
and a database refresh would have N pods racing to rescan the same fleet and N
digests reaching the customer. That is the exact failure `RunAsLeader` exists
to prevent, and it is the same argument that put `monitorsched` behind the
lock.

So `ReportPackages` does not match. It upserts the package list and sets
`scan_pending: true`. That is all it does.

`server/internal/vulnsched` then runs inside the **existing**
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`,
`workflowsched` and the sweepers — one role, one lock. Every 60 seconds it:

1. pulls `trivy-db` if the local copy is older than six hours
2. if the pulled version differs from `vulndb_meta.db_version`, marks **every**
   server `scan_pending`
3. matches all `scan_pending` servers, clears the flag, diffs against existing
   findings
4. emits **one** digest per tick covering everything newly opened

Step 4 is why batching is structural rather than bolted on. A `trivy-db`
refresh can open several hundred findings across a fleet at once; one message
per finding would rate-limit the webhook or get the channel muted, and either
way the customer stops receiving the alerts they are paying for. The tick is
already the natural batch boundary, so **the failure cannot occur by
construction** rather than by a debounce someone has to maintain.

`scan_pending` lives on the document rather than in memory, for the same reason
`next_run_at` and `workflow_log_seq` do: a leader handover between marking and
scanning would otherwise lose it. A handover costs the new leader one re-pull
of the database.

The cost of this indirection is up to 60 seconds between an agent reporting a
changed package set and its findings updating. For vulnerability data that is
nothing, and it buys a single matching path instead of two.

---

## Components

```
agent/internal/packages/        collect installed packages + /etc/os-release
proto/                          ReportPackages RPC
server/internal/vulndb/         puller, BoltDB access, matcher
server/internal/vulnsched/      leader-owned tick: pull, scan, digest
server/internal/services/       findings, acceptance, alert rules
web/app/(app)/vulnerabilities/  fleet board; plus two server-detail tabs
```

`vulnsched` takes the dependencies it needs — `LogEvent` and the notification
dispatch — as a `vulnsched.Deps` injected from `main.go`, following
`workflowsched`. The manual rescan endpoint does not call into `vulnsched` at
all: it sets `scan_pending` on every server and lets the next tick find them,
so there is no path by which `services` imports the scheduler and no cycle to
avoid later.

---

## The wire path

A new `ReportPackages` RPC on the agent's existing hourly loop — the same
`runUpdateCheck` cadence, reusing `updates.go`'s `detectPM()`.

```protobuf
rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse);

message ReportPackagesRequest {
  string server_id     = 1;
  string agent_token   = 2;
  string hash          = 3;  // sha256 of the sorted list
  OSRelease os         = 4;
  repeated InstalledPackage packages = 5;  // omitted when only offering a hash
}

message ReportPackagesResponse {
  bool need_full = 1;  // hash differs; resend with packages populated
}
```

The agent calls once with `packages` empty. `need_full` true means the hash
differs from what the server holds, and the agent immediately calls again with
the list populated.

The agent sends a SHA-256 of its sorted package list first. If it matches what
the server already holds, the server answers `unchanged` and the ~150KB body is
never sent. A machine's package set changes rarely, so almost every hour costs
one small message, and the rare changed hour costs one extra round trip.

Folding the list into the existing 15-minute `InventoryReport` static snapshot
was rejected: it would re-send ~150KB per server every 15 minutes regardless of
change, roughly 40MB/hour of gRPC traffic on a 100-server fleet to transmit
data that is almost always identical.

---

## Data model

Four new collections. Every one carries `instance_id` except `vulndb_meta`,
which is explained below.

### `server_packages` — one document per server, not per package

```go
type ServerPackages struct {
    ID          primitive.ObjectID `bson:"_id"`
    InstanceID  primitive.ObjectID `bson:"instance_id"`
    ServerID    string             `bson:"server_id"`
    OS          OSRelease          `bson:"os"`          // family, version_id, arch
    Hash        string             `bson:"hash"`        // sha256 of the sorted list
    Packages    []InstalledPackage `bson:"packages"`
    CollectedAt time.Time          `bson:"collected_at"`
    ScanPending bool               `bson:"scan_pending"`
    ScannedAt   time.Time          `bson:"scanned_at"`
    Status      string             `bson:"status"`      // ok | unsupported
    DBVersion   int                `bson:"db_version"`  // last matched against
}

type InstalledPackage struct {
    Name       string `bson:"name"`
    Version    string `bson:"version"`     // distro version string, verbatim
    Epoch      int    `bson:"epoch,omitempty"`
    Arch       string `bson:"arch"`
    SourceName string `bson:"source_name,omitempty"`
}
```

One document rather than two thousand is what makes a report a **single atomic
upsert with no delta logic** — the hash already established that something
changed, so there is nothing to reconcile field by field. A typical Linux host
lands near 150KB, comfortably inside the 16MB document limit.

Indexes: `{instance_id, server_id}` unique, and a multikey
`{instance_id, "packages.name"}` for fleet-wide package search.

`SourceName` is not decoration. **Debian and Ubuntu advisories are keyed on the
source package**: a CVE against `openssl` covers the binaries `libssl3`,
`openssl` and `libssl-dev`, so matching on binary name alone misses two of the
three.

`OS.VersionID` selects the feed. Ubuntu 22.04 and 24.04 publish different fixed
versions for the same CVE, so a scan without it is guesswork.

### `vuln_findings` — one document per (server, CVE, package)

```go
type VulnFinding struct {
    ID         primitive.ObjectID `bson:"_id"`
    InstanceID primitive.ObjectID `bson:"instance_id"`
    ServerID   string             `bson:"server_id"`

    CVEID       string   `bson:"cve_id"`
    PackageName string   `bson:"package_name"`
    Installed   string   `bson:"installed_version"`
    FixedIn     string   `bson:"fixed_in,omitempty"`
    Severity    string   `bson:"severity"`
    CVSSScore   float64  `bson:"cvss_score,omitempty"`
    Title       string   `bson:"title,omitempty"`
    References  []string `bson:"references,omitempty"`

    State     string      `bson:"state"` // open | fixed | accepted
    FirstSeen time.Time   `bson:"first_seen"`
    LastSeen  time.Time   `bson:"last_seen"`
    FixedAt   *time.Time  `bson:"fixed_at,omitempty"`
    Accepted  *Acceptance `bson:"accepted,omitempty"`
}

type Acceptance struct {
    By     primitive.ObjectID `bson:"by"`
    Reason string             `bson:"reason"`
    Until  time.Time          `bson:"until"`
    At     time.Time          `bson:"at"`
}
```

Unique on `{instance_id, server_id, cve_id, package_name}`. That key is what
makes a rescan an idempotent upsert rather than a duplicate factory, and it is
what lets `first_seen` survive across scans. Query index
`{instance_id, state, severity}`.

**An empty `FixedIn` is a real and common state** and must never be conflated
with "not vulnerable". A CVE with no vendor fix published yet is exactly the
finding people most need to see, and also the one that most needs acceptance,
because there is nothing to patch.

Findings are **not deleted when a package is patched**. State moves to `fixed`
with `fixed_at` set, so "what did we remediate last quarter" remains
answerable — which is the question an auditor asks.

### `vulndb_meta` — singleton, deliberately unscoped

`db_version`, `pulled_at`, `last_full_scan_at`, `last_error`. It carries no
`instance_id` because the vulnerability database is a property of the
deployment, not of a tenant. Same reasoning as `migrations`.

### `vuln_alert_rules`

`instance_id`, `name`, `enabled`, `min_severity`, `tags map[string]string`,
`channel_ids []`, timestamps.

The tag filter resolves through **`services.ResolveTargets`**, not a second
matcher. That function is already the single answer to which servers a
selector touches, and an alert rule that disagreed with a workflow about what
`env:prod` means would be worse than having no filter at all.

---

## The matching engine

```
server/internal/vulndb/
  pull.go      OCI fetch → temp dir, version compare against vulndb_meta
  db.go        BoltDB open, advisory lookup by (ecosystem, source, version)
  match.go     per-family matching, severity resolution
  version.go   dispatch to deb/rpm/apk comparator by OS family
```

Dependencies: `github.com/aquasecurity/trivy-db` for the BoltDB schema, plus
`go-deb-version`, `go-rpm-version` and `go-apk-version` — each a small
standalone module doing one job. The roughly 200 lines of per-distro advisory
lookup are ours.

Importing `trivy` itself was rejected: it would pull a very large transitive
dependency tree into the server binary for one feature, and its Go API carries
no stability guarantee across minor versions. Shelling out to the `trivy`
binary against a generated SBOM was rejected for shipping a second binary in
the image and turning a library call into subprocess lifecycle, timeouts and
output-format drift.

### Why the comparators are bought rather than written

Version ordering is where this feature lives or dies, and its failure mode is
silent. `dpkg` ordering has epochs, and `~` sorts *before* the empty string, so
`3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own
segment rules and treats `~` and `^` differently again. A `strings.Compare` or
a semver parse orders `1.9` above `1.10` and reports a vulnerable fleet as
clean — a false negative, which nobody notices until it matters.

### Scanning one server

1. Load `server_packages`; resolve OS family and version to a `trivy-db`
   ecosystem.
2. **Unsupported ecosystem → record `status: unsupported`, clear the flag,
   write no findings.**
3. For each package: resolve source name, look up advisories, compare versions.
4. Upsert vulnerable results as `open`, preserving `first_seen`.
5. Any currently-`open` finding absent from this result set → `fixed`, stamp
   `fixed_at`.
6. Any `accepted` finding past its `until` → back to `open`.
7. Clear `scan_pending`, stamp `scanned_at` and `db_version`.

Steps 5 and 6 must run in that order, so a finding that is both absent and
expired settles as `fixed` rather than reopening on a package that no longer
carries it.

Step 2 matters as much as any of the matching. Arch has no feed in `trivy-db`,
so an Arch host must report **unsupported**, never "0 findings". Reporting
clean when the truth is unknown is the same class of lie as a silently stale
database, and it is the reason `vulndb_meta.pulled_at` appears on screen rather
than only in a log.

### Severity

Resolved **vendor → NVD → unknown**, in that order, never invented.

This will surface as "why is this critical CVE marked low", and the answer is
that Debian and Red Hat routinely downgrade an NVD score because the vulnerable
code path is not reachable in their build. Their rating is the accurate one for
that package, and showing NVD's above it would manufacture work that does not
need doing.

---

## Findings lifecycle

`open | fixed | accepted`.

An accepted finding is suppressed from counts and alerts until its `until`
date, then reopens automatically. A reason is required.

Acceptance with a mandatory expiry, rather than permanent dismissal, is what
keeps the feature usable in both directions. Without any acceptance mechanism,
a kernel CVE awaiting a reboot window sits red indefinitely and trains people
to ignore the page. With permanent dismissal, accepted findings accumulate
silently and nobody revisits them — the dismissal list becomes where risk goes
to be forgotten, which is precisely what an auditor asks to see.

Retention: `settings.vuln_finding_retention_days`, a `*int` on the same pattern
as `workflow_log_retention_days` — nil means 90 days, 0 means forever. Only
`fixed` findings are swept, by a `StartVulnSweeper` inside the same
`RunAsLeader("housekeeping", …)` as the existing sweepers. `open` and
`accepted` findings are never swept at any setting.

---

## Alerting

Per-org rules over the existing `notification_channels`: severity threshold,
optional tag filter, target channels.

A rescan emits one message summarising what newly opened — "12 new critical
across 4 servers" — never one message per finding. See the leader section for
why the tick boundary makes this structural.

Modelling findings as a monitor type was rejected. It would reuse monitors'
state machine and channel wiring for free, but monitors are up/down for one
endpoint with retries and hourly rollups, none of which means anything for a
CVE; most fields would be disabled in the UI and the uptime graphs would be
polluted with a signal that is not uptime.

This adds one `notify` payload type and a `vuln_digest.html.tmpl` /
`vuln_digest.txt.tmpl` pair in `shared/mail`. Note that `shared/mail` templates
are parsed in `init()`, so a mistyped field is a boot-time panic — CLAUDE.md
describes a `render_test.go` guarding against exactly this, but **that file does
not exist**; the repository has no Go tests at all, and by instruction this
feature adds none. The template pair must therefore be verified by starting the
binary and sending one digest through a real channel.

---

## Entitlement

The feature name is `vuln_scanning`, and it crosses the two services the way
every other feature does:

- **admin** carries it as a per-instance entitlement toggle, so it can later be
  priced as a catalogue `feature` component without a second migration;
- **the licence** snapshots it into `License.Features []string` at issue time;
- **the server** asks `lic.HasFeature("vuln_scanning")` and never switches on
  tier, so changing what a tier includes needs no server release.

Off on Free.

**The gate is checked at `ReportPackages`, not at display.** Gating only the UI
would still pay every write cost, and storage is the expensive half.

The agent learns of it through the existing 30-second `SyncKeys` poll:
`SyncResponse` gains a `collect_packages` bool, and the hourly loop skips
collection entirely when it is false. So an ungated instance produces no
collection, no gRPC body, no document and no storage. `ReportPackages` still
re-checks the entitlement server-side and refuses — the agent flag is an
optimisation, the server check is the boundary.

Turning the feature off does not delete existing findings; they stop being
served and stop updating. Deletion is the instance-deletion path's job.

---

## REST API

```
GET    /api/vulnerabilities              # filter: severity, state, server, tags
GET    /api/vulnerabilities/summary      # severity counts + database freshness
POST   /api/vulnerabilities/rescan       # marks all scan_pending  (owner|admin)
POST   /api/vulnerabilities/:id/accept   # reason + until          (owner|admin)
DELETE /api/vulnerabilities/:id/accept   #                         (owner|admin)
GET    /api/servers/:id/vulnerabilities
GET    /api/servers/:id/packages
GET    /api/packages/search?name=        # fleet-wide
GET,POST /api/vuln-rules · PUT,DELETE /api/vuln-rules/:id
```

Every mutating path writes an audit event, as all of them do. Acceptance is the
one decision people will be asked to justify, so `by`, `reason`, `until` and
`at` land in the audit record and not only on the document.

---

## UI

`/vulnerabilities` is a fleet board **grouped by CVE** — one row per CVE with
an affected-server count, expandable to the individual servers. The same CVE
across 40 servers is one decision, and a flat list of findings makes it look
like forty.

Server detail gains **Vulnerabilities** and **Packages** tabs. Alert rules go
on `/settings/notifications`, beside the channels they consume.

Remediation introduces no new mechanism: a finding carrying `fixed_in` renders
an **Apply updates** action calling the existing
`POST /api/servers/:id/apply-updates`, which is already `ApplyUpdatesCmd`. See
it, patch it, one place — and no second patching path to keep consistent with
the first.

Database freshness is shown wherever findings are, not tucked into settings. A
fleet scanning against a three-week-old database must say so rather than
quietly report all-clear.

---

## Verification

**No automated tests.** The repository has none today, and by explicit
instruction this feature adds none — no `*_test.go`, no frontend test files.
That is a deliberate decision by the repository owner, recorded here so the
absence reads as a choice rather than an omission.

It does change the risk profile, and the places it changes it are worth naming,
because each fails by producing a **wrong answer rather than a crash**:

- **Version comparison.** The backport case — installed `1:3.0.2-0ubuntu1.15`
  against advisory fixed-in `1:3.0.2-0ubuntu1.15` resolving to *not
  vulnerable* — plus tilde ordering (`1.0~rc1` < `1.0`), epoch dominance
  (`1:1.0` > `2.0`) and `1.9` < `1.10`. Wrong here means a vulnerable fleet
  reported clean.
- **Source-package fan-out.** One advisory against `openssl` must flag
  `libssl3`, `openssl` and `libssl-dev`. Matching on binary name alone silently
  finds one of three.
- **`first_seen` preservation.** An upsert that overwrites it makes every
  finding look discovered today, and nothing surfaces that until someone reads
  a report.
- **Fixed-before-reopen ordering.** A finding both absent from a scan and past
  its acceptance expiry must settle `fixed`, not reopen.

The implementation plan carries a manual verification table for each, to be
walked before the relevant task is committed. They are the substitute for the
tests, not a formality.

---

## Failure modes

| Failure | Behaviour |
| ------- | --------- |
| Database pull fails | Keep the last good copy and serve stale. Record `last_error`, surface `pulled_at` age. **Never clear findings** — a network blip must not read as "all fixed" |
| Unsupported distribution | `status: unsupported`, not zero findings |
| Agent stops reporting | Findings persist and `collected_at` age is shown. No auto-expiry: a silent agent is not a patched server |
| `trivy-db` schema version bumps | The puller refuses an unknown schema rather than mis-parsing it |
| ghcr anonymous rate limit | Backoff; `VANTAGE_TRIVY_DB_REF` mirrors to a private registry |
| Leadership lost mid-scan | The context is cancelled and the scan returns; `scan_pending` is still set, so the next leader picks it up |
| Instance deleted | **`server_packages` and `vuln_findings` must be added to the control plane's instance-deletion collection list.** Easy to miss, and missing it orphans a tenant's package data indefinitely |

---

## Environment variables

| Name | Required | Notes |
| ---- | -------- | ----- |
| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for air-gapped installs or to avoid the anonymous ghcr rate limit |
| `VANTAGE_VULNDB_DISABLED` | no | disables the puller and the scheduler entirely. Findings already written are still served and still marked stale |

---

## Deliberately out of scope

- **Windows.** Separate source, collector and matcher; its own spec.
- **Container image scanning.** Sub-project C; needs the container registry.
- **Compliance baseline assertions.** Sub-project D; shares this findings UI
  and nothing else.
- **Language-level dependency scanning** (npm, pip, Go modules). `trivy-db`
  covers these ecosystems, but finding the manifests on a host is a different
  collection problem from asking the package manager what is installed.
- **Automatic patching on a finding.** Remediation is one click, not zero. An
  unattended upgrade triggered by a CVE feed is a fleet-wide change driven by a
  third party's data, which is not a decision to take away from an operator.
