From fe7bc300e2a09a2d8f486d409e657a5b62f3be5a Mon Sep 17 00:00:00 2001 From: mrhid6 Date: Thu, 6 Aug 2026 10:49:10 +0100 Subject: [PATCH] docs: implementation plan for package inventory and CVE findings 17 tasks, TDD where the logic is pure. Corrects two spec claims: the server reads features via License.HasFeature rather than admin's entitlement directly, and shared/mail/render_test.go does not exist. --- ...8-06-package-inventory-and-cve-findings.md | 3124 +++++++++++++++++ ...ckage-inventory-and-cve-findings-design.md | 20 +- 2 files changed, 3139 insertions(+), 5 deletions(-) create mode 100644 docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md diff --git a/docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md b/docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md new file mode 100644 index 0000000..3b61ff6 --- /dev/null +++ b/docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md @@ -0,0 +1,3124 @@ +# Package Inventory and CVE Findings Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Agents report the packages installed on each Linux server; the control plane matches them against distribution security feeds (`trivy-db`) and raises findings that link to the existing `ApplyUpdatesCmd` patching path. + +**Architecture:** The agent collects packages on its existing hourly loop and offers a hash first, sending the ~150KB list only when the server says it differs. `ReportPackages` does **not** match — it upserts the list and sets `scan_pending: true`. A new `vulnsched` loop, running inside the existing `bus.RunAsLeader("housekeeping", …)`, pulls `trivy-db` to an ephemeral directory, scans pending servers, diffs findings and emits one batched digest per tick. All matching logic is pure functions in `server/internal/vulndb` testable without a database, a network or a real database file. + +**Tech Stack:** Go 1.26, gin, mongo-driver v2, gRPC, bbolt, Next.js 16 + TanStack Query, Tailwind 3. + +**Spec:** `docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md`. Read it before starting. Where this plan and the spec disagree, the spec wins and the plan is wrong. + +## Global Constraints + +- **Linux only.** Windows agents must not collect packages and must not appear in findings. Do not add an MSRC source, a `Get-HotFix` collector or KB supersedence logic — that is a separate spec. +- **This repository has no Go tests today.** `git ls-files | grep _test.go` returns nothing. Tests added here are the first. They must run under plain `go test ./...` from the repo root with **no database, no network, no build tags and no fixtures downloaded at test time**. +- CLAUDE.md claims `shared/mail/render_test.go` exists. It does not. Task 12 creates it. +- Every service query is scoped by `instance_id`. No unscoped lookups, ever. +- Every mutating API path writes an audit event via `services.LogEvent(instanceID, eventType, actor, serverID, keyID, details string)`. +- Licence features are read with `lic.HasFeature("vuln_scanning")`. **Never switch on `Tier`.** +- Module path prefix is `gitea.hostxtra.co.uk/mrhid6/vantage/`. +- Mongo driver is **v2**: `go.mongodb.org/mongo-driver/v2/bson`, and ObjectIDs are `bson.ObjectID`, not `primitive.ObjectID`. The spec's struct listings use `primitive.` — that is spec shorthand; **write `bson.ObjectID`**. +- Collections are reached via `db.Col("name")`. +- No component in `web/` may carry a hex colour; use Tailwind token names only. `web/` is dark-only — do not add a light theme. +- Commit messages use the `feat:` / `fix:` / `docs:` prefixes already in `git log`. +- Severity strings are lowercase and fixed: `critical`, `high`, `medium`, `low`, `unknown`. +- Finding states are exactly `open`, `fixed`, `accepted`. + +## Dependencies to add + +Run from the repo root; `go.work` covers all modules. + +```bash +cd server +go get github.com/aquasecurity/trivy-db@latest +go get github.com/knqyf263/go-deb-version@latest +go get github.com/knqyf263/go-rpm-version@latest +go get github.com/knqyf263/go-apk-version@latest +go get oras.land/oras-go/v2@latest +go get go.etcd.io/bbolt@latest +``` + +`oras-go` pulls the OCI artifact. `trivy-db` provides the BoltDB schema reader. The three version modules are small and each does one job. + +--- + +## File Structure + +**Create — agent:** +- `agent/internal/packages/packages.go` — `Collect()` dispatch by package manager +- `agent/internal/packages/parse.go` — pure output parsers, one per manager +- `agent/internal/packages/parse_test.go` — parser tests from captured output +- `agent/internal/packages/osrelease.go` — `/etc/os-release` parsing +- `agent/internal/packages/osrelease_test.go` + +**Create — server:** +- `server/internal/models/packages.go` — `ServerPackages`, `InstalledPackage`, `OSRelease` +- `server/internal/models/vuln.go` — `VulnFinding`, `Acceptance`, `VulnDBMeta`, `VulnAlertRule` +- `server/internal/vulndb/version.go` — comparator dispatch by OS family +- `server/internal/vulndb/version_test.go` — **the highest-value test in the feature** +- `server/internal/vulndb/ecosystem.go` — OS family/version → trivy-db bucket, or unsupported +- `server/internal/vulndb/ecosystem_test.go` +- `server/internal/vulndb/pull.go` — OCI fetch to temp dir +- `server/internal/vulndb/db.go` — advisory lookup over the BoltDB +- `server/internal/vulndb/match.go` — pure matching +- `server/internal/vulndb/match_test.go` +- `server/internal/vulndb/source.go` — binary → source name resolution +- `server/internal/services/packages.go` — store, hash compare, package search +- `server/internal/services/findings.go` — finding diff/state machine + acceptance +- `server/internal/services/findings_test.go` — pure state-machine tests +- `server/internal/services/vulnrules.go` — alert rule CRUD, digest recipients +- `server/internal/vulnsched/sched.go` — the leader-owned loop +- `server/internal/api/vulnerabilities.go` — REST handlers +- `shared/mail/vuln.go` — `SendVulnDigest` +- `shared/mail/templates/vuln_digest.html.tmpl`, `vuln_digest.txt.tmpl` +- `shared/mail/render_test.go` — renders every template; fails on any not covered + +**Create — web:** +- `web/app/(app)/vulnerabilities/page.tsx` — fleet board grouped by CVE +- `web/components/vulnerabilities/FindingRow.tsx` — one CVE, expandable +- `web/components/vulnerabilities/AcceptDialog.tsx` — reason + until +- `web/components/vulnerabilities/DBFreshness.tsx` — `pulled_at` age banner +- `web/components/settings/VulnAlertRulesCard.tsx` + +**Modify:** +- `proto/vantage/v1/vantage.proto` — `ReportPackages` RPC and messages; `SyncResponse.collect_packages` +- `agent/internal/sync/sync.go:370-411` — `runUpdateCheck` also reports packages +- `server/internal/grpc/server.go` — `ReportPackages` handler +- `server/internal/services/servers.go` — `SyncKeys` sets `collect_packages` +- `server/cmd/main.go:184-206` — start `vulnsched` and the sweeper inside `RunAsLeader` +- `server/cmd/main.go` (schema setup, near the other `Ensure*Indexes` calls) — `EnsureVulnIndexes` +- `server/internal/api/handlers.go` — register routes +- `shared/models/settings.go` — `VulnFindingRetentionDays *int` +- `admin/internal/models/entitlements.go` — `vuln_scanning` toggle +- `web/lib/api.ts` — types and client methods +- `web/components/Sidebar.tsx` — Vulnerabilities entry +- `web/app/(app)/servers/[id]/page.tsx` — Vulnerabilities and Packages tabs +- `web/app/(app)/settings/notifications/page.tsx` — alert rules card +- The control plane's instance-deletion collection list — add both new collections +- `CLAUDE.md`, `docsite/docs/vantage/` — document the subsystem + +--- + +### Task 1: Version comparators + +The single most important task. Its failure mode is silent: a wrong comparison reports a vulnerable fleet as clean. + +**Files:** +- Create: `server/internal/vulndb/version.go` +- Test: `server/internal/vulndb/version_test.go` + +**Interfaces:** +- Consumes: nothing. +- Produces: `vulndb.LessThan(family, a, b string) (bool, error)` and `vulndb.ErrUnsupportedFamily`. + +- [ ] **Step 1: Write the failing test** + +Create `server/internal/vulndb/version_test.go`: + +```go +package vulndb + +import "testing" + +func TestLessThan(t *testing.T) { + cases := []struct { + name string + family string + a, b string + want bool + }{ + // The backport trap. An Ubuntu package patched in place carries the + // same upstream version as the vulnerable one; only the revision + // differs. Equal versions must NOT compare as less than, or a patched + // host reports as vulnerable. + {"deb equal backport", "ubuntu", "1:3.0.2-0ubuntu1.15", "1:3.0.2-0ubuntu1.15", false}, + {"deb revision older", "ubuntu", "1:3.0.2-0ubuntu1.14", "1:3.0.2-0ubuntu1.15", true}, + {"deb revision newer", "ubuntu", "1:3.0.2-0ubuntu1.16", "1:3.0.2-0ubuntu1.15", false}, + + // Tilde sorts BEFORE the empty string: 1.0~rc1 precedes 1.0. + {"deb tilde is older", "debian", "1.0~rc1", "1.0", true}, + {"deb tilde reversed", "debian", "1.0", "1.0~rc1", false}, + + // Epoch dominates everything to its right. + {"deb epoch dominates", "debian", "2.0", "1:1.0", true}, + + // Numeric segments compare numerically, not lexically. + {"deb numeric not lexical", "debian", "1.9", "1.10", true}, + + {"rpm equal", "redhat", "1.2.3-4.el9", "1.2.3-4.el9", false}, + {"rpm release older", "redhat", "1.2.3-3.el9", "1.2.3-4.el9", true}, + {"rpm epoch dominates", "redhat", "2:1.0-1", "1:9.0-1", false}, + {"rpm numeric not lexical", "rocky", "1.9-1", "1.10-1", true}, + + {"apk equal", "alpine", "1.2.3-r0", "1.2.3-r0", false}, + {"apk release older", "alpine", "1.2.3-r0", "1.2.3-r1", true}, + {"apk numeric not lexical", "alpine", "1.9-r0", "1.10-r0", true}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, err := LessThan(tc.family, tc.a, tc.b) + if err != nil { + t.Fatalf("LessThan(%q, %q, %q): unexpected error %v", tc.family, tc.a, tc.b, err) + } + if got != tc.want { + t.Errorf("LessThan(%q, %q, %q) = %v, want %v", tc.family, tc.a, tc.b, got, tc.want) + } + }) + } +} + +func TestLessThanUnsupportedFamily(t *testing.T) { + if _, err := LessThan("arch", "1.0", "2.0"); err == nil { + t.Fatal("expected an error for an unsupported family, got nil") + } +} + +func TestLessThanUnparseableVersion(t *testing.T) { + // An unparseable version must error, never silently answer false — + // false here means "not vulnerable", which is the dangerous direction. + if _, err := LessThan("ubuntu", "not a version!!", "1.0"); err == nil { + t.Fatal("expected an error for an unparseable version, got nil") + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd server && go test ./internal/vulndb/ -run TestLessThan -v` +Expected: build failure — package `vulndb` does not exist. + +- [ ] **Step 3: Write the implementation** + +Create `server/internal/vulndb/version.go`: + +```go +// Package vulndb matches installed packages against distribution security +// advisories. +// +// Version comparison is bought rather than written. Distribution version +// ordering is subtle in ways that are invisible until they are wrong: dpkg has +// epochs and sorts "~" before the empty string, rpmvercmp has its own segment +// rules and treats "~" and "^" differently again, and any ordering that falls +// back on string comparison puts 1.10 before 1.9. Every one of those mistakes +// produces a false negative — a vulnerable host reported clean — which is the +// failure nobody notices. +package vulndb + +import ( + "errors" + "fmt" + + apk "github.com/knqyf263/go-apk-version" + deb "github.com/knqyf263/go-deb-version" + rpm "github.com/knqyf263/go-rpm-version" +) + +// ErrUnsupportedFamily means we hold no comparator for this distribution, and +// therefore cannot answer whether it is vulnerable. Callers must surface this +// as "unsupported" and must never treat it as "not vulnerable". +var ErrUnsupportedFamily = errors.New("unsupported OS family") + +// LessThan reports whether version a sorts before version b under the ordering +// rules of the given OS family. +func LessThan(family, a, b string) (bool, error) { + switch family { + case "debian", "ubuntu": + va, err := deb.NewVersion(a) + if err != nil { + return false, fmt.Errorf("parse deb version %q: %w", a, err) + } + vb, err := deb.NewVersion(b) + if err != nil { + return false, fmt.Errorf("parse deb version %q: %w", b, err) + } + return va.LessThan(vb), nil + + case "redhat", "centos", "rocky", "alma", "amazon", "oracle": + // go-rpm-version does not error; rpmvercmp is defined over arbitrary + // strings. Guard empties so a missing version cannot read as equal. + if a == "" || b == "" { + return false, fmt.Errorf("empty rpm version (a=%q b=%q)", a, b) + } + return rpm.NewVersion(a).LessThan(rpm.NewVersion(b)), nil + + case "alpine": + va, err := apk.NewVersion(a) + if err != nil { + return false, fmt.Errorf("parse apk version %q: %w", a, err) + } + vb, err := apk.NewVersion(b) + if err != nil { + return false, fmt.Errorf("parse apk version %q: %w", b, err) + } + return va.LessThan(vb), nil + + case "suse", "opensuse", "sles": + if a == "" || b == "" { + return false, fmt.Errorf("empty rpm version (a=%q b=%q)", a, b) + } + return rpm.NewVersion(a).LessThan(rpm.NewVersion(b)), nil + + default: + return false, fmt.Errorf("%w: %s", ErrUnsupportedFamily, family) + } +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd server && go test ./internal/vulndb/ -v` +Expected: PASS, all subtests. + +If `TestLessThanUnparseableVersion` fails because `go-deb-version` accepts the input, replace the input with an empty string and add an explicit empty-string guard to the `debian`/`ubuntu` branch matching the rpm one. Do not delete the test. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/vulndb/version.go server/internal/vulndb/version_test.go server/go.mod server/go.sum +git commit -m "feat: version comparators for distro package ordering" +``` + +--- + +### Task 2: Ecosystem mapping + +**Files:** +- Create: `server/internal/vulndb/ecosystem.go` +- Test: `server/internal/vulndb/ecosystem_test.go` + +**Interfaces:** +- Consumes: `vulndb.ErrUnsupportedFamily` from Task 1. +- Produces: `vulndb.Bucket(family, versionID string) (string, error)`. + +- [ ] **Step 1: Write the failing test** + +Create `server/internal/vulndb/ecosystem_test.go`: + +```go +package vulndb + +import ( + "errors" + "testing" +) + +func TestBucket(t *testing.T) { + cases := []struct { + family, versionID, want string + }{ + {"ubuntu", "22.04", "ubuntu 22.04"}, + {"ubuntu", "24.04", "ubuntu 24.04"}, + {"debian", "12", "debian 12"}, + {"alpine", "3.19.1", "alpine 3.19"}, // apk advisories are keyed on major.minor + {"rocky", "9.3", "redhat 9"}, // rebuilds share Red Hat's feed, major only + {"alma", "9.3", "redhat 9"}, + {"redhat", "9", "redhat 9"}, + } + for _, tc := range cases { + got, err := Bucket(tc.family, tc.versionID) + if err != nil { + t.Fatalf("Bucket(%q, %q): unexpected error %v", tc.family, tc.versionID, err) + } + if got != tc.want { + t.Errorf("Bucket(%q, %q) = %q, want %q", tc.family, tc.versionID, got, tc.want) + } + } +} + +func TestBucketUnsupported(t *testing.T) { + // Arch has no feed in trivy-db. It must be reported unsupported, never + // scanned to zero findings — claiming clean when the truth is unknown is + // the same class of lie as a silently stale database. + if _, err := Bucket("arch", ""); !errors.Is(err, ErrUnsupportedFamily) { + t.Fatalf("Bucket(arch) error = %v, want ErrUnsupportedFamily", err) + } +} + +func TestBucketMissingVersion(t *testing.T) { + // Ubuntu 22.04 and 24.04 publish different fixed versions for the same + // CVE, so a bucket without a version is guesswork. + if _, err := Bucket("ubuntu", ""); err == nil { + t.Fatal("expected an error when the version is missing, got nil") + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd server && go test ./internal/vulndb/ -run TestBucket -v` +Expected: FAIL — `undefined: Bucket`. + +- [ ] **Step 3: Write the implementation** + +Create `server/internal/vulndb/ecosystem.go`: + +```go +package vulndb + +import ( + "fmt" + "strings" +) + +// rhelRebuilds share Red Hat's advisory feed rather than publishing their own. +var rhelRebuilds = map[string]bool{ + "redhat": true, "centos": true, "rocky": true, "alma": true, "oracle": true, +} + +// Bucket maps an OS family and version onto the trivy-db bucket that holds its +// advisories. +// +// It returns ErrUnsupportedFamily rather than a best guess when we have no +// feed. A scan that cannot be performed must say so; reporting zero findings +// for a distribution we do not cover is indistinguishable from reporting a +// clean host, and one of those is a lie. +func Bucket(family, versionID string) (string, error) { + family = strings.ToLower(strings.TrimSpace(family)) + versionID = strings.TrimSpace(versionID) + + switch { + case family == "debian" || family == "ubuntu": + if versionID == "" { + return "", fmt.Errorf("%s requires a version id", family) + } + return family + " " + versionID, nil + + case family == "alpine": + if versionID == "" { + return "", fmt.Errorf("alpine requires a version id") + } + return "alpine " + majorMinor(versionID), nil + + case rhelRebuilds[family]: + if versionID == "" { + return "", fmt.Errorf("%s requires a version id", family) + } + return "redhat " + major(versionID), nil + + default: + return "", fmt.Errorf("%w: %s", ErrUnsupportedFamily, family) + } +} + +func major(v string) string { + if i := strings.Index(v, "."); i != -1 { + return v[:i] + } + return v +} + +func majorMinor(v string) string { + parts := strings.Split(v, ".") + if len(parts) >= 2 { + return parts[0] + "." + parts[1] + } + return v +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd server && go test ./internal/vulndb/ -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/vulndb/ecosystem.go server/internal/vulndb/ecosystem_test.go +git commit -m "feat: map OS family and version to trivy-db advisory buckets" +``` + +--- + +### Task 3: Agent-side OS release detection + +**Files:** +- Create: `agent/internal/packages/osrelease.go` +- Test: `agent/internal/packages/osrelease_test.go` + +**Interfaces:** +- Consumes: nothing. +- Produces: `packages.OSRelease{Family, VersionID, Arch string}` and `packages.ParseOSRelease(r io.Reader) (OSRelease, error)`. + +- [ ] **Step 1: Write the failing test** + +Create `agent/internal/packages/osrelease_test.go`: + +```go +package packages + +import ( + "strings" + "testing" +) + +const ubuntuOSRelease = `PRETTY_NAME="Ubuntu 22.04.4 LTS" +NAME="Ubuntu" +VERSION_ID="22.04" +VERSION="22.04.4 LTS (Jammy Jellyfish)" +ID=ubuntu +ID_LIKE=debian +HOME_URL="https://www.ubuntu.com/" +` + +const rockyOSRelease = `NAME="Rocky Linux" +VERSION="9.3 (Blue Onyx)" +ID="rocky" +ID_LIKE="rhel centos fedora" +VERSION_ID="9.3" +` + +const alpineOSRelease = `NAME="Alpine Linux" +ID=alpine +VERSION_ID=3.19.1 +PRETTY_NAME="Alpine Linux v3.19" +` + +func TestParseOSRelease(t *testing.T) { + cases := []struct { + name string + input string + family, versionID string + }{ + {"ubuntu", ubuntuOSRelease, "ubuntu", "22.04"}, + {"rocky quoted id", rockyOSRelease, "rocky", "9.3"}, + {"alpine unquoted", alpineOSRelease, "alpine", "3.19.1"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, err := ParseOSRelease(strings.NewReader(tc.input)) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got.Family != tc.family { + t.Errorf("Family = %q, want %q", got.Family, tc.family) + } + if got.VersionID != tc.versionID { + t.Errorf("VersionID = %q, want %q", got.VersionID, tc.versionID) + } + }) + } +} + +func TestParseOSReleaseMissingID(t *testing.T) { + if _, err := ParseOSRelease(strings.NewReader("NAME=\"Something\"\n")); err == nil { + t.Fatal("expected an error when ID is absent, got nil") + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd agent && go test ./internal/packages/ -v` +Expected: build failure — package does not exist. + +- [ ] **Step 3: Write the implementation** + +Create `agent/internal/packages/osrelease.go`: + +```go +package packages + +import ( + "bufio" + "errors" + "io" + "os" + "runtime" + "strings" +) + +// OSRelease identifies the distribution well enough to select an advisory +// feed. VersionID is not optional: Ubuntu 22.04 and 24.04 publish different +// fixed versions for the same CVE. +type OSRelease struct { + Family string + VersionID string + Arch string +} + +// ParseOSRelease reads the os-release format: KEY=value, one per line, with +// values optionally double-quoted, and # comments. +func ParseOSRelease(r io.Reader) (OSRelease, error) { + out := OSRelease{Arch: runtime.GOARCH} + sc := bufio.NewScanner(r) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + key, val, ok := strings.Cut(line, "=") + if !ok { + continue + } + val = strings.Trim(strings.TrimSpace(val), `"'`) + switch strings.TrimSpace(key) { + case "ID": + out.Family = strings.ToLower(val) + case "VERSION_ID": + out.VersionID = val + } + } + if err := sc.Err(); err != nil { + return OSRelease{}, err + } + if out.Family == "" { + return OSRelease{}, errors.New("os-release has no ID") + } + return out, nil +} + +// DetectOS reads /etc/os-release. +func DetectOS() (OSRelease, error) { + f, err := os.Open("/etc/os-release") + if err != nil { + return OSRelease{}, err + } + defer f.Close() + return ParseOSRelease(f) +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd agent && go test ./internal/packages/ -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add agent/internal/packages/osrelease.go agent/internal/packages/osrelease_test.go +git commit -m "feat: agent parses /etc/os-release for distro identification" +``` + +--- + +### Task 4: Agent-side package collection + +**Files:** +- Create: `agent/internal/packages/parse.go`, `agent/internal/packages/packages.go` +- Test: `agent/internal/packages/parse_test.go` + +**Interfaces:** +- Consumes: `packages.OSRelease` from Task 3. +- Produces: `packages.Package{Name, Version, Arch, SourceName string; Epoch int}`, `packages.Collect() (OSRelease, []Package, error)`, `packages.Hash([]Package) string`. + +**Why `SourceName` matters:** Debian and Ubuntu advisories are keyed on the **source** package. A CVE against `openssl` covers the binaries `libssl3`, `openssl` and `libssl-dev`; matching on binary name alone misses two of the three. + +- [ ] **Step 1: Write the failing test** + +Create `agent/internal/packages/parse_test.go`: + +```go +package packages + +import "testing" + +// Captured from: dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${source:Package}\n' +const dpkgOutput = "libssl3\t3.0.2-0ubuntu1.15\tamd64\topenssl\n" + + "openssl\t3.0.2-0ubuntu1.15\tamd64\topenssl\n" + + "bash\t5.1-6ubuntu1\tamd64\tbash\n" + +// Captured from: rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n' +const rpmOutput = "openssl-libs\t1\t3.0.7-24.el9\tx86_64\topenssl-3.0.7-24.el9.src.rpm\n" + + "bash\t(none)\t5.1.8-6.el9\tx86_64\tbash-5.1.8-6.el9.src.rpm\n" + +// Captured from: apk info -v +const apkOutput = "musl-1.2.4_git20230717-r4\nopenssl-3.1.4-r5\nbusybox-1.36.1-r15\n" + +func TestParseDpkg(t *testing.T) { + got := ParseDpkg(dpkgOutput) + if len(got) != 3 { + t.Fatalf("got %d packages, want 3", len(got)) + } + if got[0].Name != "libssl3" || got[0].Version != "3.0.2-0ubuntu1.15" { + t.Errorf("first package = %+v", got[0]) + } + // The whole point of collecting the source name. + if got[0].SourceName != "openssl" { + t.Errorf("SourceName = %q, want %q", got[0].SourceName, "openssl") + } + if got[2].SourceName != "bash" { + t.Errorf("SourceName = %q, want %q", got[2].SourceName, "bash") + } +} + +func TestParseRPM(t *testing.T) { + got := ParseRPM(rpmOutput) + if len(got) != 2 { + t.Fatalf("got %d packages, want 2", len(got)) + } + if got[0].Name != "openssl-libs" || got[0].Epoch != 1 { + t.Errorf("first package = %+v", got[0]) + } + if got[0].SourceName != "openssl" { + t.Errorf("SourceName = %q, want %q", got[0].SourceName, "openssl") + } + // "(none)" is rpm's way of saying no epoch, and must become 0, not fail. + if got[1].Epoch != 0 { + t.Errorf("Epoch = %d, want 0", got[1].Epoch) + } +} + +func TestParseAPK(t *testing.T) { + got := ParseAPK(apkOutput) + if len(got) != 3 { + t.Fatalf("got %d packages, want 3", len(got)) + } + if got[1].Name != "openssl" || got[1].Version != "3.1.4-r5" { + t.Errorf("second package = %+v", got[1]) + } + // A name containing digits and underscores must not be split early. + if got[0].Name != "musl" || got[0].Version != "1.2.4_git20230717-r4" { + t.Errorf("first package = %+v", got[0]) + } +} + +func TestHashIsOrderIndependent(t *testing.T) { + a := []Package{{Name: "b", Version: "2"}, {Name: "a", Version: "1"}} + b := []Package{{Name: "a", Version: "1"}, {Name: "b", Version: "2"}} + if Hash(a) != Hash(b) { + t.Fatal("Hash must not depend on input ordering") + } +} + +func TestHashChangesWithVersion(t *testing.T) { + a := []Package{{Name: "a", Version: "1"}} + b := []Package{{Name: "a", Version: "2"}} + if Hash(a) == Hash(b) { + t.Fatal("Hash must change when a version changes") + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd agent && go test ./internal/packages/ -run "TestParse|TestHash" -v` +Expected: FAIL — `undefined: ParseDpkg`. + +- [ ] **Step 3: Write the implementation** + +Create `agent/internal/packages/parse.go`: + +```go +package packages + +import ( + "crypto/sha256" + "encoding/hex" + "sort" + "strconv" + "strings" +) + +// Package is one installed package as the distribution reports it. Version is +// the distribution's own version string, verbatim — never normalised, because +// the advisory feeds are keyed on exactly this form. +type Package struct { + Name string + Version string + Epoch int + Arch string + SourceName string +} + +// ParseDpkg reads tab-separated output of +// dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${source:Package}\n' +func ParseDpkg(out string) []Package { + var pkgs []Package + for _, line := range strings.Split(out, "\n") { + if strings.TrimSpace(line) == "" { + continue + } + f := strings.Split(line, "\t") + if len(f) < 3 { + continue + } + p := Package{Name: f[0], Version: f[1], Arch: f[2]} + if len(f) > 3 && f[3] != "" { + p.SourceName = f[3] + } else { + p.SourceName = p.Name + } + pkgs = append(pkgs, p) + } + return pkgs +} + +// ParseRPM reads tab-separated output of +// rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n' +func ParseRPM(out string) []Package { + var pkgs []Package + for _, line := range strings.Split(out, "\n") { + if strings.TrimSpace(line) == "" { + continue + } + f := strings.Split(line, "\t") + if len(f) < 4 { + continue + } + epoch := 0 + if f[1] != "" && f[1] != "(none)" { + if n, err := strconv.Atoi(f[1]); err == nil { + epoch = n + } + } + p := Package{Name: f[0], Epoch: epoch, Version: f[2], Arch: f[3]} + if len(f) > 4 { + p.SourceName = srcRPMName(f[4]) + } + if p.SourceName == "" { + p.SourceName = p.Name + } + pkgs = append(pkgs, p) + } + return pkgs +} + +// srcRPMName reduces "openssl-3.0.7-24.el9.src.rpm" to "openssl" by dropping +// the trailing ".src.rpm" and then the version and release segments, which are +// the last two hyphen-separated fields. +func srcRPMName(s string) string { + s = strings.TrimSuffix(s, ".src.rpm") + parts := strings.Split(s, "-") + if len(parts) <= 2 { + return s + } + return strings.Join(parts[:len(parts)-2], "-") +} + +// ParseAPK reads "apk info -v" output: one "name-version-rREV" per line. Alpine +// has no separate source package, so SourceName mirrors Name. +func ParseAPK(out string) []Package { + var pkgs []Package + for _, line := range strings.Split(out, "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + name, version := splitAPK(line) + if name == "" { + continue + } + pkgs = append(pkgs, Package{Name: name, Version: version, SourceName: name}) + } + return pkgs +} + +// splitAPK finds the version boundary from the right. The version is always the +// last two hyphen-separated fields ("-r"), which is reliable where +// scanning from the left is not: names legitimately contain digits, so "musl" in +// "musl-1.2.4_git20230717-r4" cannot be found by looking for the first digit. +func splitAPK(s string) (name, version string) { + last := strings.LastIndex(s, "-") + if last <= 0 { + return "", "" + } + prev := strings.LastIndex(s[:last], "-") + if prev <= 0 { + return "", "" + } + return s[:prev], s[prev+1:] +} + +// Hash fingerprints a package set so an unchanged set never has to be sent. +// It sorts first: the ordering of dpkg or rpm output is not guaranteed stable, +// and an ordering-sensitive hash would resend the full list every hour. +func Hash(pkgs []Package) string { + lines := make([]string, 0, len(pkgs)) + for _, p := range pkgs { + lines = append(lines, p.Name+"\x00"+strconv.Itoa(p.Epoch)+"\x00"+p.Version+"\x00"+p.Arch) + } + sort.Strings(lines) + h := sha256.New() + for _, l := range lines { + h.Write([]byte(l)) + h.Write([]byte("\n")) + } + return hex.EncodeToString(h.Sum(nil)) +} +``` + +Create `agent/internal/packages/packages.go`: + +```go +package packages + +import ( + "context" + "fmt" + "os/exec" + "runtime" + "time" +) + +const collectTimeout = 2 * time.Minute + +// Collect enumerates installed packages. Linux only: Windows agents are +// second-class by design and vulnerability scanning needs a different source, +// a different collector and a different matcher, all of which are out of scope. +func Collect() (OSRelease, []Package, error) { + if runtime.GOOS != "linux" { + return OSRelease{}, nil, fmt.Errorf("package collection is linux-only, got %s", runtime.GOOS) + } + + os, err := DetectOS() + if err != nil { + return OSRelease{}, nil, fmt.Errorf("detect os: %w", err) + } + + ctx, cancel := context.WithTimeout(context.Background(), collectTimeout) + defer cancel() + + switch { + case have("dpkg-query"): + out, err := run(ctx, "dpkg-query", "-W", "-f", + `${Package}\t${Version}\t${Architecture}\t${source:Package}\n`) + if err != nil { + return os, nil, err + } + return os, ParseDpkg(out), nil + + case have("rpm"): + out, err := run(ctx, "rpm", "-qa", "--qf", + `%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n`) + if err != nil { + return os, nil, err + } + return os, ParseRPM(out), nil + + case have("apk"): + out, err := run(ctx, "apk", "info", "-v") + if err != nil { + return os, nil, err + } + return os, ParseAPK(out), nil + + default: + return os, nil, fmt.Errorf("no supported package manager found") + } +} + +func have(bin string) bool { + _, err := exec.LookPath(bin) + return err == nil +} + +func run(ctx context.Context, name string, args ...string) (string, error) { + out, err := exec.CommandContext(ctx, name, args...).Output() + if err != nil { + return "", fmt.Errorf("%s: %w", name, err) + } + return string(out), nil +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd agent && go test ./internal/packages/ -v` +Expected: PASS, all tests. + +- [ ] **Step 5: Commit** + +```bash +git add agent/internal/packages/ +git commit -m "feat: agent collects installed packages per package manager" +``` + +--- + +### Task 5: Models and indexes + +**Files:** +- Create: `server/internal/models/packages.go`, `server/internal/models/vuln.go` +- Create: `server/internal/services/vulnindexes.go` +- Modify: `server/cmd/main.go` (schema setup, beside the other `Ensure*Indexes` calls) + +**Interfaces:** +- Consumes: nothing. +- Produces: `models.ServerPackages`, `models.InstalledPackage`, `models.OSRelease`, `models.VulnFinding`, `models.Acceptance`, `models.VulnDBMeta`, `models.VulnAlertRule`, and `services.EnsureVulnIndexes() error`. + +- [ ] **Step 1: Write the models** + +Create `server/internal/models/packages.go`: + +```go +package models + +import ( + "time" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// Scan status values for ServerPackages. +const ( + ScanStatusOK = "ok" + ScanStatusUnsupported = "unsupported" +) + +type OSRelease struct { + Family string `bson:"family" json:"family"` + VersionID string `bson:"version_id" json:"version_id"` + Arch string `bson:"arch" json:"arch"` +} + +type InstalledPackage struct { + Name string `bson:"name" json:"name"` + Version string `bson:"version" json:"version"` + Epoch int `bson:"epoch,omitempty" json:"epoch,omitempty"` + Arch string `bson:"arch" json:"arch"` + SourceName string `bson:"source_name,omitempty" json:"source_name,omitempty"` +} + +// ServerPackages holds one server's whole package set in ONE document rather +// than one document per package. The hash has already established that +// something changed, so a report is a single atomic upsert with no delta logic +// to get wrong. A typical Linux host is ~2000 packages and ~150KB, comfortably +// inside the 16MB document limit. +type ServerPackages struct { + ID bson.ObjectID `bson:"_id,omitempty" json:"-"` + InstanceID string `bson:"instance_id" json:"-"` + ServerID string `bson:"server_id" json:"server_id"` + OS OSRelease `bson:"os" json:"os"` + Hash string `bson:"hash" json:"hash"` + Packages []InstalledPackage `bson:"packages" json:"packages"` + CollectedAt time.Time `bson:"collected_at" json:"collected_at"` + ScanPending bool `bson:"scan_pending" json:"scan_pending"` + ScannedAt time.Time `bson:"scanned_at,omitempty" json:"scanned_at,omitempty"` + Status string `bson:"status" json:"status"` + DBVersion int `bson:"db_version" json:"db_version"` +} +``` + +Create `server/internal/models/vuln.go`: + +```go +package models + +import ( + "time" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// Finding states. +const ( + FindingOpen = "open" + FindingFixed = "fixed" + FindingAccepted = "accepted" +) + +// Severities, lowest to highest. SeverityRank orders them. +const ( + SeverityUnknown = "unknown" + SeverityLow = "low" + SeverityMedium = "medium" + SeverityHigh = "high" + SeverityCritical = "critical" +) + +func SeverityRank(s string) int { + switch s { + case SeverityCritical: + return 4 + case SeverityHigh: + return 3 + case SeverityMedium: + return 2 + case SeverityLow: + return 1 + default: + return 0 + } +} + +// Acceptance records a decision someone will be asked to justify, so who, +// why and until when all live on the document as well as in the audit log. +type Acceptance struct { + By string `bson:"by" json:"by"` + Reason string `bson:"reason" json:"reason"` + Until time.Time `bson:"until" json:"until"` + At time.Time `bson:"at" json:"at"` +} + +// VulnFinding is one vulnerable package on one server. +// +// Findings are never deleted when a package is patched: the state moves to +// "fixed" with FixedAt stamped, which is what keeps "what did we remediate +// last quarter" answerable. +type VulnFinding struct { + ID bson.ObjectID `bson:"_id,omitempty" json:"id"` + InstanceID string `bson:"instance_id" json:"-"` + ServerID string `bson:"server_id" json:"server_id"` + + CVEID string `bson:"cve_id" json:"cve_id"` + PackageName string `bson:"package_name" json:"package_name"` + Installed string `bson:"installed_version" json:"installed_version"` + // FixedIn empty means no vendor fix has been published. That is a real and + // common state and must never be conflated with "not vulnerable" — it is + // the finding most in need of acceptance, since there is nothing to patch. + FixedIn string `bson:"fixed_in,omitempty" json:"fixed_in,omitempty"` + Severity string `bson:"severity" json:"severity"` + CVSSScore float64 `bson:"cvss_score,omitempty" json:"cvss_score,omitempty"` + Title string `bson:"title,omitempty" json:"title,omitempty"` + References []string `bson:"references,omitempty" json:"references,omitempty"` + + State string `bson:"state" json:"state"` + FirstSeen time.Time `bson:"first_seen" json:"first_seen"` + LastSeen time.Time `bson:"last_seen" json:"last_seen"` + FixedAt *time.Time `bson:"fixed_at,omitempty" json:"fixed_at,omitempty"` + Accepted *Acceptance `bson:"accepted,omitempty" json:"accepted,omitempty"` +} + +// VulnDBMeta is a singleton and deliberately carries no instance_id: the +// vulnerability database is a property of the deployment, not of a tenant. +// Same reasoning as the migrations collection. +type VulnDBMeta struct { + ID bson.ObjectID `bson:"_id,omitempty" json:"-"` + DBVersion int `bson:"db_version" json:"db_version"` + PulledAt time.Time `bson:"pulled_at" json:"pulled_at"` + LastFullScanAt time.Time `bson:"last_full_scan_at,omitempty" json:"last_full_scan_at,omitempty"` + LastError string `bson:"last_error,omitempty" json:"last_error,omitempty"` +} + +type VulnAlertRule struct { + ID bson.ObjectID `bson:"_id,omitempty" json:"id"` + InstanceID string `bson:"instance_id" json:"-"` + Name string `bson:"name" json:"name"` + Enabled bool `bson:"enabled" json:"enabled"` + MinSeverity string `bson:"min_severity" json:"min_severity"` + Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"` + ChannelIDs []string `bson:"channel_ids" json:"channel_ids"` + CreatedAt time.Time `bson:"created_at" json:"created_at"` + UpdatedAt time.Time `bson:"updated_at" json:"updated_at"` +} +``` + +- [ ] **Step 2: Write the index builder** + +Create `server/internal/services/vulnindexes.go`: + +```go +package services + +import ( + "context" + "log" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/mongo" + "go.mongodb.org/mongo-driver/v2/mongo/options" +) + +// EnsureVulnIndexes declares the indexes for package inventory and findings. +// +// It warns rather than being fatal, matching EnsureSecretIndexes and +// EnsureWorkflowIndexes: a missing index degrades these queries to a collection +// scan, which is no reason to refuse to serve the fleet. +func EnsureVulnIndexes() error { + ctx := context.Background() + + pkgIdx := []mongo.IndexModel{ + { + Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "server_id", Value: 1}}, + Options: options.Index().SetUnique(true), + }, + // Multikey, for fleet-wide package search: "who runs openssl 3.0.2?" + {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "packages.name", Value: 1}}}, + {Keys: bson.D{{Key: "scan_pending", Value: 1}}}, + } + if _, err := db.Col("server_packages").Indexes().CreateMany(ctx, pkgIdx); err != nil { + log.Printf("warning: server_packages indexes: %v", err) + } + + findingIdx := []mongo.IndexModel{ + { + // This key is what makes a rescan an idempotent upsert rather than + // a duplicate factory, and what lets first_seen survive a rescan. + Keys: bson.D{ + {Key: "instance_id", Value: 1}, + {Key: "server_id", Value: 1}, + {Key: "cve_id", Value: 1}, + {Key: "package_name", Value: 1}, + }, + Options: options.Index().SetUnique(true), + }, + {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "state", Value: 1}, {Key: "severity", Value: 1}}}, + {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "cve_id", Value: 1}}}, + } + if _, err := db.Col("vuln_findings").Indexes().CreateMany(ctx, findingIdx); err != nil { + log.Printf("warning: vuln_findings indexes: %v", err) + } + + if _, err := db.Col("vuln_alert_rules").Indexes().CreateOne(ctx, mongo.IndexModel{ + Keys: bson.D{{Key: "instance_id", Value: 1}}, + }); err != nil { + log.Printf("warning: vuln_alert_rules indexes: %v", err) + } + + return nil +} +``` + +- [ ] **Step 3: Wire the index builder into schema setup** + +In `server/cmd/main.go`, find `runSchemaSetup` (near line 75) and the block that calls the other `Ensure*Indexes` functions. Add, alongside the non-fatal ones: + +```go + if err := services.EnsureVulnIndexes(); err != nil { + log.Printf("warning: vuln indexes: %v", err) + } +``` + +- [ ] **Step 4: Verify it builds** + +Run: `cd server && go build ./... && go vet ./...` +Expected: no output, exit 0. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/models/packages.go server/internal/models/vuln.go server/internal/services/vulnindexes.go server/cmd/main.go +git commit -m "feat: models and indexes for package inventory and CVE findings" +``` + +--- + +### Task 6: Proto and the hash handshake + +**Files:** +- Modify: `proto/vantage/v1/vantage.proto` +- Regenerate: `agent/internal/grpc/pb/`, `server/internal/grpc/pb/` + +**Interfaces:** +- Consumes: nothing. +- Produces: `pb.ReportPackagesRequest`, `pb.ReportPackagesResponse{NeedFull bool}`, `pb.InstalledPackage`, `pb.OSRelease`, and `SyncResponse.CollectPackages bool`. + +- [ ] **Step 1: Add the RPC and messages** + +In `proto/vantage/v1/vantage.proto`, add to the `Vantage` service block: + +```protobuf + rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse); +``` + +And add these messages at the end of the file: + +```protobuf +// ReportPackages carries a server's installed package set. +// +// The agent calls twice at most. The first call sends only the hash; if the +// server already holds that hash it answers need_full = false and the ~150KB +// body is never sent. A machine's package set changes rarely, so almost every +// hour costs one small message. +message ReportPackagesRequest { + string server_id = 1; + string agent_token = 2; + string hash = 3; + OSRelease os = 4; + repeated InstalledPackage packages = 5; // empty on the offer call +} + +message ReportPackagesResponse { + bool need_full = 1; +} + +message OSRelease { + string family = 1; + string version_id = 2; + string arch = 3; +} + +message InstalledPackage { + string name = 1; + string version = 2; + int32 epoch = 3; + string arch = 4; + string source_name = 5; +} +``` + +- [ ] **Step 2: Add the collect flag to SyncResponse** + +Find `message SyncResponse` in the same file and add a field using the next free number in that message (check the file — do not reuse a number): + +```protobuf + // collect_packages tells the agent whether this instance's licence grants + // vulnerability scanning. False means do not collect at all: no gRPC body, + // no document, no storage. The server re-checks on ReportPackages — this + // flag is the optimisation, the server check is the boundary. + bool collect_packages = ; +``` + +- [ ] **Step 3: Regenerate** + +Run the project's existing protoc generation command. If unsure, find it with: + +```bash +grep -rn "protoc" --include=Makefile --include="*.sh" --include="*.yml" . | head +``` + +Generated output must land in both `agent/internal/grpc/pb/` and `server/internal/grpc/pb/`. + +- [ ] **Step 4: Verify both modules build** + +Run: `cd agent && go build ./... && cd ../server && go build ./...` +Expected: exit 0 both times. + +- [ ] **Step 5: Commit** + +```bash +git add proto/ agent/internal/grpc/pb/ server/internal/grpc/pb/ +git commit -m "feat: ReportPackages RPC with hash short-circuit" +``` + +--- + +### Task 7: Agent reports packages + +**Files:** +- Modify: `agent/internal/sync/sync.go:370-411` (`runUpdateCheck`) +- Modify: `agent/internal/grpc/client.go` — add the client method + +**Interfaces:** +- Consumes: `packages.Collect`, `packages.Hash` (Task 4); `pb.ReportPackagesRequest` (Task 6). +- Produces: `grpcclient.Client.ReportPackages(req *pb.ReportPackagesRequest) (bool, error)` returning `need_full`. + +- [ ] **Step 1: Add the client method** + +In `agent/internal/grpc/client.go`, following the shape of the existing `ReportUpdates`: + +```go +// ReportPackages sends a package report and returns whether the server wants +// the full list. +func (c *Client) ReportPackages(req *pb.ReportPackagesRequest) (bool, error) { + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + resp, err := c.client.ReportPackages(ctx, req) + if err != nil { + return false, err + } + return resp.GetNeedFull(), nil +} +``` + +Match the surrounding file's timeout and context conventions if they differ. + +- [ ] **Step 2: Extend the hourly loop** + +In `agent/internal/sync/sync.go`, inside `runUpdateCheck`'s `doCheck` closure, after the existing `client.ReportUpdates(...)` call succeeds, add: + +```go + reportPackages(client, cfg) +``` + +Then add this function to the same file: + +```go +// reportPackages offers a hash of the installed package set and sends the full +// list only if the server does not already hold it. It runs on the same hourly +// cadence as the update check because a package set changes on roughly the same +// schedule, and reusing the loop means one timer rather than two. +func reportPackages(client *grpcclient.Client, cfg *config.Config) { + if runtime.GOOS != "linux" { + return + } + if !collectPackagesEnabled() { + return + } + + osrel, pkgs, err := packages.Collect() + if err != nil { + log.Printf("package collection error: %v", err) + return + } + + pbOS := &pb.OSRelease{ + Family: osrel.Family, + VersionId: osrel.VersionID, + Arch: osrel.Arch, + } + hash := packages.Hash(pkgs) + + needFull, err := client.ReportPackages(&pb.ReportPackagesRequest{ + ServerId: cfg.ServerID, + AgentToken: cfg.AgentToken, + Hash: hash, + Os: pbOS, + }) + if err != nil { + log.Printf("ReportPackages offer error: %v", err) + return + } + if !needFull { + return + } + + pbPkgs := make([]*pb.InstalledPackage, 0, len(pkgs)) + for _, p := range pkgs { + pbPkgs = append(pbPkgs, &pb.InstalledPackage{ + Name: p.Name, + Version: p.Version, + Epoch: int32(p.Epoch), + Arch: p.Arch, + SourceName: p.SourceName, + }) + } + + if _, err := client.ReportPackages(&pb.ReportPackagesRequest{ + ServerId: cfg.ServerID, + AgentToken: cfg.AgentToken, + Hash: hash, + Os: pbOS, + Packages: pbPkgs, + }); err != nil { + log.Printf("ReportPackages full error: %v", err) + return + } + log.Printf("reported %d installed packages", len(pkgs)) +} +``` + +- [ ] **Step 3: Store the collect flag from SyncKeys** + +`SyncResponse.collect_packages` arrives on the 30s key poll. Add a package-level flag to `agent/internal/sync/sync.go`, guarded because the poll loop and the hourly loop are different goroutines: + +```go +var collectPackagesFlag atomic.Bool + +func collectPackagesEnabled() bool { return collectPackagesFlag.Load() } +``` + +In `poll()` (around line 94), after a successful `SyncKeys` response, add: + +```go + collectPackagesFlag.Store(resp.GetCollectPackages()) +``` + +Add `"sync/atomic"`, `"runtime"` and the `packages` import to the file's import block. + +- [ ] **Step 4: Verify it builds and vets** + +Run: `cd agent && go build ./... && go vet ./... && go test ./...` +Expected: exit 0; the Task 3 and 4 tests still pass. + +- [ ] **Step 5: Commit** + +```bash +git add agent/internal/sync/sync.go agent/internal/grpc/client.go +git commit -m "feat: agent reports installed packages on the hourly loop" +``` + +--- + +### Task 8: Server stores package reports + +**Files:** +- Create: `server/internal/services/packages.go` +- Modify: `server/internal/grpc/server.go` — add the `ReportPackages` handler + +**Interfaces:** +- Consumes: `models.ServerPackages` (Task 5), `pb.ReportPackagesRequest` (Task 6). +- Produces: `services.HasPackageHash(instanceID, serverID, hash string) (bool, error)`, `services.StorePackages(instanceID, serverID string, os models.OSRelease, hash string, pkgs []models.InstalledPackage) error`, `services.ListPackages(instanceID, serverID string) (*models.ServerPackages, error)`, `services.SearchPackages(instanceID, name string) ([]PackageHit, error)`. + +**Critical:** `StorePackages` must **not** match. It upserts and sets `scan_pending: true`. Matching inline would require every replica to hold the 50MB database and would have N replicas racing on a database refresh. + +- [ ] **Step 1: Write the service** + +Create `server/internal/services/packages.go`: + +```go +package services + +import ( + "context" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/mongo" + "go.mongodb.org/mongo-driver/v2/mongo/options" +) + +// HasPackageHash reports whether we already hold this exact package set, which +// is what lets the agent skip sending ~150KB it has already sent. +func HasPackageHash(instanceID, serverID, hash string) (bool, error) { + err := db.Col("server_packages").FindOne(context.Background(), bson.M{ + "instance_id": instanceID, + "server_id": serverID, + "hash": hash, + }, options.FindOne().SetProjection(bson.M{"_id": 1})).Err() + if err == mongo.ErrNoDocuments { + return false, nil + } + return err == nil, err +} + +// StorePackages replaces a server's package set and marks it for scanning. +// +// It deliberately does NOT match against the vulnerability database. Matching +// happens in vulnsched, on the leader, for two reasons: every replica would +// otherwise need the ~50MB database resident, and a database refresh would have +// N replicas racing to rescan the same fleet and sending N digests. +func StorePackages(instanceID, serverID string, os models.OSRelease, hash string, pkgs []models.InstalledPackage) error { + now := time.Now() + _, err := db.Col("server_packages").UpdateOne(context.Background(), + bson.M{"instance_id": instanceID, "server_id": serverID}, + bson.M{"$set": bson.M{ + "os": os, + "hash": hash, + "packages": pkgs, + "collected_at": now, + "scan_pending": true, + }}, + options.UpdateOne().SetUpsert(true), + ) + return err +} + +func ListPackages(instanceID, serverID string) (*models.ServerPackages, error) { + var sp models.ServerPackages + err := db.Col("server_packages").FindOne(context.Background(), bson.M{ + "instance_id": instanceID, + "server_id": serverID, + }).Decode(&sp) + if err == mongo.ErrNoDocuments { + return nil, nil + } + if err != nil { + return nil, err + } + return &sp, nil +} + +type PackageHit struct { + ServerID string `json:"server_id"` + Name string `json:"name"` + Version string `json:"version"` +} + +// SearchPackages answers "which servers run package X" across the fleet — the +// question people actually ask during an incident. +func SearchPackages(instanceID, name string) ([]PackageHit, error) { + ctx := context.Background() + cur, err := db.Col("server_packages").Find(ctx, bson.M{ + "instance_id": instanceID, + "packages.name": name, + }, options.Find().SetProjection(bson.M{"server_id": 1, "packages": 1})) + if err != nil { + return nil, err + } + defer cur.Close(ctx) + + var docs []models.ServerPackages + if err := cur.All(ctx, &docs); err != nil { + return nil, err + } + + hits := []PackageHit{} + for _, d := range docs { + for _, p := range d.Packages { + if p.Name == name { + hits = append(hits, PackageHit{ServerID: d.ServerID, Name: p.Name, Version: p.Version}) + } + } + } + return hits, nil +} +``` + +- [ ] **Step 2: Add the gRPC handler** + +In `server/internal/grpc/server.go`, following the shape of the existing `ReportUpdates` (near line 87): + +```go +func (s *vantageServer) ReportPackages(ctx context.Context, req *pb.ReportPackagesRequest) (*pb.ReportPackagesResponse, error) { + srv, err := services.ValidateAgentToken(req.GetServerId(), req.GetAgentToken()) + if err != nil { + return nil, status.Error(codes.Unauthenticated, "invalid agent credentials") + } + + // The agent flag is an optimisation; this check is the boundary. + if !services.VulnScanningEnabled(srv.InstanceID) { + return &pb.ReportPackagesResponse{NeedFull: false}, nil + } + + // The offer call: no packages, just a hash. + if len(req.GetPackages()) == 0 { + known, err := services.HasPackageHash(srv.InstanceID, req.GetServerId(), req.GetHash()) + if err != nil { + return nil, status.Error(codes.Internal, "package hash lookup failed") + } + return &pb.ReportPackagesResponse{NeedFull: !known}, nil + } + + pkgs := make([]models.InstalledPackage, 0, len(req.GetPackages())) + for _, p := range req.GetPackages() { + pkgs = append(pkgs, models.InstalledPackage{ + Name: p.GetName(), + Version: p.GetVersion(), + Epoch: int(p.GetEpoch()), + Arch: p.GetArch(), + SourceName: p.GetSourceName(), + }) + } + + os := models.OSRelease{ + Family: req.GetOs().GetFamily(), + VersionID: req.GetOs().GetVersionId(), + Arch: req.GetOs().GetArch(), + } + + if err := services.StorePackages(srv.InstanceID, req.GetServerId(), os, req.GetHash(), pkgs); err != nil { + return nil, status.Error(codes.Internal, "failed to store packages") + } + return &pb.ReportPackagesResponse{NeedFull: false}, nil +} +``` + +Match the exact return type and error conventions of the neighbouring handlers — in particular how `ValidateAgentToken` is called and what it returns in this file. + +- [ ] **Step 3: Add the licence gate helper** + +Create it in `server/internal/services/packages.go`: + +```go +// VulnScanningEnabled reports whether this instance's licence grants +// vulnerability scanning. It reads the feature by name and never switches on +// tier, so changing what a tier includes needs no server release. +func VulnScanningEnabled(instanceID string) bool { + lic, err := GetLicense(instanceID) + if err != nil || lic == nil { + return false + } + return lic.HasFeature("vuln_scanning") +} +``` + +Find the existing licence accessor in `server/internal/services/` (the one `RequireActiveLicense` in `server/internal/api/licence.go` uses) and call **that** rather than inventing `GetLicense`. Adjust the name and signature to match what is actually there. + +- [ ] **Step 4: Verify it builds** + +Run: `cd server && go build ./... && go vet ./...` +Expected: exit 0. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/services/packages.go server/internal/grpc/server.go +git commit -m "feat: store agent package reports and mark them for scanning" +``` + +--- + +### Task 9: Serve the collect flag from SyncKeys + +**Files:** +- Modify: `server/internal/grpc/server.go` — `SyncKeys` (near line 47) + +**Interfaces:** +- Consumes: `services.VulnScanningEnabled` (Task 8). +- Produces: `SyncResponse.CollectPackages` populated on every poll. + +- [ ] **Step 1: Set the flag** + +In `SyncKeys`, where the `SyncResponse` is constructed, add: + +```go + resp.CollectPackages = services.VulnScanningEnabled(srv.InstanceID) +``` + +using whatever the response variable is actually called in that function. + +- [ ] **Step 2: Verify it builds** + +Run: `cd server && go build ./...` +Expected: exit 0. + +- [ ] **Step 3: Commit** + +```bash +git add server/internal/grpc/server.go +git commit -m "feat: tell agents whether to collect packages via SyncKeys" +``` + +--- + +### Task 10: trivy-db puller and advisory lookup + +**Files:** +- Create: `server/internal/vulndb/pull.go`, `server/internal/vulndb/db.go` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `vulndb.Pull(ctx context.Context, dir string) (version int, err error)`, `vulndb.Open(dir string) (*Store, error)`, `(*Store).Advisories(bucket, srcName string) ([]Advisory, error)`, `(*Store).Vulnerability(cveID string) (VulnInfo, error)`, `(*Store).Close() error`. + +**No test for this task.** It is I/O against an external registry and a real BoltDB file; the global constraints forbid network in tests. Task 11 tests the matching logic behind a `Store` interface instead. + +- [ ] **Step 1: Write the puller** + +Create `server/internal/vulndb/pull.go`: + +```go +package vulndb + +import ( + "archive/tar" + "compress/gzip" + "context" + "fmt" + "io" + "os" + "path/filepath" + "strings" +) + +// DefaultRef is the published trivy-db OCI artifact, rebuilt every six hours. +const DefaultRef = "ghcr.io/aquasecurity/trivy-db:2" + +// Ref returns the artifact reference, honouring VANTAGE_TRIVY_DB_REF so an +// air-gapped deployment can mirror the artifact into its own registry, and so +// a busy deployment can avoid the anonymous ghcr rate limit. +func Ref() string { + if v := os.Getenv("VANTAGE_TRIVY_DB_REF"); v != "" { + return v + } + return DefaultRef +} + +// Disabled reports whether the puller and scheduler are switched off entirely. +// Findings already written are still served, and still marked stale. +func Disabled() bool { + return strings.EqualFold(os.Getenv("VANTAGE_VULNDB_DISABLED"), "true") +} + +// Pull fetches the trivy-db artifact into dir and returns the schema version +// recorded in its metadata. +// +// Implementation notes for the engineer: +// - Use oras.land/oras-go/v2 with a remote repository for Ref(). +// - The artifact has a single layer, a gzipped tar containing "db/trivy.db" +// and "db/metadata.json". +// - Extract both into dir, writing to a temp path and renaming into place, so +// a failed pull cannot leave a half-written database that Open would accept. +// - Read metadata.json for "Version" (an int) and return it. +// - REFUSE an unknown schema version rather than mis-parsing it: if the +// version is not one this code was written against, return an error. +func Pull(ctx context.Context, dir string) (int, error) { + // See the notes above. The extraction helper below is the part worth + // getting right; the oras fetch is mechanical. + return 0, fmt.Errorf("not implemented") +} + +// SupportedSchema is the trivy-db schema version this code understands. +const SupportedSchema = 2 + +// extractTarGz writes the artifact layer into dir. Paths are cleaned and +// checked so a crafted archive cannot write outside dir. +func extractTarGz(r io.Reader, dir string) error { + gz, err := gzip.NewReader(r) + if err != nil { + return err + } + defer gz.Close() + + tr := tar.NewReader(gz) + for { + hdr, err := tr.Next() + if err == io.EOF { + return nil + } + if err != nil { + return err + } + if hdr.Typeflag != tar.TypeReg { + continue + } + name := filepath.Base(hdr.Name) // flatten; the archive is two files + if name == "." || name == ".." || name == "" { + continue + } + dst := filepath.Join(dir, name) + if !strings.HasPrefix(dst, filepath.Clean(dir)+string(os.PathSeparator)) { + return fmt.Errorf("archive entry escapes destination: %q", hdr.Name) + } + f, err := os.Create(dst) + if err != nil { + return err + } + if _, err := io.Copy(f, tr); err != nil { + f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + } +} +``` + +Then replace the `Pull` body with the real oras implementation, following the notes in its doc comment. Verify by running it once by hand against the real registry before moving on. + +- [ ] **Step 2: Write the advisory store** + +Create `server/internal/vulndb/db.go`: + +```go +package vulndb + +import ( + trivydb "github.com/aquasecurity/trivy-db/pkg/db" +) + +// Advisory is one fixed-version statement for one source package. +type Advisory struct { + CVEID string + // FixedVersion empty means no vendor fix has been published. It is a real + // state, not an absence of data, and callers must treat it as vulnerable. + FixedVersion string + Severity string +} + +// VulnInfo is the CVE's own metadata, shared across every server it affects. +type VulnInfo struct { + Title string + Severity string + CVSSScore float64 + References []string +} + +// Store reads a pulled trivy-db. +type Store struct { + cfg trivydb.Config +} + +// Open opens the database in dir read-only. +func Open(dir string) (*Store, error) { + if err := trivydb.Init(dir); err != nil { + return nil, err + } + return &Store{cfg: trivydb.Config{}}, nil +} + +func (s *Store) Close() error { return trivydb.Close() } + +// Advisories returns every advisory for a source package in a bucket. +func (s *Store) Advisories(bucket, srcName string) ([]Advisory, error) { + raw, err := s.cfg.GetAdvisories(bucket, srcName) + if err != nil { + return nil, err + } + out := make([]Advisory, 0, len(raw)) + for _, a := range raw { + out = append(out, Advisory{ + CVEID: a.VulnerabilityID, + FixedVersion: a.FixedVersion, + Severity: severityName(int(a.Severity)), + }) + } + return out, nil +} + +// Vulnerability returns a CVE's shared metadata. +func (s *Store) Vulnerability(cveID string) (VulnInfo, error) { + v, err := s.cfg.GetVulnerability(cveID) + if err != nil { + return VulnInfo{}, err + } + return VulnInfo{ + Title: v.Title, + Severity: severityName(int(v.Severity)), + References: v.References, + }, nil +} + +// severityName maps trivy-db's integer severity onto our lowercase strings. +func severityName(n int) string { + switch n { + case 4: + return "critical" + case 3: + return "high" + case 2: + return "medium" + case 1: + return "low" + default: + return "unknown" + } +} +``` + +Verify the `trivy-db` API names against the version `go get` actually resolved — `GetAdvisories`, `GetVulnerability`, `Init` and `Close` are stable but their exact signatures may differ. Adjust the wrappers, not the `Advisory`/`VulnInfo` shapes, which later tasks depend on. + +- [ ] **Step 3: Verify it builds** + +Run: `cd server && go build ./... && go test ./internal/vulndb/ -v` +Expected: build succeeds; Task 1 and 2 tests still pass. + +- [ ] **Step 4: Commit** + +```bash +git add server/internal/vulndb/pull.go server/internal/vulndb/db.go +git commit -m "feat: pull trivy-db and read its advisories" +``` + +--- + +### Task 11: The matcher + +**Files:** +- Create: `server/internal/vulndb/match.go` +- Test: `server/internal/vulndb/match_test.go` + +**Interfaces:** +- Consumes: `vulndb.LessThan` (Task 1), `vulndb.Bucket` (Task 2), `vulndb.Advisory` (Task 10), `models.InstalledPackage` (Task 5). +- Produces: `vulndb.AdvisorySource` (an interface), `vulndb.Match(src AdvisorySource, os models.OSRelease, pkgs []models.InstalledPackage) ([]Result, error)`, `vulndb.Result`. + +The interface exists so this task's tests need no real database file — the constraint that tests touch no network and no database is what forces the seam, and the seam is worth having anyway. + +- [ ] **Step 1: Write the failing test** + +Create `server/internal/vulndb/match_test.go`: + +```go +package vulndb + +import ( + "errors" + "testing" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +// fakeSource is an in-memory AdvisorySource keyed by "bucket\x00srcName". +type fakeSource map[string][]Advisory + +func (f fakeSource) Advisories(bucket, srcName string) ([]Advisory, error) { + return f[bucket+"\x00"+srcName], nil +} + +func ubuntu() models.OSRelease { + return models.OSRelease{Family: "ubuntu", VersionID: "22.04", Arch: "amd64"} +} + +func TestMatchBackportedFixIsNotVulnerable(t *testing.T) { + // The trap this whole feature is built around: the installed version equals + // the fixed version, so the host is patched even though NVD lists the + // upstream 3.0.2 as vulnerable. + src := fakeSource{ + "ubuntu 22.04\x00openssl": { + {CVEID: "CVE-2023-0286", FixedVersion: "1:3.0.2-0ubuntu1.15", Severity: "high"}, + }, + } + pkgs := []models.InstalledPackage{ + {Name: "libssl3", Version: "1:3.0.2-0ubuntu1.15", SourceName: "openssl"}, + } + + got, err := Match(src, ubuntu(), pkgs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 0 { + t.Fatalf("got %d results, want 0 — a backported fix must not be reported vulnerable: %+v", len(got), got) + } +} + +func TestMatchOlderRevisionIsVulnerable(t *testing.T) { + src := fakeSource{ + "ubuntu 22.04\x00openssl": { + {CVEID: "CVE-2023-0286", FixedVersion: "1:3.0.2-0ubuntu1.15", Severity: "high"}, + }, + } + pkgs := []models.InstalledPackage{ + {Name: "libssl3", Version: "1:3.0.2-0ubuntu1.14", SourceName: "openssl"}, + } + + got, err := Match(src, ubuntu(), pkgs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 1 { + t.Fatalf("got %d results, want 1", len(got)) + } + if got[0].CVEID != "CVE-2023-0286" { + t.Errorf("CVEID = %q", got[0].CVEID) + } + // The finding must name the BINARY package, which is what an operator sees + // installed, while the lookup used the source package. + if got[0].PackageName != "libssl3" { + t.Errorf("PackageName = %q, want libssl3", got[0].PackageName) + } + if got[0].FixedIn != "1:3.0.2-0ubuntu1.15" { + t.Errorf("FixedIn = %q", got[0].FixedIn) + } +} + +func TestMatchSourceLookupCoversEveryBinary(t *testing.T) { + // One advisory against the source package openssl must flag all three of + // its binaries. Matching on binary name alone would find only one. + src := fakeSource{ + "ubuntu 22.04\x00openssl": { + {CVEID: "CVE-2023-0286", FixedVersion: "1:3.0.2-0ubuntu1.15", Severity: "high"}, + }, + } + pkgs := []models.InstalledPackage{ + {Name: "libssl3", Version: "1:3.0.2-0ubuntu1.14", SourceName: "openssl"}, + {Name: "openssl", Version: "1:3.0.2-0ubuntu1.14", SourceName: "openssl"}, + {Name: "libssl-dev", Version: "1:3.0.2-0ubuntu1.14", SourceName: "openssl"}, + } + + got, err := Match(src, ubuntu(), pkgs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 3 { + t.Fatalf("got %d results, want 3 — one per affected binary", len(got)) + } +} + +func TestMatchNoFixedVersionIsVulnerable(t *testing.T) { + // An advisory with no fix published is the finding people most need to see. + src := fakeSource{ + "ubuntu 22.04\x00bash": { + {CVEID: "CVE-2024-0001", FixedVersion: "", Severity: "medium"}, + }, + } + pkgs := []models.InstalledPackage{ + {Name: "bash", Version: "5.1-6ubuntu1", SourceName: "bash"}, + } + + got, err := Match(src, ubuntu(), pkgs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 1 { + t.Fatalf("got %d results, want 1", len(got)) + } + if got[0].FixedIn != "" { + t.Errorf("FixedIn = %q, want empty", got[0].FixedIn) + } +} + +func TestMatchUnsupportedFamily(t *testing.T) { + _, err := Match(fakeSource{}, models.OSRelease{Family: "arch"}, nil) + if !errors.Is(err, ErrUnsupportedFamily) { + t.Fatalf("error = %v, want ErrUnsupportedFamily", err) + } +} + +func TestMatchFallsBackToPackageNameWhenSourceMissing(t *testing.T) { + // Alpine has no separate source package; SourceName mirrors Name. A report + // from an older agent may omit it entirely, and that must still match. + src := fakeSource{ + "alpine 3.19\x00openssl": { + {CVEID: "CVE-2024-0002", FixedVersion: "3.1.4-r6", Severity: "high"}, + }, + } + pkgs := []models.InstalledPackage{{Name: "openssl", Version: "3.1.4-r5"}} + + got, err := Match(src, models.OSRelease{Family: "alpine", VersionID: "3.19.1"}, pkgs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 1 { + t.Fatalf("got %d results, want 1", len(got)) + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd server && go test ./internal/vulndb/ -run TestMatch -v` +Expected: FAIL — `undefined: Match`. + +- [ ] **Step 3: Write the implementation** + +Create `server/internal/vulndb/match.go`: + +```go +package vulndb + +import ( + "fmt" + "log" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" +) + +// AdvisorySource is the advisory lookup the matcher needs. *Store satisfies it; +// tests substitute an in-memory map, which is what keeps the matching logic +// testable with no database file and no network. +type AdvisorySource interface { + Advisories(bucket, srcName string) ([]Advisory, error) +} + +// Result is one vulnerable package on one server, before it becomes a finding. +type Result struct { + CVEID string + PackageName string // the BINARY package, which is what is installed + Installed string + FixedIn string + Severity string +} + +// Match returns every advisory that the installed packages do not satisfy. +// +// Vulnerable means: no fix has been published, or the installed version sorts +// strictly before the fixed version under the distribution's own ordering. +func Match(src AdvisorySource, os models.OSRelease, pkgs []models.InstalledPackage) ([]Result, error) { + bucket, err := Bucket(os.Family, os.VersionID) + if err != nil { + return nil, err + } + + var out []Result + for _, p := range pkgs { + // Debian and Ubuntu advisories are keyed on the source package: one + // advisory against "openssl" covers libssl3, openssl and libssl-dev. + srcName := p.SourceName + if srcName == "" { + srcName = p.Name + } + + advs, err := src.Advisories(bucket, srcName) + if err != nil { + return nil, fmt.Errorf("advisories for %s: %w", srcName, err) + } + + for _, a := range advs { + // No published fix. Vulnerable, and the finding most in need of + // acceptance, since there is nothing to patch. + if a.FixedVersion == "" { + out = append(out, Result{ + CVEID: a.CVEID, PackageName: p.Name, + Installed: p.Version, Severity: a.Severity, + }) + continue + } + + older, err := LessThan(os.Family, p.Version, a.FixedVersion) + if err != nil { + // Skip this one advisory rather than failing the whole server: + // one unparseable version must not blind us to every other CVE + // on the host. Log it — a silent skip is a silent false + // negative, which is the direction that hurts. + log.Printf("vulndb: compare %s %s vs %s: %v", p.Name, p.Version, a.FixedVersion, err) + continue + } + if older { + out = append(out, Result{ + CVEID: a.CVEID, PackageName: p.Name, + Installed: p.Version, FixedIn: a.FixedVersion, Severity: a.Severity, + }) + } + } + } + return out, nil +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd server && go test ./internal/vulndb/ -v` +Expected: PASS, every test. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/vulndb/match.go server/internal/vulndb/match_test.go +git commit -m "feat: match installed packages against distro advisories" +``` + +--- + +### Task 12: Finding state machine + +**Files:** +- Create: `server/internal/services/findings.go` +- Test: `server/internal/services/findings_test.go` + +**Interfaces:** +- Consumes: `vulndb.Result` (Task 11), `models.VulnFinding` (Task 5). +- Produces: `services.DiffFindings(existing []models.VulnFinding, results []vulndb.Result, now time.Time) FindingDiff`, `services.FindingDiff{Upserts []models.VulnFinding, FixedIDs []bson.ObjectID, ReopenIDs []bson.ObjectID, NewlyOpened []models.VulnFinding}`. + +`DiffFindings` is a **pure function** — no database — which is what makes the state machine testable. The Mongo writes live in a separate thin `ApplyFindingDiff`. + +- [ ] **Step 1: Write the failing test** + +Create `server/internal/services/findings_test.go`: + +```go +package services + +import ( + "testing" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb" + "go.mongodb.org/mongo-driver/v2/bson" +) + +func now() time.Time { return time.Date(2026, 8, 6, 12, 0, 0, 0, time.UTC) } + +func TestDiffOpensNewFinding(t *testing.T) { + d := DiffFindings(nil, []vulndb.Result{ + {CVEID: "CVE-1", PackageName: "openssl", Installed: "1.0", FixedIn: "1.1", Severity: "high"}, + }, now()) + + if len(d.Upserts) != 1 { + t.Fatalf("got %d upserts, want 1", len(d.Upserts)) + } + if d.Upserts[0].State != models.FindingOpen { + t.Errorf("State = %q, want open", d.Upserts[0].State) + } + if len(d.NewlyOpened) != 1 { + t.Errorf("got %d newly opened, want 1 — this is what the digest reports", len(d.NewlyOpened)) + } +} + +func TestDiffPreservesFirstSeen(t *testing.T) { + // Easy to break with an upsert that overwrites, and nothing notices until a + // report claims everything was discovered yesterday. + first := now().Add(-30 * 24 * time.Hour) + existing := []models.VulnFinding{{ + ID: bson.NewObjectID(), CVEID: "CVE-1", PackageName: "openssl", + State: models.FindingOpen, FirstSeen: first, LastSeen: first, + }} + + d := DiffFindings(existing, []vulndb.Result{ + {CVEID: "CVE-1", PackageName: "openssl", Installed: "1.0", FixedIn: "1.1", Severity: "high"}, + }, now()) + + if len(d.Upserts) != 1 { + t.Fatalf("got %d upserts, want 1", len(d.Upserts)) + } + if !d.Upserts[0].FirstSeen.Equal(first) { + t.Errorf("FirstSeen = %v, want %v", d.Upserts[0].FirstSeen, first) + } + if !d.Upserts[0].LastSeen.Equal(now()) { + t.Errorf("LastSeen = %v, want %v", d.Upserts[0].LastSeen, now()) + } + // Already open, so not newly opened — it must not re-alert every scan. + if len(d.NewlyOpened) != 0 { + t.Errorf("got %d newly opened, want 0", len(d.NewlyOpened)) + } +} + +func TestDiffMarksAbsentFindingFixed(t *testing.T) { + id := bson.NewObjectID() + existing := []models.VulnFinding{{ + ID: id, CVEID: "CVE-1", PackageName: "openssl", State: models.FindingOpen, + }} + + d := DiffFindings(existing, nil, now()) + + if len(d.FixedIDs) != 1 || d.FixedIDs[0] != id { + t.Fatalf("FixedIDs = %v, want [%v]", d.FixedIDs, id) + } + if len(d.Upserts) != 0 { + t.Errorf("got %d upserts, want 0", len(d.Upserts)) + } +} + +func TestDiffReopensExpiredAcceptance(t *testing.T) { + id := bson.NewObjectID() + expired := now().Add(-24 * time.Hour) + existing := []models.VulnFinding{{ + ID: id, CVEID: "CVE-1", PackageName: "openssl", State: models.FindingAccepted, + Accepted: &models.Acceptance{Reason: "reboot window", Until: expired}, + }} + + d := DiffFindings(existing, []vulndb.Result{ + {CVEID: "CVE-1", PackageName: "openssl", Installed: "1.0", FixedIn: "1.1", Severity: "high"}, + }, now()) + + if len(d.ReopenIDs) != 1 || d.ReopenIDs[0] != id { + t.Fatalf("ReopenIDs = %v, want [%v]", d.ReopenIDs, id) + } +} + +func TestDiffLeavesLiveAcceptanceAlone(t *testing.T) { + existing := []models.VulnFinding{{ + ID: bson.NewObjectID(), CVEID: "CVE-1", PackageName: "openssl", State: models.FindingAccepted, + Accepted: &models.Acceptance{Reason: "no fix yet", Until: now().Add(24 * time.Hour)}, + }} + + d := DiffFindings(existing, []vulndb.Result{ + {CVEID: "CVE-1", PackageName: "openssl", Installed: "1.0", Severity: "high"}, + }, now()) + + if len(d.ReopenIDs) != 0 { + t.Errorf("ReopenIDs = %v, want empty", d.ReopenIDs) + } + // An accepted finding must not appear in the digest. + if len(d.NewlyOpened) != 0 { + t.Errorf("got %d newly opened, want 0", len(d.NewlyOpened)) + } +} + +func TestDiffAbsentAndExpiredSettlesFixed(t *testing.T) { + // Ordering matters: a finding that is BOTH absent from the scan and past its + // acceptance expiry must settle as fixed, not reopen on a package that no + // longer carries it. + id := bson.NewObjectID() + existing := []models.VulnFinding{{ + ID: id, CVEID: "CVE-1", PackageName: "openssl", State: models.FindingAccepted, + Accepted: &models.Acceptance{Until: now().Add(-24 * time.Hour)}, + }} + + d := DiffFindings(existing, nil, now()) + + if len(d.FixedIDs) != 1 { + t.Fatalf("FixedIDs = %v, want one entry", d.FixedIDs) + } + if len(d.ReopenIDs) != 0 { + t.Errorf("ReopenIDs = %v, want empty", d.ReopenIDs) + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd server && go test ./internal/services/ -run TestDiff -v` +Expected: FAIL — `undefined: DiffFindings`. + +- [ ] **Step 3: Write the implementation** + +Create `server/internal/services/findings.go`: + +```go +package services + +import ( + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb" + "go.mongodb.org/mongo-driver/v2/bson" +) + +// FindingDiff is what one server's scan changes. +type FindingDiff struct { + Upserts []models.VulnFinding + FixedIDs []bson.ObjectID + ReopenIDs []bson.ObjectID + // NewlyOpened is what the digest reports: findings that were not open + // before this scan. A finding that was already open must not re-alert every + // tick, or the digest becomes noise and stops being read. + NewlyOpened []models.VulnFinding +} + +func findingKey(cveID, pkg string) string { return cveID + "\x00" + pkg } + +// DiffFindings computes the state changes for one server's scan. +// +// Pure by design: no database, no clock of its own. That is what makes the +// state machine — the part with the subtle ordering — testable. +func DiffFindings(existing []models.VulnFinding, results []vulndb.Result, now time.Time) FindingDiff { + var d FindingDiff + + byKey := make(map[string]models.VulnFinding, len(existing)) + for _, f := range existing { + byKey[findingKey(f.CVEID, f.PackageName)] = f + } + + seen := make(map[string]bool, len(results)) + for _, r := range results { + key := findingKey(r.CVEID, r.PackageName) + seen[key] = true + + prev, had := byKey[key] + + f := models.VulnFinding{ + CVEID: r.CVEID, + PackageName: r.PackageName, + Installed: r.Installed, + FixedIn: r.FixedIn, + Severity: r.Severity, + State: models.FindingOpen, + FirstSeen: now, + LastSeen: now, + } + + if had { + f.ID = prev.ID + f.FirstSeen = prev.FirstSeen + + // A live acceptance survives the scan untouched: it is suppressed + // from counts and alerts until its expiry, then reopens on its own. + if prev.State == models.FindingAccepted && prev.Accepted != nil { + if now.Before(prev.Accepted.Until) { + continue + } + d.ReopenIDs = append(d.ReopenIDs, prev.ID) + continue + } + + if prev.State != models.FindingOpen { + d.NewlyOpened = append(d.NewlyOpened, f) + } + } else { + d.NewlyOpened = append(d.NewlyOpened, f) + } + + d.Upserts = append(d.Upserts, f) + } + + // Anything we hold that this scan did not produce is fixed. This runs after + // the loop above, so a finding that is both absent and past its acceptance + // expiry settles as fixed rather than reopening on a package that no longer + // carries it. + for _, f := range existing { + if seen[findingKey(f.CVEID, f.PackageName)] { + continue + } + if f.State == models.FindingFixed { + continue + } + d.FixedIDs = append(d.FixedIDs, f.ID) + } + + return d +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd server && go test ./internal/services/ -v` +Expected: PASS, every test. + +- [ ] **Step 5: Add the persistence half** + +Append to `server/internal/services/findings.go`: + +```go +// ApplyFindingDiff writes a diff. Thin on purpose — the logic worth testing is +// all in DiffFindings. +func ApplyFindingDiff(ctx context.Context, instanceID, serverID string, d FindingDiff, now time.Time) error { + col := db.Col("vuln_findings") + + for _, f := range d.Upserts { + f.InstanceID = instanceID + f.ServerID = serverID + _, err := col.UpdateOne(ctx, + bson.M{ + "instance_id": instanceID, + "server_id": serverID, + "cve_id": f.CVEID, + "package_name": f.PackageName, + }, + bson.M{ + "$set": bson.M{ + "installed_version": f.Installed, + "fixed_in": f.FixedIn, + "severity": f.Severity, + "state": models.FindingOpen, + "last_seen": now, + }, + // first_seen is only written when the document is created, so a + // rescan cannot move it forward. + "$setOnInsert": bson.M{ + "instance_id": instanceID, + "server_id": serverID, + "cve_id": f.CVEID, + "package_name": f.PackageName, + "first_seen": f.FirstSeen, + }, + "$unset": bson.M{"fixed_at": "", "accepted": ""}, + }, + options.UpdateOne().SetUpsert(true), + ) + if err != nil { + return err + } + } + + if len(d.FixedIDs) > 0 { + if _, err := col.UpdateMany(ctx, + bson.M{"_id": bson.M{"$in": d.FixedIDs}}, + bson.M{"$set": bson.M{"state": models.FindingFixed, "fixed_at": now}}, + ); err != nil { + return err + } + } + + if len(d.ReopenIDs) > 0 { + if _, err := col.UpdateMany(ctx, + bson.M{"_id": bson.M{"$in": d.ReopenIDs}}, + bson.M{"$set": bson.M{"state": models.FindingOpen, "last_seen": now}, "$unset": bson.M{"accepted": ""}}, + ); err != nil { + return err + } + } + + return nil +} +``` + +Add `"context"`, the `db` import and `"go.mongodb.org/mongo-driver/v2/mongo/options"` to the file's imports. + +- [ ] **Step 6: Verify and commit** + +Run: `cd server && go build ./... && go test ./... -v` + +```bash +git add server/internal/services/findings.go server/internal/services/findings_test.go +git commit -m "feat: finding state machine with acceptance expiry" +``` + +--- + +### Task 13: The scheduler + +**Files:** +- Create: `server/internal/vulnsched/sched.go` +- Modify: `server/cmd/main.go:184-206` + +**Interfaces:** +- Consumes: `vulndb.Pull`, `vulndb.Open` (Task 10), `vulndb.Match` (Task 11), `services.DiffFindings`, `services.ApplyFindingDiff` (Task 12). +- Produces: `vulnsched.Start(ctx context.Context, deps Deps)`, `vulnsched.Deps{LogEvent func(...), SendDigest func(instanceID string, newly []models.VulnFinding)}`. + +**Model this file on `server/internal/workflowsched/sched.go`.** Read it first — same `Start`/`tick` shape, same `Deps` injection, same "return when the context is cancelled" contract. + +- [ ] **Step 1: Write the scheduler** + +Create `server/internal/vulnsched/sched.go`: + +```go +// Package vulnsched owns the vulnerability scan loop. +// +// It runs inside bus.RunAsLeader("housekeeping", …) alongside monitorsched, +// workflowsched and the sweepers: one role, one lock. N replicas each running +// this loop would mean N copies of the ~50MB database resident, N rescans of +// the same fleet on every database refresh, and N digests reaching the +// customer for one set of findings. +package vulnsched + +import ( + "context" + "log" + "os" + "time" + + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" + "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb" + "go.mongodb.org/mongo-driver/v2/bson" +) + +const ( + tickInterval = 60 * time.Second + // trivy-db is rebuilt every six hours; pulling more often buys nothing. + dbMaxAge = 6 * time.Hour +) + +// Deps are injected from main.go rather than imported, following +// workflowsched. It keeps this package's reach explicit and reviewable. +type Deps struct { + LogEvent func(instanceID, eventType, actor, serverID, keyID, details string) + SendDigest func(instanceID string, newly []models.VulnFinding) +} + +type scheduler struct { + deps Deps + dir string + store *vulndb.Store + version int + pulled time.Time +} + +func Start(ctx context.Context, deps Deps) { + if vulndb.Disabled() { + log.Println("vulnsched: disabled by VANTAGE_VULNDB_DISABLED") + return + } + + dir, err := os.MkdirTemp("", "vantage-vulndb-") + if err != nil { + log.Printf("vulnsched: temp dir: %v", err) + return + } + + s := &scheduler{deps: deps, dir: dir} + + go func() { + defer os.RemoveAll(dir) + defer s.closeStore() + + ticker := time.NewTicker(tickInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + s.tick(ctx) + } + } + }() +} + +func (s *scheduler) tick(ctx context.Context) { + if err := s.ensureDB(ctx); err != nil { + // Keep the last good database and carry on scanning against it. A + // network blip must never clear findings or read as "all fixed". + log.Printf("vulnsched: database unavailable: %v", err) + recordDBError(ctx, err) + if s.store == nil { + return + } + } + s.scanPending(ctx) +} + +// ensureDB pulls a fresh database when the local copy is stale, and marks the +// whole fleet for rescanning when the version changes — which is what makes a +// newly published CVE flag existing servers within a minute rather than at the +// next agent report. +func (s *scheduler) ensureDB(ctx context.Context) error { + if s.store != nil && time.Since(s.pulled) < dbMaxAge { + return nil + } + + version, err := vulndb.Pull(ctx, s.dir) + if err != nil { + return err + } + + s.closeStore() + store, err := vulndb.Open(s.dir) + if err != nil { + return err + } + s.store = store + s.pulled = time.Now() + + changed := version != s.version + s.version = version + + _, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{}, + bson.M{"$set": bson.M{"db_version": version, "pulled_at": s.pulled}, "$unset": bson.M{"last_error": ""}}, + mongoUpsert(), + ) + + if changed { + res, err := db.Col("server_packages").UpdateMany(ctx, + bson.M{"status": bson.M{"$ne": models.ScanStatusUnsupported}}, + bson.M{"$set": bson.M{"scan_pending": true}}, + ) + if err != nil { + log.Printf("vulnsched: mark fleet pending: %v", err) + } else { + log.Printf("vulnsched: database version %d, %d servers marked for rescan", version, res.ModifiedCount) + } + } + return nil +} + +func (s *scheduler) scanPending(ctx context.Context) { + cur, err := db.Col("server_packages").Find(ctx, bson.M{"scan_pending": true}) + if err != nil { + log.Printf("vulnsched: find pending: %v", err) + return + } + defer cur.Close(ctx) + + var pending []models.ServerPackages + if err := cur.All(ctx, &pending); err != nil { + log.Printf("vulnsched: decode pending: %v", err) + return + } + + // Newly opened findings are collected across the whole tick and sent as one + // digest per instance. A database refresh can open several hundred findings + // at once; one message per finding would rate-limit the webhook or get the + // channel muted, and either way the alerts stop being read. + newly := map[string][]models.VulnFinding{} + + for _, sp := range pending { + if ctx.Err() != nil { + return // leadership lost; scan_pending is still set, so the next leader picks it up + } + opened := s.scanOne(ctx, sp) + newly[sp.InstanceID] = append(newly[sp.InstanceID], opened...) + } + + for instanceID, findings := range newly { + if len(findings) > 0 && s.deps.SendDigest != nil { + s.deps.SendDigest(instanceID, findings) + } + } +} + +func (s *scheduler) scanOne(ctx context.Context, sp models.ServerPackages) []models.VulnFinding { + now := time.Now() + + results, err := vulndb.Match(s.store, sp.OS, sp.Packages) + if err != nil { + // We hold no feed for this distribution, so we cannot answer whether it + // is vulnerable. Say "unsupported" — reporting zero findings here would + // be indistinguishable from reporting a clean host, and one of those is + // a lie. + status := models.ScanStatusUnsupported + if !isUnsupported(err) { + log.Printf("vulnsched: scan %s: %v", sp.ServerID, err) + status = sp.Status + } + clearPending(ctx, sp.ID, status, s.version, now) + return nil + } + + existing, err := services.ListFindings(ctx, sp.InstanceID, sp.ServerID) + if err != nil { + log.Printf("vulnsched: list findings %s: %v", sp.ServerID, err) + return nil + } + + diff := services.DiffFindings(existing, results, now) + if err := services.ApplyFindingDiff(ctx, sp.InstanceID, sp.ServerID, diff, now); err != nil { + log.Printf("vulnsched: apply diff %s: %v", sp.ServerID, err) + return nil + } + + clearPending(ctx, sp.ID, models.ScanStatusOK, s.version, now) + + for i := range diff.NewlyOpened { + diff.NewlyOpened[i].ServerID = sp.ServerID + } + return diff.NewlyOpened +} + +func (s *scheduler) closeStore() { + if s.store != nil { + _ = s.store.Close() + s.store = nil + } +} +``` + +Then add the three small helpers the file uses — `clearPending`, `recordDBError`, `isUnsupported` (which wraps `errors.Is(err, vulndb.ErrUnsupportedFamily)`) and `mongoUpsert` (returning `options.UpdateOne().SetUpsert(true)`) — in the same file, and add `services.ListFindings(ctx, instanceID, serverID) ([]models.VulnFinding, error)` to `server/internal/services/findings.go`. + +- [ ] **Step 2: Wire it into main.go** + +In `server/cmd/main.go`, inside the `bus.RunAsLeader(ctx, "housekeeping", …)` block (line 184), after the `workflowsched.Start(...)` call: + +```go + vulnsched.Start(jobCtx, vulnsched.Deps{ + LogEvent: services.LogEvent, + SendDigest: services.SendVulnDigest, + }) + services.StartVulnSweeper(jobCtx) +``` + +Add the `vulnsched` import. `services.SendVulnDigest` and `services.StartVulnSweeper` arrive in Tasks 14 and 15 — if implementing strictly in order, stub them as no-ops in `server/internal/services/vulnrules.go` now and fill them in there. + +- [ ] **Step 3: Verify it builds** + +Run: `cd server && go build ./... && go vet ./... && go test ./...` +Expected: exit 0. + +- [ ] **Step 4: Commit** + +```bash +git add server/internal/vulnsched/ server/cmd/main.go server/internal/services/findings.go +git commit -m "feat: leader-owned vulnerability scan loop" +``` + +--- + +### Task 14: Alert rules and the digest + +**Files:** +- Create: `server/internal/services/vulnrules.go`, `shared/mail/vuln.go` +- Create: `shared/mail/templates/vuln_digest.html.tmpl`, `shared/mail/templates/vuln_digest.txt.tmpl` +- Create: `shared/mail/render_test.go` + +**Interfaces:** +- Consumes: `models.VulnAlertRule`, `models.VulnFinding` (Task 5), `services.ResolveTargets`. +- Produces: `services.SendVulnDigest(instanceID string, newly []models.VulnFinding)`, `services.ListVulnRules/CreateVulnRule/UpdateVulnRule/DeleteVulnRule`, `mail.Sender.SendVulnDigest(to string, data VulnDigestData) error`. + +**The tag filter must go through `services.ResolveTargets`.** It is already the single answer to which servers a selector touches; a rule that disagreed with a workflow about what `env:prod` means would be worse than no filter. + +- [ ] **Step 1: Write the rule service and dispatch** + +Create `server/internal/services/vulnrules.go` with CRUD for `vuln_alert_rules` following the shape of the existing notification-channel service, plus: + +```go +// SendVulnDigest delivers one message per rule per tick — never one per +// finding. See vulnsched for why the batch boundary is the tick. +func SendVulnDigest(instanceID string, newly []models.VulnFinding) { + rules, err := ListVulnRules(instanceID) + if err != nil { + log.Printf("vuln digest: list rules: %v", err) + return + } + + for _, rule := range rules { + if !rule.Enabled { + continue + } + + matched := filterBySeverity(newly, rule.MinSeverity) + if len(matched) == 0 { + continue + } + + if len(rule.Tags) > 0 { + allowed, err := ResolveTargets(instanceID, nil, rule.Tags) + if err != nil { + log.Printf("vuln digest: resolve targets: %v", err) + continue + } + matched = filterByServers(matched, allowed) + if len(matched) == 0 { + continue + } + } + + for _, chID := range rule.ChannelIDs { + dispatchVulnDigest(instanceID, chID, rule.Name, matched) + } + } +} + +func filterBySeverity(findings []models.VulnFinding, min string) []models.VulnFinding { + floor := models.SeverityRank(min) + out := make([]models.VulnFinding, 0, len(findings)) + for _, f := range findings { + if models.SeverityRank(f.Severity) >= floor { + out = append(out, f) + } + } + return out +} + +func filterByServers(findings []models.VulnFinding, allowed []string) []models.VulnFinding { + set := make(map[string]bool, len(allowed)) + for _, id := range allowed { + set[id] = true + } + out := make([]models.VulnFinding, 0, len(findings)) + for _, f := range findings { + if set[f.ServerID] { + out = append(out, f) + } + } + return out +} +``` + +Implement `dispatchVulnDigest` by following how `server/internal/notify` already dispatches a monitor alert to a channel by ID, with a summary line of the form `"12 new critical, 4 new high across 6 servers"`. Verify `ResolveTargets`' actual signature in `server/internal/services/targets.go` and adjust the call. + +- [ ] **Step 2: Write the mail templates** + +Create `shared/mail/templates/vuln_digest.txt.tmpl`. **`subject` is defined in the txt file only** — `html/template` would escape an ampersand in an instance name, and mail clients show subjects verbatim: + +``` +{{define "subject"}}{{.Count}} new {{if eq .Count 1}}vulnerability{{else}}vulnerabilities{{end}} on {{.InstanceName}}{{end}} +{{define "title"}}New vulnerabilities detected{{end}} +{{define "pill"}}{{.TopSeverity}}{{end}} +{{define "body"}} +{{template "lead" .Summary}} + +{{range .Rows}}- {{.CVEID}} ({{.Severity}}) — {{.PackageName}} on {{.ServerName}}{{if .FixedIn}}, fixed in {{.FixedIn}}{{else}}, no fix published{{end}} +{{end}} +{{if .More}}...and {{.More}} more.{{end}} + +Scanned against vulnerability database pulled {{.DBAge}} ago. +{{end}} +``` + +Create `shared/mail/templates/vuln_digest.html.tmpl` defining `title`, `pill` and `body` only, composing the existing `p`, `lead`, `button`, `rows` and `chip` helpers from `layout.html.tmpl`. Read an existing pair — `monitor.go`'s templates — and match them exactly. **Carry no new hex colours:** every colour in the email system lives in `layout.html.tmpl`. + +Create `shared/mail/vuln.go` with `VulnDigestData` and `SendVulnDigest`, following `shared/mail/monitor.go`. + +- [ ] **Step 3: Write the render test CLAUDE.md already claims exists** + +Create `shared/mail/render_test.go`. Templates are parsed in `init()`, so a mistyped field is a boot-time panic; this is the only thing standing between that and production. + +```go +package mail + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// cases must name every template pair under templates/. The final subtest +// fails if a template exists that no case covers. +var cases = map[string]any{ + // One entry per template base name, with a fully populated data value. + // Fill this in from the SendX methods in this package — every template + // that exists must appear here. +} + +func TestEveryTemplateRenders(t *testing.T) { + for name, data := range cases { + t.Run(name, func(t *testing.T) { + if _, _, _, err := render(name, data); err != nil { + t.Fatalf("render %s: %v", name, err) + } + }) + } +} + +func TestNoTemplateIsUncovered(t *testing.T) { + entries, err := os.ReadDir("templates") + if err != nil { + t.Fatalf("read templates dir: %v", err) + } + for _, e := range entries { + name := e.Name() + if !strings.HasSuffix(name, ".txt.tmpl") || strings.HasPrefix(name, "layout") { + continue + } + base := filepath.Base(strings.TrimSuffix(name, ".txt.tmpl")) + if _, ok := cases[base]; !ok { + t.Errorf("template %q has no case in cases — add one", base) + } + } +} +``` + +Adjust `render` to the actual unexported render function's name and signature in `shared/mail/render.go`, and populate `cases` with every existing template plus `vuln_digest`. + +- [ ] **Step 4: Run the tests** + +Run: `cd shared && go test ./mail/ -v` +Expected: PASS. If a template fails to render, fix the template — that failure is the test doing its job. + +- [ ] **Step 5: Commit** + +```bash +git add server/internal/services/vulnrules.go shared/mail/ +git commit -m "feat: vulnerability alert rules and batched digest" +``` + +--- + +### Task 15: REST API, retention setting and sweeper + +**Files:** +- Create: `server/internal/api/vulnerabilities.go` +- Modify: `shared/models/settings.go`, `server/internal/api/handlers.go`, `server/internal/services/findings.go` + +**Interfaces:** +- Consumes: everything from Tasks 8, 12 and 14. +- Produces: the routes below, `models.Settings.VulnFindingRetentionDays *int`, `services.VulnFindingRetentionDays(*Settings) int`, `services.StartVulnSweeper(ctx)`. + +- [ ] **Step 1: Add the retention setting** + +In `shared/models/settings.go`, beside `WorkflowLogRetentionDays`: + +```go + // VulnFindingRetentionDays is a pointer for the same reason + // WorkflowLogRetentionDays is: absent must mean the default, not zero. + // Nil is 90 days, 0 is forever. Only "fixed" findings are ever swept. + VulnFindingRetentionDays *int `bson:"vuln_finding_retention_days,omitempty" json:"vuln_finding_retention_days,omitempty"` +``` + +- [ ] **Step 2: Add the sweeper** + +In `server/internal/services/findings.go`, following `StartAuditSweeper`: + +```go +// StartVulnSweeper deletes old FIXED findings. Open and accepted findings are +// never swept at any setting: retention is about history, and an unresolved +// vulnerability is not history. +func StartVulnSweeper(ctx context.Context) { + go func() { + ticker := time.NewTicker(6 * time.Hour) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + sweepFixedFindings(ctx) + } + } + }() +} +``` + +Implement `sweepFixedFindings` to read each instance's setting, resolve nil to 90 and skip 0, then delete `{instance_id, state: "fixed", fixed_at: {$lt: cutoff}}`. Model it on `StartAuditSweeper` in the same package. + +- [ ] **Step 3: Write the handlers** + +Create `server/internal/api/vulnerabilities.go` with these handlers, following the conventions in `server/internal/api/handlers.go` — `auth.InstanceID(c)` for scoping, `actorFromCtx(c)` for the audit actor: + +- `listVulnerabilities` — `GET /api/vulnerabilities`, query params `severity`, `state`, `server`, plus tag filters. Groups by CVE in the response: `{cve_id, severity, title, server_count, servers: [...]}`. +- `vulnerabilitySummary` — `GET /api/vulnerabilities/summary`, returns severity counts for open findings plus `{db_version, pulled_at, last_error}` from `vulndb_meta`. +- `rescanVulnerabilities` — `POST /api/vulnerabilities/rescan`, sets `scan_pending: true` on every `server_packages` document for the instance. Audit event `vuln.rescan`. +- `acceptFinding` — `POST /api/vulnerabilities/:id/accept`, body `{reason string, until time}`. **Reject an empty reason and a past `until` with 400.** Sets state `accepted` with the `Acceptance` block. Audit event `vuln.accepted` with the reason and expiry in the details. +- `unacceptFinding` — `DELETE /api/vulnerabilities/:id/accept`, back to `open`, audit event `vuln.unaccepted`. +- `listServerVulnerabilities` — `GET /api/servers/:id/vulnerabilities`. +- `getServerPackages` — `GET /api/servers/:id/packages`. +- `searchPackages` — `GET /api/packages/search?name=`, calls `services.SearchPackages`. +- Alert rule CRUD — `GET,POST /api/vuln-rules`, `PUT,DELETE /api/vuln-rules/:id`. + +- [ ] **Step 4: Register the routes** + +In `server/internal/api/handlers.go`, inside the `apiGroup` block: + +```go + apiGroup.GET("/vulnerabilities", listVulnerabilities) + apiGroup.GET("/vulnerabilities/summary", vulnerabilitySummary) + apiGroup.POST("/vulnerabilities/rescan", auth.RequireRole("owner", "admin"), rescanVulnerabilities) + apiGroup.POST("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), acceptFinding) + apiGroup.DELETE("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), unacceptFinding) + apiGroup.GET("/servers/:id/vulnerabilities", listServerVulnerabilities) + apiGroup.GET("/servers/:id/packages", getServerPackages) + apiGroup.GET("/packages/search", searchPackages) + apiGroup.GET("/vuln-rules", listVulnRules) + apiGroup.POST("/vuln-rules", auth.RequireRole("owner", "admin"), createVulnRule) + apiGroup.PUT("/vuln-rules/:id", auth.RequireRole("owner", "admin"), updateVulnRule) + apiGroup.DELETE("/vuln-rules/:id", auth.RequireRole("owner", "admin"), deleteVulnRule) +``` + +Check `auth.RequireRole`'s actual variadic signature before using it with two arguments. + +- [ ] **Step 5: Verify and commit** + +Run: `cd server && go build ./... && go vet ./... && go test ./...` + +```bash +git add server/internal/api/ shared/models/settings.go server/internal/services/findings.go +git commit -m "feat: vulnerability REST API, retention setting and sweeper" +``` + +--- + +### Task 16: Web UI + +**Files:** +- Create: `web/app/(app)/vulnerabilities/page.tsx`, `web/components/vulnerabilities/{FindingRow,AcceptDialog,DBFreshness}.tsx`, `web/components/settings/VulnAlertRulesCard.tsx` +- Modify: `web/lib/api.ts`, `web/components/Sidebar.tsx`, `web/app/(app)/servers/[id]/page.tsx`, `web/app/(app)/settings/notifications/page.tsx` + +**Interfaces:** +- Consumes: the Task 15 routes. +- Produces: no Go interfaces. + +No tests: `web/` has no frontend test infrastructure and adding one is out of scope for this feature. Verify by running the app. + +- [ ] **Step 1: Add API client types and methods** + +In `web/lib/api.ts`, add `VulnFinding`, `VulnGroup`, `VulnSummary`, `ServerPackages`, `VulnAlertRule` types mirroring the Go JSON tags exactly, and one client method per Task 15 route. Follow the file's existing conventions. + +- [ ] **Step 2: Build the fleet board** + +Create `web/app/(app)/vulnerabilities/page.tsx`. **Grouped by CVE, one row per CVE with an affected-server count, expandable to the servers.** The same CVE across 40 servers is one decision; a flat list of findings makes it look like forty. + +Requirements: +- Filter bar: severity, state (default `open`), tag filter reusing `TagFilterBar`'s pattern from the servers page. +- `DBFreshness` banner at the top, always visible, showing `pulled_at` age and `last_error` if set. A fleet scanning against a three-week-old database must say so rather than quietly report all-clear. +- A server whose `status` is `unsupported` renders as **"unsupported"**, never as zero findings. +- State never reads by colour alone: every severity pill carries a distinct shape and a text label, matching the existing monitor pills. +- Findings with no `fixed_in` show "no fix published" rather than an empty cell. +- Use Tailwind token names only. No hex values. + +- [ ] **Step 3: Build the accept dialog** + +Create `web/components/vulnerabilities/AcceptDialog.tsx`: required reason textarea, required `until` date defaulting to 30 days out, and copy stating the finding reopens automatically on that date. + +- [ ] **Step 4: Add the remediation action** + +On a finding with `fixed_in`, render an **Apply updates** button calling the existing `POST /api/servers/:id/apply-updates`. No new endpoint and no new mechanism — see it, patch it, one place. + +- [ ] **Step 5: Add the server-detail tabs and nav** + +Add **Vulnerabilities** and **Packages** tabs to `web/app/(app)/servers/[id]/page.tsx`, following the existing tab pattern in that file. Add a **Vulnerabilities** entry to `web/components/Sidebar.tsx`. Mount `VulnAlertRulesCard` on `web/app/(app)/settings/notifications/page.tsx` beside the channels. + +- [ ] **Step 6: Verify** + +Run: `cd web && npm run lint && npm run build` +Expected: exit 0, no type errors. + +- [ ] **Step 7: Commit** + +```bash +git add web/ +git commit -m "feat: vulnerability findings UI" +``` + +--- + +### Task 17: Instance deletion, entitlement and documentation + +**Files:** +- Modify: the control plane's instance-deletion collection list, `admin/internal/models/entitlements.go`, `CLAUDE.md`, `docsite/docs/vantage/` + +- [ ] **Step 1: Add the new collections to instance deletion** + +Find the instance-deletion routine — it is the one place that knows which collections carry `instance_id`: + +```bash +cd server && grep -rn "instance_id" internal/services/ | grep -i "delete\|purge\|reap" +``` + +Add `server_packages` and `vuln_findings` to its collection list. **Missing this orphans a tenant's package data indefinitely**, and it is the easiest thing in this feature to forget. + +- [ ] **Step 2: Add the admin entitlement toggle** + +In `admin/internal/models/entitlements.go`, add `vuln_scanning` to the feature toggles, following whatever pattern the existing per-instance toggles use. It must be **off for both Free plans**. Confirm the licence issue path copies it into `License.Features`. + +- [ ] **Step 3: Verify the whole tree builds and tests pass** + +Run from the repo root: + +```bash +go build ./... && go vet ./... && go test ./... +``` + +Expected: exit 0. `go test ./...` must pass with no database and no network. + +- [ ] **Step 4: Document it** + +Add a **Package inventory and CVE findings** subsystem section to `CLAUDE.md` covering: the backport trap and why matching uses distro feeds; why only the leader matches; that findings go `fixed` rather than being deleted; the two new environment variables. Add a user-facing page under `docsite/docs/vantage/`. + +While in `CLAUDE.md`, correct the `shared/mail/render_test.go` claim — that file did not exist before Task 14 created it. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "docs: document package inventory and CVE findings" +``` + +--- + +## Self-review notes + +Checked against the spec, section by section: + +- Backport trap → Tasks 1 and 11, with the equal-version case asserted in both. +- trivy-db as the source, ephemeral storage, `VANTAGE_TRIVY_DB_REF` → Task 10. +- Leader-only matching, `scan_pending` persisted, tick-boundary batching → Tasks 8 and 13. +- Hash short-circuit wire path → Tasks 6 and 7. +- All four collections and their indexes → Task 5. +- `SourceName` covering every binary of a source package → Tasks 4 and 11. +- Unsupported distro never reads as clean → Tasks 2, 13 and 16. +- Severity vendor → NVD → unknown → Task 10 (`severityName`) and Task 11. +- Findings lifecycle with acceptance expiry, ordering of fixed-before-reopen → Task 12. +- Alert rules through `ResolveTargets`, one digest per tick → Task 14. +- Entitlement gated at collection, `HasFeature` not tier → Tasks 8, 9 and 17. +- REST API, retention, sweeper → Task 15. +- CVE-grouped board, DB freshness on screen, remediation via the existing endpoint → Task 16. +- Instance-deletion collection list → Task 17. + +**Known gaps the executing engineer must close:** + +- `vulndb.Pull` ships as a stub with implementation notes rather than working oras code (Task 10, Step 1). It is the one piece that cannot be written blind, because the exact `oras-go` v2 call shape depends on the resolved version. Verify it by hand against the real registry before Task 13. +- Several call sites are written against signatures that must be confirmed in the actual files first, and each says so at the point of use: the licence accessor (Task 8), `ValidateAgentToken`'s return shape (Task 8), `ResolveTargets` (Task 14), `auth.RequireRole`'s variadic form (Task 15), the `trivy-db` API names (Task 10), and the unexported render function in `shared/mail` (Task 14). +- The `shared/mail` `cases` map (Task 14, Step 3) must be populated by reading the existing `SendX` methods. It is deliberately left empty in the plan because inventing entries for templates I have not read would produce a test that fails for the wrong reason. diff --git a/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md b/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md index 1b5024a..b8644a6 100644 --- a/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md +++ b/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md @@ -392,16 +392,26 @@ CVE; most fields would be disabled in the UI and the uptime graphs would be polluted with a signal that is not uptime. This adds one `notify` payload type and a `vuln_digest.html.tmpl` / -`vuln_digest.txt.tmpl` pair in `shared/mail`, since `render_test.go` fails on -any template no case covers. +`vuln_digest.txt.tmpl` pair in `shared/mail`. Note that `shared/mail` templates +are parsed in `init()`, so a mistyped field is a boot-time panic — CLAUDE.md +describes a `render_test.go` guarding against exactly this, but **that file does +not exist**; the repository has no Go tests at all today. The implementation +plan adds it alongside this feature's own tests. --- ## Entitlement -`entitlement.features.vuln_scanning`, a boolean alongside the existing -per-instance feature toggles, so it can later be priced as a catalogue -`feature` component without a second migration. Off on Free. +The feature name is `vuln_scanning`, and it crosses the two services the way +every other feature does: + +- **admin** carries it as a per-instance entitlement toggle, so it can later be + priced as a catalogue `feature` component without a second migration; +- **the licence** snapshots it into `License.Features []string` at issue time; +- **the server** asks `lic.HasFeature("vuln_scanning")` and never switches on + tier, so changing what a tier includes needs no server release. + +Off on Free. **The gate is checked at `ReportPackages`, not at display.** Gating only the UI would still pay every write cost, and storage is the expensive half.