Compare commits
48
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6dced22499 | ||
|
|
5cee53dc5f | ||
|
|
81248bb159 | ||
|
|
6354d54de8 | ||
|
|
da6d64f95c | ||
|
|
9ba3d4a61f | ||
|
|
eee236a072 | ||
|
|
9df89e2db4 | ||
|
|
f60c509b47 | ||
|
|
84dfcfeac7 | ||
|
|
5dda3b5c4a | ||
|
|
db64320bd8 | ||
|
|
583f60771c | ||
|
|
a92c3190c2 | ||
|
|
3a6d24fe0e | ||
|
|
c277ecff44 | ||
|
|
bd690c94c3 | ||
|
|
a22fdf197e | ||
|
|
bd24b03cac | ||
|
|
3afc4ab012 | ||
|
|
d1ac3e98ce | ||
|
|
5bba54f3e5 | ||
|
|
fe7bc300e2 | ||
|
|
00c03c365d | ||
|
|
dc8dd3dd58 | ||
|
|
85a8865892 | ||
|
|
50a9ac5fdc | ||
|
|
3388d2f895 | ||
|
|
3a77fc2abd | ||
|
|
3d59836d0c | ||
|
|
d9184312aa | ||
|
|
b9802e6b04 | ||
|
|
c2635ed51a | ||
|
|
b21ac05547 | ||
|
|
484b620867 | ||
|
|
439bc2ed7d | ||
|
|
a1e6986a64 | ||
|
|
d0e1cc4ad6 | ||
|
|
b877024365 | ||
|
|
2de7ac116b | ||
|
|
fa1fd14ed1 | ||
|
|
d1b3cd2f74 | ||
|
|
e00a0da5d9 | ||
|
|
fef0b7c7a1 | ||
|
|
efd29dc259 | ||
|
|
13cd41d202 | ||
|
|
3530ce6cb7 | ||
|
|
09522c2566 |
+138
-1
@@ -124,6 +124,86 @@ A library of reusable **steps** (bash or PowerShell scripts with declared inputs
|
||||
|
||||
Default steps are seeded per org at boot (`SeedDefaultSteps`) from `VANTAGE_DEFAULT_STEPS_DIR`, which `server/Dockerfile` bakes to `/opt/default-steps` from the repo's `default_steps/`. Deliberately **not** under `/data` — that is a bind mount, so the library would be editable from the host. Adding a step there means committing a file and rebuilding, which is why `default_steps/` is in the `server` rebuild trigger. **Steps with `source: "default"` are read-only**: `UpdateStep`/`DeleteStep` refuse with `ErrDefaultStep` (409), because seeding rewrites them on every boot, so an edit would silently revert and a delete would come back. `web/` mirrors this — the step modal opens read-only, Delete is hidden, and the designer's per-step script override is `readOnly` for a default library step — but as elsewhere, the API is the boundary and the UI is the courtesy. Seeding writes straight to the collection rather than through `UpdateStep`, so the guard does not lock out the seeder. Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever).
|
||||
|
||||
### Scheduled workflows
|
||||
|
||||
A workflow may carry `schedule{enabled, cron, tz}` — standard **5-field** cron
|
||||
and an IANA zone name, both validated at save time. `next_run_at` is
|
||||
**persisted on the document, not held in memory**: a leader handover between
|
||||
computing an occurrence and firing it would otherwise lose it or fire it twice,
|
||||
the same argument that put `workflow_log_seq` in MongoDB.
|
||||
|
||||
`server/internal/workflowsched` ticks every 30s inside the **existing**
|
||||
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched` and the sweepers —
|
||||
one role, one lock. **The atomic claim, not the lock, is what prevents a double
|
||||
fire**: the `UpdateOne` matches on the document *and* its current `next_run_at`
|
||||
while setting the recomputed one, so a second process reaching the same workflow
|
||||
matches nothing and does nothing. The lock only makes it cheap.
|
||||
|
||||
`workflowsched` **must not import `services`** — `services` already imports it
|
||||
for `SetSchedule`'s call to `NextOccurrence`, and Go has no cycles.
|
||||
`TriggerWorkflow` and `LogEvent` are therefore injected as `workflowsched.Deps`
|
||||
from `main.go`. Firing goes through the same `TriggerWorkflow` a person uses,
|
||||
with `"schedule"` as the actor, so there is no second dispatch path and the run
|
||||
detail page needed no changes.
|
||||
|
||||
`main.go` imports `_ "time/tzdata"`, and it is load-bearing: `server/Dockerfile`
|
||||
runs on `scratch`, which ships no zone database, so without it
|
||||
`time.LoadLocation("Europe/London")` fails and every schedule silently falls
|
||||
back to UTC — an hour wrong for half the year, in the direction nobody notices
|
||||
until a maintenance window lands in business hours. It works on a developer
|
||||
machine either way, which is exactly why it gets forgotten.
|
||||
|
||||
Skips are recorded and surfaced, not just logged: past the 1h grace window is
|
||||
`missed`, an active run is `already_running`, and a schedule that no longer
|
||||
parses is disabled rather than left spinning the loop every 30 seconds forever.
|
||||
|
||||
### Server tags and workflow targeting
|
||||
|
||||
A server carries `tags map[string]string` — lowercase `[a-z0-9_-]`, key ≤32,
|
||||
value ≤64, 20 per server, `sys:` reserved. **There is no `tags` collection**: a
|
||||
tag is a property of a server, not an entity, so `KnownTags` aggregates over
|
||||
`servers` rather than reading a registry that would need reference counting to
|
||||
know when a tag stopped existing. `PUT /api/servers/:id/tags` replaces the whole
|
||||
map — last-write-wins over a small map beats merge semantics between two people
|
||||
editing one server. The index is `{instance_id: 1, "tags.$**": 1}`, wildcard
|
||||
because the queried key is chosen by the user at request time and cannot be named
|
||||
in advance; `EnsureServerIndexes` warns rather than being fatal, since a missing
|
||||
index degrades tag filtering to a scan of a small collection and is no reason to
|
||||
refuse to serve the fleet list.
|
||||
|
||||
`services.ResolveTargets` is the **single** answer to which servers a workflow
|
||||
touches — the run path and validation both go through it, so the readout and the
|
||||
dispatch cannot disagree. It is the distinct union of `target_server_ids` and
|
||||
`target_tags` (AND across keys), ordered by the fleet rather than by the
|
||||
arguments, so two runs naming the same servers differently are still comparable
|
||||
line by line. **An empty selector matches nothing** on purpose: "matches
|
||||
everything" turns a cleared field in the designer into a fleet-wide run. Both
|
||||
empty is `ErrNoTargets` (400), not a success over zero servers. Offline servers
|
||||
are **not** filtered out — the dispatcher already answers 503 per server, and a
|
||||
patch run that silently omits an unreachable machine is worse than one that
|
||||
visibly fails on it.
|
||||
|
||||
**Both halves of the selector are edited in `EditWorkflowModal`** — the named
|
||||
servers in a `DualListBox`, the tag rows directly beneath it — and saved
|
||||
together by one `updateWorkflow`. The designer's Targets panel is **read-only**:
|
||||
it reports the count and the tags and links to Edit. Splitting the two halves
|
||||
across two screens meant a workflow's reach was decided in two places with no
|
||||
one view showing both.
|
||||
|
||||
`web/lib/targets.ts` **duplicates the match logic in TypeScript** to draw the
|
||||
resolved count without a round trip, since the browser already holds the fleet.
|
||||
It is a second implementation of `UnionTargets` / `MatchesTags` and must change
|
||||
in the same commit as the Go one — the same shape of hazard as the mirrored
|
||||
token blocks. It is a shared module rather than inline in a component because
|
||||
the logic had already been written twice, and the second copy — the workflows
|
||||
list — counted `target_server_ids` alone, so a **tag-only workflow reported zero
|
||||
targets** while running fine.
|
||||
|
||||
The server picker is a hand-built two-pane list, not `<select multiple>`: a
|
||||
native multi-select paints its selected rows with the platform highlight colour,
|
||||
which cannot be restyled across browsers and lands outside the token palette on
|
||||
a dark ground.
|
||||
|
||||
### Monitors
|
||||
|
||||
HTTP, TCP, ICMP and TLS checks. Each monitor has a `runner`: `"server"` (executed by the server-side scheduler) or a `server_id` (pushed to that agent, which runs it locally and reports results). Consecutive failures beyond `retries` flip state to `down`, open an `Incident`, and notify. Hourly `Rollup` documents back the uptime graphs.
|
||||
@@ -235,6 +315,51 @@ a Service cannot address the one pod holding a console listener.
|
||||
|
||||
Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`).
|
||||
|
||||
### Package inventory and CVE findings
|
||||
|
||||
Agents report their installed packages hourly; the control plane matches them
|
||||
against distribution security feeds and raises findings that link to the
|
||||
existing `ApplyUpdatesCmd` patching path. Gated by the `vuln_scanning` licence
|
||||
feature, **checked at collection rather than display** — an ungated instance
|
||||
stores no inventory, and storage is the expensive half.
|
||||
|
||||
**Matching uses distribution feeds, never NVD version ranges.** Distributions
|
||||
backport security fixes without changing the upstream version: Ubuntu's
|
||||
`openssl 3.0.2-0ubuntu1.15` is patched against CVE-2023-0286 while NVD still
|
||||
calls 3.0.2 vulnerable. Matching on NVD would report a fully patched fleet as
|
||||
critical, and once the first report is mostly wrong nobody reads the second.
|
||||
`trivy-db` is those feeds pre-merged; `server/internal/vulndb` pulls it as an
|
||||
OCI artifact to an ephemeral directory. Version comparison is bought from
|
||||
`go-deb-version`/`go-rpm-version`/`go-apk-version` because dpkg epochs, `~`
|
||||
sorting before the empty string, and `rpmvercmp` are each a silent false
|
||||
negative waiting to happen.
|
||||
|
||||
**Only the leader matches.** `ReportPackages` upserts the list and sets
|
||||
`scan_pending`; it does not scan. `vulnsched` runs inside the existing
|
||||
`bus.RunAsLeader("housekeeping", …)` and does the matching, because otherwise
|
||||
every replica needs the ~50MB database resident and a database refresh has N
|
||||
replicas rescanning the same fleet and sending N digests. The tick is also the
|
||||
digest's batch boundary, which is what makes "one message, not five hundred"
|
||||
structural rather than a debounce someone maintains.
|
||||
|
||||
Findings are **never deleted when a package is patched** — the state moves to
|
||||
`fixed`, so "what did we remediate last quarter" stays answerable. Acceptance
|
||||
requires a reason and an expiry, and reopens automatically: permanent dismissal
|
||||
is where risk goes to be forgotten. An unsupported distribution reports
|
||||
`status: unsupported`, never "0 findings"; claiming clean when the truth is
|
||||
unknown is the same lie as a silently stale database, which is why
|
||||
`vulndb_meta.pulled_at` is on screen rather than only in a log.
|
||||
|
||||
**`server/Dockerfile`'s runtime stage is `scratch`, so it carries an explicitly
|
||||
copied `/tmp`.** The scheduler unpacks the database to a temporary directory,
|
||||
and a scratch image has none — the failure is `vulnsched: temp dir: stat /tmp:
|
||||
no such file or directory`, logged once at boot while every other subsystem
|
||||
runs normally, so the only symptom is a fleet that never reports a finding.
|
||||
|
||||
Two environment variables: `VANTAGE_TRIVY_DB_REF` mirrors the artifact for
|
||||
air-gapped installs, and `VANTAGE_VULNDB_DISABLED` switches the puller and
|
||||
scheduler off entirely.
|
||||
|
||||
### Agent self-update
|
||||
|
||||
`UpdateAgentCmd` carries a target version and Gitea base URL; the agent downloads and replaces itself.
|
||||
@@ -447,6 +572,12 @@ channels GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:i
|
||||
secrets GET,POST /secrets · GET,PUT,DELETE /secrets/:group
|
||||
POST /secrets/:group/reveal · DELETE /secrets/:group/:key
|
||||
console POST /console/connect · GET /console/tunnel (websocket)
|
||||
vulns GET /vulnerabilities · GET /vulnerabilities/summary
|
||||
POST /vulnerabilities/rescan (owner|admin)
|
||||
POST,DELETE /vulnerabilities/:id/accept (owner|admin)
|
||||
GET /servers/:id/vulnerabilities · GET /servers/:id/packages
|
||||
GET /packages/search?name=
|
||||
GET,POST /vuln-rules · PUT,DELETE /vuln-rules/:id (owner|admin)
|
||||
audit GET /audit
|
||||
agent GET /agent/latest-version
|
||||
settings GET,PUT /settings · POST /settings/secrets-token (owner|admin)
|
||||
@@ -527,7 +658,7 @@ Paddle is merchant of record; `admin/internal/paddle` is a thin REST client (no
|
||||
|
||||
## MongoDB Collections
|
||||
|
||||
`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `migrations`
|
||||
`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `migrations`
|
||||
|
||||
Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`.
|
||||
|
||||
@@ -542,6 +673,10 @@ Notes that are not obvious from the structs:
|
||||
- `auth_providers.provider_id` is a short random identifier, not the Mongo `_id`: it appears in the callback URL a customer pastes into their IdP, and an `_id` there would publish a database key. `callback_notice` marks a provider migrated from the old single-provider shape, whose redirect URI therefore changed.
|
||||
- `workflow_log_lines` is keyed `(run_id, server_id, seq)` — the index is not an optimisation, every read is a range scan over it. `workflow_log_seq` holds one counter document per `run_id/server_id`, which is what lets two pods interleave into one ordered log. Neither carries `instance_id`: they are reached only through a run, and a run is already scoped.
|
||||
- `users.auth_source` is `local`, `oidc` or `hq`. An `hq` user was projected from a Vantage HQ account and carries `hq_user_id`; HQ owns its role, password and existence.
|
||||
- `server_packages` holds a server's whole package set in **one** document, not one per package. The hash already established that something changed, so a report is a single atomic upsert with no delta logic to get wrong; ~2000 packages is ~150KB, well inside the 16MB limit. `scan_pending` lives on the document rather than in memory so a leader handover cannot lose it.
|
||||
- `vuln_findings` is unique on `(instance_id, server_id, cve_id, package_name)`. That key is what makes a rescan an idempotent upsert rather than a duplicate factory, and what lets `first_seen` survive one. An empty `fixed_in` means no vendor fix exists — a real state, never "not vulnerable".
|
||||
- `vulndb_meta` is a singleton and deliberately carries **no** `instance_id`: the vulnerability database is a property of the deployment, not a tenant. Same reasoning as `migrations`, and the reason it is absent from `services.ScopedCollections`.
|
||||
- **`services.ScopedCollections` is the canonical registry of tenant-scoped collections**, and `scopedCollectionsForPurge` derives instance deletion from it rather than keeping a second list. A new collection carrying `instance_id` must be added there or its rows outlive the instance.
|
||||
|
||||
Admin's own database is separate and holds `accounts` · `admin_instances` · `licenses` · `subscriptions` · `plans` · `catalogue` · `entitlements` · `paddle_events` · `staff_users` · `customer_users` · `instance_members` · `admin_audit`. `paddle_events` is the webhook idempotency log, unique on `event_id`: an event is claimed there before processing, and a duplicate of a handled event is a 200 no-op. `instance_members` is unique on `(instance_id, customer_user_id)` — one person holds at most one user in one instance, which makes a grant idempotent-by-refusal rather than silently doubling a projection. It is an _index_ of the control-plane rows, not the authority (see "Grants project, they do not federate"). Admin has no migrations collection; `models.Backfill` runs on every boot and is idempotent by filtering on the absence of what it writes.
|
||||
|
||||
@@ -634,6 +769,8 @@ Windows: MSI built by CI (WiX), or `installer/setup.ps1` registering the agent a
|
||||
| `POD_IP` | no | this pod's own address, set by the Helm chart from the downward API. **Takes precedence over `PROXY_ADVERTISE_HOST`** — a console relay listener belongs to one replica, and a Service address names all of them |
|
||||
| `VANTAGE_MIGRATE_ONLY` | no | run schema setup (migrations, index builders, default-step seeding) and exit without serving. `GRPC_HOST` is not required in this mode. Set by the Helm chart's pre-upgrade Job |
|
||||
| `VANTAGE_SKIP_MIGRATIONS` | no | serve without running schema setup, on the assumption a Job already did. Set by the chart's Deployment whenever `server.migrationJob.enabled`. Unset under Compose, where one process still migrates and then serves |
|
||||
| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for an air-gapped install, or to avoid the anonymous ghcr rate limit |
|
||||
| `VANTAGE_VULNDB_DISABLED` | no | `true` disables the vulnerability database puller and scan loop entirely. Findings already written are still served, and still shown as stale |
|
||||
| `FREE_INSTANCE_REAP_AFTER` | no | duration past a Free licence's expiry before the instance and all its data are deleted. **Empty disables the reaper, and empty is the default.** Set to `336h` in `docker-compose.site.yml` only — a self-hosted deployment must never reap. Must match admin's value, which only names the date in warning emails |
|
||||
|
||||
**sitesvc** (`deploy/docker-compose.site.yml` only):
|
||||
@@ -67,7 +67,8 @@ func (r CatalogueRow) Priced(env string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// SeedCatalogue inserts the sixteen rows the four PAID plans need.
|
||||
// SeedCatalogue inserts the twenty rows the four PAID plans need: a base, a
|
||||
// server limit, and one row per feature key.
|
||||
//
|
||||
// The two Free plans get no rows at all, and that absence is what keeps Free
|
||||
// outside Paddle: with nothing to price, no checkout can be built for it. Do not
|
||||
@@ -84,6 +85,7 @@ func SeedCatalogue(ctx context.Context) error {
|
||||
{Kind: KindLimit, Deployment: deployment, Tier: tier, LimitKey: LimitKeyServers},
|
||||
{Kind: KindFeature, Deployment: deployment, Tier: tier, FeatureKey: license.FeatureConsole},
|
||||
{Kind: KindFeature, Deployment: deployment, Tier: tier, FeatureKey: license.FeatureOIDC},
|
||||
{Kind: KindFeature, Deployment: deployment, Tier: tier, FeatureKey: license.FeatureVulnScanning},
|
||||
}
|
||||
for _, r := range rows {
|
||||
filter := bson.M{
|
||||
|
||||
@@ -12,6 +12,7 @@ import { StatePill } from "@/components/StatePill";
|
||||
import { PageFrame, RailCard, RailFacts } from "@/components/PageFrame";
|
||||
import { PageHeader } from "@/components/PageHeader";
|
||||
import { formatDate, licenceState, limitLabel } from "@/lib/format";
|
||||
import { featureLabel } from "@/lib/features";
|
||||
|
||||
export default function InstancePage() {
|
||||
const id = String(useParams().id);
|
||||
@@ -104,7 +105,11 @@ export default function InstancePage() {
|
||||
},
|
||||
{
|
||||
label: "Features",
|
||||
value: lic.features.join(", ") || "none",
|
||||
// Labelled, not raw keys: this is
|
||||
// the customer's own licence, and
|
||||
// "vuln_scanning" is not a name
|
||||
// anyone bought.
|
||||
value: lic.features.map(featureLabel).join(", ") || "none",
|
||||
},
|
||||
]}
|
||||
/>
|
||||
|
||||
@@ -6,25 +6,16 @@ import Link from "next/link";
|
||||
import { useMutation, useQuery } from "@tanstack/react-query";
|
||||
import { ApiError, api, lineItemsFor, type CatalogueRow, type CheckoutOptions, type Deployment, type Plan, type Term, type Tier } from "@/lib/api";
|
||||
import { initPaddle, previewPrices, type PricePreview } from "@/lib/paddle";
|
||||
import { featureDesc, featureLabel } from "@/lib/features";
|
||||
|
||||
/* Tiers in the order a customer reads them, cheapest first. */
|
||||
const TIER_ORDER: Tier[] = ["free", "professional", "enterprise"];
|
||||
|
||||
const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
|
||||
/* Human labels for feature keys. The catalogue names them by key; this is the
|
||||
* one place the customer-facing wording lives. */
|
||||
const FEATURE_LABEL: Record<string, string> = {
|
||||
console: "Browser console",
|
||||
oidc: "Single sign-on",
|
||||
};
|
||||
const FEATURE_DESC: Record<string, string> = {
|
||||
console: "In-browser SSH, RDP and VNC sessions",
|
||||
oidc: "OIDC sign-in for your whole team",
|
||||
};
|
||||
function featureLabel(key: string) {
|
||||
return FEATURE_LABEL[key] ?? key;
|
||||
}
|
||||
/* Feature wording lives in lib/features.ts, shared with the staff
|
||||
* configurator. It was duplicated here and there, and the two copies had
|
||||
* already drifted. */
|
||||
|
||||
interface Choice {
|
||||
tier: Tier;
|
||||
@@ -294,7 +285,7 @@ export function PurchaseForm() {
|
||||
{featureKeys.map((key) => {
|
||||
const st = featureStateFor(plan, rows, options.env, choice.term, key);
|
||||
return (
|
||||
<Row key={key} title={featureLabel(key)} desc={FEATURE_DESC[key] ?? ""} dim={st === "absent"}>
|
||||
<Row key={key} title={featureLabel(key)} desc={featureDesc(key)} dim={st === "absent"}>
|
||||
{st === "included" ? (
|
||||
<span className="text-[0.72rem] font-semibold uppercase tracking-[0.06em] text-valid">Included</span>
|
||||
) : st === "absent" ? (
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import { useMemo } from "react";
|
||||
import type { CatalogueRow, Deployment, Plan, Term, Tier } from "@/lib/api";
|
||||
import { featureLabel } from "@/lib/features";
|
||||
|
||||
export interface PlanChoice {
|
||||
tier: Tier;
|
||||
@@ -159,7 +160,7 @@ export default function PlanConfigurator({
|
||||
})
|
||||
}
|
||||
/>
|
||||
<span>{key === "console" ? "Browser console" : "Single sign-on"}</span>
|
||||
<span>{featureLabel(key)}</span>
|
||||
<span className="text-[0.72rem] text-ink-3">
|
||||
{priced ? "paid add-on" : "included"}
|
||||
</span>
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
/* Human wording for licence feature keys.
|
||||
*
|
||||
* One place, because there were two and they disagreed: the staff configurator
|
||||
* rendered every key that was not "console" as "Single sign-on", so adding a
|
||||
* third feature silently mislabelled the checkbox that grants it. A map with a
|
||||
* fallback degrades to the raw key, which is ugly but never wrong.
|
||||
*
|
||||
* Keys must match shared/license/license.go. */
|
||||
export const FEATURE_LABEL: Record<string, string> = {
|
||||
console: "Browser console",
|
||||
oidc: "Single sign-on",
|
||||
vuln_scanning: "Vulnerability scanning",
|
||||
};
|
||||
|
||||
export const FEATURE_DESC: Record<string, string> = {
|
||||
console: "In-browser SSH, RDP and VNC sessions",
|
||||
oidc: "OIDC sign-in for your whole team",
|
||||
vuln_scanning: "Package inventory matched against distribution security advisories",
|
||||
};
|
||||
|
||||
export function featureLabel(key: string): string {
|
||||
return FEATURE_LABEL[key] ?? key;
|
||||
}
|
||||
|
||||
export function featureDesc(key: string): string {
|
||||
return FEATURE_DESC[key] ?? "";
|
||||
}
|
||||
@@ -80,7 +80,11 @@ func (c *Client) Register(serverID, preRegToken, hostname, ipAddress, osInfo str
|
||||
return resp.AgentToken, nil
|
||||
}
|
||||
|
||||
func (c *Client) SyncKeys(serverID, agentToken, version string) ([]string, error) {
|
||||
// SyncKeys returns the whole response rather than just the keys: the poll now
|
||||
// also carries CollectPackages, and a second RPC purely to learn one boolean
|
||||
// would be a message every 30 seconds for a value that changes at most when a
|
||||
// licence does.
|
||||
func (c *Client) SyncKeys(serverID, agentToken, version string) (*pb.SyncResponse, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -92,7 +96,21 @@ func (c *Client) SyncKeys(serverID, agentToken, version string) ([]string, error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp.PublicKeys, nil
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// ReportPackages sends a package report and returns whether the server wants
|
||||
// the full list. Given a longer deadline than the other unary calls because the
|
||||
// full body is ~150KB on a slow link.
|
||||
func (c *Client) ReportPackages(req *pb.ReportPackagesRequest) (bool, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
resp, err := c.client.ReportPackages(ctx, req)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return resp.NeedFull, nil
|
||||
}
|
||||
|
||||
func (c *Client) UploadGeneratedKey(serverID, agentToken, publicKey, privateKey, label string) (string, error) {
|
||||
|
||||
@@ -30,6 +30,45 @@ type SyncRequest struct {
|
||||
|
||||
type SyncResponse struct {
|
||||
PublicKeys []string `json:"public_keys"`
|
||||
// CollectPackages tells the agent whether this instance's licence grants
|
||||
// vulnerability scanning. Absent decodes as false, which is the safe
|
||||
// direction: an older server leaves agents collecting nothing.
|
||||
CollectPackages bool `json:"collect_packages,omitempty"`
|
||||
}
|
||||
|
||||
type OSRelease struct {
|
||||
Family string `json:"family"`
|
||||
// VersionId is not optional: Ubuntu 22.04 and 24.04 publish different fixed
|
||||
// versions for the same CVE, so a scan without it is guesswork.
|
||||
VersionId string `json:"version_id"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Epoch int32 `json:"epoch,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
// SourceName is what the Debian and Ubuntu feeds are keyed on: one advisory
|
||||
// against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
SourceName string `json:"source_name,omitempty"`
|
||||
}
|
||||
|
||||
// ReportPackagesRequest carries a server's installed package set.
|
||||
//
|
||||
// The agent calls twice at most: first with Packages empty, offering only the
|
||||
// hash. If the server already holds it, NeedFull is false and the ~150KB body
|
||||
// is never sent.
|
||||
type ReportPackagesRequest struct {
|
||||
ServerId string `json:"server_id"`
|
||||
AgentToken string `json:"agent_token"`
|
||||
Hash string `json:"hash"`
|
||||
Os OSRelease `json:"os"`
|
||||
Packages []InstalledPackage `json:"packages,omitempty"`
|
||||
}
|
||||
|
||||
type ReportPackagesResponse struct {
|
||||
NeedFull bool `json:"need_full"`
|
||||
}
|
||||
|
||||
type UploadKeyRequest struct {
|
||||
@@ -337,6 +376,7 @@ type VantageClient interface {
|
||||
SyncKeys(ctx context.Context, in *SyncRequest, opts ...grpc.CallOption) (*SyncResponse, error)
|
||||
UploadGeneratedKey(ctx context.Context, in *UploadKeyRequest, opts ...grpc.CallOption) (*UploadKeyResponse, error)
|
||||
ReportUpdates(ctx context.Context, in *ReportUpdatesRequest, opts ...grpc.CallOption) (*ReportUpdatesResponse, error)
|
||||
ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error)
|
||||
ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error)
|
||||
SyncMonitors(ctx context.Context, in *SyncMonitorsRequest, opts ...grpc.CallOption) (*SyncMonitorsResponse, error)
|
||||
ReportChecks(ctx context.Context, in *ReportChecksRequest, opts ...grpc.CallOption) (*ReportChecksResponse, error)
|
||||
@@ -396,6 +436,14 @@ func (c *keyManagerClient) ReportUpdates(ctx context.Context, in *ReportUpdatesR
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error) {
|
||||
out := new(ReportPackagesResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportPackages", in, out, opts...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error) {
|
||||
out := new(InventoryReportResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportInventory", in, out, opts...); err != nil {
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package packages
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"runtime"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// OSRelease identifies the distribution well enough to select an advisory
|
||||
// feed. VersionID is not optional: Ubuntu 22.04 and 24.04 publish different
|
||||
// fixed versions for the same CVE.
|
||||
type OSRelease struct {
|
||||
Family string
|
||||
VersionID string
|
||||
Arch string
|
||||
}
|
||||
|
||||
// ParseOSRelease reads the os-release format: KEY=value, one per line, with
|
||||
// values optionally quoted, and # comments.
|
||||
//
|
||||
// The quote stripping handles both ID=ubuntu and ID="rocky", which real
|
||||
// distributions both emit.
|
||||
func ParseOSRelease(r io.Reader) (OSRelease, error) {
|
||||
out := OSRelease{Arch: runtime.GOARCH}
|
||||
sc := bufio.NewScanner(r)
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
key, val, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
val = strings.Trim(strings.TrimSpace(val), `"'`)
|
||||
switch strings.TrimSpace(key) {
|
||||
case "ID":
|
||||
out.Family = strings.ToLower(val)
|
||||
case "VERSION_ID":
|
||||
out.VersionID = val
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return OSRelease{}, err
|
||||
}
|
||||
if out.Family == "" {
|
||||
return OSRelease{}, errors.New("os-release has no ID")
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// DetectOS reads /etc/os-release.
|
||||
func DetectOS() (OSRelease, error) {
|
||||
f, err := os.Open("/etc/os-release")
|
||||
if err != nil {
|
||||
return OSRelease{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
return ParseOSRelease(f)
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package packages
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"time"
|
||||
)
|
||||
|
||||
const collectTimeout = 2 * time.Minute
|
||||
|
||||
// Collect enumerates installed packages. Linux only: Windows agents are
|
||||
// second-class by design, and vulnerability scanning there needs a different
|
||||
// source, a different collector and a different matcher, all out of scope.
|
||||
//
|
||||
// The format strings below are raw string literals on purpose. The "\t" and
|
||||
// "\n" reach dpkg-query and rpm as two characters each, and those tools do the
|
||||
// interpreting themselves — Go must not consume the escapes first.
|
||||
func Collect() (OSRelease, []Package, error) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return OSRelease{}, nil, fmt.Errorf("package collection is linux-only, got %s", runtime.GOOS)
|
||||
}
|
||||
|
||||
osrel, err := DetectOS()
|
||||
if err != nil {
|
||||
return OSRelease{}, nil, fmt.Errorf("detect os: %w", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), collectTimeout)
|
||||
defer cancel()
|
||||
|
||||
switch {
|
||||
case have("dpkg-query"):
|
||||
out, err := run(ctx, "dpkg-query", "-W", "-f",
|
||||
`${Package}\t${Version}\t${Architecture}\t${source:Package}\n`)
|
||||
if err != nil {
|
||||
return osrel, nil, err
|
||||
}
|
||||
return osrel, ParseDpkg(out), nil
|
||||
|
||||
case have("rpm"):
|
||||
out, err := run(ctx, "rpm", "-qa", "--qf",
|
||||
`%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n`)
|
||||
if err != nil {
|
||||
return osrel, nil, err
|
||||
}
|
||||
return osrel, ParseRPM(out), nil
|
||||
|
||||
case have("apk"):
|
||||
out, err := run(ctx, "apk", "info", "-v")
|
||||
if err != nil {
|
||||
return osrel, nil, err
|
||||
}
|
||||
return osrel, ParseAPK(out), nil
|
||||
|
||||
default:
|
||||
return osrel, nil, fmt.Errorf("no supported package manager found")
|
||||
}
|
||||
}
|
||||
|
||||
func have(bin string) bool {
|
||||
_, err := exec.LookPath(bin)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func run(ctx context.Context, name string, args ...string) (string, error) {
|
||||
out, err := exec.CommandContext(ctx, name, args...).Output()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package packages
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Package is one installed package as the distribution reports it. Version is
|
||||
// the distribution's own version string, verbatim — never normalised, because
|
||||
// the advisory feeds are keyed on exactly this form.
|
||||
type Package struct {
|
||||
Name string
|
||||
Version string
|
||||
Epoch int
|
||||
Arch string
|
||||
SourceName string
|
||||
}
|
||||
|
||||
// ParseDpkg reads tab-separated output of
|
||||
// dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${source:Package}\n'
|
||||
//
|
||||
// SourceName is why the fourth column is requested at all: Debian and Ubuntu
|
||||
// advisories are keyed on the SOURCE package, so one CVE against "openssl"
|
||||
// covers the binaries libssl3, openssl and libssl-dev. Matching on binary name
|
||||
// alone finds one of the three.
|
||||
func ParseDpkg(out string) []Package {
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
continue
|
||||
}
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
p := Package{Name: f[0], Version: f[1], Arch: f[2]}
|
||||
if len(f) > 3 && f[3] != "" {
|
||||
p.SourceName = f[3]
|
||||
} else {
|
||||
p.SourceName = p.Name
|
||||
}
|
||||
pkgs = append(pkgs, p)
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// ParseRPM reads tab-separated output of
|
||||
// rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n'
|
||||
func ParseRPM(out string) []Package {
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
continue
|
||||
}
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 4 {
|
||||
continue
|
||||
}
|
||||
epoch := 0
|
||||
// rpm prints "(none)" rather than omitting the field when a package has
|
||||
// no epoch. That must become 0, not fail the line.
|
||||
if f[1] != "" && f[1] != "(none)" {
|
||||
if n, err := strconv.Atoi(f[1]); err == nil {
|
||||
epoch = n
|
||||
}
|
||||
}
|
||||
p := Package{Name: f[0], Epoch: epoch, Version: f[2], Arch: f[3]}
|
||||
if len(f) > 4 {
|
||||
p.SourceName = srcRPMName(f[4])
|
||||
}
|
||||
if p.SourceName == "" {
|
||||
p.SourceName = p.Name
|
||||
}
|
||||
pkgs = append(pkgs, p)
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// srcRPMName reduces "openssl-3.0.7-24.el9.src.rpm" to "openssl" by dropping
|
||||
// the trailing ".src.rpm" and then the version and release segments, which are
|
||||
// the last two hyphen-separated fields.
|
||||
func srcRPMName(s string) string {
|
||||
s = strings.TrimSuffix(s, ".src.rpm")
|
||||
parts := strings.Split(s, "-")
|
||||
if len(parts) <= 2 {
|
||||
return s
|
||||
}
|
||||
return strings.Join(parts[:len(parts)-2], "-")
|
||||
}
|
||||
|
||||
// ParseAPK reads "apk info -v" output: one "name-version-rREV" per line.
|
||||
// Alpine has no separate source package, so SourceName mirrors Name.
|
||||
func ParseAPK(out string) []Package {
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
name, version := splitAPK(line)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
pkgs = append(pkgs, Package{Name: name, Version: version, SourceName: name})
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// splitAPK finds the version boundary from the RIGHT. The version is always the
|
||||
// last two hyphen-separated fields ("<version>-r<rev>"), which is reliable
|
||||
// where scanning from the left is not: package names legitimately contain
|
||||
// digits and underscores, so "musl" in "musl-1.2.4_git20230717-r4" cannot be
|
||||
// found by looking for the first digit.
|
||||
func splitAPK(s string) (name, version string) {
|
||||
last := strings.LastIndex(s, "-")
|
||||
if last <= 0 {
|
||||
return "", ""
|
||||
}
|
||||
prev := strings.LastIndex(s[:last], "-")
|
||||
if prev <= 0 {
|
||||
return "", ""
|
||||
}
|
||||
return s[:prev], s[prev+1:]
|
||||
}
|
||||
|
||||
// Hash fingerprints a package set so an unchanged set never has to be sent.
|
||||
//
|
||||
// It sorts first: the ordering of dpkg or rpm output is not guaranteed stable,
|
||||
// and an ordering-sensitive hash would resend the full ~150KB list every hour
|
||||
// for no reason — a cost visible only as traffic.
|
||||
func Hash(pkgs []Package) string {
|
||||
lines := make([]string, 0, len(pkgs))
|
||||
for _, p := range pkgs {
|
||||
lines = append(lines, p.Name+"\x00"+strconv.Itoa(p.Epoch)+"\x00"+p.Version+"\x00"+p.Arch)
|
||||
}
|
||||
sort.Strings(lines)
|
||||
h := sha256.New()
|
||||
for _, l := range lines {
|
||||
h.Write([]byte(l))
|
||||
h.Write([]byte("\n"))
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"runtime"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/config"
|
||||
grpcclient "gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/grpc"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/grpc/pb"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/packages"
|
||||
)
|
||||
|
||||
// collectPackagesFlag is written by the 30s key poll and read by the hourly
|
||||
// package loop — two different goroutines, hence the atomic.
|
||||
//
|
||||
// It defaults to false, so an agent that has not yet completed a poll, or is
|
||||
// talking to a server too old to send the field, collects nothing. Off is the
|
||||
// safe default: collecting without a licence costs the customer storage they
|
||||
// are not paying for.
|
||||
var collectPackagesFlag atomic.Bool
|
||||
|
||||
// firstPoll closes once a SyncKeys response has set the flag above.
|
||||
//
|
||||
// Without it the boot-time package report loses a race it can only lose: the
|
||||
// hourly loop starts before the first poll, reads a flag that is still false by
|
||||
// construction, and skips — so a freshly installed agent reports no packages for
|
||||
// an hour and the server shows nothing to scan.
|
||||
// How long the boot package report waits for that first poll. Two poll
|
||||
// intervals plus slack: long enough to cover one failed attempt, short enough
|
||||
// that a dead control plane does not hold the OS-update report hostage.
|
||||
const firstPollWait = 90 * time.Second
|
||||
|
||||
var (
|
||||
firstPoll = make(chan struct{})
|
||||
firstPollOnce sync.Once
|
||||
)
|
||||
|
||||
func markFirstPoll() { firstPollOnce.Do(func() { close(firstPoll) }) }
|
||||
|
||||
// waitFirstPoll blocks until the flag is known, or gives up. The wait is
|
||||
// bounded because this loop also reports OS updates, which do not depend on the
|
||||
// flag at all — a control plane that cannot be polled must not silence those too.
|
||||
func waitFirstPoll(ctx context.Context, limit time.Duration) {
|
||||
t := time.NewTimer(limit)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-firstPoll:
|
||||
case <-t.C:
|
||||
log.Printf("package collection: no SyncKeys response within %s, collecting nothing this round", limit)
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
|
||||
func collectPackagesEnabled() bool { return collectPackagesFlag.Load() }
|
||||
|
||||
// reportPackages offers a hash of the installed package set and sends the full
|
||||
// list only if the server does not already hold it.
|
||||
//
|
||||
// It runs on the same hourly cadence as the update check because a package set
|
||||
// changes on roughly the same schedule, and reusing that loop means one timer
|
||||
// rather than two.
|
||||
func reportPackages(client *grpcclient.Client, cfg *config.Config) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return
|
||||
}
|
||||
if !collectPackagesEnabled() {
|
||||
return
|
||||
}
|
||||
|
||||
osrel, pkgs, err := packages.Collect()
|
||||
if err != nil {
|
||||
log.Printf("package collection error: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
pbOS := pb.OSRelease{
|
||||
Family: osrel.Family,
|
||||
VersionId: osrel.VersionID,
|
||||
Arch: osrel.Arch,
|
||||
}
|
||||
hash := packages.Hash(pkgs)
|
||||
|
||||
// The offer: hash only, no body. On an unchanged host this is the whole
|
||||
// exchange, which is the point of the handshake.
|
||||
needFull, err := client.ReportPackages(&pb.ReportPackagesRequest{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
Hash: hash,
|
||||
Os: pbOS,
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("ReportPackages offer error: %v", err)
|
||||
return
|
||||
}
|
||||
if !needFull {
|
||||
return
|
||||
}
|
||||
|
||||
pbPkgs := make([]pb.InstalledPackage, len(pkgs))
|
||||
for i, p := range pkgs {
|
||||
pbPkgs[i] = pb.InstalledPackage{
|
||||
Name: p.Name,
|
||||
Version: p.Version,
|
||||
Epoch: int32(p.Epoch),
|
||||
Arch: p.Arch,
|
||||
SourceName: p.SourceName,
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := client.ReportPackages(&pb.ReportPackagesRequest{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
Hash: hash,
|
||||
Os: pbOS,
|
||||
Packages: pbPkgs,
|
||||
}); err != nil {
|
||||
log.Printf("ReportPackages full error: %v", err)
|
||||
return
|
||||
}
|
||||
log.Printf("reported %d installed packages", len(pkgs))
|
||||
}
|
||||
@@ -92,11 +92,19 @@ func Run(ctx context.Context, cfg *config.Config, version string) error {
|
||||
}
|
||||
|
||||
func poll(client *grpcclient.Client, cfg *config.Config, version string) error {
|
||||
desired, err := client.SyncKeys(cfg.ServerID, cfg.AgentToken, version)
|
||||
resp, err := client.SyncKeys(cfg.ServerID, cfg.AgentToken, version)
|
||||
if err != nil {
|
||||
return fmt.Errorf("SyncKeys: %w", err)
|
||||
}
|
||||
|
||||
// Stored atomically: the hourly package loop reads this from another
|
||||
// goroutine. Absent on the wire decodes as false, so an older server leaves
|
||||
// collection off rather than on.
|
||||
collectPackagesFlag.Store(resp.CollectPackages)
|
||||
markFirstPoll()
|
||||
|
||||
desired := resp.PublicKeys
|
||||
|
||||
if runtime.GOOS != "linux" {
|
||||
return nil
|
||||
}
|
||||
@@ -395,8 +403,17 @@ func runUpdateCheck(ctx context.Context, cfg *config.Config) {
|
||||
return
|
||||
}
|
||||
log.Printf("reported %d available OS updates", len(pkgs))
|
||||
|
||||
// Same hourly cadence, same connection. A package set changes on
|
||||
// roughly the schedule available updates do, so this needs no timer of
|
||||
// its own.
|
||||
reportPackages(client, cfg)
|
||||
}
|
||||
|
||||
// The boot round only: after this the flag has long been set, and every
|
||||
// later tick is an hour past a poll that runs every 30s.
|
||||
waitFirstPoll(ctx, firstPollWait)
|
||||
|
||||
doCheck()
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Apply Package Updates",
|
||||
"description": "Apply all pending OS package updates. Supports apt, dnf, yum, zypper, apk and pacman.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nif command -v apt-get >/dev/null 2>&1; then\n export DEBIAN_FRONTEND=noninteractive\n apt-get update -qq && apt-get -y -qq upgrade\nelif command -v dnf >/dev/null 2>&1; then\n dnf -y upgrade\nelif command -v yum >/dev/null 2>&1; then\n yum -y update\nelif command -v zypper >/dev/null 2>&1; then\n zypper --non-interactive update\nelif command -v apk >/dev/null 2>&1; then\n apk update && apk upgrade\nelif command -v pacman >/dev/null 2>&1; then\n pacman -Syu --noconfirm\nelse\n echo \"no supported package manager found\"\n exit 1\nfi\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"package update failed\"\n exit 1\nfi\necho \"packages up to date\"\n# Debian and Ubuntu drop this file when a new kernel or libc needs a restart.\n# Reported rather than acted on: rebooting a fleet is a decision, not a detail.\nif [ -f /var/run/reboot-required ]; then\n echo \"REBOOT_REQUIRED=true\" >> $WORKFLOW_ENV\n echo \"a reboot is required to finish applying updates\"\nelse\n echo \"REBOOT_REQUIRED=false\" >> $WORKFLOW_ENV\nfi",
|
||||
"declared_outputs": [
|
||||
"REBOOT_REQUIRED"
|
||||
],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Check Port Is Listening",
|
||||
"description": "Fail unless something is listening on a TCP port.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nhost=\"${host:-127.0.0.1}\"\nif command -v nc >/dev/null 2>&1; then\n nc -z -w 5 \"$host\" \"$port\" >/dev/null 2>&1\n ok=$?\nelse\n # bash builds /dev/tcp in, so this needs nothing installed.\n timeout 5 bash -c \"cat < /dev/null > /dev/tcp/$host/$port\" >/dev/null 2>&1\n ok=$?\nfi\nif [ $ok -ne 0 ]; then\n echo \"PORT_OPEN=false\" >> $WORKFLOW_ENV\n echo \"nothing listening on $host:$port\"\n exit 1\nfi\necho \"PORT_OPEN=true\" >> $WORKFLOW_ENV\necho \"$host:$port is open\"",
|
||||
"declared_outputs": [
|
||||
"PORT_OPEN"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "host",
|
||||
"default": "127.0.0.1",
|
||||
"description": "host to test"
|
||||
},
|
||||
{
|
||||
"name": "port",
|
||||
"default": "",
|
||||
"description": "TCP port to test"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Copy File/Directory",
|
||||
"description": "Copy a file or directory, preserving mode, ownership and timestamps.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nif [ ! -e \"$source\" ]; then\n echo \"source $source does not exist\"\n exit 1\nfi\ncp -a \"$source\" \"$destination\" || { echo \"failed to copy $source to $destination\"; exit 1; }\necho \"copied $source to $destination\"\necho \"DEST_PATH=$destination\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"DEST_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "source",
|
||||
"default": "",
|
||||
"description": "path to copy from"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"default": "",
|
||||
"description": "path to copy to"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Create Directory",
|
||||
"description": "Create a directory, including any missing parents.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nmkdir -p \"$path\" || { echo \"failed to create $path\"; exit 1; }\nif [ -n \"${mode:-}\" ]; then\n chmod \"$mode\" \"$path\" || { echo \"failed to set mode $mode on $path\"; exit 1; }\nfi\necho \"created $path\"\necho \"DIR_PATH=$path\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"DIR_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "directory to create"
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"default": "",
|
||||
"description": "optional octal mode, e.g. 0750"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Delete File/Directory",
|
||||
"description": "Delete a path. Refuses the root filesystem and an empty value.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\n# A step that runs as root on every server in a selector has to refuse the\n# one input that would wipe the fleet. An unset variable expands to empty,\n# so the empty case is the accident this actually guards against.\ncase \"$path\" in\n \"\"|\"/\"|\"/.\"|\"/..\")\n echo \"refusing to delete '$path'\"\n exit 1\n ;;\nesac\nif [ ! -e \"$path\" ]; then\n echo \"$path does not exist, nothing to do\"\n exit 0\nfi\nrm -rf \"$path\" || { echo \"failed to delete $path\"; exit 1; }\necho \"deleted $path\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "path to delete"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Disk Usage Report",
|
||||
"description": "Report usage for a mount point and fail past a threshold.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nmount=\"${mountPoint:-/}\"\nlimit=\"${maxPercent:-90}\"\ndf -h \"$mount\"\nused=$(df --output=pcent \"$mount\" | tail -1 | tr -dc \"0-9\")\navail=$(df -h --output=avail \"$mount\" | tail -1 | tr -d \" \")\necho \"DISK_USED_PERCENT=$used\" >> $WORKFLOW_ENV\necho \"DISK_AVAILABLE=$avail\" >> $WORKFLOW_ENV\nif [ \"$used\" -ge \"$limit\" ]; then\n echo \"$mount is ${used}% full, at or over the ${limit}% limit\"\n exit 1\nfi\necho \"$mount is ${used}% full, ${avail} available\"",
|
||||
"declared_outputs": [
|
||||
"DISK_USED_PERCENT",
|
||||
"DISK_AVAILABLE"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "mountPoint",
|
||||
"default": "/",
|
||||
"description": "mount point to measure"
|
||||
},
|
||||
{
|
||||
"name": "maxPercent",
|
||||
"default": "90",
|
||||
"description": "fail at or above this percentage"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Docker Compose Pull and Up",
|
||||
"description": "Pull the latest images for a compose project and recreate its containers.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\ncd \"$projectDir\" || { echo \"no such directory: $projectDir\"; exit 1; }\nif docker compose version >/dev/null 2>&1; then\n dc=\"docker compose\"\nelif command -v docker-compose >/dev/null 2>&1; then\n dc=\"docker-compose\"\nelse\n echo \"docker compose is not installed\"\n exit 1\nfi\n$dc pull || { echo \"pull failed\"; exit 1; }\n$dc up -d --remove-orphans || { echo \"up failed\"; exit 1; }\n$dc ps",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "projectDir",
|
||||
"default": "",
|
||||
"description": "directory holding docker-compose.yml"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Download File (Linux)",
|
||||
"description": "Download a file over HTTP to a local path",
|
||||
"interpreter": "bash",
|
||||
"script": "out=$(mktemp -p ./)\necho \"Downloading file from $url\"\nwget -q $url -O $out\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to download file from url\"\n exit 1\nfi\necho \"FILE_PATH=$out\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": ["FILE_PATH"],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Enable Linux Service",
|
||||
"description": "Enable a systemd unit so it starts on boot.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\necho \"enabling service $serviceName\"\nsystemctl enable \"$serviceName\" || { echo \"failed to enable $serviceName\"; exit 1; }\necho \"$serviceName enabled\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "systemd unit to enable"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Extract Archive",
|
||||
"description": "Extract a tar, tar.gz, tar.bz2, tar.xz or zip archive into a directory.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\ndest=\"${destination:-.}\"\nif [ ! -f \"$archive\" ]; then\n echo \"archive $archive does not exist\"\n exit 1\nfi\nmkdir -p \"$dest\"\ncase \"$archive\" in\n *.tar.gz|*.tgz) tar -xzf \"$archive\" -C \"$dest\" ;;\n *.tar.bz2|*.tbz2) tar -xjf \"$archive\" -C \"$dest\" ;;\n *.tar.xz|*.txz) tar -xJf \"$archive\" -C \"$dest\" ;;\n *.tar) tar -xf \"$archive\" -C \"$dest\" ;;\n *.zip)\n command -v unzip >/dev/null 2>&1 || { echo \"unzip is not installed\"; exit 1; }\n unzip -oq \"$archive\" -d \"$dest\"\n ;;\n *)\n echo \"unsupported archive type: $archive\"\n exit 1\n ;;\nesac\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to extract $archive\"\n exit 1\nfi\necho \"extracted $archive into $dest\"\necho \"EXTRACT_DIR=$dest\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"EXTRACT_DIR"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "archive",
|
||||
"default": "",
|
||||
"description": "archive file to extract"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"default": ".",
|
||||
"description": "directory to extract into"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,12 +1,10 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Get Host Name",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "HOSTNAME=$(hostname)\necho $HOSTNAME\necho \"HOSTNAME=$HOSTNAME\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"HOSTNAME"
|
||||
],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Get Host Name",
|
||||
"description": "Gets the agents hostname",
|
||||
"interpreter": "bash",
|
||||
"script": "HOSTNAME=$(hostname)\necho $HOSTNAME\necho \"HOSTNAME=$HOSTNAME\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": ["HOSTNAME"],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "HTTP Health Check",
|
||||
"description": "Request a URL and fail unless it answers with the expected status.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nexpected=\"${expectedStatus:-200}\"\nattempts=\"${retries:-3}\"\ndelay=\"${retryDelay:-5}\"\nstatus=\"\"\ni=1\n# Retries live in the script rather than in on_failure: a service coming up\n# after a restart wants a few seconds, not a whole step re-dispatched.\nwhile [ \"$i\" -le \"$attempts\" ]; do\n status=$(curl -s -o /dev/null -w \"%{http_code}\" --max-time 10 \"$url\" || echo \"000\")\n echo \"attempt $i: $url returned $status\"\n if [ \"$status\" = \"$expected\" ]; then\n break\n fi\n i=$(( i + 1 ))\n if [ \"$i\" -le \"$attempts\" ]; then sleep \"$delay\"; fi\ndone\necho \"HTTP_STATUS=$status\" >> $WORKFLOW_ENV\nif [ \"$status\" != \"$expected\" ]; then\n echo \"$url returned $status, expected $expected\"\n exit 1\nfi\necho \"$url is healthy\"",
|
||||
"declared_outputs": [
|
||||
"HTTP_STATUS"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": "URL to request"
|
||||
},
|
||||
{
|
||||
"name": "expectedStatus",
|
||||
"default": "200",
|
||||
"description": "HTTP status that counts as healthy"
|
||||
},
|
||||
{
|
||||
"name": "retries",
|
||||
"default": "3",
|
||||
"description": "how many attempts before failing"
|
||||
},
|
||||
{
|
||||
"name": "retryDelay",
|
||||
"default": "5",
|
||||
"description": "seconds between attempts"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,16 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "List Directory",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "if [ ! -e $path ]; then\n echo \"file or directory doesn't exist: $path\"\n exit 1\nfi\nls -l $path",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "./",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "List Directory",
|
||||
"description": "Lists the files in the specified path",
|
||||
"interpreter": "bash",
|
||||
"script": "if [ ! -e $path ]; then\n echo \"file or directory doesn't exist: $path\"\n exit 1\nfi\nls -l $path",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "./",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Memory Usage Report",
|
||||
"description": "Report memory usage as a percentage of total.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nfree -h\ntotal=$(free -m | awk \"/^Mem:/ {print \\$2}\")\nused=$(free -m | awk \"/^Mem:/ {print \\$3}\")\npct=$(( used * 100 / total ))\necho \"MEM_USED_PERCENT=$pct\" >> $WORKFLOW_ENV\necho \"MEM_USED_MB=$used\" >> $WORKFLOW_ENV\necho \"memory ${pct}% used (${used}MB of ${total}MB)\"",
|
||||
"declared_outputs": [
|
||||
"MEM_USED_PERCENT",
|
||||
"MEM_USED_MB"
|
||||
],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Reboot Server",
|
||||
"description": "Schedule a reboot a minute out, so the step reports success before the machine goes down.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\ndelay=\"${delayMinutes:-1}\"\n# Scheduled rather than immediate on purpose: `shutdown -r now` kills the\n# agent before it can report, and the run records a failure on a server\n# that did exactly what it was told.\necho \"rebooting in $delay minute(s)\"\nshutdown -r \"+$delay\" \"Reboot requested by Vantage\" || { echo \"failed to schedule a reboot\"; exit 1; }",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "delayMinutes",
|
||||
"default": "1",
|
||||
"description": "minutes to wait before rebooting"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Restart Linux Service",
|
||||
"description": "Restart a systemd unit and fail if it does not come back up.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\necho \"restarting service $serviceName\"\nsystemctl restart \"$serviceName\" || { echo \"failed to restart $serviceName\"; exit 1; }\nsystemctl is-active --quiet \"$serviceName\" || {\n echo \"$serviceName did not come back up\"\n systemctl status \"$serviceName\" --no-pager --lines=20 || true\n exit 1\n}\necho \"$serviceName is active\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "systemd unit to restart, e.g. nginx"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Linux Service Status",
|
||||
"description": "Report whether a systemd unit is active and enabled. Does not fail on a stopped unit.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nstate=$(systemctl is-active \"$serviceName\" 2>/dev/null || true)\nenabled=$(systemctl is-enabled \"$serviceName\" 2>/dev/null || true)\necho \"$serviceName: state=$state enabled=$enabled\"\necho \"SERVICE_STATE=$state\" >> $WORKFLOW_ENV\necho \"SERVICE_ENABLED=$enabled\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"SERVICE_STATE",
|
||||
"SERVICE_ENABLED"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "systemd unit to inspect"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Set Permissions and Ownership",
|
||||
"description": "Set the mode and optionally the owner of a path.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nif [ ! -e \"$path\" ]; then\n echo \"$path does not exist\"\n exit 1\nfi\nrecurse=\"\"\nif [ \"${recursive:-false}\" = \"true\" ]; then\n recurse=\"-R\"\nfi\nif [ -n \"${mode:-}\" ]; then\n chmod $recurse \"$mode\" \"$path\" || { echo \"failed to set mode\"; exit 1; }\n echo \"set mode $mode on $path\"\nfi\nif [ -n \"${owner:-}\" ]; then\n chown $recurse \"$owner\" \"$path\" || { echo \"failed to set owner\"; exit 1; }\n echo \"set owner $owner on $path\"\nfi",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "path to change"
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"default": "",
|
||||
"description": "octal mode, e.g. 0640"
|
||||
},
|
||||
{
|
||||
"name": "owner",
|
||||
"default": "",
|
||||
"description": "owner, e.g. root:root"
|
||||
},
|
||||
{
|
||||
"name": "recursive",
|
||||
"default": "false",
|
||||
"description": "true to apply recursively"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Start Linux Service",
|
||||
"description": "",
|
||||
"description": "Start a linux systemd service",
|
||||
"interpreter": "bash",
|
||||
"script": "echo \"starting service $serviceName\"\nsystemctl start $serviceName\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to start service\"\n exit 1\nfi",
|
||||
"declared_outputs": [],
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Stop Linux Service",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "echo \"stopping service $serviceName\"\nsystemctl stop $serviceName\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to stop service\"\n exit 1\nfi",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Stop Linux Service",
|
||||
"description": "Stops a linux systemd service",
|
||||
"interpreter": "bash",
|
||||
"script": "echo \"stopping service $serviceName\"\nsystemctl stop $serviceName\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to stop service\"\n exit 1\nfi",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Tail Log File",
|
||||
"description": "Print the last N lines of a file, for reading a log after a deployment step.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nlines=\"${lines:-50}\"\nif [ ! -f \"$path\" ]; then\n echo \"$path does not exist\"\n exit 1\nfi\necho \"last $lines lines of $path:\"\ntail -n \"$lines\" \"$path\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "log file to read"
|
||||
},
|
||||
{
|
||||
"name": "lines",
|
||||
"default": "50",
|
||||
"description": "how many lines to print"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "TLS Certificate Expiry",
|
||||
"description": "Report days remaining on a TLS certificate and fail under a threshold.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nport=\"${port:-443}\"\nmin=\"${minDays:-14}\"\ncommand -v openssl >/dev/null 2>&1 || { echo \"openssl is not installed\"; exit 1; }\n# -servername sends SNI, without which a shared host returns the wrong\n# certificate and the expiry reported here belongs to someone else.\nend=$(echo | openssl s_client -servername \"$host\" -connect \"$host:$port\" 2>/dev/null \\\n | openssl x509 -noout -enddate | cut -d= -f2)\nif [ -z \"$end\" ]; then\n echo \"could not read a certificate from $host:$port\"\n exit 1\nfi\nendEpoch=$(date -d \"$end\" +%s)\nnowEpoch=$(date +%s)\ndays=$(( (endEpoch - nowEpoch) / 86400 ))\necho \"CERT_DAYS_REMAINING=$days\" >> $WORKFLOW_ENV\necho \"CERT_EXPIRES=$end\" >> $WORKFLOW_ENV\necho \"$host:$port expires in $days days ($end)\"\nif [ \"$days\" -lt \"$min\" ]; then\n echo \"fewer than $min days remaining\"\n exit 1\nfi",
|
||||
"declared_outputs": [
|
||||
"CERT_DAYS_REMAINING",
|
||||
"CERT_EXPIRES"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "host",
|
||||
"default": "",
|
||||
"description": "hostname to check"
|
||||
},
|
||||
{
|
||||
"name": "port",
|
||||
"default": "443",
|
||||
"description": "TLS port"
|
||||
},
|
||||
{
|
||||
"name": "minDays",
|
||||
"default": "14",
|
||||
"description": "fail below this many days remaining"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,18 +0,0 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "WGET",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "out=$(mktemp -p ./)\necho \"Downloading file from $url\"\nwget -q $url -O $out\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to download file from url\"\n exit 1\nfi\necho \"FILE_PATH=$out\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"FILE_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Download File (Windows)",
|
||||
"description": "Download a file over HTTP to a local path.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$url = $env:url\n$dest = if ($env:destination) { $env:destination } else { Join-Path $env:TEMP ([System.IO.Path]::GetFileName($url)) }\nWrite-Output \"downloading $url\"\ntry {\n # -UseBasicParsing keeps this working on Server Core, where the IE\n # engine Invoke-WebRequest otherwise reaches for is not installed.\n Invoke-WebRequest -Uri $url -OutFile $dest -UseBasicParsing\n} catch {\n Write-Output \"failed to download: $_\"\n exit 1\n}\nWrite-Output \"saved to $dest\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"FILE_PATH=$dest\"",
|
||||
"declared_outputs": [
|
||||
"FILE_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": "URL to download"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"default": "",
|
||||
"description": "where to save it; defaults to a file in TEMP"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Reboot Windows Server",
|
||||
"description": "Schedule a reboot a minute out, so the step reports success before the machine goes down.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$delay = if ($env:delaySeconds) { [int]$env:delaySeconds } else { 60 }\nWrite-Output \"rebooting in $delay second(s)\"\n& shutdown.exe /r /t $delay /c \"Reboot requested by Vantage\"\nif ($LASTEXITCODE -ne 0) {\n Write-Output \"failed to schedule a reboot\"\n exit 1\n}",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "delaySeconds",
|
||||
"default": "60",
|
||||
"description": "seconds to wait before rebooting"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Restart Windows Service",
|
||||
"description": "Restart a Windows service and fail if it does not come back up.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$name = $env:serviceName\nWrite-Output \"restarting service $name\"\ntry {\n Restart-Service -Name $name -Force\n} catch {\n Write-Output \"failed to restart ${name}: $_\"\n exit 1\n}\n$svc = Get-Service -Name $name\nif ($svc.Status -ne \"Running\") {\n Write-Output \"$name is $($svc.Status), not Running\"\n exit 1\n}\nWrite-Output \"$name is running\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "Windows service name to restart"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Windows Disk Report",
|
||||
"description": "Report free space on a drive and fail past a usage threshold.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$letter = if ($env:driveLetter) { $env:driveLetter } else { \"C\" }\n$limit = if ($env:maxPercent) { [int]$env:maxPercent } else { 90 }\n$d = Get-PSDrive -Name $letter -ErrorAction SilentlyContinue\nif ($null -eq $d) {\n Write-Output \"drive $letter not found\"\n exit 1\n}\n$total = $d.Used + $d.Free\n$pct = [math]::Round(($d.Used / $total) * 100)\n$freeGb = [math]::Round($d.Free / 1GB, 1)\nWrite-Output \"${letter}: is $pct% full, $freeGb GB free\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"DISK_USED_PERCENT=$pct\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"DISK_FREE_GB=$freeGb\"\nif ($pct -ge $limit) {\n Write-Output \"at or over the $limit% limit\"\n exit 1\n}",
|
||||
"declared_outputs": [
|
||||
"DISK_USED_PERCENT",
|
||||
"DISK_FREE_GB"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "driveLetter",
|
||||
"default": "C",
|
||||
"description": "drive letter, without a colon"
|
||||
},
|
||||
{
|
||||
"name": "maxPercent",
|
||||
"default": "90",
|
||||
"description": "fail at or above this percentage"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Windows Service Status",
|
||||
"description": "Report a Windows service's status and start type. Does not fail on a stopped service.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$name = $env:serviceName\n$svc = Get-Service -Name $name -ErrorAction SilentlyContinue\nif ($null -eq $svc) {\n Write-Output \"$name is not installed\"\n Add-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_STATE=missing\"\n Add-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_START_TYPE=none\"\n exit 0\n}\nWrite-Output \"${name}: $($svc.Status), start type $($svc.StartType)\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_STATE=$($svc.Status)\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_START_TYPE=$($svc.StartType)\"",
|
||||
"declared_outputs": [
|
||||
"SERVICE_STATE",
|
||||
"SERVICE_START_TYPE"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "Windows service name to inspect"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,235 @@
|
||||
# Server tags and scheduled workflows
|
||||
|
||||
Date: 2026-08-04
|
||||
|
||||
Two features, designed together because the second is worth much less without
|
||||
the first. Tags make a target set describable; schedules make it recur. A
|
||||
nightly job that patches "everything tagged `env:staging`" needs both halves,
|
||||
and neither half is large on its own.
|
||||
|
||||
---
|
||||
|
||||
## Part A — Server tags
|
||||
|
||||
### Model
|
||||
|
||||
`models.Server` gains one field:
|
||||
|
||||
```go
|
||||
Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"`
|
||||
```
|
||||
|
||||
Keys and values are lowercase `[a-z0-9_-]`. Keys are capped at 32 characters,
|
||||
values at 64, and a server holds at most 20 tags. Validation lives in the
|
||||
service layer rather than the handler, so the tag endpoint, the server-create
|
||||
path and anything added later cannot disagree about what a valid tag is.
|
||||
|
||||
There is **no `tags` collection.** A tag is a property of a server, not an
|
||||
entity with a lifecycle: a registry would need reference counting to know when
|
||||
a tag stopped existing, and garbage collection to act on it, which is work
|
||||
bought for nothing. The list of known keys and values that the UI offers for
|
||||
autocomplete is a distinct aggregation over `servers`, cached for 60 seconds —
|
||||
the same treatment org lookups already get.
|
||||
|
||||
No reserved keys ship in this change. If inventory-derived tags (`os`, `arch`)
|
||||
are added later they take a `sys:` key prefix, so a user tag written today can
|
||||
never collide with a system tag invented tomorrow.
|
||||
|
||||
Index: `{instance_id: 1, "tags.$**": 1}` — a wildcard index over the tag
|
||||
subdocument, because the queried key is chosen by the user at request time and
|
||||
cannot be named in advance.
|
||||
|
||||
### API
|
||||
|
||||
```
|
||||
PUT /api/servers/:id/tags # replace the whole map
|
||||
GET /api/servers/tags # known keys and values, for pickers
|
||||
GET /api/servers?tag=env:prod # repeatable; AND across keys
|
||||
```
|
||||
|
||||
`PUT` replaces the entire map rather than patching one tag. A tag set is small
|
||||
enough that sending all of it is free, and last-write-wins over a whole map is
|
||||
easier to reason about than merge semantics between two people editing the same
|
||||
server. The audit event records the map before and after.
|
||||
|
||||
`?tag=` is repeatable and ANDs: `?tag=env:prod&tag=role:web` matches servers
|
||||
carrying both. A malformed value (no colon, unknown characters) is a 400 rather
|
||||
than a silent empty result — a filter that matches nothing and a filter that is
|
||||
nonsense look identical in a list, and only one of them is the user's fault.
|
||||
|
||||
### Targeting
|
||||
|
||||
`models.Workflow` gains `TargetTags map[string]string` beside the existing
|
||||
`TargetServerIDs`. One function in `services` resolves them:
|
||||
|
||||
```go
|
||||
ResolveTargets(ctx, instanceID string, ids []string, tags map[string]string) ([]Server, error)
|
||||
```
|
||||
|
||||
- Result is the **distinct union** of the explicit IDs and the tag matches.
|
||||
- Tag matching ANDs across keys.
|
||||
- Offline servers are included. The dispatcher already answers 503 per server,
|
||||
and a patch run that silently omits an unreachable machine is worse than one
|
||||
that visibly fails on it.
|
||||
- Empty IDs **and** empty tags returns `ErrNoTargets` (400). A workflow that
|
||||
matches nothing must say so rather than report success over zero servers.
|
||||
|
||||
The resolved set is snapshotted into `WorkflowRun.ServerRuns` exactly as today.
|
||||
History records what actually ran, not what the selector would match when the
|
||||
run is later read back — the same reason `steps_snapshot` exists.
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Server detail**: tag chips in the header with an inline editor. Keys
|
||||
autocomplete from `GET /api/servers/tags`, values autocomplete per key.
|
||||
- **`/servers`**: a filter bar that reads and writes the same `?tag=` query
|
||||
params the API takes, so a filtered fleet view is a URL someone can send.
|
||||
- **Workflow designer**: a target section holding both inputs, with a live
|
||||
"runs on 14 servers" readout that lists them on hover. The union model costs
|
||||
us the at-a-glance answer to "what will this touch"; this readout buys it
|
||||
back, and it is the reason the union is acceptable.
|
||||
|
||||
---
|
||||
|
||||
## Part B — Scheduled workflows
|
||||
|
||||
### Model
|
||||
|
||||
```go
|
||||
type Schedule struct {
|
||||
Enabled bool `bson:"enabled" json:"enabled"`
|
||||
Cron string `bson:"cron" json:"cron"` // 5-field
|
||||
TZ string `bson:"tz" json:"tz"` // IANA name
|
||||
}
|
||||
|
||||
type Skip struct {
|
||||
Reason string `bson:"reason" json:"reason"` // "missed" | "already_running"
|
||||
Due time.Time `bson:"due" json:"due"`
|
||||
At time.Time `bson:"at" json:"at"`
|
||||
}
|
||||
```
|
||||
|
||||
On `Workflow`:
|
||||
|
||||
```go
|
||||
Schedule *Schedule `bson:"schedule,omitempty"`
|
||||
NextRunAt *time.Time `bson:"next_run_at,omitempty"` // UTC, indexed
|
||||
LastRunAt *time.Time `bson:"last_run_at,omitempty"`
|
||||
LastSkipped *Skip `bson:"last_skipped,omitempty"`
|
||||
```
|
||||
|
||||
`next_run_at` is **persisted, not held in memory.** A leader handover between
|
||||
computing the next occurrence and firing it would otherwise either lose the
|
||||
occurrence or fire it twice. Coordination state has to live where every replica
|
||||
can see it — the same argument that put `workflow_log_seq` in MongoDB.
|
||||
|
||||
Cron parsing uses `robfig/cron/v3`'s **parser only** — `Parse` and
|
||||
`Next(time)`. Its scheduler and goroutines are not used; the loop below is ours
|
||||
and has to be, because it runs under the leader lock.
|
||||
|
||||
**Alpine ships no tzdata.** `server/Dockerfile` builds a slim image, so
|
||||
`time.LoadLocation("Europe/London")` returns an error and every schedule
|
||||
falls back to UTC — an hour wrong for half the year, in the direction nobody
|
||||
notices until a maintenance window lands in business hours. `main` therefore
|
||||
imports `_ "time/tzdata"`, embedding the database in the binary. Zone names are
|
||||
also validated at save time, so an unknown zone is a 400 rather than a surprise
|
||||
at 2am.
|
||||
|
||||
### Scheduler
|
||||
|
||||
A new `server/internal/workflowsched` package, started inside the **existing**
|
||||
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, `StartReaper`
|
||||
and the sweepers. One role, one lock. It takes the same cancellable context and
|
||||
returns the instant leadership is lost.
|
||||
|
||||
The loop ticks every 30 seconds:
|
||||
|
||||
1. `find({schedule.enabled: true, next_run_at: {$lte: now}})`.
|
||||
2. **Claim atomically.** `findOneAndUpdate` matching the document *and* its
|
||||
current `next_run_at`, setting the recomputed next occurrence. A process
|
||||
that reaches the same document after another has claimed it matches nothing
|
||||
and does nothing. The claim is what makes this correct; the leader lock only
|
||||
makes it cheap.
|
||||
3. **Grace check.** If `now - due > 1h`, record
|
||||
`last_skipped{reason: "missed"}`, write an audit event, and do not run. A
|
||||
job missed by ten minutes during a deploy should still run; one missed by
|
||||
two days should not fire at lunchtime.
|
||||
4. **Overlap check.** If a run for this workflow is still active, record
|
||||
`last_skipped{reason: "already_running"}`, audit, and do not run. A patch
|
||||
workflow must never run twice at once, and a silent skip is how a week goes
|
||||
by before anyone notices nothing ran.
|
||||
5. Otherwise start the run through the **same** `RunWorkflow` path a person
|
||||
uses, with `TriggeredBy: "schedule"`.
|
||||
|
||||
Step 5 is the design. A scheduled run is an ordinary run with a different
|
||||
trigger: no second dispatch path, no second snapshot format, and the run detail
|
||||
page needs no changes to display one.
|
||||
|
||||
### API
|
||||
|
||||
```
|
||||
PUT /api/workflows/:id/schedule # {enabled, cron, tz}
|
||||
GET /api/workflows/:id/schedule/preview?cron=…&tz=… # next 3 occurrences
|
||||
```
|
||||
|
||||
`PUT` validates the expression and the zone, then computes and stores
|
||||
`next_run_at`. The preview endpoint exists so the browser and the scheduler
|
||||
agree on what a cron string means — a client-side cron parser that disagrees
|
||||
with the server by one field is a bug found in production, at night.
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Workflow page**: a schedule card with preset buttons (hourly, nightly at
|
||||
HH:MM, weekly on DAY at HH:MM) that write cron underneath, a raw cron field
|
||||
for anything else, a timezone select, and the next three occurrences rendered
|
||||
from the preview endpoint in mono.
|
||||
- **Workflows list**: a schedule chip and the next run as relative time.
|
||||
- **Skips are surfaced**, not just stored: a warning line reading
|
||||
"Skipped Sun 02:00 — previous run still active". Recording a reason nobody
|
||||
reads is the same as not recording one.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
**Notification on scheduled-run failure.** It needs the monitor channel
|
||||
machinery pointed at workflow outcomes and its own answer to what counts as
|
||||
failure — a non-zero exit on a step with `on_failure: continue` is not
|
||||
obviously an alert. Visibility in this change is the run list and the recorded
|
||||
skip reason. Excluded deliberately, not overlooked.
|
||||
|
||||
**Tag-scoped permissions.** Roles stay instance-wide. Tags describe servers;
|
||||
they do not yet gate who may act on them.
|
||||
|
||||
**Inventory-derived tags.** Reserved via the `sys:` prefix, not implemented.
|
||||
|
||||
---
|
||||
|
||||
## Migration and compatibility
|
||||
|
||||
No migration is required. `Tags`, `TargetTags` and `Schedule` are all
|
||||
`omitempty` and absent means what it meant before: no tags, no selector, no
|
||||
schedule. Existing workflows keep their explicit server lists and behave
|
||||
identically.
|
||||
|
||||
The wildcard tag index and the `next_run_at` index are declared by a new
|
||||
`EnsureServerIndexes`, following the convention `EnsureSecretIndexes` and
|
||||
`EnsureWorkflowIndexes` already set: it warns rather than aborting boot,
|
||||
because a missing index degrades
|
||||
tag filtering to a collection scan on a small collection rather than breaking
|
||||
the fleet list.
|
||||
|
||||
## Testing
|
||||
|
||||
- `ResolveTargets`: union deduplicates; AND across tag keys; empty/empty
|
||||
returns `ErrNoTargets`; offline servers are included.
|
||||
- Tag validation: charset, length caps, tag count cap, malformed `?tag=` is a
|
||||
400.
|
||||
- Schedule validation: bad cron and unknown zone both 400; `next_run_at` is
|
||||
computed in the stored zone, verified across a DST boundary.
|
||||
- Scheduler claim: two concurrent claims of the same due workflow start exactly
|
||||
one run.
|
||||
- Grace window: due 10 minutes ago runs; due 2 hours ago records `missed`.
|
||||
- Overlap: an active run yields `already_running` and no second run.
|
||||
- Preview endpoint and the scheduler agree on the next occurrence for a table
|
||||
of expressions, including a DST-crossing one.
|
||||
@@ -0,0 +1,538 @@
|
||||
# Package inventory and CVE findings
|
||||
|
||||
Date: 2026-08-06
|
||||
|
||||
Agents report the packages installed on each server. The control plane matches
|
||||
them against distro security feeds and raises findings that link straight to
|
||||
the patching path that already exists. A finding nobody can fix today can be
|
||||
accepted with a reason and an expiry date rather than sitting red forever.
|
||||
|
||||
This is one of four sub-projects sketched together and deliberately separated:
|
||||
|
||||
| # | Sub-project | Depends on |
|
||||
| - | ----------- | ---------- |
|
||||
| A | **Package inventory + CVE findings** — this spec | nothing |
|
||||
| B | Container/service registry | nothing |
|
||||
| C | Container image scanning | A and B |
|
||||
| D | Compliance profiles (baseline assertions) | shares A's findings UI only |
|
||||
|
||||
A and B are independent of one another. C is the joiner and must not be
|
||||
designed before both exist. D shares a page with A and nothing else — a
|
||||
different collector, a different evaluation model and a different remediation
|
||||
story — so folding it in here would double the size for no shared machinery.
|
||||
|
||||
Scope of this spec is **A, Linux only.** Windows needs a separate source
|
||||
(MSRC CVRF), a separate collector (`Get-HotFix` plus registry) and a KB
|
||||
supersedence matcher that shares no code with the Linux path. That matches the
|
||||
existing position that Windows agents are second-class by design, and the six
|
||||
package managers `updates.go` already detects cover the whole Linux surface.
|
||||
|
||||
---
|
||||
|
||||
## The trap this design is built around
|
||||
|
||||
Distributions **backport** security fixes without changing the upstream
|
||||
version. Ubuntu ships `openssl 3.0.2-0ubuntu1.15` patched against
|
||||
CVE-2023-0286; NVD says version 3.0.2 is vulnerable. Matching installed
|
||||
versions against NVD or CPE ranges therefore reports a fleet full of criticals
|
||||
that are all already fixed.
|
||||
|
||||
That is not merely noisy. It is fatal to the feature: once the first report is
|
||||
mostly wrong, nobody reads the second one, and a genuine finding is lost in the
|
||||
noise it created. Everything below follows from refusing to make that mistake.
|
||||
|
||||
The correct source is the **distribution's own security feed**, keyed on the
|
||||
distribution's own version string — Debian and Ubuntu OVAL/USN, Red Hat OVAL
|
||||
v2, Alpine secdb. `trivy-db` is those feeds pre-merged into one BoltDB
|
||||
artifact, rebuilt every six hours and published as an OCI artifact.
|
||||
|
||||
---
|
||||
|
||||
## Where the vulnerability data comes from
|
||||
|
||||
`trivy-db`, pulled server-side from `ghcr.io/aquasecurity/trivy-db:2`.
|
||||
|
||||
The alternative considered was querying OSV.dev per scan, which needs no
|
||||
storage and no puller. It was rejected on two counts: it requires outbound
|
||||
internet on every scan, which breaks air-gapped installs; and it sends the
|
||||
package list of a customer's entire fleet to a third party. The audience most
|
||||
likely to buy vulnerability scanning is the audience least willing to do that.
|
||||
|
||||
The blob is roughly 50MB, read-only, reproducible, and identified by a version
|
||||
number. **It is not stored in Mongo and not written to `/data`** —
|
||||
`server.persistence` defaults to off and nothing writes to `/data` any more.
|
||||
It does not need durable storage: whichever pod needs it pulls it to its own
|
||||
ephemeral temp directory. Nothing shared, nothing to back up, nothing to
|
||||
migrate.
|
||||
|
||||
`VANTAGE_TRIVY_DB_REF` overrides the default reference so a customer can mirror
|
||||
the artifact into their own registry. It also covers the anonymous ghcr rate
|
||||
limit, which the six-hourly pull cadence already makes unlikely to bite.
|
||||
|
||||
---
|
||||
|
||||
## Only the leader matches
|
||||
|
||||
This is the crux, and it falls out of the replica model already in the
|
||||
codebase.
|
||||
|
||||
Two things trigger matching, and they happen on different pods:
|
||||
|
||||
1. a fleet-wide rescan when `trivy-db` updates — naturally the leader's job
|
||||
2. a server's package list changing — handled by whichever pod holds *that
|
||||
agent's* command stream
|
||||
|
||||
If (2) matched inline, **every replica would need the 50MB database resident**,
|
||||
and a database refresh would have N pods racing to rescan the same fleet and N
|
||||
digests reaching the customer. That is the exact failure `RunAsLeader` exists
|
||||
to prevent, and it is the same argument that put `monitorsched` behind the
|
||||
lock.
|
||||
|
||||
So `ReportPackages` does not match. It upserts the package list and sets
|
||||
`scan_pending: true`. That is all it does.
|
||||
|
||||
`server/internal/vulnsched` then runs inside the **existing**
|
||||
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`,
|
||||
`workflowsched` and the sweepers — one role, one lock. Every 60 seconds it:
|
||||
|
||||
1. pulls `trivy-db` if the local copy is older than six hours
|
||||
2. if the pulled version differs from `vulndb_meta.db_version`, marks **every**
|
||||
server `scan_pending`
|
||||
3. matches all `scan_pending` servers, clears the flag, diffs against existing
|
||||
findings
|
||||
4. emits **one** digest per tick covering everything newly opened
|
||||
|
||||
Step 4 is why batching is structural rather than bolted on. A `trivy-db`
|
||||
refresh can open several hundred findings across a fleet at once; one message
|
||||
per finding would rate-limit the webhook or get the channel muted, and either
|
||||
way the customer stops receiving the alerts they are paying for. The tick is
|
||||
already the natural batch boundary, so **the failure cannot occur by
|
||||
construction** rather than by a debounce someone has to maintain.
|
||||
|
||||
`scan_pending` lives on the document rather than in memory, for the same reason
|
||||
`next_run_at` and `workflow_log_seq` do: a leader handover between marking and
|
||||
scanning would otherwise lose it. A handover costs the new leader one re-pull
|
||||
of the database.
|
||||
|
||||
The cost of this indirection is up to 60 seconds between an agent reporting a
|
||||
changed package set and its findings updating. For vulnerability data that is
|
||||
nothing, and it buys a single matching path instead of two.
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
```
|
||||
agent/internal/packages/ collect installed packages + /etc/os-release
|
||||
proto/ ReportPackages RPC
|
||||
server/internal/vulndb/ puller, BoltDB access, matcher
|
||||
server/internal/vulnsched/ leader-owned tick: pull, scan, digest
|
||||
server/internal/services/ findings, acceptance, alert rules
|
||||
web/app/(app)/vulnerabilities/ fleet board; plus two server-detail tabs
|
||||
```
|
||||
|
||||
`vulnsched` takes the dependencies it needs — `LogEvent` and the notification
|
||||
dispatch — as a `vulnsched.Deps` injected from `main.go`, following
|
||||
`workflowsched`. The manual rescan endpoint does not call into `vulnsched` at
|
||||
all: it sets `scan_pending` on every server and lets the next tick find them,
|
||||
so there is no path by which `services` imports the scheduler and no cycle to
|
||||
avoid later.
|
||||
|
||||
---
|
||||
|
||||
## The wire path
|
||||
|
||||
A new `ReportPackages` RPC on the agent's existing hourly loop — the same
|
||||
`runUpdateCheck` cadence, reusing `updates.go`'s `detectPM()`.
|
||||
|
||||
```protobuf
|
||||
rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse);
|
||||
|
||||
message ReportPackagesRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3; // sha256 of the sorted list
|
||||
OSRelease os = 4;
|
||||
repeated InstalledPackage packages = 5; // omitted when only offering a hash
|
||||
}
|
||||
|
||||
message ReportPackagesResponse {
|
||||
bool need_full = 1; // hash differs; resend with packages populated
|
||||
}
|
||||
```
|
||||
|
||||
The agent calls once with `packages` empty. `need_full` true means the hash
|
||||
differs from what the server holds, and the agent immediately calls again with
|
||||
the list populated.
|
||||
|
||||
The agent sends a SHA-256 of its sorted package list first. If it matches what
|
||||
the server already holds, the server answers `unchanged` and the ~150KB body is
|
||||
never sent. A machine's package set changes rarely, so almost every hour costs
|
||||
one small message, and the rare changed hour costs one extra round trip.
|
||||
|
||||
Folding the list into the existing 15-minute `InventoryReport` static snapshot
|
||||
was rejected: it would re-send ~150KB per server every 15 minutes regardless of
|
||||
change, roughly 40MB/hour of gRPC traffic on a 100-server fleet to transmit
|
||||
data that is almost always identical.
|
||||
|
||||
---
|
||||
|
||||
## Data model
|
||||
|
||||
Four new collections. Every one carries `instance_id` except `vulndb_meta`,
|
||||
which is explained below.
|
||||
|
||||
### `server_packages` — one document per server, not per package
|
||||
|
||||
```go
|
||||
type ServerPackages struct {
|
||||
ID primitive.ObjectID `bson:"_id"`
|
||||
InstanceID primitive.ObjectID `bson:"instance_id"`
|
||||
ServerID string `bson:"server_id"`
|
||||
OS OSRelease `bson:"os"` // family, version_id, arch
|
||||
Hash string `bson:"hash"` // sha256 of the sorted list
|
||||
Packages []InstalledPackage `bson:"packages"`
|
||||
CollectedAt time.Time `bson:"collected_at"`
|
||||
ScanPending bool `bson:"scan_pending"`
|
||||
ScannedAt time.Time `bson:"scanned_at"`
|
||||
Status string `bson:"status"` // ok | unsupported
|
||||
DBVersion int `bson:"db_version"` // last matched against
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `bson:"name"`
|
||||
Version string `bson:"version"` // distro version string, verbatim
|
||||
Epoch int `bson:"epoch,omitempty"`
|
||||
Arch string `bson:"arch"`
|
||||
SourceName string `bson:"source_name,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
One document rather than two thousand is what makes a report a **single atomic
|
||||
upsert with no delta logic** — the hash already established that something
|
||||
changed, so there is nothing to reconcile field by field. A typical Linux host
|
||||
lands near 150KB, comfortably inside the 16MB document limit.
|
||||
|
||||
Indexes: `{instance_id, server_id}` unique, and a multikey
|
||||
`{instance_id, "packages.name"}` for fleet-wide package search.
|
||||
|
||||
`SourceName` is not decoration. **Debian and Ubuntu advisories are keyed on the
|
||||
source package**: a CVE against `openssl` covers the binaries `libssl3`,
|
||||
`openssl` and `libssl-dev`, so matching on binary name alone misses two of the
|
||||
three.
|
||||
|
||||
`OS.VersionID` selects the feed. Ubuntu 22.04 and 24.04 publish different fixed
|
||||
versions for the same CVE, so a scan without it is guesswork.
|
||||
|
||||
### `vuln_findings` — one document per (server, CVE, package)
|
||||
|
||||
```go
|
||||
type VulnFinding struct {
|
||||
ID primitive.ObjectID `bson:"_id"`
|
||||
InstanceID primitive.ObjectID `bson:"instance_id"`
|
||||
ServerID string `bson:"server_id"`
|
||||
|
||||
CVEID string `bson:"cve_id"`
|
||||
PackageName string `bson:"package_name"`
|
||||
Installed string `bson:"installed_version"`
|
||||
FixedIn string `bson:"fixed_in,omitempty"`
|
||||
Severity string `bson:"severity"`
|
||||
CVSSScore float64 `bson:"cvss_score,omitempty"`
|
||||
Title string `bson:"title,omitempty"`
|
||||
References []string `bson:"references,omitempty"`
|
||||
|
||||
State string `bson:"state"` // open | fixed | accepted
|
||||
FirstSeen time.Time `bson:"first_seen"`
|
||||
LastSeen time.Time `bson:"last_seen"`
|
||||
FixedAt *time.Time `bson:"fixed_at,omitempty"`
|
||||
Accepted *Acceptance `bson:"accepted,omitempty"`
|
||||
}
|
||||
|
||||
type Acceptance struct {
|
||||
By primitive.ObjectID `bson:"by"`
|
||||
Reason string `bson:"reason"`
|
||||
Until time.Time `bson:"until"`
|
||||
At time.Time `bson:"at"`
|
||||
}
|
||||
```
|
||||
|
||||
Unique on `{instance_id, server_id, cve_id, package_name}`. That key is what
|
||||
makes a rescan an idempotent upsert rather than a duplicate factory, and it is
|
||||
what lets `first_seen` survive across scans. Query index
|
||||
`{instance_id, state, severity}`.
|
||||
|
||||
**An empty `FixedIn` is a real and common state** and must never be conflated
|
||||
with "not vulnerable". A CVE with no vendor fix published yet is exactly the
|
||||
finding people most need to see, and also the one that most needs acceptance,
|
||||
because there is nothing to patch.
|
||||
|
||||
Findings are **not deleted when a package is patched**. State moves to `fixed`
|
||||
with `fixed_at` set, so "what did we remediate last quarter" remains
|
||||
answerable — which is the question an auditor asks.
|
||||
|
||||
### `vulndb_meta` — singleton, deliberately unscoped
|
||||
|
||||
`db_version`, `pulled_at`, `last_full_scan_at`, `last_error`. It carries no
|
||||
`instance_id` because the vulnerability database is a property of the
|
||||
deployment, not of a tenant. Same reasoning as `migrations`.
|
||||
|
||||
### `vuln_alert_rules`
|
||||
|
||||
`instance_id`, `name`, `enabled`, `min_severity`, `tags map[string]string`,
|
||||
`channel_ids []`, timestamps.
|
||||
|
||||
The tag filter resolves through **`services.ResolveTargets`**, not a second
|
||||
matcher. That function is already the single answer to which servers a
|
||||
selector touches, and an alert rule that disagreed with a workflow about what
|
||||
`env:prod` means would be worse than having no filter at all.
|
||||
|
||||
---
|
||||
|
||||
## The matching engine
|
||||
|
||||
```
|
||||
server/internal/vulndb/
|
||||
pull.go OCI fetch → temp dir, version compare against vulndb_meta
|
||||
db.go BoltDB open, advisory lookup by (ecosystem, source, version)
|
||||
match.go per-family matching, severity resolution
|
||||
version.go dispatch to deb/rpm/apk comparator by OS family
|
||||
```
|
||||
|
||||
Dependencies: `github.com/aquasecurity/trivy-db` for the BoltDB schema, plus
|
||||
`go-deb-version`, `go-rpm-version` and `go-apk-version` — each a small
|
||||
standalone module doing one job. The roughly 200 lines of per-distro advisory
|
||||
lookup are ours.
|
||||
|
||||
Importing `trivy` itself was rejected: it would pull a very large transitive
|
||||
dependency tree into the server binary for one feature, and its Go API carries
|
||||
no stability guarantee across minor versions. Shelling out to the `trivy`
|
||||
binary against a generated SBOM was rejected for shipping a second binary in
|
||||
the image and turning a library call into subprocess lifecycle, timeouts and
|
||||
output-format drift.
|
||||
|
||||
### Why the comparators are bought rather than written
|
||||
|
||||
Version ordering is where this feature lives or dies, and its failure mode is
|
||||
silent. `dpkg` ordering has epochs, and `~` sorts *before* the empty string, so
|
||||
`3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own
|
||||
segment rules and treats `~` and `^` differently again. A `strings.Compare` or
|
||||
a semver parse orders `1.9` above `1.10` and reports a vulnerable fleet as
|
||||
clean — a false negative, which nobody notices until it matters.
|
||||
|
||||
### Scanning one server
|
||||
|
||||
1. Load `server_packages`; resolve OS family and version to a `trivy-db`
|
||||
ecosystem.
|
||||
2. **Unsupported ecosystem → record `status: unsupported`, clear the flag,
|
||||
write no findings.**
|
||||
3. For each package: resolve source name, look up advisories, compare versions.
|
||||
4. Upsert vulnerable results as `open`, preserving `first_seen`.
|
||||
5. Any currently-`open` finding absent from this result set → `fixed`, stamp
|
||||
`fixed_at`.
|
||||
6. Any `accepted` finding past its `until` → back to `open`.
|
||||
7. Clear `scan_pending`, stamp `scanned_at` and `db_version`.
|
||||
|
||||
Steps 5 and 6 must run in that order, so a finding that is both absent and
|
||||
expired settles as `fixed` rather than reopening on a package that no longer
|
||||
carries it.
|
||||
|
||||
Step 2 matters as much as any of the matching. Arch has no feed in `trivy-db`,
|
||||
so an Arch host must report **unsupported**, never "0 findings". Reporting
|
||||
clean when the truth is unknown is the same class of lie as a silently stale
|
||||
database, and it is the reason `vulndb_meta.pulled_at` appears on screen rather
|
||||
than only in a log.
|
||||
|
||||
### Severity
|
||||
|
||||
Resolved **vendor → NVD → unknown**, in that order, never invented.
|
||||
|
||||
This will surface as "why is this critical CVE marked low", and the answer is
|
||||
that Debian and Red Hat routinely downgrade an NVD score because the vulnerable
|
||||
code path is not reachable in their build. Their rating is the accurate one for
|
||||
that package, and showing NVD's above it would manufacture work that does not
|
||||
need doing.
|
||||
|
||||
---
|
||||
|
||||
## Findings lifecycle
|
||||
|
||||
`open | fixed | accepted`.
|
||||
|
||||
An accepted finding is suppressed from counts and alerts until its `until`
|
||||
date, then reopens automatically. A reason is required.
|
||||
|
||||
Acceptance with a mandatory expiry, rather than permanent dismissal, is what
|
||||
keeps the feature usable in both directions. Without any acceptance mechanism,
|
||||
a kernel CVE awaiting a reboot window sits red indefinitely and trains people
|
||||
to ignore the page. With permanent dismissal, accepted findings accumulate
|
||||
silently and nobody revisits them — the dismissal list becomes where risk goes
|
||||
to be forgotten, which is precisely what an auditor asks to see.
|
||||
|
||||
Retention: `settings.vuln_finding_retention_days`, a `*int` on the same pattern
|
||||
as `workflow_log_retention_days` — nil means 90 days, 0 means forever. Only
|
||||
`fixed` findings are swept, by a `StartVulnSweeper` inside the same
|
||||
`RunAsLeader("housekeeping", …)` as the existing sweepers. `open` and
|
||||
`accepted` findings are never swept at any setting.
|
||||
|
||||
---
|
||||
|
||||
## Alerting
|
||||
|
||||
Per-org rules over the existing `notification_channels`: severity threshold,
|
||||
optional tag filter, target channels.
|
||||
|
||||
A rescan emits one message summarising what newly opened — "12 new critical
|
||||
across 4 servers" — never one message per finding. See the leader section for
|
||||
why the tick boundary makes this structural.
|
||||
|
||||
Modelling findings as a monitor type was rejected. It would reuse monitors'
|
||||
state machine and channel wiring for free, but monitors are up/down for one
|
||||
endpoint with retries and hourly rollups, none of which means anything for a
|
||||
CVE; most fields would be disabled in the UI and the uptime graphs would be
|
||||
polluted with a signal that is not uptime.
|
||||
|
||||
This adds one `notify` payload type and a `vuln_digest.html.tmpl` /
|
||||
`vuln_digest.txt.tmpl` pair in `shared/mail`. Note that `shared/mail` templates
|
||||
are parsed in `init()`, so a mistyped field is a boot-time panic — CLAUDE.md
|
||||
describes a `render_test.go` guarding against exactly this, but **that file does
|
||||
not exist**; the repository has no Go tests at all, and by instruction this
|
||||
feature adds none. The template pair must therefore be verified by starting the
|
||||
binary and sending one digest through a real channel.
|
||||
|
||||
---
|
||||
|
||||
## Entitlement
|
||||
|
||||
The feature name is `vuln_scanning`, and it crosses the two services the way
|
||||
every other feature does:
|
||||
|
||||
- **admin** carries it as a per-instance entitlement toggle, so it can later be
|
||||
priced as a catalogue `feature` component without a second migration;
|
||||
- **the licence** snapshots it into `License.Features []string` at issue time;
|
||||
- **the server** asks `lic.HasFeature("vuln_scanning")` and never switches on
|
||||
tier, so changing what a tier includes needs no server release.
|
||||
|
||||
Off on Free.
|
||||
|
||||
**The gate is checked at `ReportPackages`, not at display.** Gating only the UI
|
||||
would still pay every write cost, and storage is the expensive half.
|
||||
|
||||
The agent learns of it through the existing 30-second `SyncKeys` poll:
|
||||
`SyncResponse` gains a `collect_packages` bool, and the hourly loop skips
|
||||
collection entirely when it is false. So an ungated instance produces no
|
||||
collection, no gRPC body, no document and no storage. `ReportPackages` still
|
||||
re-checks the entitlement server-side and refuses — the agent flag is an
|
||||
optimisation, the server check is the boundary.
|
||||
|
||||
Turning the feature off does not delete existing findings; they stop being
|
||||
served and stop updating. Deletion is the instance-deletion path's job.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
```
|
||||
GET /api/vulnerabilities # filter: severity, state, server, tags
|
||||
GET /api/vulnerabilities/summary # severity counts + database freshness
|
||||
POST /api/vulnerabilities/rescan # marks all scan_pending (owner|admin)
|
||||
POST /api/vulnerabilities/:id/accept # reason + until (owner|admin)
|
||||
DELETE /api/vulnerabilities/:id/accept # (owner|admin)
|
||||
GET /api/servers/:id/vulnerabilities
|
||||
GET /api/servers/:id/packages
|
||||
GET /api/packages/search?name= # fleet-wide
|
||||
GET,POST /api/vuln-rules · PUT,DELETE /api/vuln-rules/:id
|
||||
```
|
||||
|
||||
Every mutating path writes an audit event, as all of them do. Acceptance is the
|
||||
one decision people will be asked to justify, so `by`, `reason`, `until` and
|
||||
`at` land in the audit record and not only on the document.
|
||||
|
||||
---
|
||||
|
||||
## UI
|
||||
|
||||
`/vulnerabilities` is a fleet board **grouped by CVE** — one row per CVE with
|
||||
an affected-server count, expandable to the individual servers. The same CVE
|
||||
across 40 servers is one decision, and a flat list of findings makes it look
|
||||
like forty.
|
||||
|
||||
Server detail gains **Vulnerabilities** and **Packages** tabs. Alert rules go
|
||||
on `/settings/notifications`, beside the channels they consume.
|
||||
|
||||
Remediation introduces no new mechanism: a finding carrying `fixed_in` renders
|
||||
an **Apply updates** action calling the existing
|
||||
`POST /api/servers/:id/apply-updates`, which is already `ApplyUpdatesCmd`. See
|
||||
it, patch it, one place — and no second patching path to keep consistent with
|
||||
the first.
|
||||
|
||||
Database freshness is shown wherever findings are, not tucked into settings. A
|
||||
fleet scanning against a three-week-old database must say so rather than
|
||||
quietly report all-clear.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
**No automated tests.** The repository has none today, and by explicit
|
||||
instruction this feature adds none — no `*_test.go`, no frontend test files.
|
||||
That is a deliberate decision by the repository owner, recorded here so the
|
||||
absence reads as a choice rather than an omission.
|
||||
|
||||
It does change the risk profile, and the places it changes it are worth naming,
|
||||
because each fails by producing a **wrong answer rather than a crash**:
|
||||
|
||||
- **Version comparison.** The backport case — installed `1:3.0.2-0ubuntu1.15`
|
||||
against advisory fixed-in `1:3.0.2-0ubuntu1.15` resolving to *not
|
||||
vulnerable* — plus tilde ordering (`1.0~rc1` < `1.0`), epoch dominance
|
||||
(`1:1.0` > `2.0`) and `1.9` < `1.10`. Wrong here means a vulnerable fleet
|
||||
reported clean.
|
||||
- **Source-package fan-out.** One advisory against `openssl` must flag
|
||||
`libssl3`, `openssl` and `libssl-dev`. Matching on binary name alone silently
|
||||
finds one of three.
|
||||
- **`first_seen` preservation.** An upsert that overwrites it makes every
|
||||
finding look discovered today, and nothing surfaces that until someone reads
|
||||
a report.
|
||||
- **Fixed-before-reopen ordering.** A finding both absent from a scan and past
|
||||
its acceptance expiry must settle `fixed`, not reopen.
|
||||
|
||||
The implementation plan carries a manual verification table for each, to be
|
||||
walked before the relevant task is committed. They are the substitute for the
|
||||
tests, not a formality.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Behaviour |
|
||||
| ------- | --------- |
|
||||
| Database pull fails | Keep the last good copy and serve stale. Record `last_error`, surface `pulled_at` age. **Never clear findings** — a network blip must not read as "all fixed" |
|
||||
| Unsupported distribution | `status: unsupported`, not zero findings |
|
||||
| Agent stops reporting | Findings persist and `collected_at` age is shown. No auto-expiry: a silent agent is not a patched server |
|
||||
| `trivy-db` schema version bumps | The puller refuses an unknown schema rather than mis-parsing it |
|
||||
| ghcr anonymous rate limit | Backoff; `VANTAGE_TRIVY_DB_REF` mirrors to a private registry |
|
||||
| Leadership lost mid-scan | The context is cancelled and the scan returns; `scan_pending` is still set, so the next leader picks it up |
|
||||
| Instance deleted | **`server_packages` and `vuln_findings` must be added to the control plane's instance-deletion collection list.** Easy to miss, and missing it orphans a tenant's package data indefinitely |
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Name | Required | Notes |
|
||||
| ---- | -------- | ----- |
|
||||
| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for air-gapped installs or to avoid the anonymous ghcr rate limit |
|
||||
| `VANTAGE_VULNDB_DISABLED` | no | disables the puller and the scheduler entirely. Findings already written are still served and still marked stale |
|
||||
|
||||
---
|
||||
|
||||
## Deliberately out of scope
|
||||
|
||||
- **Windows.** Separate source, collector and matcher; its own spec.
|
||||
- **Container image scanning.** Sub-project C; needs the container registry.
|
||||
- **Compliance baseline assertions.** Sub-project D; shares this findings UI
|
||||
and nothing else.
|
||||
- **Language-level dependency scanning** (npm, pip, Go modules). `trivy-db`
|
||||
covers these ecosystems, but finding the manifests on a host is a different
|
||||
collection problem from asking the package manager what is installed.
|
||||
- **Automatic patching on a finding.** Remediation is one click, not zero. An
|
||||
unattended upgrade triggered by a CVE feed is a fleet-wide change driven by a
|
||||
third party's data, which is not a decision to take away from an operator.
|
||||
@@ -0,0 +1,441 @@
|
||||
# Workload registry
|
||||
|
||||
Date: 2026-08-06
|
||||
|
||||
Agents enumerate what each server actually runs — Docker containers, the
|
||||
compose stacks grouping them, and systemd services — and report it to the
|
||||
control plane. Containers and units can be started, stopped and restarted from
|
||||
the UI, and a bounded snapshot of their logs can be read without opening a
|
||||
console.
|
||||
|
||||
This is **sub-project B** of the four sketched in
|
||||
`2026-08-06-package-inventory-and-cve-findings-design.md`:
|
||||
|
||||
| # | Sub-project | Depends on |
|
||||
| - | ----------- | ---------- |
|
||||
| A | Package inventory + CVE findings — its own spec | nothing |
|
||||
| B | **Workload registry** — this spec | nothing |
|
||||
| C | Container image scanning | A and B |
|
||||
| D | Compliance profiles | shares A's findings UI only |
|
||||
|
||||
A and B are independent. C is the joiner and must not be designed before both
|
||||
exist: it needs B's image list and A's findings model.
|
||||
|
||||
**Workload** is the domain word throughout: one container or one systemd unit.
|
||||
It gives the collection, the commands and the page a single honest name rather
|
||||
than saying "container or service" in every identifier.
|
||||
|
||||
Scope is **Linux only**, matching sub-project A and the existing position that
|
||||
Windows agents are second-class by design. Docker runs on Windows; systemd does
|
||||
not, and half a feature per platform is worse than a clear line.
|
||||
|
||||
---
|
||||
|
||||
## What this is for
|
||||
|
||||
The control plane can manage a fleet's keys, run workflows across it and watch
|
||||
its endpoints, but it has no idea what any of those servers actually *runs*.
|
||||
"Restart nginx on that box" means opening a console. "Which of these 80 servers
|
||||
is still on the old image" is unanswerable.
|
||||
|
||||
---
|
||||
|
||||
## Reporting and refresh are one path
|
||||
|
||||
The agent reports on its own 60-second ticker through a `ReportWorkloads` RPC,
|
||||
using the same hash short-circuit as the package report: it offers a SHA-256 of
|
||||
the sorted workload list, and sends the body only when the server does not
|
||||
already hold that hash. An unchanged list costs one small message, which on a
|
||||
60-second cadence is the common case by a wide margin.
|
||||
|
||||
The on-demand refresh **does not return data**. `RefreshWorkloadsCmd` carries
|
||||
no payload back; it makes the agent report immediately through the normal RPC,
|
||||
and the UI refetches the stored document.
|
||||
|
||||
That is deliberate. A refresh that returned workloads inline would be a second
|
||||
writer for the same collection, arriving by a different route, with its own
|
||||
serialisation and its own opportunity to disagree with the periodic one. One
|
||||
writer, one shape; the refresh is a nudge, not a channel.
|
||||
|
||||
Opening a server's Workloads tab dispatches a refresh, so what is on screen is
|
||||
live rather than up to a minute stale. That matters because the page has a
|
||||
Restart button on it: a stale list is not merely a wrong impression, it is a
|
||||
wrong action aimed at a container that already died.
|
||||
|
||||
## What does answer back
|
||||
|
||||
Two operations genuinely return something:
|
||||
|
||||
| Command | Answers with |
|
||||
| ------- | ------------ |
|
||||
| `ControlWorkloadCmd{kind, id, action}` | the existing `CommandResult` — ok or error |
|
||||
| `WorkloadLogsCmd{kind, id, tail}` | a new `WorkloadLogsResult{command_id, text, truncated}` |
|
||||
|
||||
Both ride the proven path: `commandDispatcher.send()` for request and ack, and
|
||||
a `WorkloadResults` registry mirroring `StepResults.Await`/`Deliver` over the
|
||||
bus. **`Await` must subscribe before the command is dispatched** — the pod
|
||||
driving the request is usually not the pod holding the agent's stream, and a
|
||||
fast agent otherwise answers into a channel nobody has joined. This is not a
|
||||
new hazard; it is the one `stepresults.go` already documents.
|
||||
|
||||
```protobuf
|
||||
rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse);
|
||||
|
||||
message ReportWorkloadsRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3;
|
||||
bool docker_ok = 4;
|
||||
string docker_error = 5;
|
||||
bool systemd_ok = 6;
|
||||
string systemd_error = 7;
|
||||
repeated Workload workloads = 8; // empty on the offer call
|
||||
}
|
||||
|
||||
message ReportWorkloadsResponse {
|
||||
bool need_full = 1;
|
||||
}
|
||||
|
||||
// ServerCommand gains three variants.
|
||||
message RefreshWorkloadsCmd {}
|
||||
|
||||
message ControlWorkloadCmd {
|
||||
string kind = 1; // "container" | "unit"
|
||||
string id = 2;
|
||||
string action = 3; // "start" | "stop" | "restart"
|
||||
}
|
||||
|
||||
message WorkloadLogsCmd {
|
||||
string kind = 1;
|
||||
string id = 2;
|
||||
int32 tail = 3;
|
||||
}
|
||||
|
||||
// AgentMessage gains one variant.
|
||||
message WorkloadLogsResult {
|
||||
string command_id = 1;
|
||||
string text = 2;
|
||||
bool truncated = 3;
|
||||
string error = 4;
|
||||
}
|
||||
```
|
||||
|
||||
The offer-then-send handshake is the package report's, unchanged: the agent
|
||||
calls once with `workloads` empty, and resends with the body only if the
|
||||
response sets `need_full`.
|
||||
|
||||
An agent whose stream no pod holds gets a 503 from the dispatcher, as
|
||||
everything else does. Commands are not queued: a command whose owner died must
|
||||
fail loudly rather than be delivered to nobody while the operator is told it
|
||||
worked.
|
||||
|
||||
---
|
||||
|
||||
## Not gated by licence
|
||||
|
||||
Unlike CVE scanning, this reads as core fleet management rather than a premium
|
||||
add-on, so v1 ships to every instance with no entitlement check.
|
||||
|
||||
If that changes it is a one-line `HasFeature` check at `ReportWorkloads`,
|
||||
gating collection rather than display — the same placement and the same
|
||||
reasoning as sub-project A, where gating the UI alone would still pay every
|
||||
write cost.
|
||||
|
||||
---
|
||||
|
||||
## Data model
|
||||
|
||||
One new collection, `server_workloads`, one document per server, mirroring
|
||||
`server_packages`.
|
||||
|
||||
```go
|
||||
type ServerWorkloads struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
Hash string `bson:"hash" json:"hash"`
|
||||
Workloads []Workload `bson:"workloads" json:"workloads"`
|
||||
CollectedAt time.Time `bson:"collected_at" json:"collected_at"`
|
||||
|
||||
DockerOK bool `bson:"docker_ok" json:"docker_ok"`
|
||||
DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"`
|
||||
SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"`
|
||||
SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"`
|
||||
}
|
||||
|
||||
type Workload struct {
|
||||
Kind string `bson:"kind" json:"kind"` // "container" | "unit"
|
||||
ID string `bson:"id" json:"id"` // container id, or unit name
|
||||
Name string `bson:"name" json:"name"`
|
||||
State string `bson:"state" json:"state"`
|
||||
Health string `bson:"health,omitempty" json:"health,omitempty"`
|
||||
Image string `bson:"image,omitempty" json:"image,omitempty"`
|
||||
Stack string `bson:"stack,omitempty" json:"stack,omitempty"`
|
||||
Ports []string `bson:"ports,omitempty" json:"ports,omitempty"`
|
||||
Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"`
|
||||
StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
|
||||
Protected bool `bson:"protected" json:"protected"`
|
||||
}
|
||||
```
|
||||
|
||||
`State` is normalised across the two kinds: containers report `running`,
|
||||
`exited`, `paused`, `restarting`, `created`; units report `active`, `inactive`,
|
||||
`failed`, `activating`. They are deliberately **not** collapsed into a shared
|
||||
vocabulary — a failed unit and an exited container mean different things, and
|
||||
flattening them would lose the distinction the operator needs.
|
||||
|
||||
Indexes: `{instance_id, server_id}` unique, plus multikey
|
||||
`{instance_id, "workloads.image"}` for the fleet-wide "which servers run image
|
||||
X" query.
|
||||
|
||||
### Why the OK/Error pairs exist
|
||||
|
||||
A host with no Docker installed and a host where Docker is installed and
|
||||
running nothing both produce an empty list. One should read "not in use here",
|
||||
the other "nothing running", and only the second deserves any alarm.
|
||||
|
||||
The error strings separate a third case the booleans alone cannot: Docker
|
||||
installed with the daemon down. "Not installed" and "installed but not
|
||||
responding" are different problems with different fixes, and collapsing them
|
||||
into one false boolean throws away the only thing that tells them apart.
|
||||
|
||||
### Why `Protected` is reported rather than derived
|
||||
|
||||
The agent already knows which unit and container it is. Sending that up lets
|
||||
the UI render the action disabled with a reason instead of offering a button
|
||||
whose refusal is already known.
|
||||
|
||||
The field is the courtesy; the agent's own check is the boundary. See the
|
||||
control section.
|
||||
|
||||
### No history
|
||||
|
||||
A workload list is state, not a record. Nobody asks what containers ran last
|
||||
Tuesday, and keeping it would grow a collection per server per minute in
|
||||
exchange for a question nobody has.
|
||||
|
||||
---
|
||||
|
||||
## Collectors
|
||||
|
||||
### Docker: two commands, no English parsing
|
||||
|
||||
```
|
||||
docker ps -aq
|
||||
docker inspect --format '{{json .}}' <ids…>
|
||||
```
|
||||
|
||||
Not `docker ps --format '{{json .}}'` alone. That reports health and uptime
|
||||
inside a human `Status` string — `"Up 2 hours (healthy)"` — and anything built
|
||||
on it is parsing English that is localised, reworded between releases, and
|
||||
silently different for a paused or restarting container. `inspect` returns
|
||||
`State.Health.Status`, `State.StartedAt` and `RestartCount` as typed fields.
|
||||
Two execs instead of one, and no parser to be wrong.
|
||||
|
||||
`RestartCount` justifies the second call by itself: a container cycling is the
|
||||
single thing this page most needs to show, and it is invisible in a list that
|
||||
only ever says "Up".
|
||||
|
||||
Compose stacks come from the `com.docker.compose.project` label. **No YAML is
|
||||
read from disk** — the label is what Docker itself treats as authoritative, and
|
||||
a compose file on disk may not be what is actually running.
|
||||
|
||||
Docker absent, or a socket that cannot be reached, sets `DockerOK: false`. It
|
||||
is not an error and produces no log line: most servers in a fleet built around
|
||||
SSH key management will not have Docker, and treating the normal case as a
|
||||
fault makes the feature look broken on the majority of the estate.
|
||||
|
||||
### systemd: filtered on purpose
|
||||
|
||||
```
|
||||
systemctl list-units --type=service --state=running,failed --no-legend --plain --no-pager
|
||||
systemctl list-unit-files --type=service --state=enabled --no-legend --plain --no-pager
|
||||
```
|
||||
|
||||
Two calls because "running or failed" and "enabled but stopped" are different
|
||||
questions, and an enabled unit that is not running is exactly the one worth
|
||||
seeing.
|
||||
|
||||
Excluded by prefix: `systemd-`, `user@`, `session-`, `init.scope`. A typical
|
||||
host carries 300+ units, the platform's own accounting for most of them.
|
||||
Listing all of them buries the ten anyone cares about — the same failure mode
|
||||
as an unfiltered vulnerability report, and the same fix.
|
||||
|
||||
Column output rather than `--output=json`: the JSON flag requires systemd 246+,
|
||||
and this fleet includes older stable distributions. The column format has been
|
||||
stable considerably longer than the JSON one has existed.
|
||||
|
||||
---
|
||||
|
||||
## Control actions
|
||||
|
||||
```
|
||||
container: docker {start|stop|restart} <id>
|
||||
unit: systemctl {start|stop|restart} <unit>
|
||||
```
|
||||
|
||||
Owner or admin only. Every action writes an audit event naming the actor, the
|
||||
server and the target.
|
||||
|
||||
### The protected set
|
||||
|
||||
Computed agent-side: `vantage-agent.service`, plus the container ID read from
|
||||
`/proc/self/cgroup` should the agent ever be run inside a container.
|
||||
|
||||
The agent refuses those before doing anything. As with the console relay
|
||||
hardcoding `127.0.0.1` agent-side, **the control plane may name a target, but
|
||||
the agent decides what it will do to itself**. A server-side denylist alone
|
||||
would be bypassed by the next dispatch path someone adds, and the failure is
|
||||
unrecoverable from the UI: a server that stops its own agent goes offline, and
|
||||
the way back is SSH or physical access — precisely what this feature exists to
|
||||
avoid needing.
|
||||
|
||||
### Timeouts
|
||||
|
||||
`docker stop` waits on a container that may ignore SIGTERM. `systemctl stop`
|
||||
on a unit with a long `TimeoutStopSec` blocks for exactly as long as that says.
|
||||
Both run under a 90-second context, and a timeout returns a real error rather
|
||||
than an ack implying success.
|
||||
|
||||
---
|
||||
|
||||
## Logs
|
||||
|
||||
```
|
||||
container: docker logs --tail 500 --timestamps <id>
|
||||
unit: journalctl -u <unit> -n 500 --no-pager --output=short-iso
|
||||
```
|
||||
|
||||
Capped at **500 lines and 256KB, whichever binds first**, with `truncated` set
|
||||
so the UI can say so. Two caps because 500 lines of a container emitting 4KB
|
||||
JSON blobs is 2MB, and a line count alone does not stop it — the same reasoning
|
||||
that gave workflow logs both a per-line and a per-run cap.
|
||||
|
||||
Live following is deliberately absent. The browser console already offers a
|
||||
real terminal on the same server, where `docker logs -f` works properly with
|
||||
its own scrollback and cancellation. Building a second streaming path — a
|
||||
relay listener, proxy bus keys, a WebSocket upgrade and a cancellation story
|
||||
for a follow nobody closed — to duplicate that would be a large amount of
|
||||
machinery aimed at a capability already shipped. A bounded snapshot answers
|
||||
"why did this restart", which is the question that sends people to the console
|
||||
in the first place.
|
||||
|
||||
### Log reads are owner or admin only, and audited
|
||||
|
||||
Unlike workflow logs, these cannot be masked. A workflow's logs can be masked
|
||||
because the run injected the secrets and therefore knows their values. A
|
||||
container's stdout is arbitrary and may contain credentials nobody declared —
|
||||
a connection string in a startup banner, a token in a stack trace.
|
||||
|
||||
So log reads sit behind the same role check as control actions and are audited.
|
||||
A member who can see the fleet cannot read its logs. This is a deliberate
|
||||
access decision, not an oversight, and it is why log reading is not simply
|
||||
folded in with the read-only snapshot endpoints.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
```
|
||||
GET /api/servers/:id/workloads # stored snapshot
|
||||
POST /api/servers/:id/workloads/refresh # dispatch, then refetch
|
||||
POST /api/servers/:id/workloads/:wid/action # {"action":"start|stop|restart"} (owner|admin)
|
||||
GET /api/servers/:id/workloads/:wid/logs?tail= # (owner|admin)
|
||||
GET /api/workloads?image=&stack=&state= # fleet-wide
|
||||
```
|
||||
|
||||
`:wid` is a container ID or a unit name, URL-encoded. Unit names carry dots and
|
||||
`@`, which are legal in a path segment but not worth relying on unencoded.
|
||||
|
||||
`tail` is clamped to the 500-line cap server-side; a client asking for more
|
||||
gets 500, not an error.
|
||||
|
||||
---
|
||||
|
||||
## UI
|
||||
|
||||
Server detail gains a **Workloads** tab, ordered compose stacks first — grouped
|
||||
under the stack name — then loose containers, then units.
|
||||
|
||||
That ordering is not cosmetic. A stack is one thing to an operator even when it
|
||||
is six containers, and a flat list turns one decision into six rows. It is the
|
||||
same argument that groups the vulnerabilities board by CVE rather than by
|
||||
finding.
|
||||
|
||||
A `/workloads` fleet view answers "which servers run image X", which is the
|
||||
reason the snapshot is stored at all rather than fetched on demand and
|
||||
discarded.
|
||||
|
||||
Three rules that follow directly from the model:
|
||||
|
||||
- **Protected rows render their actions disabled, with the reason**, rather
|
||||
than offering a button whose refusal is already known.
|
||||
- **`DockerOK: false` reads "Docker not in use on this server"**, never an
|
||||
empty list, and `DockerError` when present is shown as a distinct problem.
|
||||
- State never reads by colour alone: every pill carries a distinct shape and a
|
||||
text label, matching the existing monitor and severity pills.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
**No automated tests.** The repository has none today, and by explicit
|
||||
instruction this feature adds none — no `*_test.go`, no frontend test files.
|
||||
A deliberate decision by the repository owner, recorded so the absence reads as
|
||||
a choice rather than an omission.
|
||||
|
||||
The behaviours that would otherwise have been tested are the ones that fail
|
||||
quietly, and the implementation plan carries a manual check for each:
|
||||
|
||||
- **Parser output against real command output.** `docker inspect` must yield
|
||||
`RestartCount`, health and the compose label as `Stack`; the `systemctl`
|
||||
exclusion filter must drop `systemd-*` and `user@*` while keeping
|
||||
`nginx.service`. Both are verified against a live host rather than a fixture.
|
||||
- **Protected-set computation.** `vantage-agent.service` marked,
|
||||
`nginx.service` not. Getting this wrong in the permissive direction lets a
|
||||
server stop its own agent, which is unrecoverable from the UI.
|
||||
- **Hash order-independence.** An ordering-sensitive hash resends the full list
|
||||
every 60 seconds, which is invisible except as traffic.
|
||||
- **Log capping in both directions.** 600 lines in → 500 out with `truncated`;
|
||||
a 300KB blob of fewer than 500 lines → capped, `truncated`. The second is the
|
||||
case a line-count-only implementation silently fails, and it fails by sending
|
||||
megabytes rather than by erroring.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Behaviour |
|
||||
| ------- | --------- |
|
||||
| Docker not installed | `DockerOK: false`, no error, UI reads "not in use" |
|
||||
| Docker installed, daemon down | `DockerOK: false` **plus** `DockerError` — different message, different fix |
|
||||
| Agent offline | 503 from the existing dispatcher. No queueing: a command whose owner died must fail loudly |
|
||||
| Action on a protected workload | Agent refuses; API answers 409 naming the reason |
|
||||
| `stop` exceeds its timeout | Real error surfaced, never a hopeful ack. Snapshot refreshed afterwards |
|
||||
| Container removed between snapshot and action | Docker's "No such container" surfaced and a refresh dispatched — this is what on-demand refresh is for |
|
||||
| Log exceeds either cap | Truncated, flagged, and stated in the UI |
|
||||
| Instance deleted | **`server_workloads` must be added to the control plane's instance-deletion collection list**, alongside sub-project A's two collections |
|
||||
|
||||
---
|
||||
|
||||
## Deliberately out of scope
|
||||
|
||||
- **Live log following.** The console already does it. See the logs section.
|
||||
- **Creating, deleting or updating containers and units.** This is a control
|
||||
and visibility surface, not a deployment tool — workflows already exist for
|
||||
changing what a server runs, with snapshots, audit and rollback.
|
||||
- **`docker exec` into a container.** The console reaches the host; exec from
|
||||
the control plane is a second remote-execution path with its own audit and
|
||||
authorisation story, and it belongs in its own spec if anywhere.
|
||||
- **Kubernetes and containerd.** The Docker collector shells to the `docker`
|
||||
CLI, so a node whose runtime is containerd or CRI-O reports nothing from it —
|
||||
`DockerOK: false`, correctly, since Docker genuinely is not in use. Covering
|
||||
those runtimes means a `crictl`/`nerdctl` collector, and talking to a
|
||||
Kubernetes API server is a different subsystem again. Neither is v1.
|
||||
- **Podman as a supported runtime.** Its `docker`-compatible CLI means an
|
||||
aliased install will largely work, and that is a happy accident rather than a
|
||||
claim: nothing here is tested against Podman and its `RestartCount` and
|
||||
compose-label behaviour are not verified.
|
||||
- **Windows.** No systemd, and a different container story.
|
||||
- **Image vulnerability scanning.** Sub-project C, which needs this spec's
|
||||
image list and sub-project A's findings model.
|
||||
@@ -10,67 +10,54 @@ HQ portal.
|
||||
|
||||
## What a licence is
|
||||
|
||||
A signed file. It carries the instance UUID it belongs to, the tier, the server
|
||||
A signed file. It carries the instance ID it belongs to, the tier, the server
|
||||
allowance, feature toggles and an expiry. The control plane verifies the
|
||||
signature locally checking a licence never contacts HQ, and a running instance
|
||||
does not need HQ to be reachable.
|
||||
signature locally.
|
||||
|
||||
Signing happens in exactly one place, in HQ. The control plane can only verify.
|
||||
A running instance does not need HQ to be reachable.
|
||||
|
||||
## 1. Find your instance UUID
|
||||
:::info One Free per account, per deployment.
|
||||
The limit is enforced per account **and** deployment, so a Free cloud instance does not stop you claiming Free on a self-hosted install.
|
||||
:::
|
||||
|
||||
In the control plane, go to **Settings → Licence**. The instance UUID is shown
|
||||
there. It is the identity your licence binds to.
|
||||
## 1. Find your instance ID
|
||||
|
||||
## 2. Link the install to your HQ account
|
||||
In the control plane, go to **Settings → Licence**. The instance ID is shown there.
|
||||
|
||||
## 2. Create a free license
|
||||
|
||||
1. Sign in at [Vantage HQ](https://vantage-hq.hostxtra.co.uk). If you have no
|
||||
account, see [Accounts and signup](../hq/accounts-and-signup.md).
|
||||
2. Choose **Link an instance**.
|
||||
3. Paste the instance UUID and give it a name you will recognise.
|
||||
2. Click on the **Buy A Plan** button.
|
||||
3. Click on **Self Hosted** then click on the **Free** plan, then finally Paste the instance ID and give it a name you will recognise.
|
||||
|
||||
Linking claims the UUID for your account. A UUID already linked elsewhere is
|
||||
refused with a conflict rather than silently moved.
|
||||
You will then see the new instance on the **Overview** page.
|
||||
|
||||
## 3. Claim Free
|
||||
## 3. Downloading the free license
|
||||
|
||||
With the instance linked, choose **Claim Free** on it. HQ issues a Free licence
|
||||
bound to that UUID and hands it back.
|
||||
With the instance created go to the **Overview** page and expand the new instance.
|
||||
|
||||
:::info One Free per account, per deployment
|
||||
The limit is enforced per account **and** deployment, so a Free cloud instance
|
||||
does not stop you claiming Free on a self-hosted install. Both the friendly
|
||||
pre-check and the issuer apply the same rule deliberately, because a
|
||||
pre-check stricter than the issuer would refuse something that would actually
|
||||
have worked.
|
||||
:::
|
||||
Click on the **View Instance Settings** button. You can then click on the **Download License** or the **Copy to clipboard** button.
|
||||
|
||||
## 4. Install the licence
|
||||
|
||||
Download the licence from HQ and paste it in the control plane at
|
||||
**Settings → Licence**.
|
||||
|
||||
The instance validates the signature, checks the UUID matches its own, and
|
||||
The instance validates the signature, checks the ID matches its own, and
|
||||
starts reporting the tier, allowance and expiry.
|
||||
|
||||
:::warning Cloud instances cannot paste a licence
|
||||
On a cloud instance `POST /license` answers `409 cloud_managed`, and the UI
|
||||
hides the form entirely. A cloud licence is written directly by HQ. This is not
|
||||
a restriction the injection path has to work around it writes to the database,
|
||||
not through the endpoint.
|
||||
:::info Cloud instances do **not** require installing the license as this is done automatically.
|
||||
:::
|
||||
|
||||
## Renewing
|
||||
|
||||
Free licences are renewable from HQ within a renewal window near expiry;
|
||||
outside that window the renew call refuses. See [Free tier](../hq/free-tier.md).
|
||||
|
||||
Pasting a licence keeps working while the current one is expired that endpoint
|
||||
is exempt from the licence check, because it is the way out of degraded mode.
|
||||
outside that window you cannot renew early. See [Free tier](../hq/free-tier.md).
|
||||
|
||||
## Moving the install to new hardware
|
||||
|
||||
Rebuilding produces a new instance UUID, and a licence binds to a UUID. Use
|
||||
Rebuilding produces a new instance ID, and a licence binds to a ID. Use
|
||||
**Relink** in HQ to move the licence across. The number of relinks per term is
|
||||
capped; the portal shows how many you have left.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ title: First login
|
||||
sidebar_label: First login
|
||||
---
|
||||
|
||||
A fresh install has no users and no organisation. The first visit creates both.
|
||||
A fresh install has no users and no instance. The first visit creates both.
|
||||
|
||||
## 1. Bootstrap
|
||||
|
||||
@@ -13,48 +13,49 @@ Open the control plane in a browser. Because no user exists, you land on
|
||||
|
||||
Fill in:
|
||||
|
||||
| Field | Notes |
|
||||
| ----------------- | ----------------------------------------------------------------- |
|
||||
| Organisation name | Display name. Shown throughout the UI |
|
||||
| Slug | Lowercase, used in the hostname on cloud. Some names are reserved |
|
||||
| Your name | |
|
||||
| Email | Becomes your sign-in identity |
|
||||
| Password | Stored bcrypt-hashed |
|
||||
| Field | Notes |
|
||||
| ------------- | ------------------------------------- |
|
||||
| Instance name | Display name. Shown throughout the UI |
|
||||
| Email | Becomes your sign-in identity |
|
||||
| Password | Stored bcrypt-hashed |
|
||||
|
||||
Submitting creates the organisation and its **owner** you.
|
||||
Submitting creates the instance and its **owner** you.
|
||||
|
||||
:::warning Bootstrap works exactly once
|
||||
The endpoint is open only while the database has no users. As soon as the first
|
||||
one exists, `/setup` redirects to the login page and the bootstrap endpoint
|
||||
refuses. There is no second chance to create the first owner, so record the
|
||||
credentials before you close the tab.
|
||||
one exists, There is no second chance to create the first owner, so record the
|
||||
credentials before you continue.
|
||||
:::
|
||||
|
||||
## 2. Sign in
|
||||
## 2. Copy the Instance ID
|
||||
|
||||
You are taken to `/login`. Sign in with the email and password you just set.
|
||||
Once you have finished setup you will see the successfully created page.
|
||||
|
||||
Sessions are an opaque 32-byte token in the `km_session` cookie, with the body
|
||||
held in Redis for 24 hours. Restarting Redis signs everyone out and loses
|
||||
nothing else.
|
||||
This will show the Instance ID. You will need this ID when creating a license in the HQ.
|
||||
|
||||
## 3. Look around
|
||||
## 3. Sign in
|
||||
|
||||
You land on the fleet dashboard, which is empty. The sidebar is the whole
|
||||
Click the continue to sign in button on the successful setup page.
|
||||
|
||||
You will be taken to `/login`. Sign in with the email and password you just set.
|
||||
|
||||
## 4. Look around
|
||||
|
||||
You land on the servers dashboard, which is empty. The sidebar is the whole
|
||||
product:
|
||||
|
||||
| Section | What it does |
|
||||
| --------- | ----------------------------------------- |
|
||||
| Servers | The fleet enrol, inspect, console, update |
|
||||
| Keys | SSH public keys and their assignments |
|
||||
| Workflows | Compose and run scripted work |
|
||||
| Steps | The reusable step library |
|
||||
| Monitors | HTTP, TCP, ICMP and TLS checks |
|
||||
| Secrets | The encrypted vault |
|
||||
| Audit | Every mutating action |
|
||||
| Settings | Members, SSO, alerts, retention, licence |
|
||||
| Section | What it does |
|
||||
| --------- | ------------------------------------------ |
|
||||
| Servers | The server enrol, inspect, console, update |
|
||||
| Keys | SSH public keys and their assignments |
|
||||
| Workflows | Compose and run scripted work |
|
||||
| Steps | The reusable step library |
|
||||
| Monitors | HTTP, TCP, ICMP and TLS checks |
|
||||
| Secrets | The encrypted vault |
|
||||
| Audit | Every mutating action |
|
||||
| Settings | Members, SSO, alerts, retention, licence |
|
||||
|
||||
## 4. Add the rest of your team
|
||||
## 5. Add the rest of your team
|
||||
|
||||
Go to **Settings → Access**. Add members with a role:
|
||||
|
||||
@@ -66,12 +67,6 @@ Go to **Settings → Access**. Add members with a role:
|
||||
|
||||
Settings and organisation management require `owner` or `admin`.
|
||||
|
||||
If you would rather not manage passwords, configure single sign-on instead:
|
||||
see [Settings](../vantage/settings.md#single-sign-on). You can add more than
|
||||
one identity provider; each gets its own button on the login page, and no
|
||||
buttons appear at all until at least one provider is configured. Client
|
||||
secrets are stored encrypted.
|
||||
If you would rather not manage passwords, configure single sign-on instead: see [Settings](../vantage/settings.md#single-sign-on).
|
||||
|
||||
## Next
|
||||
|
||||
[Add your first server](./first-server.md).
|
||||
You can add more than one identity provider; each gets its own button on the login page, and no buttons appear at all until at least one provider is configured.
|
||||
|
||||
@@ -4,41 +4,39 @@ title: Add your first server
|
||||
sidebar_label: Add your first server
|
||||
---
|
||||
|
||||
Enrolling a machine means running one command on it. The control plane issues a
|
||||
Enrolling a server means running one command on it. The control plane issues a
|
||||
short-lived token, the install script fetches the agent and writes a config, and
|
||||
the machine registers itself.
|
||||
|
||||
## 1. Create the enrolment
|
||||
|
||||
In the UI, go to **Servers → Add server**. That calls `POST /api/servers/new`,
|
||||
which generates a server ID and a pre-registration token and hands back a ready
|
||||
one-liner.
|
||||
In the UI, go to **Servers → Add server** Then click the **Generate Install Command** button.
|
||||
This generates a server ID and a pre-registration token
|
||||
|
||||
:::warning The token is single-use and lives one hour
|
||||
It is the only credential in the flow, and it is spent the moment the agent
|
||||
calls `Register`. If you paste it somewhere and come back tomorrow, create a new
|
||||
enrolment instead nothing is lost by doing so.
|
||||
It is the only credential in the flow, and it is spent the moment the agent registers.
|
||||
:::
|
||||
|
||||
## 2. Run the one-liner
|
||||
|
||||
### Linux
|
||||
|
||||
Run the generated install script as root.
|
||||
|
||||
Here is an example of the install script:
|
||||
|
||||
```bash
|
||||
curl -fsSL "https://vantage.example.com/install?server_id=<id>&token=<token>" | bash
|
||||
```
|
||||
|
||||
Run it as root. The script:
|
||||
What the script does:
|
||||
|
||||
1. Detects architecture `x86_64` and `aarch64` only; anything else exits.
|
||||
2. Asks the Gitea API for the newest `agent/v*` release.
|
||||
3. Downloads the binary and `checksums.txt`, and **verifies the SHA-256**,
|
||||
aborting on a mismatch.
|
||||
4. Installs to `/usr/local/bin/vantage-agent`, mode `0755`.
|
||||
5. Writes `/etc/vantage/config.yaml` (directory `0700`, file `0600`) containing
|
||||
the server ID, the pre-registration token and the gRPC host.
|
||||
6. Writes `/etc/systemd/system/vantage-agent.service` with `Restart=always`, and
|
||||
runs `systemctl enable --now vantage-agent`.
|
||||
2. Downloads the binary and `checksums.txt`, and **verifies the SHA-256**, aborting on a mismatch.
|
||||
3. Installs to `/usr/local/bin/vantage-agent`, mode `0755`.
|
||||
4. Writes the config file at `/etc/vantage/config.yaml`
|
||||
1. This contains the server ID, the pre-registration token and the gRPC host.
|
||||
5. Writes the systemd service file `/etc/systemd/system/vantage-agent.service` and starts the agent.
|
||||
|
||||
### Windows
|
||||
|
||||
@@ -46,42 +44,30 @@ Run it as root. The script:
|
||||
irm "https://vantage.example.com/install.ps1?server_id=<id>&token=<token>" | iex
|
||||
```
|
||||
|
||||
Run from an elevated PowerShell. The agent is registered as a service through
|
||||
NSSM, with the config at `%ProgramData%\vantage\config.yaml`. There is also an
|
||||
MSI built by CI if you would rather deploy that.
|
||||
Run from an elevated PowerShell.
|
||||
|
||||
:::info Windows agents are second-class on purpose
|
||||
They register, heartbeat, run workflow steps and report inventory. They do
|
||||
**not** manage `authorized_keys` the key subsystem is Linux-only, and a
|
||||
Windows agent stops after the heartbeat portion of the poll.
|
||||
What the script does:
|
||||
|
||||
1. Creates the config at `%ProgramData%\vantage\config.yaml`.
|
||||
1. This contains the server ID, the pre-registration token and the gRPC host.
|
||||
2. Downloads the agent MSI from Gitea.
|
||||
3. Installs the MSI and creates the Windows service.
|
||||
4. Starts the agent.
|
||||
|
||||
:::info Windows agents do **not** manage `authorized_keys` as this is a Linux-only function.
|
||||
:::
|
||||
|
||||
## 3. Watch it come up
|
||||
|
||||
The server appears immediately as `pending`. Within one poll interval 30
|
||||
seconds it flips to `active`.
|
||||
The server appears immediately as `pending`. Within one poll interval, 30 seconds it becomes `active`.
|
||||
|
||||
On the machine:
|
||||
Check the systemd logs using the following commands:
|
||||
|
||||
```bash
|
||||
systemctl status vantage-agent
|
||||
journalctl -u vantage-agent -f
|
||||
```
|
||||
|
||||
What happens on that first run:
|
||||
|
||||
```
|
||||
1. Load /etc/vantage/config.yaml
|
||||
2. pre_reg_token present → register → save agent_token, clear pre_reg_token
|
||||
3. Reconnect with the permanent token
|
||||
4. Start: command stream · hourly update check · inventory · monitors
|
||||
5. Enter the key poll loop
|
||||
```
|
||||
|
||||
After registration the config no longer contains the pre-registration token; it
|
||||
contains a permanent agent token instead. The control plane stores only the
|
||||
SHA-256 of that token, never the token itself.
|
||||
|
||||
## 4. Confirm it works
|
||||
|
||||
Open the server's detail page. Within a minute or two you should see:
|
||||
@@ -104,7 +90,7 @@ Open the server's detail page. Within a minute or two you should see:
|
||||
A server is marked `offline` when its last-seen time passes the threshold; that
|
||||
sweep runs every two minutes, so allow for it before concluding anything.
|
||||
|
||||
## Next
|
||||
## Next Steps
|
||||
|
||||
- [Assign an SSH key](../vantage/ssh-keys.md)
|
||||
- [Run a workflow](../vantage/workflows.md)
|
||||
|
||||
@@ -4,8 +4,7 @@ title: Accounts and signup
|
||||
sidebar_label: Accounts and signup
|
||||
---
|
||||
|
||||
Vantage HQ, at `vantage-hq.hostxtra.co.uk`, is where you manage the **account**
|
||||
behind your instances: your team, your instances, their licences and billing.
|
||||
[Vantage HQ](https://vantage-hq.hostxtra.co.uk) is where you manage the **account**, your team, your instances, their licences and billing.
|
||||
|
||||
## An account is a team, not a person
|
||||
|
||||
@@ -31,26 +30,15 @@ Signup is **account-first**. Creating an account creates the account and you;
|
||||
it does not create a Vantage instance. Nothing exists in any control plane until
|
||||
you later create or link one.
|
||||
|
||||
1. Go to the signup form.
|
||||
1. Go to the [signup form](https://vantage.hostxtra.co.uk/start).
|
||||
2. Enter your name, email and a password.
|
||||
3. Check your email and click the verification link.
|
||||
|
||||
:::info Verify before you can sign in
|
||||
An unverified account gets a distinct "check your email" message rather than a
|
||||
generic authentication failure the address is already known to be yours, so
|
||||
there is nothing to protect by being vague.
|
||||
:::
|
||||
|
||||
Verification links are valid for **24 hours**. The token is 32 random bytes and
|
||||
only its SHA-256 hash is stored, so a leaked database yields no working links.
|
||||
|
||||
If the verification email cannot be sent, the signup is rolled back rather than
|
||||
left stranded retry rather than assuming a half-created account is in the way.
|
||||
Verification links are valid for **24 hours**.
|
||||
|
||||
## Signing in
|
||||
|
||||
Email and password. The session is a cookie, separate from the control plane's:
|
||||
signing in to HQ does not sign you in to an instance, and vice versa.
|
||||
Use the Email and password used in the signup form to login to the HQ, signing in to HQ does not sign you in to an instance, and vice versa.
|
||||
|
||||
## What comes next
|
||||
|
||||
@@ -65,14 +53,6 @@ signing in to HQ does not sign you in to an instance, and vice versa.
|
||||
|
||||
Three destinations: **Overview**, **People**, **Billing**.
|
||||
|
||||
Settings lives in the account menu rather than the nav, because it is your
|
||||
password rather than a place. The appearance toggle is there too.
|
||||
|
||||
Overview lists your instances. Each is one record, closed to a row and open to
|
||||
its licence contents, members and actions. It opens by default when it is your
|
||||
only instance or when it needs attention, and your manual choice is remembered.
|
||||
|
||||
There is deliberately no "your plan" card in the sidebar: tier, limits and
|
||||
expiry belong to a **licence**, and a licence belongs to one instance. An
|
||||
account with a Free cloud instance and a Professional self-hosted one has no
|
||||
single plan to show.
|
||||
- Overview lists your instances.
|
||||
- People shows all the account members and their roles.
|
||||
- Billing show the current subscriptions and subscription management.
|
||||
|
||||
+17
-50
@@ -8,56 +8,29 @@ Paid plans are billed through **Paddle**, which is the merchant of record. Your
|
||||
invoice, your card details and your tax handling are all Paddle's; HQ holds a
|
||||
customer reference and nothing sensitive.
|
||||
|
||||
Billing is **owner-only**.
|
||||
:::warning
|
||||
The Billing page requires the **owner-only** account role.
|
||||
:::
|
||||
|
||||
## Buying
|
||||
## Buying A Plan
|
||||
|
||||
Buying a plan license can be found in Vantage HQ by clicking on the **Buy a Plan** button on the **Overview** page.
|
||||
|
||||
### Cloud
|
||||
|
||||
Open the instance, change its configuration to what you want, and check out.
|
||||
Checkout runs in the browser.
|
||||
On the **Buy A Plan** page you will need to select the **Deployment** to **Cloud** then chose your **Billing** cycle (Monthly or Annually).
|
||||
|
||||
Then select your desired **Plan** and configure the features.
|
||||
|
||||
Finally specify the **Instance Name** and click the **Continue to payment** button.
|
||||
|
||||
### Self-hosted
|
||||
|
||||
**Buy self-hosted**, then bind the purchase to your install's UUID. See
|
||||
[Self-hosted instances](./self-hosted-instances.md).
|
||||
On the **Buy A Plan** page you will need to select the **Deployment** to **Self-Hosted** then chose your **Billing** cycle (Monthly or Annually).
|
||||
|
||||
## What you are buying
|
||||
Then select your desired **Plan** and configure the features.
|
||||
|
||||
A subscription's line items are the configuration: the plan base, the metered
|
||||
server count above the base, and any per-instance features. Changing the
|
||||
configuration changes the line items.
|
||||
|
||||
## Changing configuration
|
||||
|
||||
**Instance → Configuration**, adjust servers or features, and save.
|
||||
|
||||
- **Increases** take effect when the payment confirms.
|
||||
- **Reductions** are scheduled for the end of the term. The portal shows the
|
||||
date and the new value.
|
||||
|
||||
## The customer portal
|
||||
|
||||
**Billing → Manage** mints a Paddle customer-portal session where you can
|
||||
update your payment method, see invoices and cancel.
|
||||
|
||||
## How a licence follows a payment
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C["Checkout / change"] --> P["Paddle"]
|
||||
P -->|signed webhook| H["HQ"]
|
||||
H --> G["Entitlement: desired → granted"]
|
||||
G --> L["Licence signed from granted"]
|
||||
```
|
||||
|
||||
The webhook is the **only** issuing path for paid plans. It is signature
|
||||
verified, processed exactly once, and resolved from the subscription's _current_
|
||||
line items so a webhook that arrives out of order still produces the right
|
||||
answer rather than replaying a stale state.
|
||||
|
||||
A licence is signed from **granted** only. A checkout you abandon changes
|
||||
nothing.
|
||||
Finally specify the **Instance Name** and click the **Continue to payment** button.
|
||||
|
||||
## Cancelling and failed payments
|
||||
|
||||
@@ -66,18 +39,12 @@ Your licence runs to its grace-padded expiry and then lapses normally. There is
|
||||
no mid-term cut-off.
|
||||
|
||||
For a cloud Free instance, lapsing eventually leads to deletion see
|
||||
[Free tier](./free-tier.md). Paid instances are not reaped.
|
||||
[Free tier](./free-tier.md). Paid instances are not deleted.
|
||||
|
||||
## Renewals
|
||||
|
||||
At renewal the subscription bills again and the licence is reissued for the new
|
||||
term. It is also the only moment a scheduled **reduction** takes effect.
|
||||
|
||||
Self-hosted customers: download and paste the reissued licence. Cloud customers:
|
||||
nothing to do.
|
||||
|
||||
## Free is not in Paddle at all
|
||||
|
||||
Free has no subscription, no £0 line item and no Paddle record. It has its own
|
||||
renewal, in the portal. An account only acquires a Paddle customer reference
|
||||
with its first paid purchase.
|
||||
- Self-hosted customers: download and paste the reissued licence.
|
||||
- Cloud customers: the license is automatically linked to the instance.
|
||||
|
||||
@@ -25,11 +25,6 @@ features on a paid plan.
|
||||
The limit is enforced per account **and** deployment. A Free cloud instance does
|
||||
not prevent a Free self-hosted one they are separate slots.
|
||||
|
||||
## Free is outside Paddle
|
||||
|
||||
There is no subscription, no £0 line item and no invoice. Your account acquires
|
||||
a Paddle customer reference only with its first paid purchase.
|
||||
|
||||
## Renewing
|
||||
|
||||
Free licences have a term and must be renewed from the portal.
|
||||
|
||||
@@ -30,9 +30,9 @@ entitlement.
|
||||
|
||||
Two are per-instance toggles rather than tier bundles:
|
||||
|
||||
| Feature | What it enables |
|
||||
| --------- | ------------------------------------------------------------------------- |
|
||||
| `console` | The [browser console](../vantage/browser-console.md) |
|
||||
| Feature | What it enables |
|
||||
| --------- | -------------------------------------------------------------------- |
|
||||
| `console` | The [browser console](../vantage/browser-console.md) |
|
||||
| `oidc` | Per-instance [single sign-on](../vantage/settings.md#single-sign-on) |
|
||||
|
||||
No tier includes them by default; you enable them on the instances that need
|
||||
@@ -88,8 +88,3 @@ or let it be written for you (cloud).
|
||||
When you exceed your server allowance, enrolling another one is refused. The
|
||||
existing fleet is unaffected. Raise the allowance in the portal, or remove a
|
||||
server you are not using.
|
||||
|
||||
## Legacy tiers
|
||||
|
||||
An older `self_hosted` tier is mapped forward to self-hosted Professional
|
||||
wherever it appears. Nothing needs doing about it.
|
||||
|
||||
@@ -10,7 +10,6 @@ themselves on command.
|
||||
## Checking the current version
|
||||
|
||||
Each server's detail page shows the version it reported at its last sync.
|
||||
`GET /api/agent/latest-version` reports the newest release available.
|
||||
|
||||
## Updating from the UI
|
||||
|
||||
@@ -21,8 +20,6 @@ version. The agent then:
|
||||
2. Verifies the SHA-256 against `checksums.txt`.
|
||||
3. Stops itself, replaces the binary in place, and starts again.
|
||||
|
||||
`Restart=always` on the systemd unit is what makes the last step work.
|
||||
|
||||
The server briefly goes `offline` and comes back within a poll interval or two.
|
||||
|
||||
## Updating from the machine
|
||||
|
||||
@@ -52,16 +52,9 @@ cp /opt/vantage/.env /secure-location/vantage.env
|
||||
|
||||
Treat it as a credential in its own right it holds the encryption key.
|
||||
|
||||
## Run logs
|
||||
|
||||
Workflow run logs live in the `./data` bind mount, not in the database. They are
|
||||
swept on the retention schedule anyway, so most people do not back them up. If
|
||||
you keep them for compliance, set retention to `0` (forever) and include the
|
||||
directory.
|
||||
|
||||
## What a restore gives you
|
||||
|
||||
Everything: fleet, keys, assignments, workflows and their history, monitors and
|
||||
Everything: server, keys, assignments, workflows and their history, monitors and
|
||||
incidents, secrets, settings and the audit log.
|
||||
|
||||
What it does **not** do is reconcile the world. After a restore:
|
||||
|
||||
@@ -37,31 +37,9 @@ tls: true
|
||||
|
||||
:::danger This file is the credential
|
||||
`agent_token` is plaintext here and nowhere else the control plane holds only
|
||||
its SHA-256. Anyone who can read this file can act as this agent. That is why
|
||||
it is `0600` and the directory is `0700`.
|
||||
its SHA-256. Anyone who can read this file can act as this agent.
|
||||
:::
|
||||
|
||||
## Startup sequence
|
||||
|
||||
```
|
||||
1. Load the config
|
||||
2. pre_reg_token present → register → save agent_token,
|
||||
clear pre_reg_token, reconnect
|
||||
3. Start: command stream · hourly update check · inventory · monitors
|
||||
4. Enter the key poll loop
|
||||
```
|
||||
|
||||
## The poll loop
|
||||
|
||||
```
|
||||
1. Ask the control plane for the desired key state, reporting the
|
||||
agent version
|
||||
2. Non-Linux hosts stop here Windows agents register and heartbeat only
|
||||
3. Diff the desired keys against /root/.ssh/authorized_keys;
|
||||
unchanged → write nothing
|
||||
4. Changed → write a temp file, rename it over the real one, chmod 0600
|
||||
```
|
||||
|
||||
## Service management
|
||||
|
||||
### Linux
|
||||
@@ -77,21 +55,13 @@ journalctl -u vantage-agent -f
|
||||
|
||||
### Windows
|
||||
|
||||
A service registered through NSSM, or installed by the MSI that CI builds.
|
||||
A service registered through NSSM, or installed by the MSI.
|
||||
|
||||
```powershell
|
||||
Get-Service vantage-agent
|
||||
Restart-Service vantage-agent
|
||||
```
|
||||
|
||||
## Command-line flags
|
||||
|
||||
```
|
||||
vantage-agent -generate-key
|
||||
```
|
||||
|
||||
Generates a keypair locally. Normal operation takes no flags.
|
||||
|
||||
## Moving an agent to a new control plane
|
||||
|
||||
Change `server_url`, clear `agent_token`, set a fresh `pre_reg_token` from a new
|
||||
|
||||
@@ -26,6 +26,52 @@ The offline sweep runs every two minutes, so a machine that has just gone away
|
||||
takes a little while to be marked as such. That delay is intentional a single
|
||||
missed poll is not an outage.
|
||||
|
||||
## Tags
|
||||
|
||||
A tag is a `key:value` label you put on a server. Tags are how you say what a
|
||||
machine **is** `env:prod`, `role:web`, `team:core-infra` so that you can find
|
||||
it later, and so that a [workflow](./workflows.md) can target it without you
|
||||
naming it by hand.
|
||||
|
||||
There is no tag library to manage first. A tag exists because a server carries
|
||||
it, and it stops existing when the last server carrying it drops it.
|
||||
|
||||
### The rules
|
||||
|
||||
| Rule | Value |
|
||||
| ---------- | ------------------------------------------------- |
|
||||
| Characters | lowercase letters, digits, `-` and `_`, on both halves |
|
||||
| Key length | up to 32 characters |
|
||||
| Value length | up to 64 characters |
|
||||
| Per server | up to 20 tags |
|
||||
|
||||
Neither half may be empty, and keys beginning `sys:` are reserved for tags
|
||||
Vantage may derive from inventory later, so a tag you write today can never
|
||||
collide with one invented for you tomorrow.
|
||||
|
||||
Anything outside those rules is refused with a message naming the rule, rather
|
||||
than quietly saved in a shape you did not intend. Uppercase is not folded to
|
||||
lowercase for you `Env` is a mistake, not a synonym for `env`.
|
||||
|
||||
### Editing a server's tags
|
||||
|
||||
On the server detail page, **Edit** beside the tag chips. Saving replaces the
|
||||
whole set: what you see in the editor is exactly what the server will have.
|
||||
There is no per-tag merge, so if two people edit the same server at once, the
|
||||
last save wins outright rather than producing a blend of the two.
|
||||
|
||||
### Filtering the fleet
|
||||
|
||||
The **Servers** list has a picker per tag key in use. Choosing values from more
|
||||
than one key narrows the list a server must match **all** of them, not any.
|
||||
Untagged servers appear only when no filter is set.
|
||||
|
||||
:::tip A filtered fleet view is a link
|
||||
The filter lives in the URL (`/servers?tag=env:prod&tag=role:web`). Copy the
|
||||
address bar and you have sent someone the same view, not a description of how to
|
||||
reproduce it.
|
||||
:::
|
||||
|
||||
## The server detail page
|
||||
|
||||
### Keys
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
id: vulnerabilities
|
||||
title: Vulnerabilities
|
||||
sidebar_label: Vulnerabilities
|
||||
---
|
||||
|
||||
Each Linux server reports the packages it has installed. Vantage matches them
|
||||
against the security advisories published by that server's own distribution and
|
||||
raises a finding for anything not yet patched.
|
||||
|
||||
Requires the **vulnerability scanning** feature on your licence. Without it,
|
||||
agents collect nothing at all — there is no inventory stored and no findings
|
||||
page to read.
|
||||
|
||||
## What gets scanned
|
||||
|
||||
Linux servers running `apt`, `dnf`/`yum`, `apk`, `zypper` or `pacman`. Agents
|
||||
report their package list hourly, and only when it has changed since the last
|
||||
report.
|
||||
|
||||
Windows servers are not scanned.
|
||||
|
||||
Some distributions publish no machine-readable advisory feed. Those servers
|
||||
show **unsupported** on their own page rather than appearing as having no
|
||||
vulnerabilities — the two are very different answers, and only one of them is
|
||||
good news.
|
||||
|
||||
## Why versions look "wrong"
|
||||
|
||||
A finding names the version your distribution ships, not the upstream release.
|
||||
Ubuntu's `openssl 3.0.2-0ubuntu1.15` carries security fixes backported into
|
||||
what still calls itself 3.0.2, so public CVE databases listing "3.0.2" as
|
||||
vulnerable are describing upstream, not your machine.
|
||||
|
||||
Vantage matches against your distribution's own advisories, which is why a
|
||||
server can be running a version some scanners flag while Vantage correctly
|
||||
reports it as patched.
|
||||
|
||||
For the same reason a severity here may be lower than the one you find on a CVE
|
||||
website. Debian and Red Hat routinely downgrade a rating when the vulnerable
|
||||
code path is not reachable in the way they build the package. Their rating is
|
||||
the accurate one for the package you are actually running.
|
||||
|
||||
## The board
|
||||
|
||||
`/vulnerabilities` groups findings by CVE. One row per CVE with the number of
|
||||
affected servers, expandable to the individual servers — the same CVE across
|
||||
forty machines is one decision, not forty.
|
||||
|
||||
Severity counts at the top filter the list when clicked. The state tabs switch
|
||||
between **open**, **accepted** and **fixed**.
|
||||
|
||||
The vulnerability database's age is shown above the board. If a pull has failed
|
||||
for long enough for the data to be stale, that becomes a warning: a low count
|
||||
against three-week-old data is not the same as a low count.
|
||||
|
||||
## Fixing something
|
||||
|
||||
A finding with a known fixed version gets an **Apply updates** button, which
|
||||
runs the same OS update the server page offers. There is no separate patching
|
||||
mechanism.
|
||||
|
||||
Vantage never patches automatically. An unattended upgrade triggered by a third
|
||||
party's data feed is a fleet-wide change nobody chose.
|
||||
|
||||
## Accepting a finding
|
||||
|
||||
Some findings cannot be fixed today: a kernel CVE waiting on a reboot window,
|
||||
or one with no vendor fix published at all.
|
||||
|
||||
**Accept** hides a finding from counts and alerts until a date you choose, with
|
||||
a reason that is recorded in the audit log along with your name. On that date it
|
||||
reopens by itself.
|
||||
|
||||
The expiry is required. A dismissal with no end date is how a finding gets
|
||||
forgotten, and it is exactly what an auditor will ask to see.
|
||||
|
||||
## Alerts
|
||||
|
||||
Alert rules live with your notification channels, under
|
||||
**Settings → Notification Channels**. A rule has a minimum severity, an optional
|
||||
server tag filter, and one or more channels.
|
||||
|
||||
A rule sends **one digest per scan** summarising what newly opened — never one
|
||||
message per finding. A database refresh can open several hundred findings at
|
||||
once, and a message each would flood the channel.
|
||||
|
||||
Findings that were already open do not re-alert.
|
||||
|
||||
## Fleet-wide package search
|
||||
|
||||
`GET /api/packages/search?name=openssl` answers which servers run a given
|
||||
package and at what version, across the whole fleet. Useful during an incident
|
||||
before a finding exists for it.
|
||||
@@ -76,7 +76,8 @@ the API is the boundary; the UI is the courtesy.
|
||||
2. Add steps in order from the library.
|
||||
3. Set inputs per step.
|
||||
4. Set failure behaviour per step.
|
||||
5. Choose target servers.
|
||||
5. Choose targets: named servers, a tag selector, or both. See
|
||||
[Targeting](#targeting).
|
||||
|
||||
### Failure behaviour
|
||||
|
||||
@@ -92,6 +93,48 @@ A workflow can override a step's script or its secret references without
|
||||
touching the library entry. This is how you adapt a default step, and it is
|
||||
scoped to that workflow.
|
||||
|
||||
## Targeting
|
||||
|
||||
A workflow names servers two ways, and it can use both at once:
|
||||
|
||||
- **Target servers** an explicit list you pick from the fleet.
|
||||
- **Target tags** a `key:value` selector matched against
|
||||
[server tags](./servers.md#tags). More than one key ANDs: a server must carry
|
||||
every pair to match.
|
||||
|
||||
A run goes to the **union** of the two, with duplicates removed. A server that is
|
||||
both named explicitly and matched by the selector runs once, not twice. This is
|
||||
what lets a workflow say "every production web server, plus this one box I am
|
||||
watching" without maintaining a list.
|
||||
|
||||
The designer shows the resolved count as you edit, so you can see how many
|
||||
machines a change to the selector just added or removed before you save.
|
||||
|
||||
:::warning An empty selector matches nothing
|
||||
Clearing the tag selector does not mean "all servers". A workflow with no named
|
||||
servers and no tags matches nothing and is refused at run time rather than
|
||||
reported as a success over zero machines.
|
||||
|
||||
The alternative reading, where an empty field means the whole fleet, turns a
|
||||
cleared box into a fleet-wide run. That is not a mistake anyone should be able to
|
||||
make by deleting text.
|
||||
:::
|
||||
|
||||
Tags are read **at run time**, not when you save. Tag a new machine `env:prod`
|
||||
and the next run of an `env:prod` workflow includes it, with nothing to update on
|
||||
the workflow itself. The same is true in reverse: removing a tag removes the
|
||||
machine from every workflow that selected on it.
|
||||
|
||||
### Offline servers are still targeted
|
||||
|
||||
A server matched by tag is dispatched to even if its agent is offline, and that
|
||||
step fails visibly on that machine. Vantage does not quietly shrink your target
|
||||
list to the machines that happened to be reachable a patch run that skipped
|
||||
three servers and reported success is worse than one that failed on three and
|
||||
said so.
|
||||
|
||||
Re-run the workflow once they are back, or fix the agent first.
|
||||
|
||||
## Running
|
||||
|
||||
**Run** snapshots the resolved steps into the run record and dispatches each step
|
||||
@@ -104,6 +147,57 @@ A run shows the script that actually executed, not the current library version.
|
||||
|
||||
Targets run **in parallel**; steps within one server run **in order**.
|
||||
|
||||
## Schedules
|
||||
|
||||
A workflow can carry a schedule, and Vantage will start it the same way a person
|
||||
would — the same dispatch, the same snapshot, the same run page. A scheduled run
|
||||
is an ordinary run with `schedule` recorded as who triggered it.
|
||||
|
||||
Open a workflow, choose **Edit**, and tick **Run on a schedule**. The expression
|
||||
is standard five-field cron:
|
||||
|
||||
```
|
||||
minute hour day-of-month month day-of-week
|
||||
```
|
||||
|
||||
The presets write cron underneath, so you can start from one and adjust:
|
||||
|
||||
| Preset | Cron |
|
||||
| ------------------- | ----------- |
|
||||
| Hourly | `0 * * * *` |
|
||||
| Nightly, 02:00 | `0 2 * * *` |
|
||||
| Weekly, Sun 02:00 | `0 2 * * 0` |
|
||||
| Monthly, 1st 02:00 | `0 2 1 * *` |
|
||||
|
||||
There is no seconds field and no `@daily`-style shorthand. The next three
|
||||
occurrences are shown as you type, and they are computed by the server rather
|
||||
than the browser, so what you see is exactly what will fire.
|
||||
|
||||
### Timezones
|
||||
|
||||
A schedule stores an IANA timezone by name — `Europe/London`, not an offset.
|
||||
That is what makes a 02:00 job stay at 02:00 across a daylight-saving change
|
||||
instead of drifting an hour for half the year. An unknown zone is refused when
|
||||
you save it, not at 2am.
|
||||
|
||||
### Overlaps are skipped, not queued
|
||||
|
||||
If a run of the same workflow is still going when the next occurrence comes
|
||||
round, the occurrence is **skipped** and the reason recorded. It is not queued
|
||||
behind the running one. A patch workflow that takes longer than its interval
|
||||
should fall behind visibly rather than pile up.
|
||||
|
||||
### Missed occurrences
|
||||
|
||||
If the control plane was not running when an occurrence was due, it still fires
|
||||
when the control plane comes back — but only within **one hour** of the due
|
||||
time. Anything older is recorded as missed and dropped. A job missed by ten
|
||||
minutes during an upgrade should still run; one missed by two days should not
|
||||
suddenly fire at lunchtime.
|
||||
|
||||
Either kind of skip is shown on the workflow's schedule panel, with the time it
|
||||
was due and why it did not run.
|
||||
|
||||
## Watching a run
|
||||
|
||||
Step stdout and stderr stream back as chunks, are appended to a log file on the
|
||||
|
||||
@@ -26,6 +26,7 @@ const sidebars: SidebarsConfig = {
|
||||
"vantage/ssh-keys",
|
||||
"vantage/workflows",
|
||||
"vantage/monitors",
|
||||
"vantage/vulnerabilities",
|
||||
"vantage/notification-channels",
|
||||
"vantage/secrets",
|
||||
"vantage/browser-console",
|
||||
|
||||
+50
@@ -1,22 +1,69 @@
|
||||
cloud.google.com/go/compute v1.25.1/go.mod h1:oopOIR53ly6viBYxaDhBfJwzUAxf1zE//uf3IB011ls=
|
||||
cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=
|
||||
github.com/Intevation/gval v1.3.0/go.mod h1:xmGyGpP5be12EL0P12h+dqiYG8qn2j3PJxIgkoOHO5o=
|
||||
github.com/Intevation/jsonpath v0.2.1/go.mod h1:WnZ8weMmwAx/fAO3SutjYFU+v7DFreNYnibV7CiaYIw=
|
||||
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
|
||||
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
|
||||
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
|
||||
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986/go.mod h1:NT+jyeCzXk6vXR5MTkdn4z64TgGfE5HMLC8qfj5unl8=
|
||||
github.com/aquasecurity/go-gem-version v0.0.0-20201115065557-8eed6fe000ce/go.mod h1:HXgVzOPvXhVGLJs4ZKO817idqr/xhwsTcj17CLYY74s=
|
||||
github.com/aquasecurity/go-npm-version v0.0.1/go.mod h1:hxbJZtKlO4P8sZ9nztizR6XLoE33O+BkPmuYQ4ACyz0=
|
||||
github.com/aquasecurity/go-pep440-version v0.0.1/go.mod h1:3naPe+Bp6wi3n4l5iBFCZgS0JG8vY6FT0H4NGhFJ+i4=
|
||||
github.com/aquasecurity/go-version v0.0.1/go.mod h1:s1UU6/v2hctXcOa3OLwfj5d9yoXHa3ahf+ipSwEvGT0=
|
||||
github.com/briandowns/spinner v1.23.0/go.mod h1:rPG4gmXeN3wQV/TsAY4w8lPdIM6RX3yqeBQJSrbXjuE=
|
||||
github.com/bytedance/sonic/loader v0.3.0/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI=
|
||||
github.com/census-instrumentation/opencensus-proto v0.4.1/go.mod h1:4T9NM4+4Vw91VeyqjLS6ao50K5bOcLKN6Q42XnYaRYw=
|
||||
github.com/cheggaaa/pb/v3 v3.1.7/go.mod h1:/Ji89zfVPeC/u5j8ukD0MBPHt2bzTYp74lQ7KlgFWTQ=
|
||||
github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50/go.mod h1:5e1+Vvlzido69INQaVO6d87Qn543Xr6nooe9Kz7oBFM=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
|
||||
github.com/envoyproxy/go-control-plane v0.12.0/go.mod h1:ZBTaoJ23lqITozF0M6G4/IragXCQKCnYbmlmtHvwRG0=
|
||||
github.com/envoyproxy/protoc-gen-validate v1.0.4/go.mod h1:qys6tmnRsYrQqIhm2bvKZH4Blx/1gTIZ2UKVY1M+Yew=
|
||||
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
|
||||
github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
|
||||
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
|
||||
github.com/goccy/go-yaml v1.19.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
|
||||
github.com/gocsaf/csaf/v3 v3.1.1/go.mod h1:EpUCrQg69i+Y66MphmQvVbcj333GFLjXOYHg1zoXVso=
|
||||
github.com/golang/glog v1.2.0/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
|
||||
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
|
||||
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
|
||||
github.com/josephburnett/jd/v2 v2.3.0/go.mod h1:0I5+gbo7y8diuajJjm79AF44eqTheSJy1K7DSbIUFAQ=
|
||||
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
|
||||
github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
|
||||
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
|
||||
github.com/masahiro331/go-mvn-version v0.0.0-20250131095131-f4974fa13b8a/go.mod h1:jZ3F25l7DbD7l7DcA8aj7eo1EZ84nbzcQHBB4lCSrI8=
|
||||
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
|
||||
github.com/oklog/ulid/v2 v2.1.1 h1:suPZ4ARWLOJLegGFiZZ1dFAkqzhMjL3J1TzI+5wHz8s=
|
||||
github.com/oklog/ulid/v2 v2.1.1/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ=
|
||||
github.com/package-url/packageurl-go v0.1.3/go.mod h1:nKAWB8E6uk1MHqiS/lQb9pYBGH2+mdJ2PJc2s50dQY0=
|
||||
github.com/pandatix/go-cvss v0.6.2/go.mod h1:jDXYlQBZrc8nvrMUVVvTG8PhmuShOnKrxP53nOFkt8Q=
|
||||
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
||||
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/samber/lo v1.50.0 h1:XrG0xOeHs+4FQ8gJR97zDz5uOFMW7OwFWiFVzqopKgY=
|
||||
github.com/samber/lo v1.50.0/go.mod h1:RjZyNk6WSnUFRKK6EyOhsRJMqft3G+pg7dCWHQCWvsc=
|
||||
github.com/samber/oops v1.18.1 h1:qjhZbqbdyhWBKntkY8sxrDNKA8b4c5VHlmI1rli7X7M=
|
||||
github.com/samber/oops v1.18.1/go.mod h1:xYqvimigkKV70HyLXiBZJFpIWi2CGcc6Xx7eV+2HycI=
|
||||
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1/go.mod h1:uToXkOrWAZ6/Oc07xWQrPOhJotwFIyu2bBVN41fcDUY=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
|
||||
github.com/urfave/cli v1.22.16/go.mod h1:EeJR6BKodywf4zciqrdw6hpCPk68JO9z5LazXZMn5Po=
|
||||
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
|
||||
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
|
||||
go.mongodb.org/mongo-driver/v2 v2.5.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0=
|
||||
go.opentelemetry.io/otel v1.34.0 h1:zRLXxLCgL1WyKsPVrgbSdMN4c0FMkDAskSTQP+0hdUY=
|
||||
go.opentelemetry.io/otel v1.34.0/go.mod h1:OWFPOQ+h4G8xpyjgqo4SxJYdDQ/qmRH+wivy7zzx9oI=
|
||||
go.opentelemetry.io/otel/trace v1.34.0 h1:+ouXS2V8Rd4hp4580a8q23bg0azF2nI8cqLYnC8mh/k=
|
||||
go.opentelemetry.io/otel/trace v1.34.0/go.mod h1:Svm7lSjQD7kG7KJ/MUHPVXSDGz2OX4h0M2jHBhmSfRE=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.3/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
|
||||
golang.org/x/crypto v0.33.0/go.mod h1:bVdXmD7IV/4GdElGPozy6U7lWdRXA4qyRVGJV57uQ5M=
|
||||
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
||||
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
|
||||
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
|
||||
golang.org/x/exp v0.0.0-20251125195548-87e1e737ad39/go.mod h1:46edojNIoXTNOhySWIWdix628clX9ODXwPsQuG6hsK0=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
|
||||
golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
@@ -32,10 +79,13 @@ golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
|
||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
|
||||
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
|
||||
golang.org/x/time v0.8.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90=
|
||||
google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237/go.mod h1:Z5Iiy3jtmioajWHDGFk7CeugTyHtPvMHA4UTmUkyalE=
|
||||
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
|
||||
google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
|
||||
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
|
||||
k8s.io/utils v0.0.0-20201110183641-67b214c5f920/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
|
||||
|
||||
@@ -9,6 +9,7 @@ service Vantage {
|
||||
rpc SyncKeys(SyncRequest) returns (SyncResponse);
|
||||
rpc UploadGeneratedKey(UploadKeyRequest) returns (UploadKeyResponse);
|
||||
rpc ReportUpdates(ReportUpdatesRequest) returns (ReportUpdatesResponse);
|
||||
rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse);
|
||||
rpc ReportInventory(InventoryReport) returns (InventoryReportResponse);
|
||||
rpc SyncMonitors(SyncMonitorsRequest) returns (SyncMonitorsResponse);
|
||||
rpc ReportChecks(ReportChecksRequest) returns (ReportChecksResponse);
|
||||
@@ -37,6 +38,51 @@ message SyncRequest {
|
||||
|
||||
message SyncResponse {
|
||||
repeated string public_keys = 1;
|
||||
|
||||
// collect_packages tells the agent whether this instance's licence grants
|
||||
// vulnerability scanning. False means do not collect at all: no gRPC body,
|
||||
// no document, no storage. The server re-checks on ReportPackages — this
|
||||
// flag is the optimisation, the server check is the boundary.
|
||||
//
|
||||
// Absent reads as false, which is the safe direction: an old server that
|
||||
// does not send it leaves agents collecting nothing.
|
||||
bool collect_packages = 2;
|
||||
}
|
||||
|
||||
// ReportPackages carries a server's installed package set.
|
||||
//
|
||||
// The agent calls twice at most. The first call sends only the hash; if the
|
||||
// server already holds that hash it answers need_full = false and the ~150KB
|
||||
// body is never sent. A machine's package set changes rarely, so almost every
|
||||
// hour costs one small message.
|
||||
message ReportPackagesRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3;
|
||||
OSRelease os = 4;
|
||||
repeated InstalledPackage packages = 5; // empty on the offer call
|
||||
}
|
||||
|
||||
message ReportPackagesResponse {
|
||||
bool need_full = 1;
|
||||
}
|
||||
|
||||
message OSRelease {
|
||||
string family = 1;
|
||||
// version_id is not optional: Ubuntu 22.04 and 24.04 publish different fixed
|
||||
// versions for the same CVE, so a scan without it is guesswork.
|
||||
string version_id = 2;
|
||||
string arch = 3;
|
||||
}
|
||||
|
||||
message InstalledPackage {
|
||||
string name = 1;
|
||||
string version = 2;
|
||||
int32 epoch = 3;
|
||||
string arch = 4;
|
||||
// source_name is what the Debian and Ubuntu feeds are keyed on: one advisory
|
||||
// against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
string source_name = 5;
|
||||
}
|
||||
|
||||
message UploadKeyRequest {
|
||||
|
||||
@@ -18,10 +18,19 @@ ARG VERSION=dev
|
||||
RUN cd server && CGO_ENABLED=0 GOOS=linux go build \
|
||||
-ldflags="-s -w -X main.Version=${VERSION}" -o /vantage-server ./cmd
|
||||
|
||||
# Staged so the scratch image below can have a /tmp. It cannot mkdir one
|
||||
# itself — scratch has no shell — and os.MkdirTemp fails outright without it.
|
||||
RUN mkdir -p /staging/tmp && chmod 1777 /staging/tmp
|
||||
|
||||
# Runtime stage
|
||||
FROM scratch
|
||||
|
||||
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
|
||||
|
||||
# vulndb unpacks the ~50MB trivy-db here. Without it the scheduler stops at
|
||||
# "temp dir: stat /tmp: no such file or directory" and no scanning happens,
|
||||
# while everything else in the process runs perfectly well.
|
||||
COPY --from=builder /staging/tmp /tmp
|
||||
COPY --from=builder /vantage-server /vantage-server
|
||||
|
||||
COPY default_steps/ /opt/default-steps/
|
||||
|
||||
@@ -11,6 +11,12 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
// Embeds the IANA zone database in the binary. Load-bearing: server/Dockerfile
|
||||
// builds on Alpine, which ships no zoneinfo, so without this
|
||||
// time.LoadLocation("Europe/London") fails in production and every workflow
|
||||
// schedule silently falls back to UTC.
|
||||
_ "time/tzdata"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/api"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/bus"
|
||||
@@ -18,6 +24,8 @@ import (
|
||||
grpcserver "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/monitorsched"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulnsched"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
@@ -112,6 +120,10 @@ func runSchemaSetup() {
|
||||
log.Printf("warning: failed to ensure secret indexes: %v", err)
|
||||
}
|
||||
|
||||
if err := services.EnsureServerIndexes(); err != nil {
|
||||
log.Printf("warning: failed to ensure server indexes: %v", err)
|
||||
}
|
||||
|
||||
if err := services.EnsureSettingsIndexes(); err != nil {
|
||||
log.Fatalf("failed to ensure settings indexes: %v", err)
|
||||
}
|
||||
@@ -120,6 +132,10 @@ func runSchemaSetup() {
|
||||
log.Printf("warning: failed to ensure workflow indexes: %v", err)
|
||||
}
|
||||
|
||||
if err := services.EnsureVulnIndexes(); err != nil {
|
||||
log.Printf("warning: failed to ensure vuln indexes: %v", err)
|
||||
}
|
||||
|
||||
if instanceIDs, err := services.ListInstanceIDs(); err != nil {
|
||||
log.Printf("warning: failed to list instances for default step seeding: %v", err)
|
||||
} else {
|
||||
@@ -175,6 +191,16 @@ func serve() {
|
||||
services.StartAuditSweeper(jobCtx)
|
||||
services.StartReaper(jobCtx)
|
||||
monitorsched.Start(jobCtx)
|
||||
workflowsched.Start(jobCtx, workflowsched.Deps{
|
||||
TriggerWorkflow: services.TriggerWorkflow,
|
||||
LogEvent: services.LogEvent,
|
||||
})
|
||||
|
||||
vulnsched.Start(jobCtx, vulnsched.Deps{
|
||||
LogEvent: services.LogEvent,
|
||||
SendDigest: services.SendVulnDigest,
|
||||
})
|
||||
services.StartVulnSweeper(jobCtx)
|
||||
|
||||
ticker := time.NewTicker(2 * time.Minute)
|
||||
defer ticker.Stop()
|
||||
|
||||
+22
-2
@@ -3,20 +3,41 @@ module gitea.hostxtra.co.uk/mrhid6/vantage/server
|
||||
go 1.26
|
||||
|
||||
require (
|
||||
github.com/aquasecurity/trivy-db v0.0.0-20260713131703-4be526083c54
|
||||
github.com/coreos/go-oidc/v3 v3.18.0
|
||||
github.com/gin-gonic/gin v1.10.0
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f
|
||||
github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23
|
||||
github.com/knqyf263/go-rpm-version v0.0.0-20240918084003-2afd7dc6a38f
|
||||
github.com/opencontainers/image-spec v1.1.1
|
||||
github.com/redis/go-redis/v9 v9.20.1
|
||||
github.com/robfig/cron/v3 v3.0.1
|
||||
github.com/wwt/guac v1.3.2
|
||||
go.mongodb.org/mongo-driver/v2 v2.8.0
|
||||
golang.org/x/crypto v0.54.0
|
||||
golang.org/x/oauth2 v0.36.0
|
||||
google.golang.org/grpc v1.64.0
|
||||
oras.land/oras-go/v2 v2.6.2
|
||||
)
|
||||
|
||||
require github.com/hyperboloide/lk v0.0.0-20251220053519-b291812e3216 // indirect
|
||||
require (
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/hyperboloide/lk v0.0.0-20251220053519-b291812e3216 // indirect
|
||||
github.com/oklog/ulid/v2 v2.1.1 // indirect
|
||||
github.com/opencontainers/go-digest v1.0.0 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/samber/lo v1.50.0 // indirect
|
||||
github.com/samber/oops v1.18.1 // indirect
|
||||
github.com/stretchr/objx v0.5.2 // indirect
|
||||
github.com/stretchr/testify v1.11.1 // indirect
|
||||
go.etcd.io/bbolt v1.4.3 // indirect
|
||||
go.opentelemetry.io/otel v1.34.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.34.0 // indirect
|
||||
)
|
||||
|
||||
require (
|
||||
gitea.hostxtra.co.uk/mrhid6/vantage/shared v0.0.0
|
||||
github.com/bytedance/sonic v1.11.6 // indirect
|
||||
github.com/bytedance/sonic/loader v0.1.1 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
@@ -38,7 +59,6 @@ require (
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||
gitea.hostxtra.co.uk/mrhid6/vantage/shared v0.0.0
|
||||
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
|
||||
github.com/sirupsen/logrus v1.4.2 // indirect
|
||||
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
|
||||
|
||||
+42
-1
@@ -1,3 +1,7 @@
|
||||
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986 h1:2a30xLN2sUZcMXl50hg+PJCIDdJgIvIbVcKqLJ/ZrtM=
|
||||
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986/go.mod h1:NT+jyeCzXk6vXR5MTkdn4z64TgGfE5HMLC8qfj5unl8=
|
||||
github.com/aquasecurity/trivy-db v0.0.0-20260713131703-4be526083c54 h1:4CZNoDkNfcuACevZeDraACGmP1+L0nKkRY52+jV8k1M=
|
||||
github.com/aquasecurity/trivy-db v0.0.0-20260713131703-4be526083c54/go.mod h1:iIEV2oGuZScvfyX2SMIn78iVMNnepgo0QuJJh/srgVI=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
@@ -35,6 +39,8 @@ github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBEx
|
||||
github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
|
||||
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
|
||||
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
|
||||
github.com/goccy/go-yaml v1.19.0 h1:EmkZ9RIsX+Uq4DYFowegAuJo8+xdX3T/2dwNPXbxEYE=
|
||||
github.com/goccy/go-yaml v1.19.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||
@@ -52,9 +58,17 @@ github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6K
|
||||
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f h1:GvCU5GXhHq+7LeOzx/haG7HSIZokl3/0GkoUFzsRJjg=
|
||||
github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f/go.mod h1:q59u9px8b7UTj0nIjEjvmTWekazka6xIt6Uogz5Dm+8=
|
||||
github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23 h1:dWzdsqjh1p2gNtRKqNwuBvKqMNwnLOPLzVZT1n6DK7s=
|
||||
github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23/go.mod h1:lUaIXCWzf7BRKTY5iEcrYy1TfgbYLYVIS/B2vPkJzOc=
|
||||
github.com/knqyf263/go-rpm-version v0.0.0-20240918084003-2afd7dc6a38f h1:xt29M2T6STgldg+WEP51gGePQCsQvklmP2eIhPIBK3g=
|
||||
github.com/knqyf263/go-rpm-version v0.0.0-20240918084003-2afd7dc6a38f/go.mod h1:i4sF0l1fFnY1aiw08QQSwVAFxHEm311Me3WsU/X7nL0=
|
||||
github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
|
||||
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
@@ -64,21 +78,38 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
|
||||
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
|
||||
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
|
||||
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
|
||||
github.com/oklog/ulid/v2 v2.1.1 h1:suPZ4ARWLOJLegGFiZZ1dFAkqzhMjL3J1TzI+5wHz8s=
|
||||
github.com/oklog/ulid/v2 v2.1.1/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ=
|
||||
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
|
||||
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
|
||||
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
|
||||
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
|
||||
github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
|
||||
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
|
||||
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.20.1 h1:sfCU6A8P3dXbKyWes02uxA2baehGux9dZHfEKtsTB1w=
|
||||
github.com/redis/go-redis/v9 v9.20.1/go.mod h1:v/M13XI1PVCDcm01VtPFOADfZtHf8YW3baQf57KlIkA=
|
||||
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
|
||||
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
|
||||
github.com/samber/lo v1.50.0 h1:XrG0xOeHs+4FQ8gJR97zDz5uOFMW7OwFWiFVzqopKgY=
|
||||
github.com/samber/lo v1.50.0/go.mod h1:RjZyNk6WSnUFRKK6EyOhsRJMqft3G+pg7dCWHQCWvsc=
|
||||
github.com/samber/oops v1.18.1 h1:qjhZbqbdyhWBKntkY8sxrDNKA8b4c5VHlmI1rli7X7M=
|
||||
github.com/samber/oops v1.18.1/go.mod h1:xYqvimigkKV70HyLXiBZJFpIWi2CGcc6Xx7eV+2HycI=
|
||||
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
|
||||
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
|
||||
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
|
||||
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
@@ -104,8 +135,14 @@ github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfS
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
|
||||
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
|
||||
go.mongodb.org/mongo-driver/v2 v2.8.0 h1:CxWDGQYY8QQwNjAl/aq2sfWakdnWZynnqJ9F4DhHbP8=
|
||||
go.mongodb.org/mongo-driver/v2 v2.8.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0=
|
||||
go.opentelemetry.io/otel v1.34.0 h1:zRLXxLCgL1WyKsPVrgbSdMN4c0FMkDAskSTQP+0hdUY=
|
||||
go.opentelemetry.io/otel v1.34.0/go.mod h1:OWFPOQ+h4G8xpyjgqo4SxJYdDQ/qmRH+wivy7zzx9oI=
|
||||
go.opentelemetry.io/otel/trace v1.34.0 h1:+ouXS2V8Rd4hp4580a8q23bg0azF2nI8cqLYnC8mh/k=
|
||||
go.opentelemetry.io/otel/trace v1.34.0/go.mod h1:Svm7lSjQD7kG7KJ/MUHPVXSDGz2OX4h0M2jHBhmSfRE=
|
||||
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
|
||||
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
|
||||
golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
|
||||
@@ -154,10 +191,14 @@ google.golang.org/grpc v1.64.0 h1:KH3VH9y/MgNQg1dE7b3XfVK0GsPSIzJwdF617gUSbvY=
|
||||
google.golang.org/grpc v1.64.0/go.mod h1:oxjF8E3FBnjp+/gVFYdWacaLDx9na1aqy9oovLpxQYg=
|
||||
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
|
||||
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
|
||||
oras.land/oras-go/v2 v2.6.2 h1:N04RXngAp1LJKTG6ifz3xHPipasEkWr+hFmInja5YKo=
|
||||
oras.land/oras-go/v2 v2.6.2/go.mod h1:PlTtg4JTDJkDe8yVHpM2wz7/YDc00GVas+i4jAW2TZ4=
|
||||
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
|
||||
|
||||
@@ -51,6 +51,9 @@ func RegisterRoutes(r *gin.Engine) {
|
||||
apiGroup.POST("/license", auth.RequireRole("owner"), postLicence)
|
||||
|
||||
apiGroup.GET("/servers", listServers)
|
||||
// Static segment, registered alongside /servers/:id exactly as
|
||||
// /servers/new already is — gin resolves statics ahead of wildcards.
|
||||
apiGroup.GET("/servers/tags", listKnownTags)
|
||||
apiGroup.POST("/servers", createServer)
|
||||
apiGroup.GET("/servers/new", newServer)
|
||||
apiGroup.POST("/servers/new", newServer)
|
||||
@@ -59,6 +62,7 @@ func RegisterRoutes(r *gin.Engine) {
|
||||
apiGroup.POST("/servers/:id/generate-key", generateKey)
|
||||
apiGroup.POST("/servers/:id/update-agent", updateAgent)
|
||||
apiGroup.POST("/servers/:id/apply-updates", applyUpdates)
|
||||
apiGroup.PUT("/servers/:id/tags", putServerTags)
|
||||
|
||||
apiGroup.GET("/agent/latest-version", getLatestAgentVersion)
|
||||
|
||||
@@ -115,11 +119,29 @@ func RegisterRoutes(r *gin.Engine) {
|
||||
providers.POST("/:id/ack-notice", ackAuthProviderNotice)
|
||||
}
|
||||
apiGroup.GET("/auth/presets", auth.RequireRole("owner", "admin"), listAuthPresets)
|
||||
|
||||
apiGroup.GET("/vulnerabilities", listVulnerabilities)
|
||||
apiGroup.GET("/vulnerabilities/summary", vulnerabilitySummary)
|
||||
apiGroup.POST("/vulnerabilities/rescan", auth.RequireRole("owner", "admin"), rescanVulnerabilities)
|
||||
apiGroup.POST("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), acceptFinding)
|
||||
apiGroup.DELETE("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), unacceptFinding)
|
||||
apiGroup.GET("/servers/:id/vulnerabilities", listServerVulnerabilities)
|
||||
apiGroup.GET("/servers/:id/packages", getServerPackages)
|
||||
apiGroup.GET("/packages/search", searchPackages)
|
||||
apiGroup.GET("/vuln-rules", listVulnRules)
|
||||
apiGroup.POST("/vuln-rules", auth.RequireRole("owner", "admin"), createVulnRule)
|
||||
apiGroup.PUT("/vuln-rules/:id", auth.RequireRole("owner", "admin"), updateVulnRule)
|
||||
apiGroup.DELETE("/vuln-rules/:id", auth.RequireRole("owner", "admin"), deleteVulnRule)
|
||||
}
|
||||
}
|
||||
|
||||
func listServers(c *gin.Context) {
|
||||
servers, err := services.ListServers(auth.InstanceID(c))
|
||||
sel, err := services.ParseTagFilters(c.QueryArray("tag"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
servers, err := services.ListServersFiltered(auth.InstanceID(c), sel)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
@@ -127,6 +149,47 @@ func listServers(c *gin.Context) {
|
||||
c.JSON(http.StatusOK, servers)
|
||||
}
|
||||
|
||||
func listKnownTags(c *gin.Context) {
|
||||
tags, err := services.KnownTags(auth.InstanceID(c))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, tags)
|
||||
}
|
||||
|
||||
func putServerTags(c *gin.Context) {
|
||||
var body struct {
|
||||
Tags map[string]string `json:"tags"`
|
||||
}
|
||||
if err := c.ShouldBindJSON(&body); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid body"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
serverID := c.Param("id")
|
||||
|
||||
before, err := services.GetServer(instanceID, serverID)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
|
||||
return
|
||||
}
|
||||
|
||||
if err := services.SetServerTags(instanceID, serverID, body.Tags); err != nil {
|
||||
if errors.Is(err, services.ErrInvalidTag) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "server.tags_updated", actorFromCtx(c), serverID, "",
|
||||
fmt.Sprintf("tags %v -> %v", before.Tags, body.Tags))
|
||||
c.JSON(http.StatusOK, gin.H{"tags": body.Tags})
|
||||
}
|
||||
|
||||
func createServer(c *gin.Context) {
|
||||
s, token, err := services.CreateServer(auth.InstanceID(c))
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// vulnGroup is one CVE across every server it affects.
|
||||
//
|
||||
// The board groups by CVE rather than listing findings flat: the same CVE on
|
||||
// forty servers is one decision, and a flat list makes it look like forty.
|
||||
type vulnGroup struct {
|
||||
CVEID string `json:"cve_id"`
|
||||
Severity string `json:"severity"`
|
||||
Title string `json:"title,omitempty"`
|
||||
ServerCount int `json:"server_count"`
|
||||
Findings []models.VulnFinding `json:"findings"`
|
||||
}
|
||||
|
||||
func listVulnerabilities(c *gin.Context) {
|
||||
findings, err := services.ListInstanceFindings(auth.InstanceID(c), services.FindingFilter{
|
||||
Severity: c.Query("severity"),
|
||||
State: c.DefaultQuery("state", models.FindingOpen),
|
||||
ServerID: c.Query("server"),
|
||||
Tags: tagsFromQuery(c),
|
||||
})
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, groupByCVE(findings))
|
||||
}
|
||||
|
||||
// groupByCVE collapses findings into one row per CVE, most severe first.
|
||||
func groupByCVE(findings []models.VulnFinding) []vulnGroup {
|
||||
index := map[string]*vulnGroup{}
|
||||
order := []string{}
|
||||
|
||||
for _, f := range findings {
|
||||
g, ok := index[f.CVEID]
|
||||
if !ok {
|
||||
g = &vulnGroup{CVEID: f.CVEID, Severity: f.Severity, Title: f.Title}
|
||||
index[f.CVEID] = g
|
||||
order = append(order, f.CVEID)
|
||||
}
|
||||
// Several servers can disagree on severity when their distributions
|
||||
// rate the same CVE differently. The highest is shown, because that is
|
||||
// the one deciding whether anyone acts.
|
||||
if models.SeverityRank(f.Severity) > models.SeverityRank(g.Severity) {
|
||||
g.Severity = f.Severity
|
||||
}
|
||||
g.Findings = append(g.Findings, f)
|
||||
}
|
||||
|
||||
out := make([]vulnGroup, 0, len(order))
|
||||
for _, id := range order {
|
||||
g := index[id]
|
||||
servers := map[string]bool{}
|
||||
for _, f := range g.Findings {
|
||||
servers[f.ServerID] = true
|
||||
}
|
||||
g.ServerCount = len(servers)
|
||||
out = append(out, *g)
|
||||
}
|
||||
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
ri, rj := models.SeverityRank(out[i].Severity), models.SeverityRank(out[j].Severity)
|
||||
if ri != rj {
|
||||
return ri > rj
|
||||
}
|
||||
return out[i].ServerCount > out[j].ServerCount
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// tagsFromQuery reads repeated tag=key:value parameters.
|
||||
func tagsFromQuery(c *gin.Context) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, raw := range c.QueryArray("tag") {
|
||||
k, v, ok := strings.Cut(raw, ":")
|
||||
if !ok || k == "" {
|
||||
continue
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func vulnerabilitySummary(c *gin.Context) {
|
||||
counts, err := services.CountOpenFindingsBySeverity(auth.InstanceID(c))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
resp := gin.H{"counts": counts}
|
||||
|
||||
// Database freshness travels with the counts rather than living in
|
||||
// settings: a fleet scanned against a three-week-old database must say so
|
||||
// wherever its findings are read, not somewhere the reader has to go and
|
||||
// look for it.
|
||||
if meta, err := services.GetVulnDBMeta(); err == nil && meta != nil {
|
||||
resp["db_version"] = meta.DBVersion
|
||||
resp["pulled_at"] = meta.PulledAt
|
||||
resp["last_full_scan_at"] = meta.LastFullScanAt
|
||||
if meta.LastError != "" {
|
||||
resp["last_error"] = meta.LastError
|
||||
}
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, resp)
|
||||
}
|
||||
|
||||
func rescanVulnerabilities(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
|
||||
n, err := services.MarkInstanceForRescan(instanceID)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rescan", actorFromCtx(c), "", "",
|
||||
"queued "+strconv.FormatInt(n, 10)+" server(s) for rescan")
|
||||
c.JSON(http.StatusOK, gin.H{"queued": n})
|
||||
}
|
||||
|
||||
type acceptFindingRequest struct {
|
||||
Reason string `json:"reason"`
|
||||
Until time.Time `json:"until"`
|
||||
}
|
||||
|
||||
func acceptFinding(c *gin.Context) {
|
||||
var req acceptFindingRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
|
||||
// Both rejected deliberately. An acceptance with no reason is a dismissal
|
||||
// nobody can audit, and one already expired is a permanent dismissal
|
||||
// wearing an expiry — the graveyard the expiry exists to prevent.
|
||||
if strings.TrimSpace(req.Reason) == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "a reason is required"})
|
||||
return
|
||||
}
|
||||
if req.Until.IsZero() || !req.Until.After(time.Now()) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "until must be a future date"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
actor := actorFromCtx(c)
|
||||
|
||||
f, err := services.AcceptFinding(instanceID, c.Param("id"), actor, req.Reason, req.Until)
|
||||
if err != nil {
|
||||
writeFindingError(c, err)
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.accepted", actor, f.ServerID, "",
|
||||
f.CVEID+" on "+f.PackageName+" accepted until "+req.Until.Format(time.RFC3339)+": "+req.Reason)
|
||||
c.JSON(http.StatusOK, f)
|
||||
}
|
||||
|
||||
func unacceptFinding(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
actor := actorFromCtx(c)
|
||||
|
||||
f, err := services.UnacceptFinding(instanceID, c.Param("id"))
|
||||
if err != nil {
|
||||
writeFindingError(c, err)
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.unaccepted", actor, f.ServerID, "",
|
||||
f.CVEID+" on "+f.PackageName+" returned to open")
|
||||
c.JSON(http.StatusOK, f)
|
||||
}
|
||||
|
||||
func writeFindingError(c *gin.Context, err error) {
|
||||
if errors.Is(err, services.ErrFindingNotFound) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "finding not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
}
|
||||
|
||||
func listServerVulnerabilities(c *gin.Context) {
|
||||
findings, err := services.ListFindings(c.Request.Context(), auth.InstanceID(c), c.Param("id"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if findings == nil {
|
||||
findings = []models.VulnFinding{}
|
||||
}
|
||||
c.JSON(http.StatusOK, findings)
|
||||
}
|
||||
|
||||
func getServerPackages(c *gin.Context) {
|
||||
sp, err := services.ListPackages(auth.InstanceID(c), c.Param("id"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if sp == nil {
|
||||
// Not a 404: an agent that has not reported yet is the normal state for
|
||||
// the first hour after install, and is a different thing from a bad
|
||||
// server id.
|
||||
c.JSON(http.StatusOK, gin.H{"reported": false})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, sp)
|
||||
}
|
||||
|
||||
func searchPackages(c *gin.Context) {
|
||||
name := c.Query("name")
|
||||
if name == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "name is required"})
|
||||
return
|
||||
}
|
||||
hits, err := services.SearchPackages(auth.InstanceID(c), name)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, hits)
|
||||
}
|
||||
|
||||
func listVulnRules(c *gin.Context) {
|
||||
rules, err := services.ListVulnRules(auth.InstanceID(c))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, rules)
|
||||
}
|
||||
|
||||
func createVulnRule(c *gin.Context) {
|
||||
var r models.VulnAlertRule
|
||||
if err := c.ShouldBindJSON(&r); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
created, err := services.CreateVulnRule(instanceID, &r)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rule_created", actorFromCtx(c), "", "", "rule "+created.Name)
|
||||
c.JSON(http.StatusCreated, created)
|
||||
}
|
||||
|
||||
func updateVulnRule(c *gin.Context) {
|
||||
var r models.VulnAlertRule
|
||||
if err := c.ShouldBindJSON(&r); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
if err := services.UpdateVulnRule(instanceID, c.Param("id"), &r); err != nil {
|
||||
if errors.Is(err, services.ErrVulnRuleNotFound) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "rule not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rule_updated", actorFromCtx(c), "", "", "rule "+r.Name)
|
||||
c.JSON(http.StatusOK, gin.H{"status": "updated"})
|
||||
}
|
||||
|
||||
func deleteVulnRule(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
if err := services.DeleteVulnRule(instanceID, c.Param("id")); err != nil {
|
||||
if errors.Is(err, services.ErrVulnRuleNotFound) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "rule not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rule_deleted", actorFromCtx(c), "", "", "rule "+c.Param("id"))
|
||||
c.JSON(http.StatusOK, gin.H{"status": "deleted"})
|
||||
}
|
||||
@@ -13,7 +13,9 @@ import (
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
|
||||
"github.com/gin-gonic/gin"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
)
|
||||
|
||||
func registerWorkflowRoutes(g *gin.RouterGroup) {
|
||||
@@ -34,6 +36,8 @@ func registerWorkflowRoutes(g *gin.RouterGroup) {
|
||||
g.DELETE("/workflows/:id", deleteWorkflow)
|
||||
g.POST("/workflows/:id/run", runWorkflow)
|
||||
g.GET("/workflows/:id/runs", listWorkflowRuns)
|
||||
g.PUT("/workflows/:id/schedule", putWorkflowSchedule)
|
||||
g.GET("/workflows/:id/schedule/preview", previewWorkflowSchedule)
|
||||
|
||||
g.GET("/runs/:runId", getRun)
|
||||
g.POST("/runs/:runId/cancel", cancelRun)
|
||||
@@ -343,6 +347,10 @@ func deleteWorkflow(c *gin.Context) {
|
||||
func runWorkflow(c *gin.Context) {
|
||||
runID, err := services.TriggerWorkflow(auth.InstanceID(c), c.Param("id"), actorFromCtx(c))
|
||||
if err != nil {
|
||||
if errors.Is(err, services.ErrNoTargets) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "this workflow matches no servers"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
@@ -382,3 +390,51 @@ func cancelRun(c *gin.Context) {
|
||||
services.LogEvent(auth.InstanceID(c), "workflow.run_cancelled", actorFromCtx(c), "", c.Param("runId"), "run cancelled")
|
||||
c.JSON(http.StatusOK, gin.H{"cancelled": true})
|
||||
}
|
||||
|
||||
func putWorkflowSchedule(c *gin.Context) {
|
||||
var body models.Schedule
|
||||
if err := c.ShouldBindJSON(&body); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid body"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
next, err := services.SetSchedule(instanceID, c.Param("id"), &body)
|
||||
if err != nil {
|
||||
if errors.Is(err, workflowsched.ErrBadSchedule) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if errors.Is(err, mongo.ErrNoDocuments) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "workflow not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "workflow.schedule_updated", actorFromCtx(c), "", c.Param("id"),
|
||||
fmt.Sprintf("schedule %q %s enabled=%v", body.Cron, body.TZ, body.Enabled))
|
||||
c.JSON(http.StatusOK, gin.H{"schedule": body, "next_run_at": next})
|
||||
}
|
||||
|
||||
// previewWorkflowSchedule exists so the browser and the scheduler agree on
|
||||
// what a cron string means. A client-side cron parser that disagrees with the
|
||||
// server by one field is a bug found in production, at night.
|
||||
func previewWorkflowSchedule(c *gin.Context) {
|
||||
expr := c.Query("cron")
|
||||
tz := c.Query("tz")
|
||||
|
||||
occurrences := make([]time.Time, 0, 3)
|
||||
from := time.Now()
|
||||
for i := 0; i < 3; i++ {
|
||||
next, err := workflowsched.NextOccurrence(expr, tz, from)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
occurrences = append(occurrences, next)
|
||||
from = next
|
||||
}
|
||||
c.JSON(http.StatusOK, gin.H{"occurrences": occurrences})
|
||||
}
|
||||
|
||||
@@ -28,6 +28,45 @@ type SyncRequest struct {
|
||||
|
||||
type SyncResponse struct {
|
||||
PublicKeys []string `json:"public_keys"`
|
||||
// CollectPackages tells the agent whether this instance's licence grants
|
||||
// vulnerability scanning. Absent decodes as false, which is the safe
|
||||
// direction: an older server leaves agents collecting nothing.
|
||||
CollectPackages bool `json:"collect_packages,omitempty"`
|
||||
}
|
||||
|
||||
type OSRelease struct {
|
||||
Family string `json:"family"`
|
||||
// VersionId is not optional: Ubuntu 22.04 and 24.04 publish different fixed
|
||||
// versions for the same CVE, so a scan without it is guesswork.
|
||||
VersionId string `json:"version_id"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Epoch int32 `json:"epoch,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
// SourceName is what the Debian and Ubuntu feeds are keyed on: one advisory
|
||||
// against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
SourceName string `json:"source_name,omitempty"`
|
||||
}
|
||||
|
||||
// ReportPackagesRequest carries a server's installed package set.
|
||||
//
|
||||
// The agent calls twice at most: first with Packages empty, offering only the
|
||||
// hash. If the server already holds it, NeedFull is false and the ~150KB body
|
||||
// is never sent.
|
||||
type ReportPackagesRequest struct {
|
||||
ServerId string `json:"server_id"`
|
||||
AgentToken string `json:"agent_token"`
|
||||
Hash string `json:"hash"`
|
||||
Os OSRelease `json:"os"`
|
||||
Packages []InstalledPackage `json:"packages,omitempty"`
|
||||
}
|
||||
|
||||
type ReportPackagesResponse struct {
|
||||
NeedFull bool `json:"need_full"`
|
||||
}
|
||||
|
||||
type UploadKeyRequest struct {
|
||||
@@ -326,6 +365,7 @@ type VantageServer interface {
|
||||
SyncKeys(context.Context, *SyncRequest) (*SyncResponse, error)
|
||||
UploadGeneratedKey(context.Context, *UploadKeyRequest) (*UploadKeyResponse, error)
|
||||
ReportUpdates(context.Context, *ReportUpdatesRequest) (*ReportUpdatesResponse, error)
|
||||
ReportPackages(context.Context, *ReportPackagesRequest) (*ReportPackagesResponse, error)
|
||||
ReportInventory(context.Context, *InventoryReport) (*InventoryReportResponse, error)
|
||||
SyncMonitors(context.Context, *SyncMonitorsRequest) (*SyncMonitorsResponse, error)
|
||||
ReportChecks(context.Context, *ReportChecksRequest) (*ReportChecksResponse, error)
|
||||
@@ -351,6 +391,10 @@ func (UnimplementedVantageServer) ReportUpdates(context.Context, *ReportUpdatesR
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportUpdates not implemented")
|
||||
}
|
||||
|
||||
func (UnimplementedVantageServer) ReportPackages(context.Context, *ReportPackagesRequest) (*ReportPackagesResponse, error) {
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportPackages not implemented")
|
||||
}
|
||||
|
||||
func (UnimplementedVantageServer) ReportInventory(context.Context, *InventoryReport) (*InventoryReportResponse, error) {
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportInventory not implemented")
|
||||
}
|
||||
@@ -376,6 +420,7 @@ type VantageClient interface {
|
||||
SyncKeys(ctx context.Context, in *SyncRequest, opts ...grpc.CallOption) (*SyncResponse, error)
|
||||
UploadGeneratedKey(ctx context.Context, in *UploadKeyRequest, opts ...grpc.CallOption) (*UploadKeyResponse, error)
|
||||
ReportUpdates(ctx context.Context, in *ReportUpdatesRequest, opts ...grpc.CallOption) (*ReportUpdatesResponse, error)
|
||||
ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error)
|
||||
ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error)
|
||||
SyncMonitors(ctx context.Context, in *SyncMonitorsRequest, opts ...grpc.CallOption) (*SyncMonitorsResponse, error)
|
||||
ReportChecks(ctx context.Context, in *ReportChecksRequest, opts ...grpc.CallOption) (*ReportChecksResponse, error)
|
||||
@@ -423,6 +468,14 @@ func (c *keyManagerClient) ReportUpdates(ctx context.Context, in *ReportUpdatesR
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error) {
|
||||
out := new(ReportPackagesResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportPackages", in, out, opts...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error) {
|
||||
out := new(InventoryReportResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportInventory", in, out, opts...); err != nil {
|
||||
@@ -475,6 +528,7 @@ var Vantage_ServiceDesc = grpc.ServiceDesc{
|
||||
{MethodName: "SyncKeys", Handler: _Vantage_SyncKeys_Handler},
|
||||
{MethodName: "UploadGeneratedKey", Handler: _Vantage_UploadGeneratedKey_Handler},
|
||||
{MethodName: "ReportUpdates", Handler: _Vantage_ReportUpdates_Handler},
|
||||
{MethodName: "ReportPackages", Handler: _Vantage_ReportPackages_Handler},
|
||||
{MethodName: "ReportInventory", Handler: _Vantage_ReportInventory_Handler},
|
||||
{MethodName: "SyncMonitors", Handler: _Vantage_SyncMonitors_Handler},
|
||||
{MethodName: "ReportChecks", Handler: _Vantage_ReportChecks_Handler},
|
||||
@@ -556,6 +610,21 @@ func _Vantage_ReportUpdates_Handler(srv interface{}, ctx context.Context, dec fu
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _Vantage_ReportPackages_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(ReportPackagesRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(VantageServer).ReportPackages(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{Server: srv, FullMethod: "/vantage.v1.Vantage/ReportPackages"}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(VantageServer).ReportPackages(ctx, req.(*ReportPackagesRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _Vantage_ReportInventory_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(InventoryReport)
|
||||
if err := dec(in); err != nil {
|
||||
|
||||
@@ -63,7 +63,13 @@ func (s *vantageServer) SyncKeys(ctx context.Context, req *pb.SyncRequest) (*pb.
|
||||
return nil, status.Errorf(codes.Internal, "failed to build authorized keys: %v", err)
|
||||
}
|
||||
|
||||
return &pb.SyncResponse{PublicKeys: keys}, nil
|
||||
// Carried on the 30s key poll rather than its own RPC: the agent needs it
|
||||
// before its hourly package report, and this is the only message it already
|
||||
// receives that often.
|
||||
return &pb.SyncResponse{
|
||||
PublicKeys: keys,
|
||||
CollectPackages: services.VulnScanningEnabled(srv.InstanceID),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *vantageServer) UploadGeneratedKey(ctx context.Context, req *pb.UploadKeyRequest) (*pb.UploadKeyResponse, error) {
|
||||
@@ -104,6 +110,59 @@ func (s *vantageServer) ReportUpdates(ctx context.Context, req *pb.ReportUpdates
|
||||
return &pb.ReportUpdatesResponse{}, nil
|
||||
}
|
||||
|
||||
// ReportPackages stores a server's installed package set.
|
||||
//
|
||||
// It does NOT match against the vulnerability database. Matching happens in
|
||||
// vulnsched, on the leader: every replica would otherwise need the ~50MB
|
||||
// database resident, and a database refresh would have N replicas racing to
|
||||
// rescan the same fleet and sending N digests to the customer.
|
||||
func (s *vantageServer) ReportPackages(ctx context.Context, req *pb.ReportPackagesRequest) (*pb.ReportPackagesResponse, error) {
|
||||
srv, err := services.ValidateAgentToken(req.ServerId, req.AgentToken)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Unauthenticated, "invalid agent token")
|
||||
}
|
||||
|
||||
// The agent's collect_packages flag is an optimisation; this is the
|
||||
// boundary. An agent that ignores the flag still stores nothing.
|
||||
if !services.VulnScanningEnabled(srv.InstanceID) {
|
||||
return &pb.ReportPackagesResponse{NeedFull: false}, nil
|
||||
}
|
||||
|
||||
// The offer call: a hash and no packages. Answering NeedFull=false here is
|
||||
// what saves the ~150KB body on the overwhelming majority of reports.
|
||||
if len(req.Packages) == 0 {
|
||||
known, err := services.HasPackageHash(srv.InstanceID, srv.ServerID, req.Hash)
|
||||
if err != nil {
|
||||
log.Printf("package hash lookup for %s: %v", srv.ServerID, err)
|
||||
return nil, status.Errorf(codes.Internal, "package hash lookup failed")
|
||||
}
|
||||
return &pb.ReportPackagesResponse{NeedFull: !known}, nil
|
||||
}
|
||||
|
||||
pkgs := make([]models.InstalledPackage, len(req.Packages))
|
||||
for i, p := range req.Packages {
|
||||
pkgs[i] = models.InstalledPackage{
|
||||
Name: p.Name,
|
||||
Version: p.Version,
|
||||
Epoch: int(p.Epoch),
|
||||
Arch: p.Arch,
|
||||
SourceName: p.SourceName,
|
||||
}
|
||||
}
|
||||
|
||||
os := models.OSRelease{
|
||||
Family: req.Os.Family,
|
||||
VersionID: req.Os.VersionId,
|
||||
Arch: req.Os.Arch,
|
||||
}
|
||||
|
||||
if err := services.StorePackages(srv.InstanceID, srv.ServerID, os, req.Hash, pkgs); err != nil {
|
||||
log.Printf("store packages for %s: %v", srv.ServerID, err)
|
||||
return nil, status.Errorf(codes.Internal, "failed to store packages")
|
||||
}
|
||||
return &pb.ReportPackagesResponse{NeedFull: false}, nil
|
||||
}
|
||||
|
||||
func (s *vantageServer) ReportInventory(ctx context.Context, req *pb.InventoryReport) (*pb.InventoryReportResponse, error) {
|
||||
srv, err := services.ValidateAgentToken(req.ServerId, req.AgentToken)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
)
|
||||
|
||||
// Scan status values for ServerPackages.
|
||||
const (
|
||||
ScanStatusOK = "ok"
|
||||
ScanStatusUnsupported = "unsupported"
|
||||
)
|
||||
|
||||
type OSRelease struct {
|
||||
Family string `bson:"family" json:"family"`
|
||||
VersionID string `bson:"version_id" json:"version_id"`
|
||||
Arch string `bson:"arch" json:"arch"`
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `bson:"name" json:"name"`
|
||||
Version string `bson:"version" json:"version"`
|
||||
Epoch int `bson:"epoch,omitempty" json:"epoch,omitempty"`
|
||||
Arch string `bson:"arch" json:"arch"`
|
||||
// SourceName is what the Debian and Ubuntu feeds are keyed on. One advisory
|
||||
// against "openssl" covers the binaries libssl3, openssl and libssl-dev;
|
||||
// matching on binary name alone finds one of the three.
|
||||
SourceName string `bson:"source_name,omitempty" json:"source_name,omitempty"`
|
||||
}
|
||||
|
||||
// ServerPackages holds one server's whole package set in ONE document rather
|
||||
// than one document per package. The hash has already established that
|
||||
// something changed, so a report is a single atomic upsert with no delta logic
|
||||
// to get wrong. A typical Linux host is ~2000 packages and ~150KB, comfortably
|
||||
// inside the 16MB document limit.
|
||||
type ServerPackages struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
OS OSRelease `bson:"os" json:"os"`
|
||||
Hash string `bson:"hash" json:"hash"`
|
||||
Packages []InstalledPackage `bson:"packages" json:"packages"`
|
||||
CollectedAt time.Time `bson:"collected_at" json:"collected_at"`
|
||||
ScanPending bool `bson:"scan_pending" json:"scan_pending"`
|
||||
ScannedAt time.Time `bson:"scanned_at,omitempty" json:"scanned_at,omitempty"`
|
||||
// Status distinguishes a scanned host from one whose distribution we hold
|
||||
// no feed for. Reporting zero findings for an unsupported distribution is
|
||||
// indistinguishable from reporting a clean host, and one of those is a lie.
|
||||
Status string `bson:"status" json:"status"`
|
||||
DBVersion int `bson:"db_version" json:"db_version"`
|
||||
}
|
||||
@@ -44,24 +44,25 @@ type Inventory struct {
|
||||
}
|
||||
|
||||
type Server struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"_id,omitempty"`
|
||||
InstanceID string `bson:"instance_id" json:"instance_id"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
Hostname string `bson:"hostname" json:"hostname"`
|
||||
IPAddress string `bson:"ip_address" json:"ip_address"`
|
||||
OSInfo string `bson:"os_info" json:"os_info"`
|
||||
OSType string `bson:"os_type,omitempty" json:"os_type,omitempty"`
|
||||
ConsoleProtocols []string `bson:"console_protocols,omitempty" json:"console_protocols,omitempty"`
|
||||
SSHPort int `bson:"ssh_port,omitempty" json:"ssh_port,omitempty"`
|
||||
RDPPort int `bson:"rdp_port,omitempty" json:"rdp_port,omitempty"`
|
||||
PreRegToken string `bson:"pre_reg_token,omitempty" json:"pre_reg_token,omitempty"`
|
||||
PreRegExpires *time.Time `bson:"pre_reg_expires,omitempty" json:"pre_reg_expires,omitempty"`
|
||||
AgentTokenHash string `bson:"agent_token_hash,omitempty" json:"-"`
|
||||
Status string `bson:"status" json:"status"`
|
||||
AgentVersion string `bson:"agent_version,omitempty" json:"agent_version,omitempty"`
|
||||
LastSeen *time.Time `bson:"last_seen,omitempty" json:"last_seen,omitempty"`
|
||||
AvailableUpdates []PackageUpdate `bson:"available_updates,omitempty" json:"available_updates,omitempty"`
|
||||
UpdatesCheckedAt *time.Time `bson:"updates_checked_at,omitempty" json:"updates_checked_at,omitempty"`
|
||||
Inventory *Inventory `bson:"inventory,omitempty" json:"inventory,omitempty"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"_id,omitempty"`
|
||||
InstanceID string `bson:"instance_id" json:"instance_id"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
Hostname string `bson:"hostname" json:"hostname"`
|
||||
IPAddress string `bson:"ip_address" json:"ip_address"`
|
||||
OSInfo string `bson:"os_info" json:"os_info"`
|
||||
OSType string `bson:"os_type,omitempty" json:"os_type,omitempty"`
|
||||
ConsoleProtocols []string `bson:"console_protocols,omitempty" json:"console_protocols,omitempty"`
|
||||
SSHPort int `bson:"ssh_port,omitempty" json:"ssh_port,omitempty"`
|
||||
RDPPort int `bson:"rdp_port,omitempty" json:"rdp_port,omitempty"`
|
||||
PreRegToken string `bson:"pre_reg_token,omitempty" json:"pre_reg_token,omitempty"`
|
||||
PreRegExpires *time.Time `bson:"pre_reg_expires,omitempty" json:"pre_reg_expires,omitempty"`
|
||||
AgentTokenHash string `bson:"agent_token_hash,omitempty" json:"-"`
|
||||
Status string `bson:"status" json:"status"`
|
||||
AgentVersion string `bson:"agent_version,omitempty" json:"agent_version,omitempty"`
|
||||
LastSeen *time.Time `bson:"last_seen,omitempty" json:"last_seen,omitempty"`
|
||||
AvailableUpdates []PackageUpdate `bson:"available_updates,omitempty" json:"available_updates,omitempty"`
|
||||
UpdatesCheckedAt *time.Time `bson:"updates_checked_at,omitempty" json:"updates_checked_at,omitempty"`
|
||||
Inventory *Inventory `bson:"inventory,omitempty" json:"inventory,omitempty"`
|
||||
Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
}
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
)
|
||||
|
||||
// Finding states.
|
||||
const (
|
||||
FindingOpen = "open"
|
||||
FindingFixed = "fixed"
|
||||
FindingAccepted = "accepted"
|
||||
)
|
||||
|
||||
// Severities. Lowercase and fixed; SeverityRank orders them.
|
||||
const (
|
||||
SeverityUnknown = "unknown"
|
||||
SeverityLow = "low"
|
||||
SeverityMedium = "medium"
|
||||
SeverityHigh = "high"
|
||||
SeverityCritical = "critical"
|
||||
)
|
||||
|
||||
// SeverityRank orders severities for threshold comparisons. An unrecognised
|
||||
// value ranks lowest rather than panicking: severity comes from a third-party
|
||||
// feed, and an unexpected string must not stop a scan.
|
||||
func SeverityRank(s string) int {
|
||||
switch s {
|
||||
case SeverityCritical:
|
||||
return 4
|
||||
case SeverityHigh:
|
||||
return 3
|
||||
case SeverityMedium:
|
||||
return 2
|
||||
case SeverityLow:
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// Acceptance records a decision someone will be asked to justify, so who, why
|
||||
// and until when all live on the document as well as in the audit log.
|
||||
type Acceptance struct {
|
||||
By string `bson:"by" json:"by"`
|
||||
Reason string `bson:"reason" json:"reason"`
|
||||
Until time.Time `bson:"until" json:"until"`
|
||||
At time.Time `bson:"at" json:"at"`
|
||||
}
|
||||
|
||||
// VulnFinding is one vulnerable package on one server.
|
||||
//
|
||||
// Findings are never deleted when a package is patched: the state moves to
|
||||
// "fixed" with FixedAt stamped, which is what keeps "what did we remediate last
|
||||
// quarter" answerable.
|
||||
type VulnFinding struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"id"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
|
||||
CVEID string `bson:"cve_id" json:"cve_id"`
|
||||
PackageName string `bson:"package_name" json:"package_name"`
|
||||
Installed string `bson:"installed_version" json:"installed_version"`
|
||||
// FixedIn empty means no vendor fix has been published. That is a real and
|
||||
// common state and must never be conflated with "not vulnerable" — it is
|
||||
// the finding most in need of acceptance, since there is nothing to patch.
|
||||
FixedIn string `bson:"fixed_in,omitempty" json:"fixed_in,omitempty"`
|
||||
Severity string `bson:"severity" json:"severity"`
|
||||
CVSSScore float64 `bson:"cvss_score,omitempty" json:"cvss_score,omitempty"`
|
||||
Title string `bson:"title,omitempty" json:"title,omitempty"`
|
||||
References []string `bson:"references,omitempty" json:"references,omitempty"`
|
||||
|
||||
State string `bson:"state" json:"state"`
|
||||
FirstSeen time.Time `bson:"first_seen" json:"first_seen"`
|
||||
LastSeen time.Time `bson:"last_seen" json:"last_seen"`
|
||||
FixedAt *time.Time `bson:"fixed_at,omitempty" json:"fixed_at,omitempty"`
|
||||
Accepted *Acceptance `bson:"accepted,omitempty" json:"accepted,omitempty"`
|
||||
}
|
||||
|
||||
// VulnDBMeta is a singleton and deliberately carries no instance_id: the
|
||||
// vulnerability database is a property of the deployment, not of a tenant.
|
||||
// Same reasoning as the migrations collection, and the reason vulndb_meta is
|
||||
// absent from ScopedCollections.
|
||||
type VulnDBMeta struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
DBVersion int `bson:"db_version" json:"db_version"`
|
||||
PulledAt time.Time `bson:"pulled_at" json:"pulled_at"`
|
||||
LastFullScanAt time.Time `bson:"last_full_scan_at,omitempty" json:"last_full_scan_at,omitempty"`
|
||||
LastError string `bson:"last_error,omitempty" json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// VulnAlertRule routes newly opened findings to notification channels.
|
||||
//
|
||||
// Tags resolve through services.ResolveTargets rather than a second matcher:
|
||||
// that function is already the single answer to which servers a selector
|
||||
// touches, and a rule disagreeing with a workflow about what env:prod means
|
||||
// would be worse than having no filter.
|
||||
type VulnAlertRule struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"id"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
Name string `bson:"name" json:"name"`
|
||||
Enabled bool `bson:"enabled" json:"enabled"`
|
||||
MinSeverity string `bson:"min_severity" json:"min_severity"`
|
||||
Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"`
|
||||
ChannelIDs []string `bson:"channel_ids" json:"channel_ids"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -44,13 +44,32 @@ type StepOverride struct {
|
||||
SecretRefs []string `bson:"secret_refs,omitempty" json:"secret_refs,omitempty"`
|
||||
}
|
||||
|
||||
type Schedule struct {
|
||||
Enabled bool `bson:"enabled" json:"enabled"`
|
||||
Cron string `bson:"cron" json:"cron"` // 5-field: minute hour dom month dow
|
||||
TZ string `bson:"tz" json:"tz"` // IANA name, e.g. Europe/London
|
||||
}
|
||||
|
||||
// Skip records why an occurrence did not run. Recording a reason nobody reads
|
||||
// is the same as not recording one, so this is surfaced in the UI.
|
||||
type Skip struct {
|
||||
Reason string `bson:"reason" json:"reason"` // "missed" | "already_running"
|
||||
Due time.Time `bson:"due" json:"due"`
|
||||
At time.Time `bson:"at" json:"at"`
|
||||
}
|
||||
|
||||
type Workflow struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
InstanceID string `bson:"instance_id" json:"instance_id"`
|
||||
WorkflowID string `bson:"workflow_id" json:"workflow_id"`
|
||||
Name string `bson:"name" json:"name"`
|
||||
TargetServerIDs []string `bson:"target_server_ids" json:"target_server_ids"`
|
||||
TargetTags map[string]string `bson:"target_tags,omitempty" json:"target_tags,omitempty"`
|
||||
Steps []WorkflowStepRef `bson:"steps" json:"steps"`
|
||||
Schedule *Schedule `bson:"schedule,omitempty" json:"schedule,omitempty"`
|
||||
NextRunAt *time.Time `bson:"next_run_at,omitempty" json:"next_run_at,omitempty"`
|
||||
LastRunAt *time.Time `bson:"last_run_at,omitempty" json:"last_run_at,omitempty"`
|
||||
LastSkipped *Skip `bson:"last_skipped,omitempty" json:"last_skipped,omitempty"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
|
||||
}
|
||||
|
||||
@@ -26,7 +26,11 @@ func (e Event) title() string {
|
||||
verb = "is DOWN"
|
||||
}
|
||||
var s string
|
||||
if e.Type == TypeServer {
|
||||
if e.Type == TypeVuln {
|
||||
// A digest is not a transition. MonitorName already carries the whole
|
||||
// headline ("12 new critical across 4 servers"), so no verb applies.
|
||||
s = fmt.Sprintf("[Vantage] %s", e.MonitorName)
|
||||
} else if e.Type == TypeServer {
|
||||
if e.NewStatus == models.StatusDown {
|
||||
verb = "went offline"
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/shared/mail"
|
||||
)
|
||||
|
||||
// TypeVuln marks an event whose subject is a batch of new vulnerability
|
||||
// findings rather than a state transition. It exists so title() does not
|
||||
// describe a digest as something going "DOWN".
|
||||
const TypeVuln = "vulnerability"
|
||||
|
||||
// VulnDigest is one batch of newly opened findings, ready to send.
|
||||
//
|
||||
// One per rule per scan, never one per finding: a database refresh can open
|
||||
// several hundred at once, and a message each would rate-limit the webhook or
|
||||
// get the channel muted — either way the alerts stop being read.
|
||||
type VulnDigest struct {
|
||||
InstanceName string
|
||||
RuleName string
|
||||
Summary string
|
||||
TopSeverity string
|
||||
Count int
|
||||
Rows []mail.VulnDigestRow
|
||||
More int
|
||||
DBAge string
|
||||
}
|
||||
|
||||
// DispatchVulnDigest delivers a digest over one channel.
|
||||
//
|
||||
// SMTP gets its own template so a vulnerability digest does not arrive dressed
|
||||
// as a monitor alert. The other four transports carry short text, so they reuse
|
||||
// the existing Event path rather than growing a second payload shape per
|
||||
// transport.
|
||||
func DispatchVulnDigest(ch models.NotificationChannel, d VulnDigest) error {
|
||||
if ch.Type == models.ChannelSMTP {
|
||||
to := ch.Config["to"]
|
||||
sender := mail.Sender{
|
||||
Host: ch.Config["host"],
|
||||
Port: ch.Config["port"],
|
||||
From: ch.Config["from"],
|
||||
Username: ch.Config["username"],
|
||||
Password: ch.Config["password"],
|
||||
}
|
||||
if !sender.Enabled() || sender.Port == "" || to == "" {
|
||||
return fmt.Errorf("smtp: missing host/port/from/to")
|
||||
}
|
||||
return sender.SendVulnDigest(to, mail.VulnDigest{
|
||||
InstanceName: d.InstanceName,
|
||||
Count: d.Count,
|
||||
TopSeverity: d.TopSeverity,
|
||||
Summary: d.Summary,
|
||||
Rows: d.Rows,
|
||||
More: d.More,
|
||||
DBAge: d.DBAge,
|
||||
})
|
||||
}
|
||||
|
||||
return Dispatch(ch, Event{
|
||||
MonitorName: d.Summary,
|
||||
Type: TypeVuln,
|
||||
Message: vulnLines(d),
|
||||
})
|
||||
}
|
||||
|
||||
// vulnLines renders the finding list for the text-only transports, capped by
|
||||
// whatever the caller already put in Rows.
|
||||
func vulnLines(d VulnDigest) string {
|
||||
var s string
|
||||
for _, r := range d.Rows {
|
||||
fix := "no fix published"
|
||||
if r.FixedIn != "" {
|
||||
fix = "fixed in " + r.FixedIn
|
||||
}
|
||||
s += fmt.Sprintf("\n• %s (%s) — %s on %s, %s", r.CVEID, r.Severity, r.PackageName, r.ServerName, fix)
|
||||
}
|
||||
if d.More > 0 {
|
||||
s += fmt.Sprintf("\n…and %d more.", d.More)
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,420 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// FindingDiff is what one server's scan changes.
|
||||
type FindingDiff struct {
|
||||
Upserts []models.VulnFinding
|
||||
FixedIDs []bson.ObjectID
|
||||
ReopenIDs []bson.ObjectID
|
||||
// NewlyOpened is what the digest reports: findings that were not open
|
||||
// before this scan. A finding that was already open must not re-alert every
|
||||
// tick, or the digest becomes noise and stops being read.
|
||||
NewlyOpened []models.VulnFinding
|
||||
}
|
||||
|
||||
func findingKey(cveID, pkg string) string { return cveID + "\x00" + pkg }
|
||||
|
||||
// DiffFindings computes the state changes for one server's scan.
|
||||
//
|
||||
// Pure by design: no database, no clock of its own. The ordering below is
|
||||
// load-bearing — see the comment above the second loop.
|
||||
func DiffFindings(existing []models.VulnFinding, results []vulndb.Result, now time.Time) FindingDiff {
|
||||
var d FindingDiff
|
||||
|
||||
byKey := make(map[string]models.VulnFinding, len(existing))
|
||||
for _, f := range existing {
|
||||
byKey[findingKey(f.CVEID, f.PackageName)] = f
|
||||
}
|
||||
|
||||
seen := make(map[string]bool, len(results))
|
||||
for _, r := range results {
|
||||
key := findingKey(r.CVEID, r.PackageName)
|
||||
seen[key] = true
|
||||
|
||||
prev, had := byKey[key]
|
||||
|
||||
f := models.VulnFinding{
|
||||
CVEID: r.CVEID,
|
||||
PackageName: r.PackageName,
|
||||
Installed: r.Installed,
|
||||
FixedIn: r.FixedIn,
|
||||
Severity: r.Severity,
|
||||
State: models.FindingOpen,
|
||||
FirstSeen: now,
|
||||
LastSeen: now,
|
||||
}
|
||||
|
||||
if had {
|
||||
f.ID = prev.ID
|
||||
// Preserved, never overwritten: an upsert that moves first_seen
|
||||
// forward makes every finding look discovered today.
|
||||
f.FirstSeen = prev.FirstSeen
|
||||
|
||||
// A live acceptance survives the scan untouched: it is suppressed
|
||||
// from counts and alerts until its expiry, then reopens on its own.
|
||||
if prev.State == models.FindingAccepted && prev.Accepted != nil {
|
||||
if now.Before(prev.Accepted.Until) {
|
||||
continue
|
||||
}
|
||||
d.ReopenIDs = append(d.ReopenIDs, prev.ID)
|
||||
continue
|
||||
}
|
||||
|
||||
if prev.State != models.FindingOpen {
|
||||
d.NewlyOpened = append(d.NewlyOpened, f)
|
||||
}
|
||||
} else {
|
||||
d.NewlyOpened = append(d.NewlyOpened, f)
|
||||
}
|
||||
|
||||
d.Upserts = append(d.Upserts, f)
|
||||
}
|
||||
|
||||
// Anything we hold that this scan did not produce is fixed. This runs AFTER
|
||||
// the loop above, and the ordering matters: a finding that is both absent
|
||||
// and past its acceptance expiry must settle as fixed rather than reopening
|
||||
// on a package that no longer carries it.
|
||||
for _, f := range existing {
|
||||
if seen[findingKey(f.CVEID, f.PackageName)] {
|
||||
continue
|
||||
}
|
||||
if f.State == models.FindingFixed {
|
||||
continue
|
||||
}
|
||||
d.FixedIDs = append(d.FixedIDs, f.ID)
|
||||
}
|
||||
|
||||
return d
|
||||
}
|
||||
|
||||
// ErrFindingNotFound is returned for a finding that does not exist in this
|
||||
// instance. Callers turn it into a 404 — never a 403, which would confirm the
|
||||
// finding exists in someone else's instance.
|
||||
var ErrFindingNotFound = errors.New("finding not found")
|
||||
|
||||
// FindingFilter narrows a fleet-wide finding query. An empty field is no
|
||||
// filter.
|
||||
type FindingFilter struct {
|
||||
Severity string
|
||||
State string
|
||||
ServerID string
|
||||
Tags map[string]string
|
||||
}
|
||||
|
||||
// ListInstanceFindings returns findings across the whole fleet.
|
||||
func ListInstanceFindings(instanceID string, f FindingFilter) ([]models.VulnFinding, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
filter := bson.M{"instance_id": instanceID}
|
||||
if f.State != "" {
|
||||
filter["state"] = f.State
|
||||
}
|
||||
if f.Severity != "" {
|
||||
filter["severity"] = f.Severity
|
||||
}
|
||||
if f.ServerID != "" {
|
||||
filter["server_id"] = f.ServerID
|
||||
}
|
||||
|
||||
// The tag selector resolves through ResolveTargets, the single answer to
|
||||
// which servers a selector touches. A second matcher here could disagree
|
||||
// with what a workflow means by env:prod.
|
||||
if len(f.Tags) > 0 {
|
||||
servers, err := ResolveTargets(instanceID, nil, f.Tags)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids := make([]string, 0, len(servers))
|
||||
for _, s := range servers {
|
||||
ids = append(ids, s.ServerID)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return []models.VulnFinding{}, nil
|
||||
}
|
||||
filter["server_id"] = bson.M{"$in": ids}
|
||||
}
|
||||
|
||||
cur, err := db.Col("vuln_findings").Find(ctx, filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
out := []models.VulnFinding{}
|
||||
if err := cur.All(ctx, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// CountOpenFindingsBySeverity powers the summary tiles. Accepted findings are
|
||||
// excluded: they are suppressed from counts until their expiry, which is the
|
||||
// whole point of accepting one.
|
||||
func CountOpenFindingsBySeverity(instanceID string) (map[string]int, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cur, err := db.Col("vuln_findings").Aggregate(ctx, []bson.M{
|
||||
{"$match": bson.M{"instance_id": instanceID, "state": models.FindingOpen}},
|
||||
{"$group": bson.M{"_id": "$severity", "n": bson.M{"$sum": 1}}},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var rows []struct {
|
||||
Severity string `bson:"_id"`
|
||||
N int `bson:"n"`
|
||||
}
|
||||
if err := cur.All(ctx, &rows); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
counts := map[string]int{}
|
||||
for _, r := range rows {
|
||||
counts[r.Severity] = r.N
|
||||
}
|
||||
return counts, nil
|
||||
}
|
||||
|
||||
// MarkInstanceForRescan flags every server in an instance for rescanning and
|
||||
// returns how many were flagged.
|
||||
//
|
||||
// It does not scan. vulnsched picks the flags up on its next tick, which keeps
|
||||
// matching on the leader and means this endpoint cannot become a second
|
||||
// scanning path.
|
||||
func MarkInstanceForRescan(instanceID string) (int64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("server_packages").UpdateMany(ctx,
|
||||
bson.M{"instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{"scan_pending": true}},
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return res.ModifiedCount, nil
|
||||
}
|
||||
|
||||
// AcceptFinding suppresses a finding until a date, with a reason.
|
||||
//
|
||||
// The expiry is mandatory at the API layer. A finding reopens on its own when
|
||||
// it passes, which is what stops the accepted list becoming where risk goes to
|
||||
// be forgotten.
|
||||
func AcceptFinding(instanceID, findingID, actor, reason string, until time.Time) (*models.VulnFinding, error) {
|
||||
id, err := bson.ObjectIDFromHex(findingID)
|
||||
if err != nil {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var f models.VulnFinding
|
||||
err = db.Col("vuln_findings").FindOneAndUpdate(ctx,
|
||||
bson.M{"_id": id, "instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{
|
||||
"state": models.FindingAccepted,
|
||||
"accepted": models.Acceptance{
|
||||
By: actor,
|
||||
Reason: reason,
|
||||
Until: until,
|
||||
At: time.Now(),
|
||||
},
|
||||
}},
|
||||
options.FindOneAndUpdate().SetReturnDocument(options.After),
|
||||
).Decode(&f)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &f, nil
|
||||
}
|
||||
|
||||
// UnacceptFinding returns an accepted finding to open before its expiry.
|
||||
func UnacceptFinding(instanceID, findingID string) (*models.VulnFinding, error) {
|
||||
id, err := bson.ObjectIDFromHex(findingID)
|
||||
if err != nil {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var f models.VulnFinding
|
||||
err = db.Col("vuln_findings").FindOneAndUpdate(ctx,
|
||||
bson.M{"_id": id, "instance_id": instanceID},
|
||||
bson.M{
|
||||
"$set": bson.M{"state": models.FindingOpen},
|
||||
"$unset": bson.M{"accepted": ""},
|
||||
},
|
||||
options.FindOneAndUpdate().SetReturnDocument(options.After),
|
||||
).Decode(&f)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &f, nil
|
||||
}
|
||||
|
||||
// ListFindings returns every finding held for one server.
|
||||
func ListFindings(ctx context.Context, instanceID, serverID string) ([]models.VulnFinding, error) {
|
||||
cur, err := db.Col("vuln_findings").Find(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var out []models.VulnFinding
|
||||
if err := cur.All(ctx, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ApplyFindingDiff writes a diff. Thin on purpose — the logic worth reading
|
||||
// twice is all in DiffFindings.
|
||||
func ApplyFindingDiff(ctx context.Context, instanceID, serverID string, d FindingDiff, now time.Time) error {
|
||||
col := db.Col("vuln_findings")
|
||||
|
||||
for _, f := range d.Upserts {
|
||||
_, err := col.UpdateOne(ctx,
|
||||
bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"cve_id": f.CVEID,
|
||||
"package_name": f.PackageName,
|
||||
},
|
||||
bson.M{
|
||||
"$set": bson.M{
|
||||
"installed_version": f.Installed,
|
||||
"fixed_in": f.FixedIn,
|
||||
"severity": f.Severity,
|
||||
"state": models.FindingOpen,
|
||||
"last_seen": now,
|
||||
},
|
||||
// first_seen is written only on insert, so a rescan cannot move
|
||||
// it forward.
|
||||
"$setOnInsert": bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"cve_id": f.CVEID,
|
||||
"package_name": f.PackageName,
|
||||
"first_seen": f.FirstSeen,
|
||||
},
|
||||
"$unset": bson.M{"fixed_at": "", "accepted": ""},
|
||||
},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if len(d.FixedIDs) > 0 {
|
||||
if _, err := col.UpdateMany(ctx,
|
||||
bson.M{"_id": bson.M{"$in": d.FixedIDs}},
|
||||
bson.M{"$set": bson.M{"state": models.FindingFixed, "fixed_at": now}},
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if len(d.ReopenIDs) > 0 {
|
||||
if _, err := col.UpdateMany(ctx,
|
||||
bson.M{"_id": bson.M{"$in": d.ReopenIDs}},
|
||||
bson.M{"$set": bson.M{"state": models.FindingOpen, "last_seen": now}, "$unset": bson.M{"accepted": ""}},
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// StartVulnSweeper deletes old FIXED findings. Open and accepted findings are
|
||||
// never swept at any setting: retention is about history, and an unresolved
|
||||
// vulnerability is not history.
|
||||
func StartVulnSweeper(ctx context.Context) {
|
||||
go func() {
|
||||
ticker := time.NewTicker(6 * time.Hour)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
sweepFixedFindings(ctx)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// defaultVulnRetentionDays is what an unset setting means. A pointer field and
|
||||
// this constant together give absent-means-90 and 0-means-forever, the same
|
||||
// shape as workflow log retention.
|
||||
const defaultVulnRetentionDays = 90
|
||||
|
||||
// sweepFixedFindings deletes fixed findings past each instance's retention.
|
||||
//
|
||||
// Only "fixed" is ever swept. An open or accepted finding is not history, it is
|
||||
// an outstanding decision, and deleting one on a timer would quietly shrink the
|
||||
// fleet's risk picture.
|
||||
func sweepFixedFindings(ctx context.Context) {
|
||||
instanceIDs, err := ListInstanceIDs()
|
||||
if err != nil {
|
||||
log.Printf("vuln sweeper: list instances: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, instanceID := range instanceIDs {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
|
||||
days := defaultVulnRetentionDays
|
||||
if s, err := GetSettings(instanceID); err == nil && s != nil && s.VulnFindingRetentionDays != nil {
|
||||
days = *s.VulnFindingRetentionDays
|
||||
}
|
||||
if days <= 0 {
|
||||
continue // 0 means keep forever
|
||||
}
|
||||
|
||||
cutoff := time.Now().AddDate(0, 0, -days)
|
||||
res, err := db.Col("vuln_findings").DeleteMany(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"state": models.FindingFixed,
|
||||
"fixed_at": bson.M{"$lt": cutoff},
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("vuln sweeper: delete for %s: %v", instanceID, err)
|
||||
continue
|
||||
}
|
||||
if res.DeletedCount > 0 {
|
||||
log.Printf("vuln sweeper: removed %d fixed findings for %s", res.DeletedCount, instanceID)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,9 @@ var ScopedCollections = []string{
|
||||
"console_sessions",
|
||||
"audit_logs",
|
||||
"auth_providers",
|
||||
"server_packages",
|
||||
"vuln_findings",
|
||||
"vuln_alert_rules",
|
||||
}
|
||||
|
||||
// collectionRenames maps the two collections whose names change. Ordered so the
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/shared/license"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// VulnScanningEnabled reports whether this instance may collect packages.
|
||||
//
|
||||
// It reads the feature by name from shared/license and never switches on tier,
|
||||
// so changing what a tier includes needs no server release. A lapsed licence
|
||||
// collects nothing: there is no point accumulating inventory an instance cannot
|
||||
// act on.
|
||||
func VulnScanningEnabled(instanceID string) bool {
|
||||
st := GetLicenseState(instanceID)
|
||||
return st.Active() && st.Feature(license.FeatureVulnScanning)
|
||||
}
|
||||
|
||||
// HasPackageHash reports whether we already hold this exact package set, which
|
||||
// is what lets the agent skip sending ~150KB it has already sent.
|
||||
func HasPackageHash(instanceID, serverID, hash string) (bool, error) {
|
||||
err := db.Col("server_packages").FindOne(context.Background(), bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"hash": hash,
|
||||
}, options.FindOne().SetProjection(bson.M{"_id": 1})).Err()
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// StorePackages replaces a server's package set and marks it for scanning.
|
||||
//
|
||||
// It deliberately does NOT match against the vulnerability database. Matching
|
||||
// happens in vulnsched, on the leader, for two reasons: every replica would
|
||||
// otherwise need the ~50MB database resident, and a database refresh would have
|
||||
// N replicas racing to rescan the same fleet and sending N digests.
|
||||
func StorePackages(instanceID, serverID string, os models.OSRelease, hash string, pkgs []models.InstalledPackage) error {
|
||||
now := time.Now()
|
||||
_, err := db.Col("server_packages").UpdateOne(context.Background(),
|
||||
bson.M{"instance_id": instanceID, "server_id": serverID},
|
||||
bson.M{"$set": bson.M{
|
||||
"os": os,
|
||||
"hash": hash,
|
||||
"packages": pkgs,
|
||||
"collected_at": now,
|
||||
"scan_pending": true,
|
||||
}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
// ListPackages returns a server's stored package set, or nil when the agent has
|
||||
// not reported yet. A missing document is not an error: an agent that has never
|
||||
// reported is the normal state for the first hour after install.
|
||||
func ListPackages(instanceID, serverID string) (*models.ServerPackages, error) {
|
||||
var sp models.ServerPackages
|
||||
err := db.Col("server_packages").FindOne(context.Background(), bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
}).Decode(&sp)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sp, nil
|
||||
}
|
||||
|
||||
// PackageHit is one server running one package.
|
||||
type PackageHit struct {
|
||||
ServerID string `json:"server_id"`
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
|
||||
// SearchPackages answers "which servers run package X" across the fleet — the
|
||||
// question people actually ask during an incident.
|
||||
//
|
||||
// The Mongo filter narrows to documents containing the name; the second pass is
|
||||
// needed because a multikey match returns the whole document, not the matching
|
||||
// array element.
|
||||
func SearchPackages(instanceID, name string) ([]PackageHit, error) {
|
||||
ctx := context.Background()
|
||||
cur, err := db.Col("server_packages").Find(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"packages.name": name,
|
||||
}, options.Find().SetProjection(bson.M{"server_id": 1, "packages": 1}))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var docs []models.ServerPackages
|
||||
if err := cur.All(ctx, &docs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
hits := []PackageHit{}
|
||||
for _, d := range docs {
|
||||
for _, p := range d.Packages {
|
||||
if p.Name == name {
|
||||
hits = append(hits, PackageHit{ServerID: d.ServerID, Name: p.Name, Version: p.Version})
|
||||
}
|
||||
}
|
||||
}
|
||||
return hits, nil
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/notify"
|
||||
"github.com/google/uuid"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
@@ -371,3 +372,46 @@ func notifyServerOffline(instanceID string, channelIDs []string, s models.Server
|
||||
}(ch)
|
||||
}
|
||||
}
|
||||
|
||||
// ListServersFiltered is ListServers with an optional tag selector. An empty
|
||||
// selector returns the whole fleet — unlike MatchesTags, where empty means
|
||||
// "nothing", because here the caller is a list view whose default is
|
||||
// "everything", not a run about to touch machines.
|
||||
func ListServersFiltered(instanceID string, sel map[string]string) ([]models.Server, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
filter := bson.M{"instance_id": instanceID}
|
||||
for k, v := range sel {
|
||||
filter["tags."+k] = v
|
||||
}
|
||||
|
||||
cur, err := db.Col("servers").Find(ctx, filter, options.Find().SetSort(bson.D{{Key: "created_at", Value: -1}}))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
servers := []models.Server{}
|
||||
if err := cur.All(ctx, &servers); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return servers, nil
|
||||
}
|
||||
|
||||
// EnsureServerIndexes declares the wildcard index over the tag subdocument.
|
||||
// It is wildcard because the queried key is chosen by the user at request time
|
||||
// and cannot be named in advance.
|
||||
//
|
||||
// Non-fatal, following EnsureSecretIndexes: a missing index degrades tag
|
||||
// filtering to a collection scan over a small collection, which is slower.
|
||||
// A fatal error here would refuse to boot the fleet list over it.
|
||||
func EnsureServerIndexes() error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
_, err := db.Col("servers").Indexes().CreateOne(ctx, mongo.IndexModel{
|
||||
Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "tags.$**", Value: 1}},
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// ErrInvalidTag is returned for any tag the rules below reject. Handlers map
|
||||
// it to 400 — a malformed tag is the caller's mistake, not a server fault.
|
||||
var ErrInvalidTag = errors.New("invalid tag")
|
||||
|
||||
const (
|
||||
maxTagKeyLen = 32
|
||||
maxTagValueLen = 64
|
||||
maxTagsPerHost = 20
|
||||
// Reserved for tags the agent may derive from inventory later. Refusing
|
||||
// it now means a user tag written today can never collide with a system
|
||||
// tag invented tomorrow.
|
||||
sysTagPrefix = "sys:"
|
||||
)
|
||||
|
||||
func validTagRunes(s string) bool {
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z':
|
||||
case r >= '0' && r <= '9':
|
||||
case r == '-' || r == '_':
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ValidateTags enforces the shape of a whole tag map. It lives in the service
|
||||
// layer rather than a handler so that every write path — the tags endpoint,
|
||||
// server create, anything added later — agrees on what a valid tag is.
|
||||
func ValidateTags(tags map[string]string) error {
|
||||
if len(tags) > maxTagsPerHost {
|
||||
return fmt.Errorf("%w: at most %d tags per server", ErrInvalidTag, maxTagsPerHost)
|
||||
}
|
||||
for k, v := range tags {
|
||||
if strings.HasPrefix(k, sysTagPrefix) {
|
||||
return fmt.Errorf("%w: keys beginning %q are reserved", ErrInvalidTag, sysTagPrefix)
|
||||
}
|
||||
if k == "" || len(k) > maxTagKeyLen || !validTagRunes(k) {
|
||||
return fmt.Errorf("%w: key %q must be 1-%d chars of a-z, 0-9, - or _", ErrInvalidTag, k, maxTagKeyLen)
|
||||
}
|
||||
if v == "" || len(v) > maxTagValueLen || !validTagRunes(v) {
|
||||
return fmt.Errorf("%w: value for %q must be 1-%d chars of a-z, 0-9, - or _", ErrInvalidTag, k, maxTagValueLen)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseTagFilters turns repeated ?tag=key:value query values into a map.
|
||||
//
|
||||
// A malformed filter is an error rather than a silently ignored value: a
|
||||
// filter that matches nothing and a filter that is nonsense look identical in
|
||||
// a list, and only one of them is the caller's fault.
|
||||
func ParseTagFilters(raw []string) (map[string]string, error) {
|
||||
out := make(map[string]string, len(raw))
|
||||
for _, r := range raw {
|
||||
k, v, found := strings.Cut(r, ":")
|
||||
if !found {
|
||||
return nil, fmt.Errorf("%w: filter %q must be key:value", ErrInvalidTag, r)
|
||||
}
|
||||
if strings.Contains(v, ":") {
|
||||
return nil, fmt.Errorf("%w: filter %q has more than one colon", ErrInvalidTag, r)
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
if err := ValidateTags(out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// SetServerTags replaces a server's whole tag map.
|
||||
//
|
||||
// Replace rather than patch: a tag set is small enough that sending all of it
|
||||
// is free, and last-write-wins over a whole map is easier to reason about than
|
||||
// merge semantics between two people editing the same server.
|
||||
func SetServerTags(instanceID, serverID string, tags map[string]string) error {
|
||||
if err := ValidateTags(tags); err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("servers").UpdateOne(ctx,
|
||||
bson.M{"server_id": serverID, "instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{"tags": tags}},
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if res.MatchedCount == 0 {
|
||||
return mongo.ErrNoDocuments
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// KnownTags returns every key in use in this instance with its distinct
|
||||
// values, for the UI's pickers. This is an aggregation rather than a
|
||||
// maintained registry: a tag is a property of a server, not an entity, and a
|
||||
// registry would need reference counting to know when a tag stopped existing.
|
||||
func KnownTags(instanceID string) (map[string][]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cur, err := db.Col("servers").Find(ctx,
|
||||
bson.M{"instance_id": instanceID, "tags": bson.M{"$exists": true}},
|
||||
options.Find().SetProjection(bson.M{"tags": 1}),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
seen := map[string]map[string]bool{}
|
||||
for cur.Next(ctx) {
|
||||
var s models.Server
|
||||
if err := cur.Decode(&s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for k, v := range s.Tags {
|
||||
if seen[k] == nil {
|
||||
seen[k] = map[string]bool{}
|
||||
}
|
||||
seen[k][v] = true
|
||||
}
|
||||
}
|
||||
if err := cur.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
out := make(map[string][]string, len(seen))
|
||||
for k, vals := range seen {
|
||||
list := make([]string, 0, len(vals))
|
||||
for v := range vals {
|
||||
list = append(list, v)
|
||||
}
|
||||
sort.Strings(list)
|
||||
out[k] = list
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
// ErrNoTargets means a workflow named no servers and matched none. Handlers
|
||||
// map it to 400: a workflow that matches nothing must say so rather than
|
||||
// report success over zero servers.
|
||||
var ErrNoTargets = errors.New("workflow has no target servers")
|
||||
|
||||
// MatchesTags reports whether srv carries every pair in sel — AND across keys.
|
||||
// An empty selector matches nothing. That is deliberate: the alternative,
|
||||
// "matches everything", turns a cleared field in the workflow designer into a
|
||||
// fleet-wide run.
|
||||
func MatchesTags(srv models.Server, sel map[string]string) bool {
|
||||
if len(sel) == 0 {
|
||||
return false
|
||||
}
|
||||
for k, v := range sel {
|
||||
if srv.Tags[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// UnionTargets returns the distinct union of the servers named by ids and
|
||||
// those matching sel, in the order they appear in all.
|
||||
//
|
||||
// Order comes from the fleet rather than the arguments so that two workflows
|
||||
// naming the same servers in a different order still run them in the same
|
||||
// order, which makes two runs comparable line by line.
|
||||
//
|
||||
// Offline servers are NOT filtered out. The dispatcher already answers 503 per
|
||||
// server, and a patch run that silently omits an unreachable machine is worse
|
||||
// than one that visibly fails on it.
|
||||
func UnionTargets(all []models.Server, ids []string, sel map[string]string) []models.Server {
|
||||
named := make(map[string]bool, len(ids))
|
||||
for _, id := range ids {
|
||||
named[id] = true
|
||||
}
|
||||
|
||||
out := make([]models.Server, 0, len(ids)+len(all))
|
||||
for _, s := range all {
|
||||
if named[s.ServerID] || MatchesTags(s, sel) {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ResolveTargets is the database-backed wrapper around UnionTargets. It is the
|
||||
// single answer to "which servers does this workflow touch", used by the run
|
||||
// path and by validation alike, so the two cannot disagree.
|
||||
func ResolveTargets(instanceID string, ids []string, sel map[string]string) ([]models.Server, error) {
|
||||
all, err := ListServers(instanceID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
matched := UnionTargets(all, ids, sel)
|
||||
if len(matched) == 0 {
|
||||
return nil, ErrNoTargets
|
||||
}
|
||||
return matched, nil
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// EnsureVulnIndexes declares the indexes for package inventory and findings.
|
||||
//
|
||||
// It warns rather than being fatal, matching EnsureSecretIndexes and
|
||||
// EnsureWorkflowIndexes: a missing index degrades these queries to a collection
|
||||
// scan, which is no reason to refuse to serve the fleet.
|
||||
func EnsureVulnIndexes() error {
|
||||
ctx := context.Background()
|
||||
|
||||
pkgIdx := []mongo.IndexModel{
|
||||
{
|
||||
Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "server_id", Value: 1}},
|
||||
Options: options.Index().SetUnique(true),
|
||||
},
|
||||
// Multikey, for fleet-wide package search: "who runs openssl 3.0.2?"
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "packages.name", Value: 1}}},
|
||||
// The scheduler's only query. Deliberately unscoped: it sweeps the whole
|
||||
// deployment on the leader, not one tenant.
|
||||
{Keys: bson.D{{Key: "scan_pending", Value: 1}}},
|
||||
}
|
||||
if _, err := db.Col("server_packages").Indexes().CreateMany(ctx, pkgIdx); err != nil {
|
||||
log.Printf("warning: server_packages indexes: %v", err)
|
||||
}
|
||||
|
||||
findingIdx := []mongo.IndexModel{
|
||||
{
|
||||
// This key is what makes a rescan an idempotent upsert rather than a
|
||||
// duplicate factory, and what lets first_seen survive a rescan.
|
||||
Keys: bson.D{
|
||||
{Key: "instance_id", Value: 1},
|
||||
{Key: "server_id", Value: 1},
|
||||
{Key: "cve_id", Value: 1},
|
||||
{Key: "package_name", Value: 1},
|
||||
},
|
||||
Options: options.Index().SetUnique(true),
|
||||
},
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "state", Value: 1}, {Key: "severity", Value: 1}}},
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "cve_id", Value: 1}}},
|
||||
}
|
||||
if _, err := db.Col("vuln_findings").Indexes().CreateMany(ctx, findingIdx); err != nil {
|
||||
log.Printf("warning: vuln_findings indexes: %v", err)
|
||||
}
|
||||
|
||||
if _, err := db.Col("vuln_alert_rules").Indexes().CreateOne(ctx, mongo.IndexModel{
|
||||
Keys: bson.D{{Key: "instance_id", Value: 1}},
|
||||
}); err != nil {
|
||||
log.Printf("warning: vuln_alert_rules indexes: %v", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/notify"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/shared/mail"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
)
|
||||
|
||||
// digestRowLimit caps how many findings a single digest lists by name. The
|
||||
// remainder is summarised as a count: a webhook payload holding six hundred
|
||||
// rows is not a notification, it is a report nobody reads in a chat client.
|
||||
const digestRowLimit = 20
|
||||
|
||||
// ErrVulnRuleNotFound is returned for a rule that does not exist in this
|
||||
// instance. Callers turn it into a 404.
|
||||
var ErrVulnRuleNotFound = fmt.Errorf("vulnerability alert rule not found")
|
||||
|
||||
func ListVulnRules(instanceID string) ([]models.VulnAlertRule, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cur, err := db.Col("vuln_alert_rules").Find(ctx, bson.M{"instance_id": instanceID})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
rules := []models.VulnAlertRule{}
|
||||
if err := cur.All(ctx, &rules); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return rules, nil
|
||||
}
|
||||
|
||||
func CreateVulnRule(instanceID string, r *models.VulnAlertRule) (*models.VulnAlertRule, error) {
|
||||
if err := validateVulnRule(instanceID, r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
r.ID = bson.NewObjectID()
|
||||
r.InstanceID = instanceID
|
||||
r.CreatedAt = time.Now()
|
||||
r.UpdatedAt = r.CreatedAt
|
||||
|
||||
if _, err := db.Col("vuln_alert_rules").InsertOne(ctx, r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func UpdateVulnRule(instanceID, ruleID string, r *models.VulnAlertRule) error {
|
||||
if err := validateVulnRule(instanceID, r); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
id, err := bson.ObjectIDFromHex(ruleID)
|
||||
if err != nil {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("vuln_alert_rules").UpdateOne(ctx,
|
||||
bson.M{"_id": id, "instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{
|
||||
"name": r.Name,
|
||||
"enabled": r.Enabled,
|
||||
"min_severity": r.MinSeverity,
|
||||
"tags": r.Tags,
|
||||
"channel_ids": r.ChannelIDs,
|
||||
"updated_at": time.Now(),
|
||||
}},
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if res.MatchedCount == 0 {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func DeleteVulnRule(instanceID, ruleID string) error {
|
||||
id, err := bson.ObjectIDFromHex(ruleID)
|
||||
if err != nil {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("vuln_alert_rules").DeleteOne(ctx, bson.M{"_id": id, "instance_id": instanceID})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if res.DeletedCount == 0 {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateVulnRule rejects a rule that could never fire, and one naming a
|
||||
// channel from another instance. The channel check reuses validateChannelIDs so
|
||||
// there is one answer to "is this channel mine".
|
||||
func validateVulnRule(instanceID string, r *models.VulnAlertRule) error {
|
||||
if r.Name == "" {
|
||||
return fmt.Errorf("name is required")
|
||||
}
|
||||
switch r.MinSeverity {
|
||||
case models.SeverityUnknown, models.SeverityLow, models.SeverityMedium,
|
||||
models.SeverityHigh, models.SeverityCritical:
|
||||
default:
|
||||
return fmt.Errorf("min_severity %q is not a severity", r.MinSeverity)
|
||||
}
|
||||
if len(r.ChannelIDs) == 0 {
|
||||
return fmt.Errorf("at least one channel is required")
|
||||
}
|
||||
return validateChannelIDs(instanceID, r.ChannelIDs)
|
||||
}
|
||||
|
||||
// SendVulnDigest delivers one message per rule per tick — never one per
|
||||
// finding. See vulnsched for why the tick is the batch boundary.
|
||||
func SendVulnDigest(instanceID string, newly []models.VulnFinding) {
|
||||
rules, err := ListVulnRules(instanceID)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: list rules: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, rule := range rules {
|
||||
if !rule.Enabled {
|
||||
continue
|
||||
}
|
||||
|
||||
matched := filterBySeverity(newly, rule.MinSeverity)
|
||||
if len(matched) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
if len(rule.Tags) > 0 {
|
||||
// ResolveTargets is already the single answer to which servers a
|
||||
// selector touches. A rule that disagreed with a workflow about
|
||||
// what env:prod means would be worse than no filter at all.
|
||||
allowed, err := ResolveTargets(instanceID, nil, rule.Tags)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: resolve targets: %v", err)
|
||||
continue
|
||||
}
|
||||
matched = filterByServers(matched, allowed)
|
||||
if len(matched) == 0 {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
dispatchVulnDigest(instanceID, rule, matched)
|
||||
}
|
||||
}
|
||||
|
||||
func filterBySeverity(findings []models.VulnFinding, min string) []models.VulnFinding {
|
||||
floor := models.SeverityRank(min)
|
||||
out := make([]models.VulnFinding, 0, len(findings))
|
||||
for _, f := range findings {
|
||||
if models.SeverityRank(f.Severity) >= floor {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// filterByServers keeps findings on servers the rule's tag selector matched.
|
||||
// ResolveTargets answers in whole server documents, so the IDs are lifted here.
|
||||
func filterByServers(findings []models.VulnFinding, allowed []models.Server) []models.VulnFinding {
|
||||
set := make(map[string]bool, len(allowed))
|
||||
for _, s := range allowed {
|
||||
set[s.ServerID] = true
|
||||
}
|
||||
out := make([]models.VulnFinding, 0, len(findings))
|
||||
for _, f := range findings {
|
||||
if set[f.ServerID] {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// dispatchVulnDigest builds one digest and sends it over each of the rule's
|
||||
// channels.
|
||||
func dispatchVulnDigest(instanceID string, rule models.VulnAlertRule, findings []models.VulnFinding) {
|
||||
channels, err := GetChannels(instanceID, rule.ChannelIDs)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: load channels for rule %s: %v", rule.Name, err)
|
||||
return
|
||||
}
|
||||
|
||||
digest := buildVulnDigest(instanceID, rule, findings)
|
||||
|
||||
for _, ch := range channels {
|
||||
if !ch.Enabled {
|
||||
continue
|
||||
}
|
||||
go func(c models.NotificationChannel) {
|
||||
if err := notify.DispatchVulnDigest(c, digest); err != nil {
|
||||
log.Printf("vuln digest: dispatch to %s (%s): %v", c.Name, c.Type, err)
|
||||
}
|
||||
}(ch)
|
||||
}
|
||||
}
|
||||
|
||||
// buildVulnDigest turns a batch of findings into one message.
|
||||
//
|
||||
// Findings are ordered most severe first so the capped list shows the ones that
|
||||
// matter rather than whichever the scan happened to produce first.
|
||||
func buildVulnDigest(instanceID string, rule models.VulnAlertRule, findings []models.VulnFinding) notify.VulnDigest {
|
||||
sorted := make([]models.VulnFinding, len(findings))
|
||||
copy(sorted, findings)
|
||||
sort.SliceStable(sorted, func(i, j int) bool {
|
||||
return models.SeverityRank(sorted[i].Severity) > models.SeverityRank(sorted[j].Severity)
|
||||
})
|
||||
|
||||
counts := map[string]int{}
|
||||
servers := map[string]bool{}
|
||||
for _, f := range sorted {
|
||||
counts[f.Severity]++
|
||||
servers[f.ServerID] = true
|
||||
}
|
||||
|
||||
names := serverNames(instanceID)
|
||||
|
||||
shown := sorted
|
||||
more := 0
|
||||
if len(shown) > digestRowLimit {
|
||||
more = len(shown) - digestRowLimit
|
||||
shown = shown[:digestRowLimit]
|
||||
}
|
||||
|
||||
rows := make([]mail.VulnDigestRow, 0, len(shown))
|
||||
for _, f := range shown {
|
||||
name := names[f.ServerID]
|
||||
if name == "" {
|
||||
name = f.ServerID
|
||||
}
|
||||
rows = append(rows, mail.VulnDigestRow{
|
||||
CVEID: f.CVEID,
|
||||
Severity: f.Severity,
|
||||
PackageName: f.PackageName,
|
||||
ServerName: name,
|
||||
FixedIn: f.FixedIn,
|
||||
})
|
||||
}
|
||||
|
||||
top := models.SeverityUnknown
|
||||
if len(sorted) > 0 {
|
||||
top = sorted[0].Severity
|
||||
}
|
||||
|
||||
instanceName := instanceID
|
||||
if inst, err := GetInstance(instanceID); err == nil && inst != nil && inst.Name != "" {
|
||||
instanceName = inst.Name
|
||||
}
|
||||
|
||||
return notify.VulnDigest{
|
||||
InstanceName: instanceName,
|
||||
RuleName: rule.Name,
|
||||
Summary: summariseCounts(counts, len(servers)),
|
||||
TopSeverity: top,
|
||||
Count: len(sorted),
|
||||
Rows: rows,
|
||||
More: more,
|
||||
DBAge: vulnDBAge(),
|
||||
}
|
||||
}
|
||||
|
||||
// summariseCounts renders "12 new critical, 4 new high across 6 servers".
|
||||
func summariseCounts(counts map[string]int, serverCount int) string {
|
||||
order := []string{
|
||||
models.SeverityCritical, models.SeverityHigh,
|
||||
models.SeverityMedium, models.SeverityLow, models.SeverityUnknown,
|
||||
}
|
||||
parts := ""
|
||||
for _, sev := range order {
|
||||
if counts[sev] == 0 {
|
||||
continue
|
||||
}
|
||||
if parts != "" {
|
||||
parts += ", "
|
||||
}
|
||||
parts += fmt.Sprintf("%d new %s", counts[sev], sev)
|
||||
}
|
||||
if parts == "" {
|
||||
parts = "new findings"
|
||||
}
|
||||
plural := "servers"
|
||||
if serverCount == 1 {
|
||||
plural = "server"
|
||||
}
|
||||
return fmt.Sprintf("%s across %d %s", parts, serverCount, plural)
|
||||
}
|
||||
|
||||
// serverNames maps server IDs to display names for one instance. A digest that
|
||||
// named raw UUIDs would be unreadable in a chat client.
|
||||
func serverNames(instanceID string) map[string]string {
|
||||
out := map[string]string{}
|
||||
servers, err := ListServers(instanceID)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: list servers: %v", err)
|
||||
return out
|
||||
}
|
||||
for _, s := range servers {
|
||||
out[s.ServerID] = s.Hostname
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// vulnDBAge renders how long ago the vulnerability database was pulled.
|
||||
//
|
||||
// It is on every digest deliberately: a fleet scanned against a three-week-old
|
||||
// database must say so rather than let the reader assume freshness.
|
||||
func vulnDBAge() string {
|
||||
meta, err := GetVulnDBMeta()
|
||||
if err != nil || meta == nil || meta.PulledAt.IsZero() {
|
||||
return "an unknown time"
|
||||
}
|
||||
d := time.Since(meta.PulledAt)
|
||||
switch {
|
||||
case d < time.Hour:
|
||||
return fmt.Sprintf("%d minutes", int(d.Minutes()))
|
||||
case d < 48*time.Hour:
|
||||
return fmt.Sprintf("%d hours", int(d.Hours()))
|
||||
default:
|
||||
return fmt.Sprintf("%d days", int(d.Hours()/24))
|
||||
}
|
||||
}
|
||||
|
||||
// GetVulnDBMeta reads the deployment-wide vulnerability database metadata.
|
||||
// It carries no instance_id: the database is a property of the deployment, not
|
||||
// of a tenant.
|
||||
func GetVulnDBMeta() (*models.VulnDBMeta, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var meta models.VulnDBMeta
|
||||
err := db.Col("vulndb_meta").FindOne(ctx, bson.M{}).Decode(&meta)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &meta, nil
|
||||
}
|
||||
@@ -22,17 +22,14 @@ func TriggerWorkflow(instanceID, workflowID, actor string) (string, error) {
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(wf.TargetServerIDs) == 0 {
|
||||
return "", fmt.Errorf("workflow has no target servers")
|
||||
targets, err := ResolveTargets(instanceID, wf.TargetServerIDs, wf.TargetTags)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(wf.Steps) == 0 {
|
||||
return "", fmt.Errorf("workflow has no steps")
|
||||
}
|
||||
|
||||
if err := validateTargetServers(instanceID, wf.TargetServerIDs); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
ctx, cancel := wfCtx()
|
||||
running := db.Col("workflow_runs").FindOne(ctx, bson.M{"instance_id": instanceID, "workflow_id": workflowID, "status": "running"})
|
||||
cancel()
|
||||
@@ -54,14 +51,10 @@ func TriggerWorkflow(instanceID, workflowID, actor string) (string, error) {
|
||||
Status: "running",
|
||||
TriggeredBy: actor,
|
||||
StartedAt: time.Now(),
|
||||
ServerRuns: make([]models.ServerRun, 0, len(wf.TargetServerIDs)),
|
||||
ServerRuns: make([]models.ServerRun, 0, len(targets)),
|
||||
}
|
||||
for _, sid := range wf.TargetServerIDs {
|
||||
hostname := sid
|
||||
if s, e := getServerByID(sid); e == nil {
|
||||
hostname = s.Hostname
|
||||
}
|
||||
sr := models.ServerRun{ServerID: sid, Hostname: hostname, Status: "queued", RunEnv: map[string]string{}}
|
||||
for _, srv := range targets {
|
||||
sr := models.ServerRun{ServerID: srv.ServerID, Hostname: srv.Hostname, Status: "queued", RunEnv: map[string]string{}}
|
||||
for _, rs := range resolved {
|
||||
sr.Steps = append(sr.Steps, models.StepRun{Order: rs.Order, Name: rs.Name, Status: "queued", OutputEnv: map[string]string{}})
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
|
||||
"github.com/google/uuid"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
@@ -60,6 +61,11 @@ func EnsureWorkflowIndexes() error {
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := db.Col("workflows").Indexes().CreateOne(ctx, mongo.IndexModel{
|
||||
Keys: bson.D{{Key: "next_run_at", Value: 1}},
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := EnsureLogIndexes(); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -249,6 +255,9 @@ func CreateWorkflow(instanceID string, w models.Workflow) (*models.Workflow, err
|
||||
if err := ValidateWorkflow(w); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ValidateTags(w.TargetTags); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateTargetServers(instanceID, w.TargetServerIDs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -265,6 +274,9 @@ func UpdateWorkflow(instanceID, id string, w models.Workflow) error {
|
||||
if err := ValidateWorkflow(w); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ValidateTags(w.TargetTags); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateTargetServers(instanceID, w.TargetServerIDs); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -272,6 +284,7 @@ func UpdateWorkflow(instanceID, id string, w models.Workflow) error {
|
||||
_, err := db.Col("workflows").UpdateOne(ctx, bson.M{"workflow_id": id, "instance_id": instanceID}, bson.M{"$set": bson.M{
|
||||
"name": w.Name,
|
||||
"target_server_ids": w.TargetServerIDs,
|
||||
"target_tags": w.TargetTags,
|
||||
"steps": w.Steps,
|
||||
"updated_at": time.Now(),
|
||||
}})
|
||||
@@ -314,3 +327,45 @@ func DeleteWorkflow(instanceID, id string) error {
|
||||
_, err := db.Col("workflows").DeleteOne(ctx, bson.M{"workflow_id": id, "instance_id": instanceID})
|
||||
return err
|
||||
}
|
||||
|
||||
// SetSchedule validates and stores a workflow's schedule, computing the first
|
||||
// occurrence. next_run_at is persisted rather than held in memory: a leader
|
||||
// handover between computing an occurrence and firing it would otherwise lose
|
||||
// it or fire it twice.
|
||||
//
|
||||
// Passing s == nil, or a disabled schedule, clears next_run_at so the
|
||||
// scheduler's query stops matching the document at all.
|
||||
func SetSchedule(instanceID, workflowID string, s *models.Schedule) (*time.Time, error) {
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
|
||||
set := bson.M{"schedule": s, "updated_at": time.Now()}
|
||||
unset := bson.M{}
|
||||
|
||||
var next *time.Time
|
||||
if s != nil && s.Enabled {
|
||||
at, err := workflowsched.NextOccurrence(s.Cron, s.TZ, time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
next = &at
|
||||
set["next_run_at"] = at
|
||||
} else {
|
||||
unset["next_run_at"] = ""
|
||||
}
|
||||
|
||||
update := bson.M{"$set": set}
|
||||
if len(unset) > 0 {
|
||||
update["$unset"] = unset
|
||||
}
|
||||
|
||||
res, err := db.Col("workflows").UpdateOne(ctx,
|
||||
bson.M{"workflow_id": workflowID, "instance_id": instanceID}, update)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if res.MatchedCount == 0 {
|
||||
return nil, mongo.ErrNoDocuments
|
||||
}
|
||||
return next, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
trivydb "github.com/aquasecurity/trivy-db/pkg/db"
|
||||
trivytypes "github.com/aquasecurity/trivy-db/pkg/types"
|
||||
)
|
||||
|
||||
// Advisory is one fixed-version statement for one source package.
|
||||
type Advisory struct {
|
||||
CVEID string
|
||||
// FixedVersion empty means no vendor fix has been published. It is a real
|
||||
// state, not an absence of data, and callers must treat it as vulnerable.
|
||||
FixedVersion string
|
||||
Severity string
|
||||
}
|
||||
|
||||
// VulnInfo is the CVE's own metadata, shared across every server it affects.
|
||||
type VulnInfo struct {
|
||||
Title string
|
||||
Severity string
|
||||
CVSSScore float64
|
||||
References []string
|
||||
}
|
||||
|
||||
// Store reads a pulled trivy-db.
|
||||
type Store struct {
|
||||
cfg trivydb.Config
|
||||
}
|
||||
|
||||
// Open opens the database in dir. trivy-db expects the directory, not the file:
|
||||
// it appends "trivy.db" itself.
|
||||
func Open(dir string) (*Store, error) {
|
||||
if err := trivydb.Init(dir); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Store{cfg: trivydb.Config{}}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Close() error { return trivydb.Close() }
|
||||
|
||||
// Advisories returns every advisory for a source package in a bucket.
|
||||
func (s *Store) Advisories(bucket, srcName string) ([]Advisory, error) {
|
||||
raw, err := s.cfg.GetAdvisories(bucket, srcName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]Advisory, 0, len(raw))
|
||||
for _, a := range raw {
|
||||
out = append(out, Advisory{
|
||||
CVEID: a.VulnerabilityID,
|
||||
FixedVersion: a.FixedVersion,
|
||||
// Advisory.Severity is trivy's numeric Severity type, unlike
|
||||
// Vulnerability.Severity which is a string. They are genuinely
|
||||
// different types in trivy-db, not an inconsistency here.
|
||||
Severity: severityFromLevel(a.Severity),
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Vulnerability returns a CVE's shared metadata.
|
||||
func (s *Store) Vulnerability(cveID string) (VulnInfo, error) {
|
||||
v, err := s.cfg.GetVulnerability(cveID)
|
||||
if err != nil {
|
||||
return VulnInfo{}, err
|
||||
}
|
||||
return VulnInfo{
|
||||
Title: v.Title,
|
||||
Severity: resolveSeverity(v),
|
||||
CVSSScore: topCVSS(v),
|
||||
References: v.References,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// resolveSeverity picks a CVE's severity: vendor, then NVD, then unknown.
|
||||
//
|
||||
// Never invented. This will surface as "why is this critical CVE marked low":
|
||||
// Debian and Red Hat routinely downgrade an NVD score because the vulnerable
|
||||
// path is not reachable in their build, and their rating is the accurate one
|
||||
// for that package. Where several vendors disagree the highest wins, because
|
||||
// under-reporting a vulnerability is the worse mistake.
|
||||
func resolveSeverity(v trivytypes.Vulnerability) string {
|
||||
best := 0
|
||||
for _, sev := range v.VendorSeverity {
|
||||
if int(sev) > best {
|
||||
best = int(sev)
|
||||
}
|
||||
}
|
||||
if best > 0 {
|
||||
return severityFromLevel(trivytypes.Severity(best))
|
||||
}
|
||||
|
||||
// Vulnerability.Severity is the deprecated NVD-derived string. Used only as
|
||||
// the fallback, which is exactly what it is good for.
|
||||
if s := strings.ToLower(strings.TrimSpace(v.Severity)); s != "" && s != "unknown" {
|
||||
return s
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
// topCVSS returns the highest V3 score any source published, or 0.
|
||||
func topCVSS(v trivytypes.Vulnerability) float64 {
|
||||
var top float64
|
||||
for _, c := range v.CVSS {
|
||||
if c.V3Score > top {
|
||||
top = c.V3Score
|
||||
}
|
||||
}
|
||||
return top
|
||||
}
|
||||
|
||||
// severityFromLevel maps trivy-db's numeric severity onto our lowercase
|
||||
// strings. The names are fixed by models.Severity* and must stay in step.
|
||||
func severityFromLevel(n trivytypes.Severity) string {
|
||||
switch int(n) {
|
||||
case 4:
|
||||
return "critical"
|
||||
case 3:
|
||||
return "high"
|
||||
case 2:
|
||||
return "medium"
|
||||
case 1:
|
||||
return "low"
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// rhelRebuilds share Red Hat's advisory feed rather than publishing their own.
|
||||
var rhelRebuilds = map[string]bool{
|
||||
"redhat": true, "centos": true, "rocky": true, "alma": true, "oracle": true,
|
||||
}
|
||||
|
||||
// Bucket maps an OS family and version onto the trivy-db bucket that holds its
|
||||
// advisories.
|
||||
//
|
||||
// It returns ErrUnsupportedFamily rather than a best guess when we have no
|
||||
// feed. A scan that cannot be performed must say so; reporting zero findings
|
||||
// for a distribution we do not cover is indistinguishable from reporting a
|
||||
// clean host, and one of those is a lie.
|
||||
func Bucket(family, versionID string) (string, error) {
|
||||
family = strings.ToLower(strings.TrimSpace(family))
|
||||
versionID = strings.TrimSpace(versionID)
|
||||
|
||||
switch {
|
||||
case family == "debian" || family == "ubuntu":
|
||||
if versionID == "" {
|
||||
return "", fmt.Errorf("%s requires a version id", family)
|
||||
}
|
||||
return family + " " + versionID, nil
|
||||
|
||||
case family == "alpine":
|
||||
if versionID == "" {
|
||||
return "", fmt.Errorf("alpine requires a version id")
|
||||
}
|
||||
return "alpine " + majorMinor(versionID), nil
|
||||
|
||||
case rhelRebuilds[family]:
|
||||
if versionID == "" {
|
||||
return "", fmt.Errorf("%s requires a version id", family)
|
||||
}
|
||||
return "redhat " + major(versionID), nil
|
||||
|
||||
default:
|
||||
return "", fmt.Errorf("%w: %s", ErrUnsupportedFamily, family)
|
||||
}
|
||||
}
|
||||
|
||||
func major(v string) string {
|
||||
if i := strings.Index(v, "."); i != -1 {
|
||||
return v[:i]
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func majorMinor(v string) string {
|
||||
parts := strings.Split(v, ".")
|
||||
if len(parts) >= 2 {
|
||||
return parts[0] + "." + parts[1]
|
||||
}
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
// AdvisorySource is the advisory lookup the matcher needs. *Store satisfies it.
|
||||
// The seam keeps the matching logic independent of how the database is opened.
|
||||
type AdvisorySource interface {
|
||||
Advisories(bucket, srcName string) ([]Advisory, error)
|
||||
}
|
||||
|
||||
// Result is one vulnerable package on one server, before it becomes a finding.
|
||||
type Result struct {
|
||||
CVEID string
|
||||
PackageName string // the BINARY package, which is what is installed
|
||||
Installed string
|
||||
FixedIn string
|
||||
Severity string
|
||||
}
|
||||
|
||||
// Match returns every advisory that the installed packages do not satisfy.
|
||||
//
|
||||
// Vulnerable means: no fix has been published, or the installed version sorts
|
||||
// strictly before the fixed version under the distribution's own ordering.
|
||||
// Equal is NOT vulnerable — that is the backported-fix case, where a
|
||||
// distribution patches in place without changing the upstream version, and
|
||||
// treating it as vulnerable reports a patched fleet as exposed.
|
||||
func Match(src AdvisorySource, os models.OSRelease, pkgs []models.InstalledPackage) ([]Result, error) {
|
||||
bucket, err := Bucket(os.Family, os.VersionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var out []Result
|
||||
for _, p := range pkgs {
|
||||
// Debian and Ubuntu advisories are keyed on the source package: one
|
||||
// advisory against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
srcName := p.SourceName
|
||||
if srcName == "" {
|
||||
srcName = p.Name
|
||||
}
|
||||
|
||||
advs, err := src.Advisories(bucket, srcName)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("advisories for %s: %w", srcName, err)
|
||||
}
|
||||
|
||||
for _, a := range advs {
|
||||
// No published fix. Vulnerable, and the finding most in need of
|
||||
// acceptance, since there is nothing to patch.
|
||||
if a.FixedVersion == "" {
|
||||
out = append(out, Result{
|
||||
CVEID: a.CVEID, PackageName: p.Name,
|
||||
Installed: p.Version, Severity: a.Severity,
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
older, err := LessThan(os.Family, p.Version, a.FixedVersion)
|
||||
if err != nil {
|
||||
// Skip this one advisory rather than failing the whole server:
|
||||
// one unparseable version must not blind us to every other CVE
|
||||
// on the host. Log it — a silent skip is a silent false
|
||||
// negative, which is the direction that hurts.
|
||||
log.Printf("vulndb: compare %s %s vs %s: %v", p.Name, p.Version, a.FixedVersion, err)
|
||||
continue
|
||||
}
|
||||
if older {
|
||||
out = append(out, Result{
|
||||
CVEID: a.CVEID, PackageName: p.Name,
|
||||
Installed: p.Version, FixedIn: a.FixedVersion, Severity: a.Severity,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
|
||||
"oras.land/oras-go/v2"
|
||||
"oras.land/oras-go/v2/registry"
|
||||
"oras.land/oras-go/v2/registry/remote"
|
||||
)
|
||||
|
||||
// DefaultRef is the published trivy-db OCI artifact, rebuilt every six hours.
|
||||
const DefaultRef = "ghcr.io/aquasecurity/trivy-db:2"
|
||||
|
||||
// SupportedSchema is the trivy-db schema version this code understands.
|
||||
//
|
||||
// A different version is refused rather than parsed on the assumption it is
|
||||
// close enough. Mis-reading the schema would not fail loudly — it would return
|
||||
// no advisories, which is indistinguishable from a clean fleet.
|
||||
const SupportedSchema = 2
|
||||
|
||||
// dbFileName and metaFileName are the two files inside the artifact layer.
|
||||
const (
|
||||
dbFileName = "trivy.db"
|
||||
metaFileName = "metadata.json"
|
||||
)
|
||||
|
||||
// Ref returns the artifact reference, honouring VANTAGE_TRIVY_DB_REF so an
|
||||
// air-gapped deployment can mirror the artifact into its own registry, and so
|
||||
// a busy deployment can avoid the anonymous ghcr rate limit.
|
||||
func Ref() string {
|
||||
if v := os.Getenv("VANTAGE_TRIVY_DB_REF"); v != "" {
|
||||
return v
|
||||
}
|
||||
return DefaultRef
|
||||
}
|
||||
|
||||
// Disabled reports whether the puller and scheduler are switched off entirely.
|
||||
// Findings already written are still served, and still marked stale.
|
||||
func Disabled() bool {
|
||||
return strings.EqualFold(os.Getenv("VANTAGE_VULNDB_DISABLED"), "true")
|
||||
}
|
||||
|
||||
// dbMetadata is the subset of trivy-db's metadata.json we read.
|
||||
type dbMetadata struct {
|
||||
Version int `json:"Version"`
|
||||
}
|
||||
|
||||
// Pull fetches the trivy-db artifact into dir and returns its schema version.
|
||||
//
|
||||
// It extracts into a staging directory and only moves the files into place once
|
||||
// both are present and the schema has been accepted. A pull that fails partway
|
||||
// therefore leaves the previous database untouched rather than a half-written
|
||||
// one that Open would happily accept and scan against.
|
||||
func Pull(ctx context.Context, dir string) (int, error) {
|
||||
ref := Ref()
|
||||
|
||||
parsed, err := registry.ParseReference(ref)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("parse reference %q: %w", ref, err)
|
||||
}
|
||||
|
||||
repo, err := remote.NewRepository(ref)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("open repository %q: %w", ref, err)
|
||||
}
|
||||
|
||||
// The tag or digest half of the reference; the repository already knows the
|
||||
// registry and path.
|
||||
target := parsed.Reference
|
||||
if target == "" {
|
||||
target = "latest"
|
||||
}
|
||||
|
||||
_, manifestBytes, err := oras.FetchBytes(ctx, repo, target, oras.DefaultFetchBytesOptions)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("fetch manifest %s: %w", ref, err)
|
||||
}
|
||||
|
||||
var man ocispec.Manifest
|
||||
if err := json.Unmarshal(manifestBytes, &man); err != nil {
|
||||
return 0, fmt.Errorf("decode manifest %s: %w", ref, err)
|
||||
}
|
||||
if len(man.Layers) == 0 {
|
||||
return 0, fmt.Errorf("artifact %s has no layers", ref)
|
||||
}
|
||||
|
||||
// Streamed rather than buffered: the layer is ~50MB and there is no reason
|
||||
// to hold it in memory on the way to disk.
|
||||
rc, err := repo.Blobs().Fetch(ctx, man.Layers[0])
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("fetch layer: %w", err)
|
||||
}
|
||||
defer rc.Close()
|
||||
|
||||
staging, err := os.MkdirTemp(dir, ".staging-")
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("staging dir: %w", err)
|
||||
}
|
||||
defer os.RemoveAll(staging)
|
||||
|
||||
if err := extractTarGz(rc, staging); err != nil {
|
||||
return 0, fmt.Errorf("extract layer: %w", err)
|
||||
}
|
||||
|
||||
metaBytes, err := os.ReadFile(filepath.Join(staging, metaFileName))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read %s: %w", metaFileName, err)
|
||||
}
|
||||
var meta dbMetadata
|
||||
if err := json.Unmarshal(metaBytes, &meta); err != nil {
|
||||
return 0, fmt.Errorf("decode %s: %w", metaFileName, err)
|
||||
}
|
||||
if meta.Version != SupportedSchema {
|
||||
return 0, fmt.Errorf("trivy-db schema %d is not supported (want %d)", meta.Version, SupportedSchema)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(filepath.Join(staging, dbFileName)); err != nil {
|
||||
return 0, fmt.Errorf("artifact has no %s: %w", dbFileName, err)
|
||||
}
|
||||
|
||||
// Both files present and the schema accepted, so it is safe to replace.
|
||||
for _, name := range []string{dbFileName, metaFileName} {
|
||||
src := filepath.Join(staging, name)
|
||||
dst := filepath.Join(dir, name)
|
||||
if err := os.Remove(dst); err != nil && !os.IsNotExist(err) {
|
||||
return 0, fmt.Errorf("remove old %s: %w", name, err)
|
||||
}
|
||||
if err := os.Rename(src, dst); err != nil {
|
||||
return 0, fmt.Errorf("install %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
return meta.Version, nil
|
||||
}
|
||||
|
||||
// extractTarGz writes the artifact layer into dir. Paths are flattened and
|
||||
// checked so a crafted archive cannot write outside dir.
|
||||
func extractTarGz(r io.Reader, dir string) error {
|
||||
gz, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer gz.Close()
|
||||
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if hdr.Typeflag != tar.TypeReg {
|
||||
continue
|
||||
}
|
||||
name := filepath.Base(hdr.Name) // flatten; the archive is two files
|
||||
if name == "." || name == ".." || name == "" {
|
||||
continue
|
||||
}
|
||||
dst := filepath.Join(dir, name)
|
||||
if !strings.HasPrefix(dst, filepath.Clean(dir)+string(os.PathSeparator)) {
|
||||
return fmt.Errorf("archive entry escapes destination: %q", hdr.Name)
|
||||
}
|
||||
f, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(f, tr); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
// Package vulndb matches installed packages against distribution security
|
||||
// advisories.
|
||||
//
|
||||
// Version comparison is bought rather than written. Distribution version
|
||||
// ordering is subtle in ways that are invisible until they are wrong: dpkg has
|
||||
// epochs and sorts "~" before the empty string, rpmvercmp has its own segment
|
||||
// rules and treats "~" and "^" differently again, and any ordering that falls
|
||||
// back on string comparison puts 1.10 before 1.9. Every one of those mistakes
|
||||
// produces a false negative — a vulnerable host reported clean — which is the
|
||||
// failure nobody notices.
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
apk "github.com/knqyf263/go-apk-version"
|
||||
deb "github.com/knqyf263/go-deb-version"
|
||||
rpm "github.com/knqyf263/go-rpm-version"
|
||||
)
|
||||
|
||||
// ErrUnsupportedFamily means we hold no comparator for this distribution, and
|
||||
// therefore cannot answer whether it is vulnerable. Callers must surface this
|
||||
// as "unsupported" and must never treat it as "not vulnerable".
|
||||
var ErrUnsupportedFamily = errors.New("unsupported OS family")
|
||||
|
||||
// LessThan reports whether version a sorts before version b under the ordering
|
||||
// rules of the given OS family.
|
||||
//
|
||||
// An unparseable or empty version is an error, never a quiet false. False here
|
||||
// means "not vulnerable", which is the dangerous direction to guess in.
|
||||
func LessThan(family, a, b string) (bool, error) {
|
||||
switch family {
|
||||
case "debian", "ubuntu":
|
||||
if a == "" || b == "" {
|
||||
return false, fmt.Errorf("empty deb version (a=%q b=%q)", a, b)
|
||||
}
|
||||
va, err := deb.NewVersion(a)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse deb version %q: %w", a, err)
|
||||
}
|
||||
vb, err := deb.NewVersion(b)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse deb version %q: %w", b, err)
|
||||
}
|
||||
return va.LessThan(vb), nil
|
||||
|
||||
case "redhat", "centos", "rocky", "alma", "amazon", "oracle", "suse", "opensuse", "sles":
|
||||
// go-rpm-version does not error; rpmvercmp is defined over arbitrary
|
||||
// strings. Guard empties so a missing version cannot read as equal.
|
||||
if a == "" || b == "" {
|
||||
return false, fmt.Errorf("empty rpm version (a=%q b=%q)", a, b)
|
||||
}
|
||||
return rpm.NewVersion(a).LessThan(rpm.NewVersion(b)), nil
|
||||
|
||||
case "alpine":
|
||||
if a == "" || b == "" {
|
||||
return false, fmt.Errorf("empty apk version (a=%q b=%q)", a, b)
|
||||
}
|
||||
va, err := apk.NewVersion(a)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse apk version %q: %w", a, err)
|
||||
}
|
||||
vb, err := apk.NewVersion(b)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse apk version %q: %w", b, err)
|
||||
}
|
||||
return va.LessThan(vb), nil
|
||||
|
||||
default:
|
||||
return false, fmt.Errorf("%w: %s", ErrUnsupportedFamily, family)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
// Package vulnsched owns the vulnerability scan loop.
|
||||
//
|
||||
// It runs inside bus.RunAsLeader("housekeeping", …) alongside monitorsched,
|
||||
// workflowsched and the sweepers: one role, one lock. N replicas each running
|
||||
// this loop would mean N copies of the ~50MB database resident, N rescans of
|
||||
// the same fleet on every database refresh, and N digests reaching the
|
||||
// customer for one set of findings.
|
||||
package vulnsched
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
const (
|
||||
tickInterval = 60 * time.Second
|
||||
// trivy-db is rebuilt every six hours; pulling more often buys nothing.
|
||||
dbMaxAge = 6 * time.Hour
|
||||
)
|
||||
|
||||
// Deps are injected from main.go rather than imported, following
|
||||
// workflowsched. It keeps this package's reach explicit and reviewable.
|
||||
type Deps struct {
|
||||
LogEvent func(instanceID, eventType, actor, serverID, keyID, details string)
|
||||
SendDigest func(instanceID string, newly []models.VulnFinding)
|
||||
}
|
||||
|
||||
type scheduler struct {
|
||||
deps Deps
|
||||
dir string
|
||||
store *vulndb.Store
|
||||
version int
|
||||
pulled time.Time
|
||||
}
|
||||
|
||||
func Start(ctx context.Context, deps Deps) {
|
||||
if vulndb.Disabled() {
|
||||
log.Println("vulnsched: disabled by VANTAGE_VULNDB_DISABLED")
|
||||
return
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "vantage-vulndb-")
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: temp dir: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
s := &scheduler{deps: deps, dir: dir}
|
||||
|
||||
go func() {
|
||||
defer os.RemoveAll(dir)
|
||||
defer s.closeStore()
|
||||
|
||||
ticker := time.NewTicker(tickInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.tick(ctx)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (s *scheduler) tick(ctx context.Context) {
|
||||
if err := s.ensureDB(ctx); err != nil {
|
||||
// Keep the last good database and carry on scanning against it. A
|
||||
// network blip must never clear findings or read as "all fixed".
|
||||
log.Printf("vulnsched: database unavailable: %v", err)
|
||||
s.recordDBError(ctx, err)
|
||||
if s.store == nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
s.scanPending(ctx)
|
||||
}
|
||||
|
||||
// ensureDB pulls a fresh database when the local copy is stale, and marks the
|
||||
// whole fleet for rescanning when the version changes — which is what makes a
|
||||
// newly published CVE flag existing servers within a minute rather than at the
|
||||
// next agent report.
|
||||
func (s *scheduler) ensureDB(ctx context.Context) error {
|
||||
if s.store != nil && time.Since(s.pulled) < dbMaxAge {
|
||||
return nil
|
||||
}
|
||||
|
||||
version, err := vulndb.Pull(ctx, s.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.closeStore()
|
||||
store, err := vulndb.Open(s.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.store = store
|
||||
s.pulled = time.Now()
|
||||
|
||||
changed := version != s.version
|
||||
s.version = version
|
||||
|
||||
_, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{},
|
||||
bson.M{"$set": bson.M{"db_version": version, "pulled_at": s.pulled}, "$unset": bson.M{"last_error": ""}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
|
||||
if changed {
|
||||
res, err := db.Col("server_packages").UpdateMany(ctx,
|
||||
bson.M{"status": bson.M{"$ne": models.ScanStatusUnsupported}},
|
||||
bson.M{"$set": bson.M{"scan_pending": true}},
|
||||
)
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: mark fleet pending: %v", err)
|
||||
} else {
|
||||
log.Printf("vulnsched: database version %d, %d servers marked for rescan", version, res.ModifiedCount)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *scheduler) recordDBError(ctx context.Context, err error) {
|
||||
_, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{},
|
||||
bson.M{"$set": bson.M{"last_error": err.Error()}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
}
|
||||
|
||||
func (s *scheduler) scanPending(ctx context.Context) {
|
||||
cur, err := db.Col("server_packages").Find(ctx, bson.M{"scan_pending": true})
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: find pending: %v", err)
|
||||
return
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var pending []models.ServerPackages
|
||||
if err := cur.All(ctx, &pending); err != nil {
|
||||
log.Printf("vulnsched: decode pending: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Newly opened findings are collected across the whole tick and sent as one
|
||||
// digest per instance. A database refresh can open several hundred findings
|
||||
// at once; one message per finding would rate-limit the webhook or get the
|
||||
// channel muted, and either way the alerts stop being read.
|
||||
newly := map[string][]models.VulnFinding{}
|
||||
|
||||
for _, sp := range pending {
|
||||
if ctx.Err() != nil {
|
||||
// Leadership lost. scan_pending is still set, so the next leader
|
||||
// picks these up — which is why it lives on the document.
|
||||
return
|
||||
}
|
||||
opened := s.scanOne(ctx, sp)
|
||||
newly[sp.InstanceID] = append(newly[sp.InstanceID], opened...)
|
||||
}
|
||||
|
||||
for instanceID, findings := range newly {
|
||||
if len(findings) > 0 && s.deps.SendDigest != nil {
|
||||
s.deps.SendDigest(instanceID, findings)
|
||||
}
|
||||
}
|
||||
|
||||
_, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{},
|
||||
bson.M{"$set": bson.M{"last_full_scan_at": time.Now()}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
}
|
||||
|
||||
func (s *scheduler) scanOne(ctx context.Context, sp models.ServerPackages) []models.VulnFinding {
|
||||
now := time.Now()
|
||||
|
||||
results, err := vulndb.Match(s.store, sp.OS, sp.Packages)
|
||||
if err != nil {
|
||||
// We hold no feed for this distribution, so we cannot answer whether it
|
||||
// is vulnerable. Say "unsupported" — reporting zero findings here would
|
||||
// be indistinguishable from reporting a clean host, and one of those is
|
||||
// a lie.
|
||||
status := models.ScanStatusUnsupported
|
||||
if !errors.Is(err, vulndb.ErrUnsupportedFamily) {
|
||||
log.Printf("vulnsched: scan %s: %v", sp.ServerID, err)
|
||||
status = sp.Status
|
||||
}
|
||||
s.clearPending(ctx, sp.ID, status, now)
|
||||
return nil
|
||||
}
|
||||
|
||||
existing, err := services.ListFindings(ctx, sp.InstanceID, sp.ServerID)
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: list findings %s: %v", sp.ServerID, err)
|
||||
return nil
|
||||
}
|
||||
|
||||
diff := services.DiffFindings(existing, results, now)
|
||||
if err := services.ApplyFindingDiff(ctx, sp.InstanceID, sp.ServerID, diff, now); err != nil {
|
||||
log.Printf("vulnsched: apply diff %s: %v", sp.ServerID, err)
|
||||
return nil
|
||||
}
|
||||
|
||||
s.clearPending(ctx, sp.ID, models.ScanStatusOK, now)
|
||||
|
||||
for i := range diff.NewlyOpened {
|
||||
diff.NewlyOpened[i].ServerID = sp.ServerID
|
||||
}
|
||||
return diff.NewlyOpened
|
||||
}
|
||||
|
||||
func (s *scheduler) clearPending(ctx context.Context, id bson.ObjectID, status string, now time.Time) {
|
||||
_, _ = db.Col("server_packages").UpdateOne(ctx,
|
||||
bson.M{"_id": id},
|
||||
bson.M{"$set": bson.M{
|
||||
"scan_pending": false,
|
||||
"status": status,
|
||||
"scanned_at": now,
|
||||
"db_version": s.version,
|
||||
}},
|
||||
)
|
||||
}
|
||||
|
||||
func (s *scheduler) closeStore() {
|
||||
if s.store != nil {
|
||||
_ = s.store.Close()
|
||||
s.store = nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
// Package workflowsched fires workflow runs on a cron schedule.
|
||||
//
|
||||
// Only robfig/cron's parser is used — Parse and Next. Its own scheduler is
|
||||
// not, because this work runs under the housekeeping leader lock and has to
|
||||
// stop the moment leadership is lost.
|
||||
package workflowsched
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/robfig/cron/v3"
|
||||
)
|
||||
|
||||
// ErrBadSchedule covers both a malformed expression and an unknown timezone.
|
||||
// Handlers map it to 400 — both are the caller's mistake, and both are much
|
||||
// cheaper to find at save time than at 2am.
|
||||
var ErrBadSchedule = errors.New("invalid schedule")
|
||||
|
||||
// Standard 5-field cron: minute hour dom month dow. Deliberately no seconds
|
||||
// field and no descriptors — a schedule a person cannot read back is a
|
||||
// schedule nobody can audit.
|
||||
var cronParser = cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)
|
||||
|
||||
func ParseSchedule(expr, tz string) (cron.Schedule, error) {
|
||||
if tz == "" {
|
||||
return nil, fmt.Errorf("%w: a timezone is required", ErrBadSchedule)
|
||||
}
|
||||
if _, err := time.LoadLocation(tz); err != nil {
|
||||
return nil, fmt.Errorf("%w: unknown timezone %q", ErrBadSchedule, tz)
|
||||
}
|
||||
sched, err := cronParser.Parse(expr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrBadSchedule, err)
|
||||
}
|
||||
return sched, nil
|
||||
}
|
||||
|
||||
// NextOccurrence returns the first firing strictly after from, computed in the
|
||||
// schedule's own zone so that a DST boundary moves the wall-clock time the way
|
||||
// a person expects rather than drifting by an hour for half the year.
|
||||
func NextOccurrence(expr, tz string, from time.Time) (time.Time, error) {
|
||||
sched, err := ParseSchedule(expr, tz)
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
loc, err := time.LoadLocation(tz)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("%w: unknown timezone %q", ErrBadSchedule, tz)
|
||||
}
|
||||
return sched.Next(from.In(loc)), nil
|
||||
}
|
||||
|
||||
type Decision string
|
||||
|
||||
const (
|
||||
Fire Decision = "fire"
|
||||
SkipMissed Decision = "missed"
|
||||
SkipRunning Decision = "already_running"
|
||||
)
|
||||
|
||||
// GraceWindow is how late an occurrence may be and still run. A job missed by
|
||||
// ten minutes during a deploy should still run; one missed by two days should
|
||||
// not fire at lunchtime.
|
||||
const GraceWindow = time.Hour
|
||||
|
||||
// Decide is the whole fire/skip policy, kept pure so it can be tested without
|
||||
// a database and read without following a loop.
|
||||
//
|
||||
// The missed check comes first: an occurrence that is already too old to run
|
||||
// should be recorded as missed regardless of what is running now, or a slow
|
||||
// run would relabel a stale occurrence as a fresh conflict.
|
||||
func Decide(due, now time.Time, runActive bool) Decision {
|
||||
if now.Sub(due) > GraceWindow {
|
||||
return SkipMissed
|
||||
}
|
||||
if runActive {
|
||||
return SkipRunning
|
||||
}
|
||||
return Fire
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package workflowsched
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
const tickInterval = 30 * time.Second
|
||||
|
||||
// Deps are the service functions the loop needs. They are injected rather than
|
||||
// imported because services already imports this package for NextOccurrence,
|
||||
// and a package cannot import its own importer.
|
||||
type Deps struct {
|
||||
TriggerWorkflow func(instanceID, workflowID, actor string) (string, error)
|
||||
LogEvent func(instanceID, eventType, actor, serverID, keyID, details string)
|
||||
}
|
||||
|
||||
// Start runs the scheduler until ctx is cancelled. It is called inside
|
||||
// bus.RunAsLeader("housekeeping", …) alongside monitorsched and the sweepers:
|
||||
// one role, one lock. N replicas each running this loop would fire every
|
||||
// scheduled workflow N times.
|
||||
func Start(ctx context.Context, deps Deps) {
|
||||
go func() {
|
||||
ticker := time.NewTicker(tickInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
tick(ctx, deps)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func tick(ctx context.Context, deps Deps) {
|
||||
now := time.Now()
|
||||
|
||||
cur, err := db.Col("workflows").Find(ctx, bson.M{
|
||||
"schedule.enabled": true,
|
||||
"next_run_at": bson.M{"$lte": now},
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("workflowsched: find due: %v", err)
|
||||
return
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var due []models.Workflow
|
||||
if err := cur.All(ctx, &due); err != nil {
|
||||
log.Printf("workflowsched: decode due: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, wf := range due {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
process(ctx, deps, wf, now)
|
||||
}
|
||||
}
|
||||
|
||||
func process(ctx context.Context, deps Deps, wf models.Workflow, now time.Time) {
|
||||
if wf.NextRunAt == nil || wf.Schedule == nil {
|
||||
return
|
||||
}
|
||||
dueAt := *wf.NextRunAt
|
||||
|
||||
next, err := NextOccurrence(wf.Schedule.Cron, wf.Schedule.TZ, now)
|
||||
if err != nil {
|
||||
// A schedule that no longer parses cannot be advanced, and leaving
|
||||
// next_run_at in the past would spin this loop every 30 seconds
|
||||
// forever. Disable it and say so.
|
||||
log.Printf("workflowsched: workflow %s has an unusable schedule, disabling: %v", wf.WorkflowID, err)
|
||||
disable(ctx, deps, wf, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// The claim. Matching on the current next_run_at as well as the id means a
|
||||
// second process reaching this document after another has claimed it
|
||||
// matches nothing and does nothing. This — not the leader lock — is what
|
||||
// makes a double fire impossible; the lock only keeps it cheap.
|
||||
res, err := db.Col("workflows").UpdateOne(ctx,
|
||||
bson.M{"workflow_id": wf.WorkflowID, "next_run_at": dueAt},
|
||||
bson.M{"$set": bson.M{"next_run_at": next}},
|
||||
)
|
||||
if err != nil {
|
||||
log.Printf("workflowsched: claim %s: %v", wf.WorkflowID, err)
|
||||
return
|
||||
}
|
||||
if res.MatchedCount == 0 {
|
||||
return // claimed elsewhere
|
||||
}
|
||||
|
||||
switch Decide(dueAt, now, hasActiveRun(ctx, wf.InstanceID, wf.WorkflowID)) {
|
||||
case SkipMissed:
|
||||
recordSkip(ctx, deps, wf, string(SkipMissed), dueAt, now)
|
||||
case SkipRunning:
|
||||
recordSkip(ctx, deps, wf, string(SkipRunning), dueAt, now)
|
||||
case Fire:
|
||||
if _, err := deps.TriggerWorkflow(wf.InstanceID, wf.WorkflowID, "schedule"); err != nil {
|
||||
log.Printf("workflowsched: trigger %s: %v", wf.WorkflowID, err)
|
||||
recordSkip(ctx, deps, wf, "error: "+err.Error(), dueAt, now)
|
||||
return
|
||||
}
|
||||
_, _ = db.Col("workflows").UpdateOne(ctx,
|
||||
bson.M{"workflow_id": wf.WorkflowID},
|
||||
bson.M{"$set": bson.M{"last_run_at": now}, "$unset": bson.M{"last_skipped": ""}},
|
||||
)
|
||||
deps.LogEvent(wf.InstanceID, "workflow.scheduled_run", "schedule", "", "",
|
||||
"workflow "+wf.Name+" started on schedule")
|
||||
}
|
||||
}
|
||||
|
||||
func hasActiveRun(ctx context.Context, instanceID, workflowID string) bool {
|
||||
err := db.Col("workflow_runs").FindOne(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"workflow_id": workflowID,
|
||||
"status": "running",
|
||||
}, options.FindOne().SetProjection(bson.M{"_id": 1})).Err()
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func recordSkip(ctx context.Context, deps Deps, wf models.Workflow, reason string, due, at time.Time) {
|
||||
_, _ = db.Col("workflows").UpdateOne(ctx,
|
||||
bson.M{"workflow_id": wf.WorkflowID},
|
||||
bson.M{"$set": bson.M{"last_skipped": models.Skip{Reason: reason, Due: due, At: at}}},
|
||||
)
|
||||
deps.LogEvent(wf.InstanceID, "workflow.schedule_skipped", "schedule", "", "",
|
||||
"workflow "+wf.Name+" skipped "+due.Format(time.RFC3339)+": "+reason)
|
||||
}
|
||||
|
||||
func disable(ctx context.Context, deps Deps, wf models.Workflow, reason string) {
|
||||
_, _ = db.Col("workflows").UpdateOne(ctx,
|
||||
bson.M{"workflow_id": wf.WorkflowID},
|
||||
bson.M{
|
||||
"$set": bson.M{"schedule.enabled": false},
|
||||
"$unset": bson.M{"next_run_at": ""},
|
||||
},
|
||||
)
|
||||
deps.LogEvent(wf.InstanceID, "workflow.schedule_disabled", "schedule", "", "",
|
||||
"workflow "+wf.Name+" schedule disabled: "+reason)
|
||||
}
|
||||
@@ -31,6 +31,10 @@ const (
|
||||
|
||||
FeatureConsole = "console" // browser SSH/RDP/VNC
|
||||
FeatureOIDC = "oidc" // per-instance single sign-on
|
||||
// FeatureVulnScanning gates package inventory collection as well as the
|
||||
// findings themselves. The gate is at collection, not display: an ungated
|
||||
// instance stores no inventory, and storage is the expensive half.
|
||||
FeatureVulnScanning = "vuln_scanning"
|
||||
)
|
||||
|
||||
// Support levels. Carried for display and enforced by nothing — there is no code
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
{{define "title"}}New vulnerabilities detected{{end}}
|
||||
{{define "pill"}}{{template "chip" (dict "label" (upper .TopSeverity) "tone" "down")}}{{end}}
|
||||
{{define "body"}}
|
||||
{{template "lead" .Summary}}
|
||||
{{template "rows" (list
|
||||
(dict "k" "Instance" "v" .InstanceName)
|
||||
(dict "k" "New findings" "v" .Count))}}
|
||||
{{range .Rows}}
|
||||
{{if .FixedIn}}{{template "well" (printf "%s (%s) — %s on %s, fixed in %s" .CVEID .Severity .PackageName .ServerName .FixedIn)}}{{else}}{{template "well" (printf "%s (%s) — %s on %s, no fix published" .CVEID .Severity .PackageName .ServerName)}}{{end}}
|
||||
{{end}}
|
||||
{{if .More}}{{template "p" (printf "…and %d more." .More)}}{{end}}
|
||||
{{template "note" (printf "Scanned against a vulnerability database pulled %s ago." .DBAge)}}
|
||||
{{end}}
|
||||
@@ -0,0 +1,12 @@
|
||||
{{define "subject"}}{{.Count}} new {{if eq .Count 1}}vulnerability{{else}}vulnerabilities{{end}} on {{.InstanceName}}{{end}}
|
||||
{{define "title"}}New vulnerabilities detected{{end}}
|
||||
{{define "pill"}}{{.TopSeverity}}{{end}}
|
||||
{{define "body"}}
|
||||
{{template "lead" .Summary}}
|
||||
|
||||
{{range .Rows}}- {{.CVEID}} ({{.Severity}}) — {{.PackageName}} on {{.ServerName}}{{if .FixedIn}}, fixed in {{.FixedIn}}{{else}}, no fix published{{end}}
|
||||
{{end}}
|
||||
{{if .More}}...and {{.More}} more.{{end}}
|
||||
|
||||
Scanned against vulnerability database pulled {{.DBAge}} ago.
|
||||
{{end}}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user