Compare commits
46
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0c15b25ecd | ||
|
|
0c21765da3 | ||
|
|
4ff8fc8d51 | ||
|
|
483053b9a2 | ||
|
|
fd4c51f3db | ||
|
|
1b351cfca4 | ||
|
|
cf9d85b3cd | ||
|
|
6a4ef5b6c6 | ||
|
|
501cf4e733 | ||
|
|
89c21d752a | ||
|
|
0e38d9d500 | ||
|
|
3511c34daa | ||
|
|
0838d1d735 | ||
|
|
d1769fc886 | ||
|
|
6dced22499 | ||
|
|
5cee53dc5f | ||
|
|
81248bb159 | ||
|
|
6354d54de8 | ||
|
|
da6d64f95c | ||
|
|
9ba3d4a61f | ||
|
|
eee236a072 | ||
|
|
9df89e2db4 | ||
|
|
f60c509b47 | ||
|
|
84dfcfeac7 | ||
|
|
5dda3b5c4a | ||
|
|
db64320bd8 | ||
|
|
583f60771c | ||
|
|
a92c3190c2 | ||
|
|
3a6d24fe0e | ||
|
|
c277ecff44 | ||
|
|
bd690c94c3 | ||
|
|
a22fdf197e | ||
|
|
bd24b03cac | ||
|
|
3afc4ab012 | ||
|
|
d1ac3e98ce | ||
|
|
5bba54f3e5 | ||
|
|
fe7bc300e2 | ||
|
|
00c03c365d | ||
|
|
dc8dd3dd58 | ||
|
|
85a8865892 | ||
|
|
50a9ac5fdc | ||
|
|
3388d2f895 | ||
|
|
3a77fc2abd | ||
|
|
3d59836d0c | ||
|
|
d9184312aa | ||
|
|
b9802e6b04 |
+188
-7
@@ -124,6 +124,39 @@ A library of reusable **steps** (bash or PowerShell scripts with declared inputs
|
||||
|
||||
Default steps are seeded per org at boot (`SeedDefaultSteps`) from `VANTAGE_DEFAULT_STEPS_DIR`, which `server/Dockerfile` bakes to `/opt/default-steps` from the repo's `default_steps/`. Deliberately **not** under `/data` — that is a bind mount, so the library would be editable from the host. Adding a step there means committing a file and rebuilding, which is why `default_steps/` is in the `server` rebuild trigger. **Steps with `source: "default"` are read-only**: `UpdateStep`/`DeleteStep` refuse with `ErrDefaultStep` (409), because seeding rewrites them on every boot, so an edit would silently revert and a delete would come back. `web/` mirrors this — the step modal opens read-only, Delete is hidden, and the designer's per-step script override is `readOnly` for a default library step — but as elsewhere, the API is the boundary and the UI is the courtesy. Seeding writes straight to the collection rather than through `UpdateStep`, so the guard does not lock out the seeder. Logs are swept by retention (`workflow_log_retention_days`; nil = 30 days, 0 = forever).
|
||||
|
||||
### Scheduled workflows
|
||||
|
||||
A workflow may carry `schedule{enabled, cron, tz}` — standard **5-field** cron
|
||||
and an IANA zone name, both validated at save time. `next_run_at` is
|
||||
**persisted on the document, not held in memory**: a leader handover between
|
||||
computing an occurrence and firing it would otherwise lose it or fire it twice,
|
||||
the same argument that put `workflow_log_seq` in MongoDB.
|
||||
|
||||
`server/internal/workflowsched` ticks every 30s inside the **existing**
|
||||
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched` and the sweepers —
|
||||
one role, one lock. **The atomic claim, not the lock, is what prevents a double
|
||||
fire**: the `UpdateOne` matches on the document *and* its current `next_run_at`
|
||||
while setting the recomputed one, so a second process reaching the same workflow
|
||||
matches nothing and does nothing. The lock only makes it cheap.
|
||||
|
||||
`workflowsched` **must not import `services`** — `services` already imports it
|
||||
for `SetSchedule`'s call to `NextOccurrence`, and Go has no cycles.
|
||||
`TriggerWorkflow` and `LogEvent` are therefore injected as `workflowsched.Deps`
|
||||
from `main.go`. Firing goes through the same `TriggerWorkflow` a person uses,
|
||||
with `"schedule"` as the actor, so there is no second dispatch path and the run
|
||||
detail page needed no changes.
|
||||
|
||||
`main.go` imports `_ "time/tzdata"`, and it is load-bearing: `server/Dockerfile`
|
||||
runs on `scratch`, which ships no zone database, so without it
|
||||
`time.LoadLocation("Europe/London")` fails and every schedule silently falls
|
||||
back to UTC — an hour wrong for half the year, in the direction nobody notices
|
||||
until a maintenance window lands in business hours. It works on a developer
|
||||
machine either way, which is exactly why it gets forgotten.
|
||||
|
||||
Skips are recorded and surfaced, not just logged: past the 1h grace window is
|
||||
`missed`, an active run is `already_running`, and a schedule that no longer
|
||||
parses is disabled rather than left spinning the loop every 30 seconds forever.
|
||||
|
||||
### Server tags and workflow targeting
|
||||
|
||||
A server carries `tags map[string]string` — lowercase `[a-z0-9_-]`, key ≤32,
|
||||
@@ -150,11 +183,26 @@ are **not** filtered out — the dispatcher already answers 503 per server, and
|
||||
patch run that silently omits an unreachable machine is worse than one that
|
||||
visibly fails on it.
|
||||
|
||||
`web/app/(app)/workflows/[id]/page.tsx` **duplicates that match logic in
|
||||
TypeScript** to draw the resolved count without a round trip, since the browser
|
||||
already holds the fleet. It is a second implementation of `UnionTargets` /
|
||||
`MatchesTags` and must change in the same commit as the Go one — the same shape
|
||||
of hazard as the mirrored token blocks.
|
||||
**Both halves of the selector are edited in `EditWorkflowModal`** — the named
|
||||
servers in a `DualListBox`, the tag rows directly beneath it — and saved
|
||||
together by one `updateWorkflow`. The designer's Targets panel is **read-only**:
|
||||
it reports the count and the tags and links to Edit. Splitting the two halves
|
||||
across two screens meant a workflow's reach was decided in two places with no
|
||||
one view showing both.
|
||||
|
||||
`web/lib/targets.ts` **duplicates the match logic in TypeScript** to draw the
|
||||
resolved count without a round trip, since the browser already holds the fleet.
|
||||
It is a second implementation of `UnionTargets` / `MatchesTags` and must change
|
||||
in the same commit as the Go one — the same shape of hazard as the mirrored
|
||||
token blocks. It is a shared module rather than inline in a component because
|
||||
the logic had already been written twice, and the second copy — the workflows
|
||||
list — counted `target_server_ids` alone, so a **tag-only workflow reported zero
|
||||
targets** while running fine.
|
||||
|
||||
The server picker is a hand-built two-pane list, not `<select multiple>`: a
|
||||
native multi-select paints its selected rows with the platform highlight colour,
|
||||
which cannot be restyled across browsers and lands outside the token palette on
|
||||
a dark ground.
|
||||
|
||||
### Monitors
|
||||
|
||||
@@ -267,6 +315,121 @@ a Service cannot address the one pod holding a console listener.
|
||||
|
||||
Agents report CPU/memory/swap/partitions/kernel — metrics every 30s, full static snapshot every 15 min. They also check for pending OS package updates hourly and can apply them on command (`ApplyUpdatesCmd`).
|
||||
|
||||
### Package inventory and CVE findings
|
||||
|
||||
Agents report their installed packages hourly; the control plane matches them
|
||||
against distribution security feeds and raises findings that link to the
|
||||
existing `ApplyUpdatesCmd` patching path. Gated by the `vuln_scanning` licence
|
||||
feature, **checked at collection rather than display** — an ungated instance
|
||||
stores no inventory, and storage is the expensive half.
|
||||
|
||||
**Matching uses distribution feeds, never NVD version ranges.** Distributions
|
||||
backport security fixes without changing the upstream version: Ubuntu's
|
||||
`openssl 3.0.2-0ubuntu1.15` is patched against CVE-2023-0286 while NVD still
|
||||
calls 3.0.2 vulnerable. Matching on NVD would report a fully patched fleet as
|
||||
critical, and once the first report is mostly wrong nobody reads the second.
|
||||
`trivy-db` is those feeds pre-merged; `server/internal/vulndb` pulls it as an
|
||||
OCI artifact to an ephemeral directory. Version comparison is bought from
|
||||
`go-deb-version`/`go-rpm-version`/`go-apk-version` because dpkg epochs, `~`
|
||||
sorting before the empty string, and `rpmvercmp` are each a silent false
|
||||
negative waiting to happen.
|
||||
|
||||
**Only the leader matches.** `ReportPackages` upserts the list and sets
|
||||
`scan_pending`; it does not scan. `vulnsched` runs inside the existing
|
||||
`bus.RunAsLeader("housekeeping", …)` and does the matching, because otherwise
|
||||
every replica needs the ~50MB database resident and a database refresh has N
|
||||
replicas rescanning the same fleet and sending N digests. The tick is also the
|
||||
digest's batch boundary, which is what makes "one message, not five hundred"
|
||||
structural rather than a debounce someone maintains.
|
||||
|
||||
Findings are **never deleted when a package is patched** — the state moves to
|
||||
`fixed`, so "what did we remediate last quarter" stays answerable. Acceptance
|
||||
requires a reason and an expiry, and reopens automatically: permanent dismissal
|
||||
is where risk goes to be forgotten. An unsupported distribution reports
|
||||
`status: unsupported`, never "0 findings"; claiming clean when the truth is
|
||||
unknown is the same lie as a silently stale database, which is why
|
||||
`vulndb_meta.pulled_at` is on screen rather than only in a log.
|
||||
|
||||
**`server/Dockerfile`'s runtime stage is `scratch`, so it carries an explicitly
|
||||
copied `/tmp`.** The scheduler unpacks the database to a temporary directory,
|
||||
and a scratch image has none — the failure is `vulnsched: temp dir: stat /tmp:
|
||||
no such file or directory`, logged once at boot while every other subsystem
|
||||
runs normally, so the only symptom is a fleet that never reports a finding.
|
||||
|
||||
Two environment variables: `VANTAGE_TRIVY_DB_REF` mirrors the artifact for
|
||||
air-gapped installs, and `VANTAGE_VULNDB_DISABLED` switches the puller and
|
||||
scheduler off entirely.
|
||||
|
||||
### Workload registry
|
||||
|
||||
A **workload** is one Docker container or one systemd unit — one word for the
|
||||
page, the collection and the commands, rather than saying "container or
|
||||
service" in every identifier. Linux only, and **not gated by licence**: this
|
||||
reads as core fleet management, so v1 ships everywhere with no `HasFeature`
|
||||
check. If that changes the check belongs at `ReportWorkloads`, gating collection
|
||||
rather than display, exactly as sub-project A does.
|
||||
|
||||
Agents collect on a 60-second ticker and report through `ReportWorkloads` with
|
||||
the **offer-then-send** handshake the package report already uses. The offer is
|
||||
identified by an explicit `full` flag, **not by an empty workloads list**: a
|
||||
host genuinely running nothing sends an empty list as its full report, and
|
||||
inferring the offer from emptiness leaves that host answering `need_full` every
|
||||
60 seconds forever and never storing anything.
|
||||
|
||||
**The on-demand refresh returns no data.** `RefreshWorkloadsCmd` carries nothing
|
||||
back; it makes the agent report through the normal RPC and the UI refetches. A
|
||||
refresh that returned workloads inline would be a second writer for
|
||||
`server_workloads`, arriving by a different route with its own serialisation and
|
||||
its own opportunity to disagree with the periodic one. One writer, one shape.
|
||||
Opening the panel dispatches a refresh because the panel has a Restart button on
|
||||
it, and a stale row is a wrong action aimed at a container that already died.
|
||||
|
||||
Two operations do answer back, both over the bus, both with `Await` called
|
||||
**before** dispatch: control actions reuse the existing `CommandResult`, and log
|
||||
reads get `WorkloadLogsResult`. `CommandStream` republishes **every**
|
||||
`CommandResult` onto `bus.ResultChannel` — publishing with no subscriber is a
|
||||
no-op, so this costs nothing and avoids a second result path.
|
||||
|
||||
**The protected set is computed agent-side and enforced agent-side.**
|
||||
`vantage-agent.service`, plus the container ID read from `/proc/self/cgroup`
|
||||
should the agent ever run in a container. As with the console relay hardcoding
|
||||
`127.0.0.1`, the control plane may name a target but the agent decides what it
|
||||
will do to itself; a server-side denylist alone would be bypassed by the next
|
||||
dispatch path someone adds, and the failure is unrecoverable from the UI. The
|
||||
reported `Protected` flag is the courtesy that greys the button; the agent's own
|
||||
check is the boundary. The API answers **409** when it fires — nothing failed.
|
||||
|
||||
Collection avoids parsing English: `docker ps -aq` then
|
||||
`docker inspect --format '{{json .}}'`, because `docker ps` reports health and
|
||||
uptime inside a human `Status` string that is localised and reworded between
|
||||
releases. Compose stacks come from the `com.docker.compose.project` label, never
|
||||
from YAML on disk — a compose file there may not be what is running. systemd
|
||||
uses **column** output, not `--output=json`, which needs systemd 246+.
|
||||
|
||||
`DockerOK`/`DockerError` are two fields because there are three states: not
|
||||
installed (common on this fleet, and not a fault), installed but not responding,
|
||||
and running nothing. The UI must render the first as "not in use here" rather
|
||||
than an empty list.
|
||||
|
||||
Logs are capped at **500 lines and 256KB, whichever binds first** — a line count
|
||||
alone does not bound size, and 500 lines of 4KB JSON is 2MB across the bus. The
|
||||
cap is mirrored in `services.MaxWorkloadLogLines` because `agent/` is a separate
|
||||
module with an `internal/` tree and the constant cannot be shared; change one,
|
||||
change the other. There is **no follow mode**: the browser console already gives
|
||||
a real terminal where `docker logs -f` works properly. Log reads and control
|
||||
actions are **owner|admin and audited**, unlike the read-only snapshot — a
|
||||
container's stdout is arbitrary and cannot be masked the way a workflow's can.
|
||||
|
||||
`server_workloads` is one document per server, mirroring `server_packages`, and
|
||||
is in `ScopedCollections` (which `scopedCollectionsForPurge` derives from). There
|
||||
is no history: a workload list is state, not a record.
|
||||
|
||||
**`proto/vantage/v1/vantage.proto` is documentation, not a generator input.**
|
||||
Both `pb` packages are hand-written JSON-tagged structs over a custom codec, and
|
||||
there are two copies — `agent/internal/grpc/pb` and `server/internal/grpc/pb`.
|
||||
A message added to one must be added to the other and to the `.proto`, in the
|
||||
same commit.
|
||||
|
||||
### Agent self-update
|
||||
|
||||
`UpdateAgentCmd` carries a target version and Gitea base URL; the agent downloads and replaces itself.
|
||||
@@ -419,6 +582,7 @@ service Vantage {
|
||||
rpc SyncKeys(SyncRequest) returns (SyncResponse);
|
||||
rpc UploadGeneratedKey(UploadKeyRequest) returns (UploadKeyResponse);
|
||||
rpc ReportUpdates(ReportUpdatesRequest) returns (ReportUpdatesResponse);
|
||||
rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse);
|
||||
rpc ReportInventory(InventoryReport) returns (InventoryReportResponse);
|
||||
rpc SyncMonitors(SyncMonitorsRequest) returns (SyncMonitorsResponse);
|
||||
rpc ReportChecks(ReportChecksRequest) returns (ReportChecksResponse);
|
||||
@@ -428,7 +592,8 @@ service Vantage {
|
||||
|
||||
`CommandStream` is the only streaming RPC: the agent authenticates once with `AgentReady`, then the server pushes `ServerCommand`s and the agent replies with `CommandResult`, `StepResult`, or `StepOutputChunk`.
|
||||
|
||||
`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`, `OpenProxyCmd`, `PingCmd`.
|
||||
`ServerCommand` variants: `GenerateKeyCmd`, `DeleteKeyCmd`, `UpdateAgentCmd`, `ApplyUpdatesCmd`, `RunStepCmd`, `CleanupWorkspaceCmd`, `OpenProxyCmd`, `PingCmd`, `RefreshWorkloadsCmd`, `ControlWorkloadCmd`,
|
||||
`WorkloadLogsCmd`.
|
||||
|
||||
**`PingCmd` is a liveness beat, and it is not redundant with gRPC keepalive.**
|
||||
The server sends one every 20s on an otherwise idle command stream; the agent
|
||||
@@ -479,6 +644,16 @@ channels GET,POST /channels · PUT,DELETE /channels/:id · POST /channels/:i
|
||||
secrets GET,POST /secrets · GET,PUT,DELETE /secrets/:group
|
||||
POST /secrets/:group/reveal · DELETE /secrets/:group/:key
|
||||
console POST /console/connect · GET /console/tunnel (websocket)
|
||||
vulns GET /vulnerabilities · GET /vulnerabilities/summary
|
||||
POST /vulnerabilities/rescan (owner|admin)
|
||||
POST,DELETE /vulnerabilities/:id/accept (owner|admin)
|
||||
GET /servers/:id/vulnerabilities · GET /servers/:id/packages
|
||||
GET /packages/search?name=
|
||||
GET,POST /vuln-rules · PUT,DELETE /vuln-rules/:id (owner|admin)
|
||||
workloads GET /workloads · GET /servers/:id/workloads
|
||||
POST /servers/:id/workloads/refresh
|
||||
POST /servers/:id/workloads/:wid/action (owner|admin)
|
||||
GET /servers/:id/workloads/:wid/logs (owner|admin)
|
||||
audit GET /audit
|
||||
agent GET /agent/latest-version
|
||||
settings GET,PUT /settings · POST /settings/secrets-token (owner|admin)
|
||||
@@ -559,7 +734,7 @@ Paddle is merchant of record; `admin/internal/paddle` is a thin REST client (no
|
||||
|
||||
## MongoDB Collections
|
||||
|
||||
`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `migrations`
|
||||
`servers` · `keys` · `assignments` · `orgs` · `users` · `auth_providers` · `settings` · `secrets` · `workflows` · `workflow_steps` · `workflow_runs` · `workflow_log_lines` · `workflow_log_seq` · `monitors` · `incidents` · `monitor_rollups` · `notification_channels` · `console_sessions` · `audit_logs` · `server_packages` · `vuln_findings` · `vuln_alert_rules` · `vulndb_meta` · `server_workloads` · `migrations`
|
||||
|
||||
Every document except `migrations` carries `org_id`. Struct definitions are the source of truth — see `server/internal/models/`.
|
||||
|
||||
@@ -574,6 +749,10 @@ Notes that are not obvious from the structs:
|
||||
- `auth_providers.provider_id` is a short random identifier, not the Mongo `_id`: it appears in the callback URL a customer pastes into their IdP, and an `_id` there would publish a database key. `callback_notice` marks a provider migrated from the old single-provider shape, whose redirect URI therefore changed.
|
||||
- `workflow_log_lines` is keyed `(run_id, server_id, seq)` — the index is not an optimisation, every read is a range scan over it. `workflow_log_seq` holds one counter document per `run_id/server_id`, which is what lets two pods interleave into one ordered log. Neither carries `instance_id`: they are reached only through a run, and a run is already scoped.
|
||||
- `users.auth_source` is `local`, `oidc` or `hq`. An `hq` user was projected from a Vantage HQ account and carries `hq_user_id`; HQ owns its role, password and existence.
|
||||
- `server_packages` holds a server's whole package set in **one** document, not one per package. The hash already established that something changed, so a report is a single atomic upsert with no delta logic to get wrong; ~2000 packages is ~150KB, well inside the 16MB limit. `scan_pending` lives on the document rather than in memory so a leader handover cannot lose it.
|
||||
- `vuln_findings` is unique on `(instance_id, server_id, cve_id, package_name)`. That key is what makes a rescan an idempotent upsert rather than a duplicate factory, and what lets `first_seen` survive one. An empty `fixed_in` means no vendor fix exists — a real state, never "not vulnerable".
|
||||
- `vulndb_meta` is a singleton and deliberately carries **no** `instance_id`: the vulnerability database is a property of the deployment, not a tenant. Same reasoning as `migrations`, and the reason it is absent from `services.ScopedCollections`.
|
||||
- **`services.ScopedCollections` is the canonical registry of tenant-scoped collections**, and `scopedCollectionsForPurge` derives instance deletion from it rather than keeping a second list. A new collection carrying `instance_id` must be added there or its rows outlive the instance.
|
||||
|
||||
Admin's own database is separate and holds `accounts` · `admin_instances` · `licenses` · `subscriptions` · `plans` · `catalogue` · `entitlements` · `paddle_events` · `staff_users` · `customer_users` · `instance_members` · `admin_audit`. `paddle_events` is the webhook idempotency log, unique on `event_id`: an event is claimed there before processing, and a duplicate of a handled event is a 200 no-op. `instance_members` is unique on `(instance_id, customer_user_id)` — one person holds at most one user in one instance, which makes a grant idempotent-by-refusal rather than silently doubling a projection. It is an _index_ of the control-plane rows, not the authority (see "Grants project, they do not federate"). Admin has no migrations collection; `models.Backfill` runs on every boot and is idempotent by filtering on the absence of what it writes.
|
||||
|
||||
@@ -666,6 +845,8 @@ Windows: MSI built by CI (WiX), or `installer/setup.ps1` registering the agent a
|
||||
| `POD_IP` | no | this pod's own address, set by the Helm chart from the downward API. **Takes precedence over `PROXY_ADVERTISE_HOST`** — a console relay listener belongs to one replica, and a Service address names all of them |
|
||||
| `VANTAGE_MIGRATE_ONLY` | no | run schema setup (migrations, index builders, default-step seeding) and exit without serving. `GRPC_HOST` is not required in this mode. Set by the Helm chart's pre-upgrade Job |
|
||||
| `VANTAGE_SKIP_MIGRATIONS` | no | serve without running schema setup, on the assumption a Job already did. Set by the chart's Deployment whenever `server.migrationJob.enabled`. Unset under Compose, where one process still migrates and then serves |
|
||||
| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for an air-gapped install, or to avoid the anonymous ghcr rate limit |
|
||||
| `VANTAGE_VULNDB_DISABLED` | no | `true` disables the vulnerability database puller and scan loop entirely. Findings already written are still served, and still shown as stale |
|
||||
| `FREE_INSTANCE_REAP_AFTER` | no | duration past a Free licence's expiry before the instance and all its data are deleted. **Empty disables the reaper, and empty is the default.** Set to `336h` in `docker-compose.site.yml` only — a self-hosted deployment must never reap. Must match admin's value, which only names the date in warning emails |
|
||||
|
||||
**sitesvc** (`deploy/docker-compose.site.yml` only):
|
||||
@@ -67,7 +67,8 @@ func (r CatalogueRow) Priced(env string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// SeedCatalogue inserts the sixteen rows the four PAID plans need.
|
||||
// SeedCatalogue inserts the twenty rows the four PAID plans need: a base, a
|
||||
// server limit, and one row per feature key.
|
||||
//
|
||||
// The two Free plans get no rows at all, and that absence is what keeps Free
|
||||
// outside Paddle: with nothing to price, no checkout can be built for it. Do not
|
||||
@@ -84,6 +85,7 @@ func SeedCatalogue(ctx context.Context) error {
|
||||
{Kind: KindLimit, Deployment: deployment, Tier: tier, LimitKey: LimitKeyServers},
|
||||
{Kind: KindFeature, Deployment: deployment, Tier: tier, FeatureKey: license.FeatureConsole},
|
||||
{Kind: KindFeature, Deployment: deployment, Tier: tier, FeatureKey: license.FeatureOIDC},
|
||||
{Kind: KindFeature, Deployment: deployment, Tier: tier, FeatureKey: license.FeatureVulnScanning},
|
||||
}
|
||||
for _, r := range rows {
|
||||
filter := bson.M{
|
||||
|
||||
@@ -12,6 +12,7 @@ import { StatePill } from "@/components/StatePill";
|
||||
import { PageFrame, RailCard, RailFacts } from "@/components/PageFrame";
|
||||
import { PageHeader } from "@/components/PageHeader";
|
||||
import { formatDate, licenceState, limitLabel } from "@/lib/format";
|
||||
import { featureLabel } from "@/lib/features";
|
||||
|
||||
export default function InstancePage() {
|
||||
const id = String(useParams().id);
|
||||
@@ -104,7 +105,11 @@ export default function InstancePage() {
|
||||
},
|
||||
{
|
||||
label: "Features",
|
||||
value: lic.features.join(", ") || "none",
|
||||
// Labelled, not raw keys: this is
|
||||
// the customer's own licence, and
|
||||
// "vuln_scanning" is not a name
|
||||
// anyone bought.
|
||||
value: lic.features.map(featureLabel).join(", ") || "none",
|
||||
},
|
||||
]}
|
||||
/>
|
||||
|
||||
@@ -6,25 +6,16 @@ import Link from "next/link";
|
||||
import { useMutation, useQuery } from "@tanstack/react-query";
|
||||
import { ApiError, api, lineItemsFor, type CatalogueRow, type CheckoutOptions, type Deployment, type Plan, type Term, type Tier } from "@/lib/api";
|
||||
import { initPaddle, previewPrices, type PricePreview } from "@/lib/paddle";
|
||||
import { featureDesc, featureLabel } from "@/lib/features";
|
||||
|
||||
/* Tiers in the order a customer reads them, cheapest first. */
|
||||
const TIER_ORDER: Tier[] = ["free", "professional", "enterprise"];
|
||||
|
||||
const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
|
||||
/* Human labels for feature keys. The catalogue names them by key; this is the
|
||||
* one place the customer-facing wording lives. */
|
||||
const FEATURE_LABEL: Record<string, string> = {
|
||||
console: "Browser console",
|
||||
oidc: "Single sign-on",
|
||||
};
|
||||
const FEATURE_DESC: Record<string, string> = {
|
||||
console: "In-browser SSH, RDP and VNC sessions",
|
||||
oidc: "OIDC sign-in for your whole team",
|
||||
};
|
||||
function featureLabel(key: string) {
|
||||
return FEATURE_LABEL[key] ?? key;
|
||||
}
|
||||
/* Feature wording lives in lib/features.ts, shared with the staff
|
||||
* configurator. It was duplicated here and there, and the two copies had
|
||||
* already drifted. */
|
||||
|
||||
interface Choice {
|
||||
tier: Tier;
|
||||
@@ -294,7 +285,7 @@ export function PurchaseForm() {
|
||||
{featureKeys.map((key) => {
|
||||
const st = featureStateFor(plan, rows, options.env, choice.term, key);
|
||||
return (
|
||||
<Row key={key} title={featureLabel(key)} desc={FEATURE_DESC[key] ?? ""} dim={st === "absent"}>
|
||||
<Row key={key} title={featureLabel(key)} desc={featureDesc(key)} dim={st === "absent"}>
|
||||
{st === "included" ? (
|
||||
<span className="text-[0.72rem] font-semibold uppercase tracking-[0.06em] text-valid">Included</span>
|
||||
) : st === "absent" ? (
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import { useMemo } from "react";
|
||||
import type { CatalogueRow, Deployment, Plan, Term, Tier } from "@/lib/api";
|
||||
import { featureLabel } from "@/lib/features";
|
||||
|
||||
export interface PlanChoice {
|
||||
tier: Tier;
|
||||
@@ -159,7 +160,7 @@ export default function PlanConfigurator({
|
||||
})
|
||||
}
|
||||
/>
|
||||
<span>{key === "console" ? "Browser console" : "Single sign-on"}</span>
|
||||
<span>{featureLabel(key)}</span>
|
||||
<span className="text-[0.72rem] text-ink-3">
|
||||
{priced ? "paid add-on" : "included"}
|
||||
</span>
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
/* Human wording for licence feature keys.
|
||||
*
|
||||
* One place, because there were two and they disagreed: the staff configurator
|
||||
* rendered every key that was not "console" as "Single sign-on", so adding a
|
||||
* third feature silently mislabelled the checkbox that grants it. A map with a
|
||||
* fallback degrades to the raw key, which is ugly but never wrong.
|
||||
*
|
||||
* Keys must match shared/license/license.go. */
|
||||
export const FEATURE_LABEL: Record<string, string> = {
|
||||
console: "Browser console",
|
||||
oidc: "Single sign-on",
|
||||
vuln_scanning: "Vulnerability scanning",
|
||||
};
|
||||
|
||||
export const FEATURE_DESC: Record<string, string> = {
|
||||
console: "In-browser SSH, RDP and VNC sessions",
|
||||
oidc: "OIDC sign-in for your whole team",
|
||||
vuln_scanning: "Package inventory matched against distribution security advisories",
|
||||
};
|
||||
|
||||
export function featureLabel(key: string): string {
|
||||
return FEATURE_LABEL[key] ?? key;
|
||||
}
|
||||
|
||||
export function featureDesc(key: string): string {
|
||||
return FEATURE_DESC[key] ?? "";
|
||||
}
|
||||
@@ -80,7 +80,11 @@ func (c *Client) Register(serverID, preRegToken, hostname, ipAddress, osInfo str
|
||||
return resp.AgentToken, nil
|
||||
}
|
||||
|
||||
func (c *Client) SyncKeys(serverID, agentToken, version string) ([]string, error) {
|
||||
// SyncKeys returns the whole response rather than just the keys: the poll now
|
||||
// also carries CollectPackages, and a second RPC purely to learn one boolean
|
||||
// would be a message every 30 seconds for a value that changes at most when a
|
||||
// licence does.
|
||||
func (c *Client) SyncKeys(serverID, agentToken, version string) (*pb.SyncResponse, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -92,7 +96,34 @@ func (c *Client) SyncKeys(serverID, agentToken, version string) ([]string, error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp.PublicKeys, nil
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// ReportPackages sends a package report and returns whether the server wants
|
||||
// the full list. Given a longer deadline than the other unary calls because the
|
||||
// full body is ~150KB on a slow link.
|
||||
func (c *Client) ReportPackages(req *pb.ReportPackagesRequest) (bool, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
resp, err := c.client.ReportPackages(ctx, req)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return resp.NeedFull, nil
|
||||
}
|
||||
|
||||
// ReportWorkloads sends a workload report and returns whether the server wants
|
||||
// the full list.
|
||||
func (c *Client) ReportWorkloads(req *pb.ReportWorkloadsRequest) (bool, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
resp, err := c.client.ReportWorkloads(ctx, req)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return resp.NeedFull, nil
|
||||
}
|
||||
|
||||
func (c *Client) UploadGeneratedKey(serverID, agentToken, publicKey, privateKey, label string) (string, error) {
|
||||
|
||||
@@ -30,6 +30,45 @@ type SyncRequest struct {
|
||||
|
||||
type SyncResponse struct {
|
||||
PublicKeys []string `json:"public_keys"`
|
||||
// CollectPackages tells the agent whether this instance's licence grants
|
||||
// vulnerability scanning. Absent decodes as false, which is the safe
|
||||
// direction: an older server leaves agents collecting nothing.
|
||||
CollectPackages bool `json:"collect_packages,omitempty"`
|
||||
}
|
||||
|
||||
type OSRelease struct {
|
||||
Family string `json:"family"`
|
||||
// VersionId is not optional: Ubuntu 22.04 and 24.04 publish different fixed
|
||||
// versions for the same CVE, so a scan without it is guesswork.
|
||||
VersionId string `json:"version_id"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Epoch int32 `json:"epoch,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
// SourceName is what the Debian and Ubuntu feeds are keyed on: one advisory
|
||||
// against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
SourceName string `json:"source_name,omitempty"`
|
||||
}
|
||||
|
||||
// ReportPackagesRequest carries a server's installed package set.
|
||||
//
|
||||
// The agent calls twice at most: first with Packages empty, offering only the
|
||||
// hash. If the server already holds it, NeedFull is false and the ~150KB body
|
||||
// is never sent.
|
||||
type ReportPackagesRequest struct {
|
||||
ServerId string `json:"server_id"`
|
||||
AgentToken string `json:"agent_token"`
|
||||
Hash string `json:"hash"`
|
||||
Os OSRelease `json:"os"`
|
||||
Packages []InstalledPackage `json:"packages,omitempty"`
|
||||
}
|
||||
|
||||
type ReportPackagesResponse struct {
|
||||
NeedFull bool `json:"need_full"`
|
||||
}
|
||||
|
||||
type UploadKeyRequest struct {
|
||||
@@ -167,6 +206,10 @@ type ServerCommand struct {
|
||||
CleanupWorkspace *CleanupWorkspaceCmd `json:"cleanup_workspace,omitempty"`
|
||||
OpenProxy *OpenProxyCmd `json:"open_proxy,omitempty"`
|
||||
Ping *PingCmd `json:"ping,omitempty"`
|
||||
|
||||
RefreshWorkloads *RefreshWorkloadsCmd `json:"refresh_workloads,omitempty"`
|
||||
ControlWorkload *ControlWorkloadCmd `json:"control_workload,omitempty"`
|
||||
WorkloadLogs *WorkloadLogsCmd `json:"workload_logs,omitempty"`
|
||||
}
|
||||
|
||||
// PingCmd is a server-originated liveness beat. It carries nothing and expects
|
||||
@@ -204,6 +247,8 @@ type AgentMessage struct {
|
||||
Result *CommandResult `json:"result,omitempty"`
|
||||
StepResult *StepResult `json:"step_result,omitempty"`
|
||||
StepOutput *StepOutputChunk `json:"step_output,omitempty"`
|
||||
|
||||
WorkloadLogsResult *WorkloadLogsResult `json:"workload_logs_result,omitempty"`
|
||||
}
|
||||
|
||||
type AgentReady struct{}
|
||||
@@ -337,6 +382,8 @@ type VantageClient interface {
|
||||
SyncKeys(ctx context.Context, in *SyncRequest, opts ...grpc.CallOption) (*SyncResponse, error)
|
||||
UploadGeneratedKey(ctx context.Context, in *UploadKeyRequest, opts ...grpc.CallOption) (*UploadKeyResponse, error)
|
||||
ReportUpdates(ctx context.Context, in *ReportUpdatesRequest, opts ...grpc.CallOption) (*ReportUpdatesResponse, error)
|
||||
ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error)
|
||||
ReportWorkloads(ctx context.Context, in *ReportWorkloadsRequest, opts ...grpc.CallOption) (*ReportWorkloadsResponse, error)
|
||||
ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error)
|
||||
SyncMonitors(ctx context.Context, in *SyncMonitorsRequest, opts ...grpc.CallOption) (*SyncMonitorsResponse, error)
|
||||
ReportChecks(ctx context.Context, in *ReportChecksRequest, opts ...grpc.CallOption) (*ReportChecksResponse, error)
|
||||
@@ -396,6 +443,14 @@ func (c *keyManagerClient) ReportUpdates(ctx context.Context, in *ReportUpdatesR
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error) {
|
||||
out := new(ReportPackagesResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportPackages", in, out, opts...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error) {
|
||||
out := new(InventoryReportResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportInventory", in, out, opts...); err != nil {
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package pb
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"google.golang.org/grpc"
|
||||
)
|
||||
|
||||
// Workload registry messages. Hand-written like the rest of this package: the
|
||||
// .proto is the contract, this file is the Go side of it, and the two must be
|
||||
// changed together.
|
||||
|
||||
// Workload is one container or one systemd unit.
|
||||
type Workload struct {
|
||||
Kind string `json:"kind"`
|
||||
Id string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
State string `json:"state"`
|
||||
Health string `json:"health,omitempty"`
|
||||
Image string `json:"image,omitempty"`
|
||||
Stack string `json:"stack,omitempty"`
|
||||
Ports []string `json:"ports,omitempty"`
|
||||
Restarts int32 `json:"restarts,omitempty"`
|
||||
StartedAt string `json:"started_at,omitempty"` // RFC3339, empty when not running
|
||||
Protected bool `json:"protected,omitempty"`
|
||||
}
|
||||
|
||||
// ReportWorkloadsRequest carries what a server is running.
|
||||
//
|
||||
// Offer-then-send, the same handshake as ReportPackages: the agent calls once
|
||||
// with Workloads empty, and resends with the body only if NeedFull is set.
|
||||
type ReportWorkloadsRequest struct {
|
||||
ServerId string `json:"server_id"`
|
||||
AgentToken string `json:"agent_token"`
|
||||
Hash string `json:"hash"`
|
||||
DockerOk bool `json:"docker_ok"`
|
||||
DockerError string `json:"docker_error,omitempty"`
|
||||
SystemdOk bool `json:"systemd_ok"`
|
||||
SystemdError string `json:"systemd_error,omitempty"`
|
||||
Workloads []Workload `json:"workloads,omitempty"` // empty on the offer call
|
||||
// Full marks the second call. It is not inferred from an empty Workloads
|
||||
// slice: a host running nothing sends an empty list as its full report.
|
||||
Full bool `json:"full,omitempty"`
|
||||
}
|
||||
|
||||
type ReportWorkloadsResponse struct {
|
||||
NeedFull bool `json:"need_full"`
|
||||
}
|
||||
|
||||
// RefreshWorkloadsCmd carries no payload back. It makes the agent report
|
||||
// immediately through ReportWorkloads, so there is exactly one writer for the
|
||||
// server_workloads collection rather than two arriving by different routes.
|
||||
type RefreshWorkloadsCmd struct{}
|
||||
|
||||
type ControlWorkloadCmd struct {
|
||||
Kind string `json:"kind"`
|
||||
Id string `json:"id"`
|
||||
Action string `json:"action"` // start | stop | restart
|
||||
}
|
||||
|
||||
type WorkloadLogsCmd struct {
|
||||
Kind string `json:"kind"`
|
||||
Id string `json:"id"`
|
||||
Tail int32 `json:"tail,omitempty"`
|
||||
}
|
||||
|
||||
type WorkloadLogsResult struct {
|
||||
CommandId string `json:"command_id"`
|
||||
Text string `json:"text,omitempty"`
|
||||
Truncated bool `json:"truncated,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportWorkloads(ctx context.Context, in *ReportWorkloadsRequest, opts ...grpc.CallOption) (*ReportWorkloadsResponse, error) {
|
||||
out := new(ReportWorkloadsResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportWorkloads", in, out, opts...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package packages
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"runtime"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// OSRelease identifies the distribution well enough to select an advisory
|
||||
// feed. VersionID is not optional: Ubuntu 22.04 and 24.04 publish different
|
||||
// fixed versions for the same CVE.
|
||||
type OSRelease struct {
|
||||
Family string
|
||||
VersionID string
|
||||
Arch string
|
||||
}
|
||||
|
||||
// ParseOSRelease reads the os-release format: KEY=value, one per line, with
|
||||
// values optionally quoted, and # comments.
|
||||
//
|
||||
// The quote stripping handles both ID=ubuntu and ID="rocky", which real
|
||||
// distributions both emit.
|
||||
func ParseOSRelease(r io.Reader) (OSRelease, error) {
|
||||
out := OSRelease{Arch: runtime.GOARCH}
|
||||
sc := bufio.NewScanner(r)
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
key, val, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
val = strings.Trim(strings.TrimSpace(val), `"'`)
|
||||
switch strings.TrimSpace(key) {
|
||||
case "ID":
|
||||
out.Family = strings.ToLower(val)
|
||||
case "VERSION_ID":
|
||||
out.VersionID = val
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return OSRelease{}, err
|
||||
}
|
||||
if out.Family == "" {
|
||||
return OSRelease{}, errors.New("os-release has no ID")
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// DetectOS reads /etc/os-release.
|
||||
func DetectOS() (OSRelease, error) {
|
||||
f, err := os.Open("/etc/os-release")
|
||||
if err != nil {
|
||||
return OSRelease{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
return ParseOSRelease(f)
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package packages
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"time"
|
||||
)
|
||||
|
||||
const collectTimeout = 2 * time.Minute
|
||||
|
||||
// Collect enumerates installed packages. Linux only: Windows agents are
|
||||
// second-class by design, and vulnerability scanning there needs a different
|
||||
// source, a different collector and a different matcher, all out of scope.
|
||||
//
|
||||
// The format strings below are raw string literals on purpose. The "\t" and
|
||||
// "\n" reach dpkg-query and rpm as two characters each, and those tools do the
|
||||
// interpreting themselves — Go must not consume the escapes first.
|
||||
func Collect() (OSRelease, []Package, error) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return OSRelease{}, nil, fmt.Errorf("package collection is linux-only, got %s", runtime.GOOS)
|
||||
}
|
||||
|
||||
osrel, err := DetectOS()
|
||||
if err != nil {
|
||||
return OSRelease{}, nil, fmt.Errorf("detect os: %w", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), collectTimeout)
|
||||
defer cancel()
|
||||
|
||||
switch {
|
||||
case have("dpkg-query"):
|
||||
out, err := run(ctx, "dpkg-query", "-W", "-f",
|
||||
`${Package}\t${Version}\t${Architecture}\t${source:Package}\t${db:Status-Status}\n`)
|
||||
if err != nil {
|
||||
return osrel, nil, err
|
||||
}
|
||||
return osrel, ParseDpkg(out), nil
|
||||
|
||||
case have("rpm"):
|
||||
out, err := run(ctx, "rpm", "-qa", "--qf",
|
||||
`%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n`)
|
||||
if err != nil {
|
||||
return osrel, nil, err
|
||||
}
|
||||
return osrel, ParseRPM(out), nil
|
||||
|
||||
case have("apk"):
|
||||
out, err := run(ctx, "apk", "info", "-v")
|
||||
if err != nil {
|
||||
return osrel, nil, err
|
||||
}
|
||||
return osrel, ParseAPK(out), nil
|
||||
|
||||
default:
|
||||
return osrel, nil, fmt.Errorf("no supported package manager found")
|
||||
}
|
||||
}
|
||||
|
||||
func have(bin string) bool {
|
||||
_, err := exec.LookPath(bin)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func run(ctx context.Context, name string, args ...string) (string, error) {
|
||||
out, err := exec.CommandContext(ctx, name, args...).Output()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
package packages
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Package is one installed package as the distribution reports it. Version is
|
||||
// the distribution's own version string, verbatim — never normalised, because
|
||||
// the advisory feeds are keyed on exactly this form.
|
||||
type Package struct {
|
||||
Name string
|
||||
Version string
|
||||
Epoch int
|
||||
Arch string
|
||||
SourceName string
|
||||
}
|
||||
|
||||
// ParseDpkg reads tab-separated output of
|
||||
// dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${source:Package}\t${db:Status-Status}\n'
|
||||
//
|
||||
// SourceName is why the fourth column is requested at all: Debian and Ubuntu
|
||||
// advisories are keyed on the SOURCE package, so one CVE against "openssl"
|
||||
// covers the binaries libssl3, openssl and libssl-dev. Matching on binary name
|
||||
// alone finds one of the three.
|
||||
//
|
||||
// The fifth column is why "rc" packages do not appear. dpkg-query -W lists
|
||||
// every package dpkg knows about, including ones removed with their config
|
||||
// files left behind — a host that has upgraded its kernel a dozen times reports
|
||||
// a dozen old linux-modules versions that are not on disk, and the oldest of
|
||||
// them sorts first and reads as the installed version. Only "installed" is
|
||||
// installed. An empty status means dpkg did not understand the field, in which
|
||||
// case the line is kept rather than the whole inventory silently vanishing.
|
||||
func ParseDpkg(out string) []Package {
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
continue
|
||||
}
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
if len(f) > 4 {
|
||||
if s := strings.TrimSpace(f[4]); s != "" && s != "installed" {
|
||||
continue
|
||||
}
|
||||
}
|
||||
p := Package{Name: f[0], Version: f[1], Arch: f[2]}
|
||||
if len(f) > 3 && f[3] != "" {
|
||||
p.SourceName = f[3]
|
||||
} else {
|
||||
p.SourceName = p.Name
|
||||
}
|
||||
pkgs = append(pkgs, p)
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// ParseRPM reads tab-separated output of
|
||||
// rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n'
|
||||
func ParseRPM(out string) []Package {
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
continue
|
||||
}
|
||||
f := strings.Split(line, "\t")
|
||||
if len(f) < 4 {
|
||||
continue
|
||||
}
|
||||
epoch := 0
|
||||
// rpm prints "(none)" rather than omitting the field when a package has
|
||||
// no epoch. That must become 0, not fail the line.
|
||||
if f[1] != "" && f[1] != "(none)" {
|
||||
if n, err := strconv.Atoi(f[1]); err == nil {
|
||||
epoch = n
|
||||
}
|
||||
}
|
||||
p := Package{Name: f[0], Epoch: epoch, Version: f[2], Arch: f[3]}
|
||||
if len(f) > 4 {
|
||||
p.SourceName = srcRPMName(f[4])
|
||||
}
|
||||
if p.SourceName == "" {
|
||||
p.SourceName = p.Name
|
||||
}
|
||||
pkgs = append(pkgs, p)
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// srcRPMName reduces "openssl-3.0.7-24.el9.src.rpm" to "openssl" by dropping
|
||||
// the trailing ".src.rpm" and then the version and release segments, which are
|
||||
// the last two hyphen-separated fields.
|
||||
func srcRPMName(s string) string {
|
||||
s = strings.TrimSuffix(s, ".src.rpm")
|
||||
parts := strings.Split(s, "-")
|
||||
if len(parts) <= 2 {
|
||||
return s
|
||||
}
|
||||
return strings.Join(parts[:len(parts)-2], "-")
|
||||
}
|
||||
|
||||
// ParseAPK reads "apk info -v" output: one "name-version-rREV" per line.
|
||||
// Alpine has no separate source package, so SourceName mirrors Name.
|
||||
func ParseAPK(out string) []Package {
|
||||
var pkgs []Package
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
name, version := splitAPK(line)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
pkgs = append(pkgs, Package{Name: name, Version: version, SourceName: name})
|
||||
}
|
||||
return pkgs
|
||||
}
|
||||
|
||||
// splitAPK finds the version boundary from the RIGHT. The version is always the
|
||||
// last two hyphen-separated fields ("<version>-r<rev>"), which is reliable
|
||||
// where scanning from the left is not: package names legitimately contain
|
||||
// digits and underscores, so "musl" in "musl-1.2.4_git20230717-r4" cannot be
|
||||
// found by looking for the first digit.
|
||||
func splitAPK(s string) (name, version string) {
|
||||
last := strings.LastIndex(s, "-")
|
||||
if last <= 0 {
|
||||
return "", ""
|
||||
}
|
||||
prev := strings.LastIndex(s[:last], "-")
|
||||
if prev <= 0 {
|
||||
return "", ""
|
||||
}
|
||||
return s[:prev], s[prev+1:]
|
||||
}
|
||||
|
||||
// Hash fingerprints a package set so an unchanged set never has to be sent.
|
||||
//
|
||||
// It sorts first: the ordering of dpkg or rpm output is not guaranteed stable,
|
||||
// and an ordering-sensitive hash would resend the full ~150KB list every hour
|
||||
// for no reason — a cost visible only as traffic.
|
||||
func Hash(pkgs []Package) string {
|
||||
lines := make([]string, 0, len(pkgs))
|
||||
for _, p := range pkgs {
|
||||
lines = append(lines, p.Name+"\x00"+strconv.Itoa(p.Epoch)+"\x00"+p.Version+"\x00"+p.Arch)
|
||||
}
|
||||
sort.Strings(lines)
|
||||
h := sha256.New()
|
||||
for _, l := range lines {
|
||||
h.Write([]byte(l))
|
||||
h.Write([]byte("\n"))
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"runtime"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/config"
|
||||
grpcclient "gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/grpc"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/grpc/pb"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/packages"
|
||||
)
|
||||
|
||||
// collectPackagesFlag is written by the 30s key poll and read by the hourly
|
||||
// package loop — two different goroutines, hence the atomic.
|
||||
//
|
||||
// It defaults to false, so an agent that has not yet completed a poll, or is
|
||||
// talking to a server too old to send the field, collects nothing. Off is the
|
||||
// safe default: collecting without a licence costs the customer storage they
|
||||
// are not paying for.
|
||||
var collectPackagesFlag atomic.Bool
|
||||
|
||||
// firstPoll closes once a SyncKeys response has set the flag above.
|
||||
//
|
||||
// Without it the boot-time package report loses a race it can only lose: the
|
||||
// hourly loop starts before the first poll, reads a flag that is still false by
|
||||
// construction, and skips — so a freshly installed agent reports no packages for
|
||||
// an hour and the server shows nothing to scan.
|
||||
// How long the boot package report waits for that first poll. Two poll
|
||||
// intervals plus slack: long enough to cover one failed attempt, short enough
|
||||
// that a dead control plane does not hold the OS-update report hostage.
|
||||
const firstPollWait = 90 * time.Second
|
||||
|
||||
var (
|
||||
firstPoll = make(chan struct{})
|
||||
firstPollOnce sync.Once
|
||||
)
|
||||
|
||||
func markFirstPoll() { firstPollOnce.Do(func() { close(firstPoll) }) }
|
||||
|
||||
// waitFirstPoll blocks until the flag is known, or gives up. The wait is
|
||||
// bounded because this loop also reports OS updates, which do not depend on the
|
||||
// flag at all — a control plane that cannot be polled must not silence those too.
|
||||
func waitFirstPoll(ctx context.Context, limit time.Duration) {
|
||||
t := time.NewTimer(limit)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-firstPoll:
|
||||
case <-t.C:
|
||||
log.Printf("package collection: no SyncKeys response within %s, collecting nothing this round", limit)
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
|
||||
func collectPackagesEnabled() bool { return collectPackagesFlag.Load() }
|
||||
|
||||
// reportPackages offers a hash of the installed package set and sends the full
|
||||
// list only if the server does not already hold it.
|
||||
//
|
||||
// It runs on the same hourly cadence as the update check because a package set
|
||||
// changes on roughly the same schedule, and reusing that loop means one timer
|
||||
// rather than two.
|
||||
func reportPackages(client *grpcclient.Client, cfg *config.Config) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return
|
||||
}
|
||||
if !collectPackagesEnabled() {
|
||||
return
|
||||
}
|
||||
|
||||
osrel, pkgs, err := packages.Collect()
|
||||
if err != nil {
|
||||
log.Printf("package collection error: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
pbOS := pb.OSRelease{
|
||||
Family: osrel.Family,
|
||||
VersionId: osrel.VersionID,
|
||||
Arch: osrel.Arch,
|
||||
}
|
||||
hash := packages.Hash(pkgs)
|
||||
|
||||
// The offer: hash only, no body. On an unchanged host this is the whole
|
||||
// exchange, which is the point of the handshake.
|
||||
needFull, err := client.ReportPackages(&pb.ReportPackagesRequest{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
Hash: hash,
|
||||
Os: pbOS,
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("ReportPackages offer error: %v", err)
|
||||
return
|
||||
}
|
||||
if !needFull {
|
||||
return
|
||||
}
|
||||
|
||||
pbPkgs := make([]pb.InstalledPackage, len(pkgs))
|
||||
for i, p := range pkgs {
|
||||
pbPkgs[i] = pb.InstalledPackage{
|
||||
Name: p.Name,
|
||||
Version: p.Version,
|
||||
Epoch: int32(p.Epoch),
|
||||
Arch: p.Arch,
|
||||
SourceName: p.SourceName,
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := client.ReportPackages(&pb.ReportPackagesRequest{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
Hash: hash,
|
||||
Os: pbOS,
|
||||
Packages: pbPkgs,
|
||||
}); err != nil {
|
||||
log.Printf("ReportPackages full error: %v", err)
|
||||
return
|
||||
}
|
||||
log.Printf("reported %d installed packages", len(pkgs))
|
||||
}
|
||||
@@ -70,6 +70,8 @@ func Run(ctx context.Context, cfg *config.Config, version string) error {
|
||||
|
||||
go runInventory(ctx, cfg)
|
||||
|
||||
go runWorkloads(ctx, cfg)
|
||||
|
||||
go monitors.Run(ctx, cfg)
|
||||
|
||||
ticker := time.NewTicker(cfg.PollInterval)
|
||||
@@ -92,11 +94,19 @@ func Run(ctx context.Context, cfg *config.Config, version string) error {
|
||||
}
|
||||
|
||||
func poll(client *grpcclient.Client, cfg *config.Config, version string) error {
|
||||
desired, err := client.SyncKeys(cfg.ServerID, cfg.AgentToken, version)
|
||||
resp, err := client.SyncKeys(cfg.ServerID, cfg.AgentToken, version)
|
||||
if err != nil {
|
||||
return fmt.Errorf("SyncKeys: %w", err)
|
||||
}
|
||||
|
||||
// Stored atomically: the hourly package loop reads this from another
|
||||
// goroutine. Absent on the wire decodes as false, so an older server leaves
|
||||
// collection off rather than on.
|
||||
collectPackagesFlag.Store(resp.CollectPackages)
|
||||
markFirstPoll()
|
||||
|
||||
desired := resp.PublicKeys
|
||||
|
||||
if runtime.GOOS != "linux" {
|
||||
return nil
|
||||
}
|
||||
@@ -339,6 +349,15 @@ func connectAndHandleStream(ctx context.Context, cfg *config.Config) error {
|
||||
if cmd.OpenProxy != nil {
|
||||
go handleOpenProxy(ctx, cfg, cmd.OpenProxy)
|
||||
}
|
||||
if cmd.RefreshWorkloads != nil {
|
||||
go handleRefreshWorkloads(cfg)
|
||||
}
|
||||
if cmd.ControlWorkload != nil {
|
||||
go handleControlWorkload(send, cfg, cmd.CommandId, cmd.ControlWorkload)
|
||||
}
|
||||
if cmd.WorkloadLogs != nil {
|
||||
go handleWorkloadLogs(send, cfg, cmd.CommandId, cmd.WorkloadLogs)
|
||||
}
|
||||
if cmd.RunStep != nil {
|
||||
go func(rc *pb.RunStepCmd, cid string) {
|
||||
emit := func(seq uint64, data []byte) {
|
||||
@@ -395,8 +414,17 @@ func runUpdateCheck(ctx context.Context, cfg *config.Config) {
|
||||
return
|
||||
}
|
||||
log.Printf("reported %d available OS updates", len(pkgs))
|
||||
|
||||
// Same hourly cadence, same connection. A package set changes on
|
||||
// roughly the schedule available updates do, so this needs no timer of
|
||||
// its own.
|
||||
reportPackages(client, cfg)
|
||||
}
|
||||
|
||||
// The boot round only: after this the flag has long been set, and every
|
||||
// later tick is an hour past a poll that runs every 30s.
|
||||
waitFirstPoll(ctx, firstPollWait)
|
||||
|
||||
doCheck()
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/config"
|
||||
grpcclient "gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/grpc"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/grpc/pb"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/agent/internal/workloads"
|
||||
)
|
||||
|
||||
// workloadInterval is the report cadence. Sixty seconds is affordable because
|
||||
// an unchanged list costs one small offer message, not the body.
|
||||
const workloadInterval = 60 * time.Second
|
||||
|
||||
// runWorkloads reports what this host runs, on its own ticker.
|
||||
func runWorkloads(ctx context.Context, cfg *config.Config) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return
|
||||
}
|
||||
|
||||
reportWorkloads(cfg)
|
||||
|
||||
ticker := time.NewTicker(workloadInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
reportWorkloads(cfg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reportWorkloads offers a hash of the current workload set and sends the full
|
||||
// list only if the server does not already hold it.
|
||||
//
|
||||
// This is the ONLY writer of the server_workloads collection. RefreshWorkloadsCmd
|
||||
// calls straight into here rather than answering with data of its own.
|
||||
func reportWorkloads(cfg *config.Config) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
res := workloads.Collect(ctx)
|
||||
hash := workloads.Hash(res.Workloads)
|
||||
|
||||
client, err := grpcclient.New(cfg.ServerURL, cfg.TLS)
|
||||
if err != nil {
|
||||
log.Printf("workload report dial error: %v", err)
|
||||
return
|
||||
}
|
||||
defer client.Close()
|
||||
|
||||
base := func() *pb.ReportWorkloadsRequest {
|
||||
return &pb.ReportWorkloadsRequest{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
Hash: hash,
|
||||
DockerOk: res.DockerOK,
|
||||
DockerError: res.DockerError,
|
||||
SystemdOk: res.SystemdOK,
|
||||
SystemdError: res.SystemdError,
|
||||
}
|
||||
}
|
||||
|
||||
// The offer: hash only, no body. On an unchanged host this is the whole
|
||||
// exchange, which is the point of the handshake.
|
||||
needFull, err := client.ReportWorkloads(base())
|
||||
if err != nil {
|
||||
log.Printf("ReportWorkloads offer error: %v", err)
|
||||
return
|
||||
}
|
||||
if !needFull {
|
||||
return
|
||||
}
|
||||
|
||||
req := base()
|
||||
req.Full = true
|
||||
req.Workloads = make([]pb.Workload, len(res.Workloads))
|
||||
for i, w := range res.Workloads {
|
||||
req.Workloads[i] = pb.Workload{
|
||||
Kind: w.Kind,
|
||||
Id: w.ID,
|
||||
Name: w.Name,
|
||||
State: w.State,
|
||||
Health: w.Health,
|
||||
Image: w.Image,
|
||||
Stack: w.Stack,
|
||||
Ports: w.Ports,
|
||||
Restarts: int32(w.Restarts),
|
||||
Protected: w.Protected,
|
||||
}
|
||||
if !w.StartedAt.IsZero() {
|
||||
req.Workloads[i].StartedAt = w.StartedAt.Format(time.RFC3339)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := client.ReportWorkloads(req); err != nil {
|
||||
log.Printf("ReportWorkloads error: %v", err)
|
||||
return
|
||||
}
|
||||
log.Printf("reported %d workload(s)", len(res.Workloads))
|
||||
}
|
||||
|
||||
// handleRefreshWorkloads makes the agent report immediately. It sends nothing
|
||||
// back beyond the stream ack: the refresh is a nudge, not a channel, so there
|
||||
// is one writer for the collection rather than two.
|
||||
func handleRefreshWorkloads(cfg *config.Config) {
|
||||
reportWorkloads(cfg)
|
||||
}
|
||||
|
||||
// handleControlWorkload starts, stops or restarts a workload and answers with
|
||||
// the ordinary CommandResult.
|
||||
//
|
||||
// The agent's own protected check inside workloads.Control is the boundary; the
|
||||
// Protected flag it reports is only there so the UI can grey the button.
|
||||
func handleControlWorkload(send func(*pb.AgentMessage) error, cfg *config.Config, commandID string, cmd *pb.ControlWorkloadCmd) {
|
||||
err := workloads.Control(context.Background(), cmd.Kind, cmd.Id, cmd.Action)
|
||||
|
||||
res := &pb.CommandResult{CommandId: commandID, Success: err == nil}
|
||||
if err != nil {
|
||||
res.Message = err.Error()
|
||||
log.Printf("workload %s %s failed (cmd=%s): %v", cmd.Action, cmd.Id, commandID, err)
|
||||
} else {
|
||||
res.Message = cmd.Action + " " + cmd.Id + " ok"
|
||||
}
|
||||
|
||||
_ = send(&pb.AgentMessage{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
Result: res,
|
||||
})
|
||||
|
||||
// Report straight away on success so the UI's refetch shows the new state
|
||||
// rather than the old one.
|
||||
if err == nil {
|
||||
reportWorkloads(cfg)
|
||||
}
|
||||
}
|
||||
|
||||
func handleWorkloadLogs(send func(*pb.AgentMessage) error, cfg *config.Config, commandID string, cmd *pb.WorkloadLogsCmd) {
|
||||
text, truncated, err := workloads.Logs(context.Background(), cmd.Kind, cmd.Id, int(cmd.Tail))
|
||||
res := &pb.WorkloadLogsResult{CommandId: commandID, Text: text, Truncated: truncated}
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
}
|
||||
|
||||
_ = send(&pb.AgentMessage{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
WorkloadLogsResult: res,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package workloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrProtected is returned for a workload the agent will not act on.
|
||||
var ErrProtected = errors.New("workload is protected")
|
||||
|
||||
// AgentUnit is the systemd unit this agent runs as.
|
||||
const AgentUnit = "vantage-agent.service"
|
||||
|
||||
// controlTimeout bounds a stop that may never finish on its own. `docker stop`
|
||||
// waits on a container that may ignore SIGTERM, and `systemctl stop` on a unit
|
||||
// with a long TimeoutStopSec blocks for exactly as long as that says. A
|
||||
// timeout must return a real error rather than an ack implying success.
|
||||
const controlTimeout = 90 * time.Second
|
||||
|
||||
// ownContainerID is read once: the container this agent runs in, if any.
|
||||
var ownContainerID = detectOwnContainer()
|
||||
|
||||
var cgroupContainerRe = regexp.MustCompile(`[0-9a-f]{64}`)
|
||||
|
||||
// detectOwnContainer returns this process's container ID, or "" on a host
|
||||
// install. The agent is normally a systemd service, so "" is the common case;
|
||||
// this exists so containerising it later cannot silently remove the guard.
|
||||
func detectOwnContainer() string {
|
||||
b, err := os.ReadFile("/proc/self/cgroup")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
if m := cgroupContainerRe.FindString(string(b)); m != "" {
|
||||
return m
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// isProtected reports whether the agent refuses to act on this workload.
|
||||
//
|
||||
// The refusal lives here, in the agent, and not in the control plane. As with
|
||||
// the console relay hardcoding 127.0.0.1 agent-side: the control plane may name
|
||||
// a target, but the agent decides what it will do to itself. A server-side
|
||||
// denylist alone would be bypassed by the next dispatch path someone adds.
|
||||
func isProtected(kind, id, name string) bool {
|
||||
if kind == "unit" {
|
||||
return id == AgentUnit || name == strings.TrimSuffix(AgentUnit, ".service")
|
||||
}
|
||||
if ownContainerID == "" {
|
||||
return false
|
||||
}
|
||||
// Container IDs are commonly abbreviated to 12 characters; compare on the
|
||||
// shorter of the two so a short id still matches a full one.
|
||||
return strings.HasPrefix(ownContainerID, id) || strings.HasPrefix(id, ownContainerID)
|
||||
}
|
||||
|
||||
// markProtected stamps the flag onto a collected list so the UI can render the
|
||||
// action disabled with a reason.
|
||||
func markProtected(wls []Workload) {
|
||||
for i := range wls {
|
||||
wls[i].Protected = isProtected(wls[i].Kind, wls[i].ID, wls[i].Name)
|
||||
}
|
||||
}
|
||||
|
||||
// Control starts, stops or restarts a workload.
|
||||
func Control(ctx context.Context, kind, id, action string) error {
|
||||
switch action {
|
||||
case "start", "stop", "restart":
|
||||
default:
|
||||
return fmt.Errorf("unknown action %q", action)
|
||||
}
|
||||
|
||||
// Checked before anything else happens, and checked here rather than only
|
||||
// on the server. See isProtected.
|
||||
if isProtected(kind, id, strings.TrimSuffix(id, ".service")) {
|
||||
return fmt.Errorf("%w: %s", ErrProtected, id)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, controlTimeout)
|
||||
defer cancel()
|
||||
|
||||
var cmd *exec.Cmd
|
||||
switch kind {
|
||||
case "container":
|
||||
cmd = exec.CommandContext(ctx, "docker", action, id)
|
||||
case "unit":
|
||||
cmd = exec.CommandContext(ctx, "systemctl", action, id)
|
||||
default:
|
||||
return fmt.Errorf("unknown workload kind %q", kind)
|
||||
}
|
||||
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
return fmt.Errorf("%s %s timed out after %s", action, id, controlTimeout)
|
||||
}
|
||||
return fmt.Errorf("%s %s: %s", action, id, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package workloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Workload is one container or one systemd unit, agent-side. It mirrors
|
||||
// models.Workload on the server.
|
||||
type Workload struct {
|
||||
Kind string
|
||||
ID string
|
||||
Name string
|
||||
State string
|
||||
Health string
|
||||
Image string
|
||||
Stack string
|
||||
Ports []string
|
||||
Restarts int
|
||||
StartedAt time.Time
|
||||
Protected bool
|
||||
}
|
||||
|
||||
const dockerTimeout = 30 * time.Second
|
||||
|
||||
// dockerInspect is the subset of `docker inspect` output we read.
|
||||
//
|
||||
// We use inspect rather than `docker ps --format '{{json .}}'` because ps
|
||||
// reports health and uptime inside a human Status string — "Up 2 hours
|
||||
// (healthy)" — and anything built on that is parsing English that is
|
||||
// localised, reworded between releases, and silently different for a paused or
|
||||
// restarting container. inspect gives typed fields instead.
|
||||
type dockerInspect struct {
|
||||
ID string `json:"Id"`
|
||||
Name string `json:"Name"`
|
||||
State struct {
|
||||
Status string `json:"Status"`
|
||||
StartedAt string `json:"StartedAt"`
|
||||
Restarting bool `json:"Restarting"`
|
||||
Health *struct {
|
||||
Status string `json:"Status"`
|
||||
} `json:"Health"`
|
||||
} `json:"State"`
|
||||
Config struct {
|
||||
Image string `json:"Image"`
|
||||
Labels map[string]string `json:"Labels"`
|
||||
} `json:"Config"`
|
||||
RestartCount int `json:"RestartCount"`
|
||||
NetworkSettings struct {
|
||||
Ports map[string][]struct {
|
||||
HostIP string `json:"HostIp"`
|
||||
HostPort string `json:"HostPort"`
|
||||
} `json:"Ports"`
|
||||
} `json:"NetworkSettings"`
|
||||
}
|
||||
|
||||
// collectDocker enumerates containers. It returns ok=false with an empty error
|
||||
// string when Docker is simply not installed — the common case on this fleet,
|
||||
// and not a fault.
|
||||
func collectDocker(ctx context.Context) ([]Workload, bool, string) {
|
||||
if _, err := exec.LookPath("docker"); err != nil {
|
||||
return nil, false, "" // not installed; not an error
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, dockerTimeout)
|
||||
defer cancel()
|
||||
|
||||
idsOut, err := exec.CommandContext(ctx, "docker", "ps", "-aq").Output()
|
||||
if err != nil {
|
||||
// Installed but not answering: a different problem with a different
|
||||
// fix, so it carries a message where "not installed" does not.
|
||||
return nil, false, "docker ps failed: " + errText(err)
|
||||
}
|
||||
|
||||
ids := strings.Fields(string(idsOut))
|
||||
if len(ids) == 0 {
|
||||
return []Workload{}, true, "" // Docker present, nothing running
|
||||
}
|
||||
|
||||
args := append([]string{"inspect", "--format", "{{json .}}"}, ids...)
|
||||
out, err := exec.CommandContext(ctx, "docker", args...).Output()
|
||||
if err != nil {
|
||||
return nil, false, "docker inspect failed: " + errText(err)
|
||||
}
|
||||
|
||||
var wls []Workload
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
var di dockerInspect
|
||||
if err := json.Unmarshal([]byte(line), &di); err != nil {
|
||||
continue
|
||||
}
|
||||
wls = append(wls, dockerToWorkload(di))
|
||||
}
|
||||
return wls, true, ""
|
||||
}
|
||||
|
||||
func dockerToWorkload(di dockerInspect) Workload {
|
||||
w := Workload{
|
||||
Kind: "container",
|
||||
ID: di.ID,
|
||||
Name: strings.TrimPrefix(di.Name, "/"),
|
||||
State: di.State.Status,
|
||||
Image: di.Config.Image,
|
||||
Restarts: di.RestartCount,
|
||||
}
|
||||
if di.State.Health != nil {
|
||||
w.Health = strings.ToLower(di.State.Health.Status)
|
||||
}
|
||||
// The compose project label is what Docker itself treats as authoritative.
|
||||
// No YAML is read from disk: a compose file there may not be what is running.
|
||||
if v := di.Config.Labels["com.docker.compose.project"]; v != "" {
|
||||
w.Stack = v
|
||||
}
|
||||
if t, err := time.Parse(time.RFC3339Nano, di.State.StartedAt); err == nil {
|
||||
w.StartedAt = t
|
||||
}
|
||||
for container, bindings := range di.NetworkSettings.Ports {
|
||||
for _, b := range bindings {
|
||||
w.Ports = append(w.Ports, b.HostIP+":"+b.HostPort+"->"+container)
|
||||
}
|
||||
}
|
||||
// Map iteration order is random; sort so a stored snapshot does not reorder
|
||||
// its own ports between two otherwise identical reports.
|
||||
sort.Strings(w.Ports)
|
||||
return w
|
||||
}
|
||||
|
||||
func errText(err error) string {
|
||||
if ee, ok := err.(*exec.ExitError); ok && len(ee.Stderr) > 0 {
|
||||
return strings.TrimSpace(string(ee.Stderr))
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package workloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// MaxLogLines and MaxLogBytes are BOTH enforced, whichever binds first.
|
||||
//
|
||||
// A line count alone does not bound size: 500 lines of a container printing
|
||||
// 4KB JSON blobs is 2MB travelling over the bus. This is the same reasoning
|
||||
// that gave workflow logs a per-line cap as well as a per-run one.
|
||||
MaxLogLines = 500
|
||||
MaxLogBytes = 256 * 1024
|
||||
|
||||
logTimeout = 60 * time.Second
|
||||
)
|
||||
|
||||
// Logs returns a bounded snapshot of a workload's recent output.
|
||||
//
|
||||
// There is no follow mode. The browser console already offers a real terminal
|
||||
// on the same server where `docker logs -f` works properly, with its own
|
||||
// scrollback and cancellation. A snapshot answers "why did this restart",
|
||||
// which is the question that sends people to the console in the first place.
|
||||
func Logs(ctx context.Context, kind, id string, tail int) (string, bool, error) {
|
||||
if tail <= 0 || tail > MaxLogLines {
|
||||
tail = MaxLogLines
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, logTimeout)
|
||||
defer cancel()
|
||||
|
||||
var cmd *exec.Cmd
|
||||
switch kind {
|
||||
case "container":
|
||||
cmd = exec.CommandContext(ctx, "docker", "logs",
|
||||
"--tail", strconv.Itoa(tail), "--timestamps", id)
|
||||
case "unit":
|
||||
cmd = exec.CommandContext(ctx, "journalctl", "-u", id,
|
||||
"-n", strconv.Itoa(tail), "--no-pager", "--output=short-iso")
|
||||
default:
|
||||
return "", false, fmt.Errorf("unknown workload kind %q", kind)
|
||||
}
|
||||
|
||||
// docker logs writes container stderr to our stderr, so both streams must
|
||||
// be captured or half the output silently disappears.
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil && len(out) == 0 {
|
||||
return "", false, fmt.Errorf("read logs for %s: %s", id, errText(err))
|
||||
}
|
||||
|
||||
text, truncated := capLog(string(out))
|
||||
return text, truncated, nil
|
||||
}
|
||||
|
||||
// capLog enforces both limits, trimming from the FRONT: the most recent lines
|
||||
// are the ones worth keeping.
|
||||
func capLog(s string) (string, bool) {
|
||||
truncated := false
|
||||
|
||||
lines := strings.Split(s, "\n")
|
||||
if len(lines) > MaxLogLines {
|
||||
lines = lines[len(lines)-MaxLogLines:]
|
||||
truncated = true
|
||||
}
|
||||
s = strings.Join(lines, "\n")
|
||||
|
||||
if len(s) > MaxLogBytes {
|
||||
s = s[len(s)-MaxLogBytes:]
|
||||
// Drop the leading partial line left by a byte-wise cut.
|
||||
if i := strings.IndexByte(s, '\n'); i >= 0 {
|
||||
s = s[i+1:]
|
||||
}
|
||||
truncated = true
|
||||
}
|
||||
|
||||
return s, truncated
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package workloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const systemdTimeout = 30 * time.Second
|
||||
|
||||
// excludedPrefixes drops the platform's own units. A typical host carries 300+
|
||||
// units and systemd accounts for most of them; listing all of them buries the
|
||||
// ten anyone cares about.
|
||||
var excludedPrefixes = []string{"systemd-", "user@", "user-", "session-", "init.scope"}
|
||||
|
||||
// collectSystemd enumerates services in two passes, because "running or
|
||||
// failed" and "enabled but stopped" are different questions — and an enabled
|
||||
// unit that is not running is exactly the one worth seeing.
|
||||
func collectSystemd(ctx context.Context) ([]Workload, bool, string) {
|
||||
if _, err := exec.LookPath("systemctl"); err != nil {
|
||||
return nil, false, ""
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, systemdTimeout)
|
||||
defer cancel()
|
||||
|
||||
// Column output rather than --output=json: the JSON flag needs systemd
|
||||
// 246+, and this fleet includes older stable distributions. The columns
|
||||
// have been stable considerably longer than the JSON has existed.
|
||||
unitsOut, err := exec.CommandContext(ctx, "systemctl",
|
||||
"list-units", "--type=service", "--state=running,failed",
|
||||
"--no-legend", "--plain", "--no-pager").Output()
|
||||
if err != nil {
|
||||
return nil, false, "systemctl list-units failed: " + errText(err)
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
var wls []Workload
|
||||
|
||||
for _, line := range strings.Split(string(unitsOut), "\n") {
|
||||
f := strings.Fields(line)
|
||||
// UNIT LOAD ACTIVE SUB DESCRIPTION…
|
||||
if len(f) < 4 {
|
||||
continue
|
||||
}
|
||||
name := f[0]
|
||||
if excluded(name) || seen[name] {
|
||||
continue
|
||||
}
|
||||
seen[name] = true
|
||||
wls = append(wls, Workload{
|
||||
Kind: "unit",
|
||||
ID: name,
|
||||
Name: strings.TrimSuffix(name, ".service"),
|
||||
State: f[2], // ACTIVE: active | failed | activating | inactive
|
||||
})
|
||||
}
|
||||
|
||||
filesOut, err := exec.CommandContext(ctx, "systemctl",
|
||||
"list-unit-files", "--type=service", "--state=enabled",
|
||||
"--no-legend", "--plain", "--no-pager").Output()
|
||||
if err == nil {
|
||||
for _, line := range strings.Split(string(filesOut), "\n") {
|
||||
f := strings.Fields(line)
|
||||
// UNIT FILE STATE [PRESET]
|
||||
if len(f) < 2 {
|
||||
continue
|
||||
}
|
||||
name := f[0]
|
||||
if excluded(name) || seen[name] {
|
||||
continue
|
||||
}
|
||||
seen[name] = true
|
||||
wls = append(wls, Workload{
|
||||
Kind: "unit",
|
||||
ID: name,
|
||||
Name: strings.TrimSuffix(name, ".service"),
|
||||
State: "inactive", // enabled but not currently running
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return wls, true, ""
|
||||
}
|
||||
|
||||
func excluded(name string) bool {
|
||||
for _, p := range excludedPrefixes {
|
||||
if strings.HasPrefix(name, p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package workloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Result is one collection pass.
|
||||
type Result struct {
|
||||
Workloads []Workload
|
||||
DockerOK bool
|
||||
DockerError string
|
||||
SystemdOK bool
|
||||
SystemdError string
|
||||
}
|
||||
|
||||
// Collect enumerates every workload on this host. Linux only.
|
||||
func Collect(ctx context.Context) Result {
|
||||
if runtime.GOOS != "linux" {
|
||||
return Result{}
|
||||
}
|
||||
|
||||
var r Result
|
||||
containers, dockerOK, dockerErr := collectDocker(ctx)
|
||||
units, systemdOK, systemdErr := collectSystemd(ctx)
|
||||
|
||||
r.DockerOK, r.DockerError = dockerOK, dockerErr
|
||||
r.SystemdOK, r.SystemdError = systemdOK, systemdErr
|
||||
r.Workloads = append(append([]Workload{}, containers...), units...)
|
||||
|
||||
markProtected(r.Workloads)
|
||||
return r
|
||||
}
|
||||
|
||||
// Hash fingerprints a workload set so an unchanged set never has to be sent.
|
||||
//
|
||||
// It sorts first: `docker ps` output ordering is not stable, and an
|
||||
// ordering-sensitive hash would resend the full list every 60 seconds forever
|
||||
// — a cost visible only as traffic.
|
||||
//
|
||||
// StartedAt is deliberately excluded: it does not change while a container
|
||||
// runs, and including it would add nothing. Restarts IS included, because a
|
||||
// container cycling is exactly the change worth reporting.
|
||||
func Hash(wls []Workload) string {
|
||||
lines := make([]string, 0, len(wls))
|
||||
for _, w := range wls {
|
||||
lines = append(lines, strings.Join([]string{
|
||||
w.Kind, w.ID, w.Name, w.State, w.Health, w.Image, w.Stack,
|
||||
strconv.Itoa(w.Restarts),
|
||||
}, "\x00"))
|
||||
}
|
||||
sort.Strings(lines)
|
||||
h := sha256.New()
|
||||
for _, l := range lines {
|
||||
h.Write([]byte(l))
|
||||
h.Write([]byte("\n"))
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Apply Package Updates",
|
||||
"description": "Apply all pending OS package updates. Supports apt, dnf, yum, zypper, apk and pacman.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nif command -v apt-get >/dev/null 2>&1; then\n export DEBIAN_FRONTEND=noninteractive\n apt-get update -qq && apt-get -y -qq upgrade\nelif command -v dnf >/dev/null 2>&1; then\n dnf -y upgrade\nelif command -v yum >/dev/null 2>&1; then\n yum -y update\nelif command -v zypper >/dev/null 2>&1; then\n zypper --non-interactive update\nelif command -v apk >/dev/null 2>&1; then\n apk update && apk upgrade\nelif command -v pacman >/dev/null 2>&1; then\n pacman -Syu --noconfirm\nelse\n echo \"no supported package manager found\"\n exit 1\nfi\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"package update failed\"\n exit 1\nfi\necho \"packages up to date\"\n# Debian and Ubuntu drop this file when a new kernel or libc needs a restart.\n# Reported rather than acted on: rebooting a fleet is a decision, not a detail.\nif [ -f /var/run/reboot-required ]; then\n echo \"REBOOT_REQUIRED=true\" >> $WORKFLOW_ENV\n echo \"a reboot is required to finish applying updates\"\nelse\n echo \"REBOOT_REQUIRED=false\" >> $WORKFLOW_ENV\nfi",
|
||||
"declared_outputs": [
|
||||
"REBOOT_REQUIRED"
|
||||
],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Check Port Is Listening",
|
||||
"description": "Fail unless something is listening on a TCP port.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nhost=\"${host:-127.0.0.1}\"\nif command -v nc >/dev/null 2>&1; then\n nc -z -w 5 \"$host\" \"$port\" >/dev/null 2>&1\n ok=$?\nelse\n # bash builds /dev/tcp in, so this needs nothing installed.\n timeout 5 bash -c \"cat < /dev/null > /dev/tcp/$host/$port\" >/dev/null 2>&1\n ok=$?\nfi\nif [ $ok -ne 0 ]; then\n echo \"PORT_OPEN=false\" >> $WORKFLOW_ENV\n echo \"nothing listening on $host:$port\"\n exit 1\nfi\necho \"PORT_OPEN=true\" >> $WORKFLOW_ENV\necho \"$host:$port is open\"",
|
||||
"declared_outputs": [
|
||||
"PORT_OPEN"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "host",
|
||||
"default": "127.0.0.1",
|
||||
"description": "host to test"
|
||||
},
|
||||
{
|
||||
"name": "port",
|
||||
"default": "",
|
||||
"description": "TCP port to test"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Copy File/Directory",
|
||||
"description": "Copy a file or directory, preserving mode, ownership and timestamps.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nif [ ! -e \"$source\" ]; then\n echo \"source $source does not exist\"\n exit 1\nfi\ncp -a \"$source\" \"$destination\" || { echo \"failed to copy $source to $destination\"; exit 1; }\necho \"copied $source to $destination\"\necho \"DEST_PATH=$destination\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"DEST_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "source",
|
||||
"default": "",
|
||||
"description": "path to copy from"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"default": "",
|
||||
"description": "path to copy to"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Create Directory",
|
||||
"description": "Create a directory, including any missing parents.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nmkdir -p \"$path\" || { echo \"failed to create $path\"; exit 1; }\nif [ -n \"${mode:-}\" ]; then\n chmod \"$mode\" \"$path\" || { echo \"failed to set mode $mode on $path\"; exit 1; }\nfi\necho \"created $path\"\necho \"DIR_PATH=$path\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"DIR_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "directory to create"
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"default": "",
|
||||
"description": "optional octal mode, e.g. 0750"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Delete File/Directory",
|
||||
"description": "Delete a path. Refuses the root filesystem and an empty value.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\n# A step that runs as root on every server in a selector has to refuse the\n# one input that would wipe the fleet. An unset variable expands to empty,\n# so the empty case is the accident this actually guards against.\ncase \"$path\" in\n \"\"|\"/\"|\"/.\"|\"/..\")\n echo \"refusing to delete '$path'\"\n exit 1\n ;;\nesac\nif [ ! -e \"$path\" ]; then\n echo \"$path does not exist, nothing to do\"\n exit 0\nfi\nrm -rf \"$path\" || { echo \"failed to delete $path\"; exit 1; }\necho \"deleted $path\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "path to delete"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Disk Usage Report",
|
||||
"description": "Report usage for a mount point and fail past a threshold.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nmount=\"${mountPoint:-/}\"\nlimit=\"${maxPercent:-90}\"\ndf -h \"$mount\"\nused=$(df --output=pcent \"$mount\" | tail -1 | tr -dc \"0-9\")\navail=$(df -h --output=avail \"$mount\" | tail -1 | tr -d \" \")\necho \"DISK_USED_PERCENT=$used\" >> $WORKFLOW_ENV\necho \"DISK_AVAILABLE=$avail\" >> $WORKFLOW_ENV\nif [ \"$used\" -ge \"$limit\" ]; then\n echo \"$mount is ${used}% full, at or over the ${limit}% limit\"\n exit 1\nfi\necho \"$mount is ${used}% full, ${avail} available\"",
|
||||
"declared_outputs": [
|
||||
"DISK_USED_PERCENT",
|
||||
"DISK_AVAILABLE"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "mountPoint",
|
||||
"default": "/",
|
||||
"description": "mount point to measure"
|
||||
},
|
||||
{
|
||||
"name": "maxPercent",
|
||||
"default": "90",
|
||||
"description": "fail at or above this percentage"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Docker Compose Pull and Up",
|
||||
"description": "Pull the latest images for a compose project and recreate its containers.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\ncd \"$projectDir\" || { echo \"no such directory: $projectDir\"; exit 1; }\nif docker compose version >/dev/null 2>&1; then\n dc=\"docker compose\"\nelif command -v docker-compose >/dev/null 2>&1; then\n dc=\"docker-compose\"\nelse\n echo \"docker compose is not installed\"\n exit 1\nfi\n$dc pull || { echo \"pull failed\"; exit 1; }\n$dc up -d --remove-orphans || { echo \"up failed\"; exit 1; }\n$dc ps",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "projectDir",
|
||||
"default": "",
|
||||
"description": "directory holding docker-compose.yml"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Download File (Linux)",
|
||||
"description": "Download a file over HTTP to a local path",
|
||||
"interpreter": "bash",
|
||||
"script": "out=$(mktemp -p ./)\necho \"Downloading file from $url\"\nwget -q $url -O $out\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to download file from url\"\n exit 1\nfi\necho \"FILE_PATH=$out\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": ["FILE_PATH"],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Enable Linux Service",
|
||||
"description": "Enable a systemd unit so it starts on boot.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\necho \"enabling service $serviceName\"\nsystemctl enable \"$serviceName\" || { echo \"failed to enable $serviceName\"; exit 1; }\necho \"$serviceName enabled\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "systemd unit to enable"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Extract Archive",
|
||||
"description": "Extract a tar, tar.gz, tar.bz2, tar.xz or zip archive into a directory.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\ndest=\"${destination:-.}\"\nif [ ! -f \"$archive\" ]; then\n echo \"archive $archive does not exist\"\n exit 1\nfi\nmkdir -p \"$dest\"\ncase \"$archive\" in\n *.tar.gz|*.tgz) tar -xzf \"$archive\" -C \"$dest\" ;;\n *.tar.bz2|*.tbz2) tar -xjf \"$archive\" -C \"$dest\" ;;\n *.tar.xz|*.txz) tar -xJf \"$archive\" -C \"$dest\" ;;\n *.tar) tar -xf \"$archive\" -C \"$dest\" ;;\n *.zip)\n command -v unzip >/dev/null 2>&1 || { echo \"unzip is not installed\"; exit 1; }\n unzip -oq \"$archive\" -d \"$dest\"\n ;;\n *)\n echo \"unsupported archive type: $archive\"\n exit 1\n ;;\nesac\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to extract $archive\"\n exit 1\nfi\necho \"extracted $archive into $dest\"\necho \"EXTRACT_DIR=$dest\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"EXTRACT_DIR"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "archive",
|
||||
"default": "",
|
||||
"description": "archive file to extract"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"default": ".",
|
||||
"description": "directory to extract into"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,12 +1,10 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Get Host Name",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "HOSTNAME=$(hostname)\necho $HOSTNAME\necho \"HOSTNAME=$HOSTNAME\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"HOSTNAME"
|
||||
],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Get Host Name",
|
||||
"description": "Gets the agents hostname",
|
||||
"interpreter": "bash",
|
||||
"script": "HOSTNAME=$(hostname)\necho $HOSTNAME\necho \"HOSTNAME=$HOSTNAME\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": ["HOSTNAME"],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "HTTP Health Check",
|
||||
"description": "Request a URL and fail unless it answers with the expected status.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nexpected=\"${expectedStatus:-200}\"\nattempts=\"${retries:-3}\"\ndelay=\"${retryDelay:-5}\"\nstatus=\"\"\ni=1\n# Retries live in the script rather than in on_failure: a service coming up\n# after a restart wants a few seconds, not a whole step re-dispatched.\nwhile [ \"$i\" -le \"$attempts\" ]; do\n status=$(curl -s -o /dev/null -w \"%{http_code}\" --max-time 10 \"$url\" || echo \"000\")\n echo \"attempt $i: $url returned $status\"\n if [ \"$status\" = \"$expected\" ]; then\n break\n fi\n i=$(( i + 1 ))\n if [ \"$i\" -le \"$attempts\" ]; then sleep \"$delay\"; fi\ndone\necho \"HTTP_STATUS=$status\" >> $WORKFLOW_ENV\nif [ \"$status\" != \"$expected\" ]; then\n echo \"$url returned $status, expected $expected\"\n exit 1\nfi\necho \"$url is healthy\"",
|
||||
"declared_outputs": [
|
||||
"HTTP_STATUS"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": "URL to request"
|
||||
},
|
||||
{
|
||||
"name": "expectedStatus",
|
||||
"default": "200",
|
||||
"description": "HTTP status that counts as healthy"
|
||||
},
|
||||
{
|
||||
"name": "retries",
|
||||
"default": "3",
|
||||
"description": "how many attempts before failing"
|
||||
},
|
||||
{
|
||||
"name": "retryDelay",
|
||||
"default": "5",
|
||||
"description": "seconds between attempts"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,16 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "List Directory",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "if [ ! -e $path ]; then\n echo \"file or directory doesn't exist: $path\"\n exit 1\nfi\nls -l $path",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "./",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "List Directory",
|
||||
"description": "Lists the files in the specified path",
|
||||
"interpreter": "bash",
|
||||
"script": "if [ ! -e $path ]; then\n echo \"file or directory doesn't exist: $path\"\n exit 1\nfi\nls -l $path",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "./",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Memory Usage Report",
|
||||
"description": "Report memory usage as a percentage of total.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nfree -h\ntotal=$(free -m | awk \"/^Mem:/ {print \\$2}\")\nused=$(free -m | awk \"/^Mem:/ {print \\$3}\")\npct=$(( used * 100 / total ))\necho \"MEM_USED_PERCENT=$pct\" >> $WORKFLOW_ENV\necho \"MEM_USED_MB=$used\" >> $WORKFLOW_ENV\necho \"memory ${pct}% used (${used}MB of ${total}MB)\"",
|
||||
"declared_outputs": [
|
||||
"MEM_USED_PERCENT",
|
||||
"MEM_USED_MB"
|
||||
],
|
||||
"declared_inputs": [],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Reboot Server",
|
||||
"description": "Schedule a reboot a minute out, so the step reports success before the machine goes down.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\ndelay=\"${delayMinutes:-1}\"\n# Scheduled rather than immediate on purpose: `shutdown -r now` kills the\n# agent before it can report, and the run records a failure on a server\n# that did exactly what it was told.\necho \"rebooting in $delay minute(s)\"\nshutdown -r \"+$delay\" \"Reboot requested by Vantage\" || { echo \"failed to schedule a reboot\"; exit 1; }",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "delayMinutes",
|
||||
"default": "1",
|
||||
"description": "minutes to wait before rebooting"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Restart Linux Service",
|
||||
"description": "Restart a systemd unit and fail if it does not come back up.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\necho \"restarting service $serviceName\"\nsystemctl restart \"$serviceName\" || { echo \"failed to restart $serviceName\"; exit 1; }\nsystemctl is-active --quiet \"$serviceName\" || {\n echo \"$serviceName did not come back up\"\n systemctl status \"$serviceName\" --no-pager --lines=20 || true\n exit 1\n}\necho \"$serviceName is active\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "systemd unit to restart, e.g. nginx"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Linux Service Status",
|
||||
"description": "Report whether a systemd unit is active and enabled. Does not fail on a stopped unit.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nstate=$(systemctl is-active \"$serviceName\" 2>/dev/null || true)\nenabled=$(systemctl is-enabled \"$serviceName\" 2>/dev/null || true)\necho \"$serviceName: state=$state enabled=$enabled\"\necho \"SERVICE_STATE=$state\" >> $WORKFLOW_ENV\necho \"SERVICE_ENABLED=$enabled\" >> $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"SERVICE_STATE",
|
||||
"SERVICE_ENABLED"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "systemd unit to inspect"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Set Permissions and Ownership",
|
||||
"description": "Set the mode and optionally the owner of a path.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nif [ ! -e \"$path\" ]; then\n echo \"$path does not exist\"\n exit 1\nfi\nrecurse=\"\"\nif [ \"${recursive:-false}\" = \"true\" ]; then\n recurse=\"-R\"\nfi\nif [ -n \"${mode:-}\" ]; then\n chmod $recurse \"$mode\" \"$path\" || { echo \"failed to set mode\"; exit 1; }\n echo \"set mode $mode on $path\"\nfi\nif [ -n \"${owner:-}\" ]; then\n chown $recurse \"$owner\" \"$path\" || { echo \"failed to set owner\"; exit 1; }\n echo \"set owner $owner on $path\"\nfi",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "path to change"
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"default": "",
|
||||
"description": "octal mode, e.g. 0640"
|
||||
},
|
||||
{
|
||||
"name": "owner",
|
||||
"default": "",
|
||||
"description": "owner, e.g. root:root"
|
||||
},
|
||||
{
|
||||
"name": "recursive",
|
||||
"default": "false",
|
||||
"description": "true to apply recursively"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Start Linux Service",
|
||||
"description": "",
|
||||
"description": "Start a linux systemd service",
|
||||
"interpreter": "bash",
|
||||
"script": "echo \"starting service $serviceName\"\nsystemctl start $serviceName\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to start service\"\n exit 1\nfi",
|
||||
"declared_outputs": [],
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Stop Linux Service",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "echo \"stopping service $serviceName\"\nsystemctl stop $serviceName\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to stop service\"\n exit 1\nfi",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Stop Linux Service",
|
||||
"description": "Stops a linux systemd service",
|
||||
"interpreter": "bash",
|
||||
"script": "echo \"stopping service $serviceName\"\nsystemctl stop $serviceName\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to stop service\"\n exit 1\nfi",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Tail Log File",
|
||||
"description": "Print the last N lines of a file, for reading a log after a deployment step.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nlines=\"${lines:-50}\"\nif [ ! -f \"$path\" ]; then\n echo \"$path does not exist\"\n exit 1\nfi\necho \"last $lines lines of $path:\"\ntail -n \"$lines\" \"$path\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "path",
|
||||
"default": "",
|
||||
"description": "log file to read"
|
||||
},
|
||||
{
|
||||
"name": "lines",
|
||||
"default": "50",
|
||||
"description": "how many lines to print"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "TLS Certificate Expiry",
|
||||
"description": "Report days remaining on a TLS certificate and fail under a threshold.",
|
||||
"interpreter": "bash",
|
||||
"script": "set -u\nport=\"${port:-443}\"\nmin=\"${minDays:-14}\"\ncommand -v openssl >/dev/null 2>&1 || { echo \"openssl is not installed\"; exit 1; }\n# -servername sends SNI, without which a shared host returns the wrong\n# certificate and the expiry reported here belongs to someone else.\nend=$(echo | openssl s_client -servername \"$host\" -connect \"$host:$port\" 2>/dev/null \\\n | openssl x509 -noout -enddate | cut -d= -f2)\nif [ -z \"$end\" ]; then\n echo \"could not read a certificate from $host:$port\"\n exit 1\nfi\nendEpoch=$(date -d \"$end\" +%s)\nnowEpoch=$(date +%s)\ndays=$(( (endEpoch - nowEpoch) / 86400 ))\necho \"CERT_DAYS_REMAINING=$days\" >> $WORKFLOW_ENV\necho \"CERT_EXPIRES=$end\" >> $WORKFLOW_ENV\necho \"$host:$port expires in $days days ($end)\"\nif [ \"$days\" -lt \"$min\" ]; then\n echo \"fewer than $min days remaining\"\n exit 1\nfi",
|
||||
"declared_outputs": [
|
||||
"CERT_DAYS_REMAINING",
|
||||
"CERT_EXPIRES"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "host",
|
||||
"default": "",
|
||||
"description": "hostname to check"
|
||||
},
|
||||
{
|
||||
"name": "port",
|
||||
"default": "443",
|
||||
"description": "TLS port"
|
||||
},
|
||||
{
|
||||
"name": "minDays",
|
||||
"default": "14",
|
||||
"description": "fail below this many days remaining"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -1,18 +0,0 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "WGET",
|
||||
"description": "",
|
||||
"interpreter": "bash",
|
||||
"script": "out=$(mktemp -p ./)\necho \"Downloading file from $url\"\nwget -q $url -O $out\nretVal=$?\nif [ $retVal -ne 0 ]; then\n echo \"failed to download file from url\"\n exit 1\nfi\necho \"FILE_PATH=$out\" \u003e\u003e $WORKFLOW_ENV",
|
||||
"declared_outputs": [
|
||||
"FILE_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Download File (Windows)",
|
||||
"description": "Download a file over HTTP to a local path.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$url = $env:url\n$dest = if ($env:destination) { $env:destination } else { Join-Path $env:TEMP ([System.IO.Path]::GetFileName($url)) }\nWrite-Output \"downloading $url\"\ntry {\n # -UseBasicParsing keeps this working on Server Core, where the IE\n # engine Invoke-WebRequest otherwise reaches for is not installed.\n Invoke-WebRequest -Uri $url -OutFile $dest -UseBasicParsing\n} catch {\n Write-Output \"failed to download: $_\"\n exit 1\n}\nWrite-Output \"saved to $dest\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"FILE_PATH=$dest\"",
|
||||
"declared_outputs": [
|
||||
"FILE_PATH"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "url",
|
||||
"default": "",
|
||||
"description": "URL to download"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"default": "",
|
||||
"description": "where to save it; defaults to a file in TEMP"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Reboot Windows Server",
|
||||
"description": "Schedule a reboot a minute out, so the step reports success before the machine goes down.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$delay = if ($env:delaySeconds) { [int]$env:delaySeconds } else { 60 }\nWrite-Output \"rebooting in $delay second(s)\"\n& shutdown.exe /r /t $delay /c \"Reboot requested by Vantage\"\nif ($LASTEXITCODE -ne 0) {\n Write-Output \"failed to schedule a reboot\"\n exit 1\n}",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "delaySeconds",
|
||||
"default": "60",
|
||||
"description": "seconds to wait before rebooting"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Restart Windows Service",
|
||||
"description": "Restart a Windows service and fail if it does not come back up.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$name = $env:serviceName\nWrite-Output \"restarting service $name\"\ntry {\n Restart-Service -Name $name -Force\n} catch {\n Write-Output \"failed to restart ${name}: $_\"\n exit 1\n}\n$svc = Get-Service -Name $name\nif ($svc.Status -ne \"Running\") {\n Write-Output \"$name is $($svc.Status), not Running\"\n exit 1\n}\nWrite-Output \"$name is running\"",
|
||||
"declared_outputs": [],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "Windows service name to restart"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Windows Disk Report",
|
||||
"description": "Report free space on a drive and fail past a usage threshold.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$letter = if ($env:driveLetter) { $env:driveLetter } else { \"C\" }\n$limit = if ($env:maxPercent) { [int]$env:maxPercent } else { 90 }\n$d = Get-PSDrive -Name $letter -ErrorAction SilentlyContinue\nif ($null -eq $d) {\n Write-Output \"drive $letter not found\"\n exit 1\n}\n$total = $d.Used + $d.Free\n$pct = [math]::Round(($d.Used / $total) * 100)\n$freeGb = [math]::Round($d.Free / 1GB, 1)\nWrite-Output \"${letter}: is $pct% full, $freeGb GB free\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"DISK_USED_PERCENT=$pct\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"DISK_FREE_GB=$freeGb\"\nif ($pct -ge $limit) {\n Write-Output \"at or over the $limit% limit\"\n exit 1\n}",
|
||||
"declared_outputs": [
|
||||
"DISK_USED_PERCENT",
|
||||
"DISK_FREE_GB"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "driveLetter",
|
||||
"default": "C",
|
||||
"description": "drive letter, without a colon"
|
||||
},
|
||||
{
|
||||
"name": "maxPercent",
|
||||
"default": "90",
|
||||
"description": "fail at or above this percentage"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "Windows Service Status",
|
||||
"description": "Report a Windows service's status and start type. Does not fail on a stopped service.",
|
||||
"interpreter": "powershell",
|
||||
"script": "$ErrorActionPreference = \"Stop\"\n$name = $env:serviceName\n$svc = Get-Service -Name $name -ErrorAction SilentlyContinue\nif ($null -eq $svc) {\n Write-Output \"$name is not installed\"\n Add-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_STATE=missing\"\n Add-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_START_TYPE=none\"\n exit 0\n}\nWrite-Output \"${name}: $($svc.Status), start type $($svc.StartType)\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_STATE=$($svc.Status)\"\nAdd-Content -Path $env:WORKFLOW_ENV -Value \"SERVICE_START_TYPE=$($svc.StartType)\"",
|
||||
"declared_outputs": [
|
||||
"SERVICE_STATE",
|
||||
"SERVICE_START_TYPE"
|
||||
],
|
||||
"declared_inputs": [
|
||||
{
|
||||
"name": "serviceName",
|
||||
"default": "",
|
||||
"description": "Windows service name to inspect"
|
||||
}
|
||||
],
|
||||
"secret_refs": []
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,538 @@
|
||||
# Package inventory and CVE findings
|
||||
|
||||
Date: 2026-08-06
|
||||
|
||||
Agents report the packages installed on each server. The control plane matches
|
||||
them against distro security feeds and raises findings that link straight to
|
||||
the patching path that already exists. A finding nobody can fix today can be
|
||||
accepted with a reason and an expiry date rather than sitting red forever.
|
||||
|
||||
This is one of four sub-projects sketched together and deliberately separated:
|
||||
|
||||
| # | Sub-project | Depends on |
|
||||
| - | ----------- | ---------- |
|
||||
| A | **Package inventory + CVE findings** — this spec | nothing |
|
||||
| B | Container/service registry | nothing |
|
||||
| C | Container image scanning | A and B |
|
||||
| D | Compliance profiles (baseline assertions) | shares A's findings UI only |
|
||||
|
||||
A and B are independent of one another. C is the joiner and must not be
|
||||
designed before both exist. D shares a page with A and nothing else — a
|
||||
different collector, a different evaluation model and a different remediation
|
||||
story — so folding it in here would double the size for no shared machinery.
|
||||
|
||||
Scope of this spec is **A, Linux only.** Windows needs a separate source
|
||||
(MSRC CVRF), a separate collector (`Get-HotFix` plus registry) and a KB
|
||||
supersedence matcher that shares no code with the Linux path. That matches the
|
||||
existing position that Windows agents are second-class by design, and the six
|
||||
package managers `updates.go` already detects cover the whole Linux surface.
|
||||
|
||||
---
|
||||
|
||||
## The trap this design is built around
|
||||
|
||||
Distributions **backport** security fixes without changing the upstream
|
||||
version. Ubuntu ships `openssl 3.0.2-0ubuntu1.15` patched against
|
||||
CVE-2023-0286; NVD says version 3.0.2 is vulnerable. Matching installed
|
||||
versions against NVD or CPE ranges therefore reports a fleet full of criticals
|
||||
that are all already fixed.
|
||||
|
||||
That is not merely noisy. It is fatal to the feature: once the first report is
|
||||
mostly wrong, nobody reads the second one, and a genuine finding is lost in the
|
||||
noise it created. Everything below follows from refusing to make that mistake.
|
||||
|
||||
The correct source is the **distribution's own security feed**, keyed on the
|
||||
distribution's own version string — Debian and Ubuntu OVAL/USN, Red Hat OVAL
|
||||
v2, Alpine secdb. `trivy-db` is those feeds pre-merged into one BoltDB
|
||||
artifact, rebuilt every six hours and published as an OCI artifact.
|
||||
|
||||
---
|
||||
|
||||
## Where the vulnerability data comes from
|
||||
|
||||
`trivy-db`, pulled server-side from `ghcr.io/aquasecurity/trivy-db:2`.
|
||||
|
||||
The alternative considered was querying OSV.dev per scan, which needs no
|
||||
storage and no puller. It was rejected on two counts: it requires outbound
|
||||
internet on every scan, which breaks air-gapped installs; and it sends the
|
||||
package list of a customer's entire fleet to a third party. The audience most
|
||||
likely to buy vulnerability scanning is the audience least willing to do that.
|
||||
|
||||
The blob is roughly 50MB, read-only, reproducible, and identified by a version
|
||||
number. **It is not stored in Mongo and not written to `/data`** —
|
||||
`server.persistence` defaults to off and nothing writes to `/data` any more.
|
||||
It does not need durable storage: whichever pod needs it pulls it to its own
|
||||
ephemeral temp directory. Nothing shared, nothing to back up, nothing to
|
||||
migrate.
|
||||
|
||||
`VANTAGE_TRIVY_DB_REF` overrides the default reference so a customer can mirror
|
||||
the artifact into their own registry. It also covers the anonymous ghcr rate
|
||||
limit, which the six-hourly pull cadence already makes unlikely to bite.
|
||||
|
||||
---
|
||||
|
||||
## Only the leader matches
|
||||
|
||||
This is the crux, and it falls out of the replica model already in the
|
||||
codebase.
|
||||
|
||||
Two things trigger matching, and they happen on different pods:
|
||||
|
||||
1. a fleet-wide rescan when `trivy-db` updates — naturally the leader's job
|
||||
2. a server's package list changing — handled by whichever pod holds *that
|
||||
agent's* command stream
|
||||
|
||||
If (2) matched inline, **every replica would need the 50MB database resident**,
|
||||
and a database refresh would have N pods racing to rescan the same fleet and N
|
||||
digests reaching the customer. That is the exact failure `RunAsLeader` exists
|
||||
to prevent, and it is the same argument that put `monitorsched` behind the
|
||||
lock.
|
||||
|
||||
So `ReportPackages` does not match. It upserts the package list and sets
|
||||
`scan_pending: true`. That is all it does.
|
||||
|
||||
`server/internal/vulnsched` then runs inside the **existing**
|
||||
`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`,
|
||||
`workflowsched` and the sweepers — one role, one lock. Every 60 seconds it:
|
||||
|
||||
1. pulls `trivy-db` if the local copy is older than six hours
|
||||
2. if the pulled version differs from `vulndb_meta.db_version`, marks **every**
|
||||
server `scan_pending`
|
||||
3. matches all `scan_pending` servers, clears the flag, diffs against existing
|
||||
findings
|
||||
4. emits **one** digest per tick covering everything newly opened
|
||||
|
||||
Step 4 is why batching is structural rather than bolted on. A `trivy-db`
|
||||
refresh can open several hundred findings across a fleet at once; one message
|
||||
per finding would rate-limit the webhook or get the channel muted, and either
|
||||
way the customer stops receiving the alerts they are paying for. The tick is
|
||||
already the natural batch boundary, so **the failure cannot occur by
|
||||
construction** rather than by a debounce someone has to maintain.
|
||||
|
||||
`scan_pending` lives on the document rather than in memory, for the same reason
|
||||
`next_run_at` and `workflow_log_seq` do: a leader handover between marking and
|
||||
scanning would otherwise lose it. A handover costs the new leader one re-pull
|
||||
of the database.
|
||||
|
||||
The cost of this indirection is up to 60 seconds between an agent reporting a
|
||||
changed package set and its findings updating. For vulnerability data that is
|
||||
nothing, and it buys a single matching path instead of two.
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
```
|
||||
agent/internal/packages/ collect installed packages + /etc/os-release
|
||||
proto/ ReportPackages RPC
|
||||
server/internal/vulndb/ puller, BoltDB access, matcher
|
||||
server/internal/vulnsched/ leader-owned tick: pull, scan, digest
|
||||
server/internal/services/ findings, acceptance, alert rules
|
||||
web/app/(app)/vulnerabilities/ fleet board; plus two server-detail tabs
|
||||
```
|
||||
|
||||
`vulnsched` takes the dependencies it needs — `LogEvent` and the notification
|
||||
dispatch — as a `vulnsched.Deps` injected from `main.go`, following
|
||||
`workflowsched`. The manual rescan endpoint does not call into `vulnsched` at
|
||||
all: it sets `scan_pending` on every server and lets the next tick find them,
|
||||
so there is no path by which `services` imports the scheduler and no cycle to
|
||||
avoid later.
|
||||
|
||||
---
|
||||
|
||||
## The wire path
|
||||
|
||||
A new `ReportPackages` RPC on the agent's existing hourly loop — the same
|
||||
`runUpdateCheck` cadence, reusing `updates.go`'s `detectPM()`.
|
||||
|
||||
```protobuf
|
||||
rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse);
|
||||
|
||||
message ReportPackagesRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3; // sha256 of the sorted list
|
||||
OSRelease os = 4;
|
||||
repeated InstalledPackage packages = 5; // omitted when only offering a hash
|
||||
}
|
||||
|
||||
message ReportPackagesResponse {
|
||||
bool need_full = 1; // hash differs; resend with packages populated
|
||||
}
|
||||
```
|
||||
|
||||
The agent calls once with `packages` empty. `need_full` true means the hash
|
||||
differs from what the server holds, and the agent immediately calls again with
|
||||
the list populated.
|
||||
|
||||
The agent sends a SHA-256 of its sorted package list first. If it matches what
|
||||
the server already holds, the server answers `unchanged` and the ~150KB body is
|
||||
never sent. A machine's package set changes rarely, so almost every hour costs
|
||||
one small message, and the rare changed hour costs one extra round trip.
|
||||
|
||||
Folding the list into the existing 15-minute `InventoryReport` static snapshot
|
||||
was rejected: it would re-send ~150KB per server every 15 minutes regardless of
|
||||
change, roughly 40MB/hour of gRPC traffic on a 100-server fleet to transmit
|
||||
data that is almost always identical.
|
||||
|
||||
---
|
||||
|
||||
## Data model
|
||||
|
||||
Four new collections. Every one carries `instance_id` except `vulndb_meta`,
|
||||
which is explained below.
|
||||
|
||||
### `server_packages` — one document per server, not per package
|
||||
|
||||
```go
|
||||
type ServerPackages struct {
|
||||
ID primitive.ObjectID `bson:"_id"`
|
||||
InstanceID primitive.ObjectID `bson:"instance_id"`
|
||||
ServerID string `bson:"server_id"`
|
||||
OS OSRelease `bson:"os"` // family, version_id, arch
|
||||
Hash string `bson:"hash"` // sha256 of the sorted list
|
||||
Packages []InstalledPackage `bson:"packages"`
|
||||
CollectedAt time.Time `bson:"collected_at"`
|
||||
ScanPending bool `bson:"scan_pending"`
|
||||
ScannedAt time.Time `bson:"scanned_at"`
|
||||
Status string `bson:"status"` // ok | unsupported
|
||||
DBVersion int `bson:"db_version"` // last matched against
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `bson:"name"`
|
||||
Version string `bson:"version"` // distro version string, verbatim
|
||||
Epoch int `bson:"epoch,omitempty"`
|
||||
Arch string `bson:"arch"`
|
||||
SourceName string `bson:"source_name,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
One document rather than two thousand is what makes a report a **single atomic
|
||||
upsert with no delta logic** — the hash already established that something
|
||||
changed, so there is nothing to reconcile field by field. A typical Linux host
|
||||
lands near 150KB, comfortably inside the 16MB document limit.
|
||||
|
||||
Indexes: `{instance_id, server_id}` unique, and a multikey
|
||||
`{instance_id, "packages.name"}` for fleet-wide package search.
|
||||
|
||||
`SourceName` is not decoration. **Debian and Ubuntu advisories are keyed on the
|
||||
source package**: a CVE against `openssl` covers the binaries `libssl3`,
|
||||
`openssl` and `libssl-dev`, so matching on binary name alone misses two of the
|
||||
three.
|
||||
|
||||
`OS.VersionID` selects the feed. Ubuntu 22.04 and 24.04 publish different fixed
|
||||
versions for the same CVE, so a scan without it is guesswork.
|
||||
|
||||
### `vuln_findings` — one document per (server, CVE, package)
|
||||
|
||||
```go
|
||||
type VulnFinding struct {
|
||||
ID primitive.ObjectID `bson:"_id"`
|
||||
InstanceID primitive.ObjectID `bson:"instance_id"`
|
||||
ServerID string `bson:"server_id"`
|
||||
|
||||
CVEID string `bson:"cve_id"`
|
||||
PackageName string `bson:"package_name"`
|
||||
Installed string `bson:"installed_version"`
|
||||
FixedIn string `bson:"fixed_in,omitempty"`
|
||||
Severity string `bson:"severity"`
|
||||
CVSSScore float64 `bson:"cvss_score,omitempty"`
|
||||
Title string `bson:"title,omitempty"`
|
||||
References []string `bson:"references,omitempty"`
|
||||
|
||||
State string `bson:"state"` // open | fixed | accepted
|
||||
FirstSeen time.Time `bson:"first_seen"`
|
||||
LastSeen time.Time `bson:"last_seen"`
|
||||
FixedAt *time.Time `bson:"fixed_at,omitempty"`
|
||||
Accepted *Acceptance `bson:"accepted,omitempty"`
|
||||
}
|
||||
|
||||
type Acceptance struct {
|
||||
By primitive.ObjectID `bson:"by"`
|
||||
Reason string `bson:"reason"`
|
||||
Until time.Time `bson:"until"`
|
||||
At time.Time `bson:"at"`
|
||||
}
|
||||
```
|
||||
|
||||
Unique on `{instance_id, server_id, cve_id, package_name}`. That key is what
|
||||
makes a rescan an idempotent upsert rather than a duplicate factory, and it is
|
||||
what lets `first_seen` survive across scans. Query index
|
||||
`{instance_id, state, severity}`.
|
||||
|
||||
**An empty `FixedIn` is a real and common state** and must never be conflated
|
||||
with "not vulnerable". A CVE with no vendor fix published yet is exactly the
|
||||
finding people most need to see, and also the one that most needs acceptance,
|
||||
because there is nothing to patch.
|
||||
|
||||
Findings are **not deleted when a package is patched**. State moves to `fixed`
|
||||
with `fixed_at` set, so "what did we remediate last quarter" remains
|
||||
answerable — which is the question an auditor asks.
|
||||
|
||||
### `vulndb_meta` — singleton, deliberately unscoped
|
||||
|
||||
`db_version`, `pulled_at`, `last_full_scan_at`, `last_error`. It carries no
|
||||
`instance_id` because the vulnerability database is a property of the
|
||||
deployment, not of a tenant. Same reasoning as `migrations`.
|
||||
|
||||
### `vuln_alert_rules`
|
||||
|
||||
`instance_id`, `name`, `enabled`, `min_severity`, `tags map[string]string`,
|
||||
`channel_ids []`, timestamps.
|
||||
|
||||
The tag filter resolves through **`services.ResolveTargets`**, not a second
|
||||
matcher. That function is already the single answer to which servers a
|
||||
selector touches, and an alert rule that disagreed with a workflow about what
|
||||
`env:prod` means would be worse than having no filter at all.
|
||||
|
||||
---
|
||||
|
||||
## The matching engine
|
||||
|
||||
```
|
||||
server/internal/vulndb/
|
||||
pull.go OCI fetch → temp dir, version compare against vulndb_meta
|
||||
db.go BoltDB open, advisory lookup by (ecosystem, source, version)
|
||||
match.go per-family matching, severity resolution
|
||||
version.go dispatch to deb/rpm/apk comparator by OS family
|
||||
```
|
||||
|
||||
Dependencies: `github.com/aquasecurity/trivy-db` for the BoltDB schema, plus
|
||||
`go-deb-version`, `go-rpm-version` and `go-apk-version` — each a small
|
||||
standalone module doing one job. The roughly 200 lines of per-distro advisory
|
||||
lookup are ours.
|
||||
|
||||
Importing `trivy` itself was rejected: it would pull a very large transitive
|
||||
dependency tree into the server binary for one feature, and its Go API carries
|
||||
no stability guarantee across minor versions. Shelling out to the `trivy`
|
||||
binary against a generated SBOM was rejected for shipping a second binary in
|
||||
the image and turning a library call into subprocess lifecycle, timeouts and
|
||||
output-format drift.
|
||||
|
||||
### Why the comparators are bought rather than written
|
||||
|
||||
Version ordering is where this feature lives or dies, and its failure mode is
|
||||
silent. `dpkg` ordering has epochs, and `~` sorts *before* the empty string, so
|
||||
`3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own
|
||||
segment rules and treats `~` and `^` differently again. A `strings.Compare` or
|
||||
a semver parse orders `1.9` above `1.10` and reports a vulnerable fleet as
|
||||
clean — a false negative, which nobody notices until it matters.
|
||||
|
||||
### Scanning one server
|
||||
|
||||
1. Load `server_packages`; resolve OS family and version to a `trivy-db`
|
||||
ecosystem.
|
||||
2. **Unsupported ecosystem → record `status: unsupported`, clear the flag,
|
||||
write no findings.**
|
||||
3. For each package: resolve source name, look up advisories, compare versions.
|
||||
4. Upsert vulnerable results as `open`, preserving `first_seen`.
|
||||
5. Any currently-`open` finding absent from this result set → `fixed`, stamp
|
||||
`fixed_at`.
|
||||
6. Any `accepted` finding past its `until` → back to `open`.
|
||||
7. Clear `scan_pending`, stamp `scanned_at` and `db_version`.
|
||||
|
||||
Steps 5 and 6 must run in that order, so a finding that is both absent and
|
||||
expired settles as `fixed` rather than reopening on a package that no longer
|
||||
carries it.
|
||||
|
||||
Step 2 matters as much as any of the matching. Arch has no feed in `trivy-db`,
|
||||
so an Arch host must report **unsupported**, never "0 findings". Reporting
|
||||
clean when the truth is unknown is the same class of lie as a silently stale
|
||||
database, and it is the reason `vulndb_meta.pulled_at` appears on screen rather
|
||||
than only in a log.
|
||||
|
||||
### Severity
|
||||
|
||||
Resolved **vendor → NVD → unknown**, in that order, never invented.
|
||||
|
||||
This will surface as "why is this critical CVE marked low", and the answer is
|
||||
that Debian and Red Hat routinely downgrade an NVD score because the vulnerable
|
||||
code path is not reachable in their build. Their rating is the accurate one for
|
||||
that package, and showing NVD's above it would manufacture work that does not
|
||||
need doing.
|
||||
|
||||
---
|
||||
|
||||
## Findings lifecycle
|
||||
|
||||
`open | fixed | accepted`.
|
||||
|
||||
An accepted finding is suppressed from counts and alerts until its `until`
|
||||
date, then reopens automatically. A reason is required.
|
||||
|
||||
Acceptance with a mandatory expiry, rather than permanent dismissal, is what
|
||||
keeps the feature usable in both directions. Without any acceptance mechanism,
|
||||
a kernel CVE awaiting a reboot window sits red indefinitely and trains people
|
||||
to ignore the page. With permanent dismissal, accepted findings accumulate
|
||||
silently and nobody revisits them — the dismissal list becomes where risk goes
|
||||
to be forgotten, which is precisely what an auditor asks to see.
|
||||
|
||||
Retention: `settings.vuln_finding_retention_days`, a `*int` on the same pattern
|
||||
as `workflow_log_retention_days` — nil means 90 days, 0 means forever. Only
|
||||
`fixed` findings are swept, by a `StartVulnSweeper` inside the same
|
||||
`RunAsLeader("housekeeping", …)` as the existing sweepers. `open` and
|
||||
`accepted` findings are never swept at any setting.
|
||||
|
||||
---
|
||||
|
||||
## Alerting
|
||||
|
||||
Per-org rules over the existing `notification_channels`: severity threshold,
|
||||
optional tag filter, target channels.
|
||||
|
||||
A rescan emits one message summarising what newly opened — "12 new critical
|
||||
across 4 servers" — never one message per finding. See the leader section for
|
||||
why the tick boundary makes this structural.
|
||||
|
||||
Modelling findings as a monitor type was rejected. It would reuse monitors'
|
||||
state machine and channel wiring for free, but monitors are up/down for one
|
||||
endpoint with retries and hourly rollups, none of which means anything for a
|
||||
CVE; most fields would be disabled in the UI and the uptime graphs would be
|
||||
polluted with a signal that is not uptime.
|
||||
|
||||
This adds one `notify` payload type and a `vuln_digest.html.tmpl` /
|
||||
`vuln_digest.txt.tmpl` pair in `shared/mail`. Note that `shared/mail` templates
|
||||
are parsed in `init()`, so a mistyped field is a boot-time panic — CLAUDE.md
|
||||
describes a `render_test.go` guarding against exactly this, but **that file does
|
||||
not exist**; the repository has no Go tests at all, and by instruction this
|
||||
feature adds none. The template pair must therefore be verified by starting the
|
||||
binary and sending one digest through a real channel.
|
||||
|
||||
---
|
||||
|
||||
## Entitlement
|
||||
|
||||
The feature name is `vuln_scanning`, and it crosses the two services the way
|
||||
every other feature does:
|
||||
|
||||
- **admin** carries it as a per-instance entitlement toggle, so it can later be
|
||||
priced as a catalogue `feature` component without a second migration;
|
||||
- **the licence** snapshots it into `License.Features []string` at issue time;
|
||||
- **the server** asks `lic.HasFeature("vuln_scanning")` and never switches on
|
||||
tier, so changing what a tier includes needs no server release.
|
||||
|
||||
Off on Free.
|
||||
|
||||
**The gate is checked at `ReportPackages`, not at display.** Gating only the UI
|
||||
would still pay every write cost, and storage is the expensive half.
|
||||
|
||||
The agent learns of it through the existing 30-second `SyncKeys` poll:
|
||||
`SyncResponse` gains a `collect_packages` bool, and the hourly loop skips
|
||||
collection entirely when it is false. So an ungated instance produces no
|
||||
collection, no gRPC body, no document and no storage. `ReportPackages` still
|
||||
re-checks the entitlement server-side and refuses — the agent flag is an
|
||||
optimisation, the server check is the boundary.
|
||||
|
||||
Turning the feature off does not delete existing findings; they stop being
|
||||
served and stop updating. Deletion is the instance-deletion path's job.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
```
|
||||
GET /api/vulnerabilities # filter: severity, state, server, tags
|
||||
GET /api/vulnerabilities/summary # severity counts + database freshness
|
||||
POST /api/vulnerabilities/rescan # marks all scan_pending (owner|admin)
|
||||
POST /api/vulnerabilities/:id/accept # reason + until (owner|admin)
|
||||
DELETE /api/vulnerabilities/:id/accept # (owner|admin)
|
||||
GET /api/servers/:id/vulnerabilities
|
||||
GET /api/servers/:id/packages
|
||||
GET /api/packages/search?name= # fleet-wide
|
||||
GET,POST /api/vuln-rules · PUT,DELETE /api/vuln-rules/:id
|
||||
```
|
||||
|
||||
Every mutating path writes an audit event, as all of them do. Acceptance is the
|
||||
one decision people will be asked to justify, so `by`, `reason`, `until` and
|
||||
`at` land in the audit record and not only on the document.
|
||||
|
||||
---
|
||||
|
||||
## UI
|
||||
|
||||
`/vulnerabilities` is a fleet board **grouped by CVE** — one row per CVE with
|
||||
an affected-server count, expandable to the individual servers. The same CVE
|
||||
across 40 servers is one decision, and a flat list of findings makes it look
|
||||
like forty.
|
||||
|
||||
Server detail gains **Vulnerabilities** and **Packages** tabs. Alert rules go
|
||||
on `/settings/notifications`, beside the channels they consume.
|
||||
|
||||
Remediation introduces no new mechanism: a finding carrying `fixed_in` renders
|
||||
an **Apply updates** action calling the existing
|
||||
`POST /api/servers/:id/apply-updates`, which is already `ApplyUpdatesCmd`. See
|
||||
it, patch it, one place — and no second patching path to keep consistent with
|
||||
the first.
|
||||
|
||||
Database freshness is shown wherever findings are, not tucked into settings. A
|
||||
fleet scanning against a three-week-old database must say so rather than
|
||||
quietly report all-clear.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
**No automated tests.** The repository has none today, and by explicit
|
||||
instruction this feature adds none — no `*_test.go`, no frontend test files.
|
||||
That is a deliberate decision by the repository owner, recorded here so the
|
||||
absence reads as a choice rather than an omission.
|
||||
|
||||
It does change the risk profile, and the places it changes it are worth naming,
|
||||
because each fails by producing a **wrong answer rather than a crash**:
|
||||
|
||||
- **Version comparison.** The backport case — installed `1:3.0.2-0ubuntu1.15`
|
||||
against advisory fixed-in `1:3.0.2-0ubuntu1.15` resolving to *not
|
||||
vulnerable* — plus tilde ordering (`1.0~rc1` < `1.0`), epoch dominance
|
||||
(`1:1.0` > `2.0`) and `1.9` < `1.10`. Wrong here means a vulnerable fleet
|
||||
reported clean.
|
||||
- **Source-package fan-out.** One advisory against `openssl` must flag
|
||||
`libssl3`, `openssl` and `libssl-dev`. Matching on binary name alone silently
|
||||
finds one of three.
|
||||
- **`first_seen` preservation.** An upsert that overwrites it makes every
|
||||
finding look discovered today, and nothing surfaces that until someone reads
|
||||
a report.
|
||||
- **Fixed-before-reopen ordering.** A finding both absent from a scan and past
|
||||
its acceptance expiry must settle `fixed`, not reopen.
|
||||
|
||||
The implementation plan carries a manual verification table for each, to be
|
||||
walked before the relevant task is committed. They are the substitute for the
|
||||
tests, not a formality.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Behaviour |
|
||||
| ------- | --------- |
|
||||
| Database pull fails | Keep the last good copy and serve stale. Record `last_error`, surface `pulled_at` age. **Never clear findings** — a network blip must not read as "all fixed" |
|
||||
| Unsupported distribution | `status: unsupported`, not zero findings |
|
||||
| Agent stops reporting | Findings persist and `collected_at` age is shown. No auto-expiry: a silent agent is not a patched server |
|
||||
| `trivy-db` schema version bumps | The puller refuses an unknown schema rather than mis-parsing it |
|
||||
| ghcr anonymous rate limit | Backoff; `VANTAGE_TRIVY_DB_REF` mirrors to a private registry |
|
||||
| Leadership lost mid-scan | The context is cancelled and the scan returns; `scan_pending` is still set, so the next leader picks it up |
|
||||
| Instance deleted | **`server_packages` and `vuln_findings` must be added to the control plane's instance-deletion collection list.** Easy to miss, and missing it orphans a tenant's package data indefinitely |
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Name | Required | Notes |
|
||||
| ---- | -------- | ----- |
|
||||
| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for air-gapped installs or to avoid the anonymous ghcr rate limit |
|
||||
| `VANTAGE_VULNDB_DISABLED` | no | disables the puller and the scheduler entirely. Findings already written are still served and still marked stale |
|
||||
|
||||
---
|
||||
|
||||
## Deliberately out of scope
|
||||
|
||||
- **Windows.** Separate source, collector and matcher; its own spec.
|
||||
- **Container image scanning.** Sub-project C; needs the container registry.
|
||||
- **Compliance baseline assertions.** Sub-project D; shares this findings UI
|
||||
and nothing else.
|
||||
- **Language-level dependency scanning** (npm, pip, Go modules). `trivy-db`
|
||||
covers these ecosystems, but finding the manifests on a host is a different
|
||||
collection problem from asking the package manager what is installed.
|
||||
- **Automatic patching on a finding.** Remediation is one click, not zero. An
|
||||
unattended upgrade triggered by a CVE feed is a fleet-wide change driven by a
|
||||
third party's data, which is not a decision to take away from an operator.
|
||||
@@ -0,0 +1,441 @@
|
||||
# Workload registry
|
||||
|
||||
Date: 2026-08-06
|
||||
|
||||
Agents enumerate what each server actually runs — Docker containers, the
|
||||
compose stacks grouping them, and systemd services — and report it to the
|
||||
control plane. Containers and units can be started, stopped and restarted from
|
||||
the UI, and a bounded snapshot of their logs can be read without opening a
|
||||
console.
|
||||
|
||||
This is **sub-project B** of the four sketched in
|
||||
`2026-08-06-package-inventory-and-cve-findings-design.md`:
|
||||
|
||||
| # | Sub-project | Depends on |
|
||||
| - | ----------- | ---------- |
|
||||
| A | Package inventory + CVE findings — its own spec | nothing |
|
||||
| B | **Workload registry** — this spec | nothing |
|
||||
| C | Container image scanning | A and B |
|
||||
| D | Compliance profiles | shares A's findings UI only |
|
||||
|
||||
A and B are independent. C is the joiner and must not be designed before both
|
||||
exist: it needs B's image list and A's findings model.
|
||||
|
||||
**Workload** is the domain word throughout: one container or one systemd unit.
|
||||
It gives the collection, the commands and the page a single honest name rather
|
||||
than saying "container or service" in every identifier.
|
||||
|
||||
Scope is **Linux only**, matching sub-project A and the existing position that
|
||||
Windows agents are second-class by design. Docker runs on Windows; systemd does
|
||||
not, and half a feature per platform is worse than a clear line.
|
||||
|
||||
---
|
||||
|
||||
## What this is for
|
||||
|
||||
The control plane can manage a fleet's keys, run workflows across it and watch
|
||||
its endpoints, but it has no idea what any of those servers actually *runs*.
|
||||
"Restart nginx on that box" means opening a console. "Which of these 80 servers
|
||||
is still on the old image" is unanswerable.
|
||||
|
||||
---
|
||||
|
||||
## Reporting and refresh are one path
|
||||
|
||||
The agent reports on its own 60-second ticker through a `ReportWorkloads` RPC,
|
||||
using the same hash short-circuit as the package report: it offers a SHA-256 of
|
||||
the sorted workload list, and sends the body only when the server does not
|
||||
already hold that hash. An unchanged list costs one small message, which on a
|
||||
60-second cadence is the common case by a wide margin.
|
||||
|
||||
The on-demand refresh **does not return data**. `RefreshWorkloadsCmd` carries
|
||||
no payload back; it makes the agent report immediately through the normal RPC,
|
||||
and the UI refetches the stored document.
|
||||
|
||||
That is deliberate. A refresh that returned workloads inline would be a second
|
||||
writer for the same collection, arriving by a different route, with its own
|
||||
serialisation and its own opportunity to disagree with the periodic one. One
|
||||
writer, one shape; the refresh is a nudge, not a channel.
|
||||
|
||||
Opening a server's Workloads tab dispatches a refresh, so what is on screen is
|
||||
live rather than up to a minute stale. That matters because the page has a
|
||||
Restart button on it: a stale list is not merely a wrong impression, it is a
|
||||
wrong action aimed at a container that already died.
|
||||
|
||||
## What does answer back
|
||||
|
||||
Two operations genuinely return something:
|
||||
|
||||
| Command | Answers with |
|
||||
| ------- | ------------ |
|
||||
| `ControlWorkloadCmd{kind, id, action}` | the existing `CommandResult` — ok or error |
|
||||
| `WorkloadLogsCmd{kind, id, tail}` | a new `WorkloadLogsResult{command_id, text, truncated}` |
|
||||
|
||||
Both ride the proven path: `commandDispatcher.send()` for request and ack, and
|
||||
a `WorkloadResults` registry mirroring `StepResults.Await`/`Deliver` over the
|
||||
bus. **`Await` must subscribe before the command is dispatched** — the pod
|
||||
driving the request is usually not the pod holding the agent's stream, and a
|
||||
fast agent otherwise answers into a channel nobody has joined. This is not a
|
||||
new hazard; it is the one `stepresults.go` already documents.
|
||||
|
||||
```protobuf
|
||||
rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse);
|
||||
|
||||
message ReportWorkloadsRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3;
|
||||
bool docker_ok = 4;
|
||||
string docker_error = 5;
|
||||
bool systemd_ok = 6;
|
||||
string systemd_error = 7;
|
||||
repeated Workload workloads = 8; // empty on the offer call
|
||||
}
|
||||
|
||||
message ReportWorkloadsResponse {
|
||||
bool need_full = 1;
|
||||
}
|
||||
|
||||
// ServerCommand gains three variants.
|
||||
message RefreshWorkloadsCmd {}
|
||||
|
||||
message ControlWorkloadCmd {
|
||||
string kind = 1; // "container" | "unit"
|
||||
string id = 2;
|
||||
string action = 3; // "start" | "stop" | "restart"
|
||||
}
|
||||
|
||||
message WorkloadLogsCmd {
|
||||
string kind = 1;
|
||||
string id = 2;
|
||||
int32 tail = 3;
|
||||
}
|
||||
|
||||
// AgentMessage gains one variant.
|
||||
message WorkloadLogsResult {
|
||||
string command_id = 1;
|
||||
string text = 2;
|
||||
bool truncated = 3;
|
||||
string error = 4;
|
||||
}
|
||||
```
|
||||
|
||||
The offer-then-send handshake is the package report's, unchanged: the agent
|
||||
calls once with `workloads` empty, and resends with the body only if the
|
||||
response sets `need_full`.
|
||||
|
||||
An agent whose stream no pod holds gets a 503 from the dispatcher, as
|
||||
everything else does. Commands are not queued: a command whose owner died must
|
||||
fail loudly rather than be delivered to nobody while the operator is told it
|
||||
worked.
|
||||
|
||||
---
|
||||
|
||||
## Not gated by licence
|
||||
|
||||
Unlike CVE scanning, this reads as core fleet management rather than a premium
|
||||
add-on, so v1 ships to every instance with no entitlement check.
|
||||
|
||||
If that changes it is a one-line `HasFeature` check at `ReportWorkloads`,
|
||||
gating collection rather than display — the same placement and the same
|
||||
reasoning as sub-project A, where gating the UI alone would still pay every
|
||||
write cost.
|
||||
|
||||
---
|
||||
|
||||
## Data model
|
||||
|
||||
One new collection, `server_workloads`, one document per server, mirroring
|
||||
`server_packages`.
|
||||
|
||||
```go
|
||||
type ServerWorkloads struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
Hash string `bson:"hash" json:"hash"`
|
||||
Workloads []Workload `bson:"workloads" json:"workloads"`
|
||||
CollectedAt time.Time `bson:"collected_at" json:"collected_at"`
|
||||
|
||||
DockerOK bool `bson:"docker_ok" json:"docker_ok"`
|
||||
DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"`
|
||||
SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"`
|
||||
SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"`
|
||||
}
|
||||
|
||||
type Workload struct {
|
||||
Kind string `bson:"kind" json:"kind"` // "container" | "unit"
|
||||
ID string `bson:"id" json:"id"` // container id, or unit name
|
||||
Name string `bson:"name" json:"name"`
|
||||
State string `bson:"state" json:"state"`
|
||||
Health string `bson:"health,omitempty" json:"health,omitempty"`
|
||||
Image string `bson:"image,omitempty" json:"image,omitempty"`
|
||||
Stack string `bson:"stack,omitempty" json:"stack,omitempty"`
|
||||
Ports []string `bson:"ports,omitempty" json:"ports,omitempty"`
|
||||
Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"`
|
||||
StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
|
||||
Protected bool `bson:"protected" json:"protected"`
|
||||
}
|
||||
```
|
||||
|
||||
`State` is normalised across the two kinds: containers report `running`,
|
||||
`exited`, `paused`, `restarting`, `created`; units report `active`, `inactive`,
|
||||
`failed`, `activating`. They are deliberately **not** collapsed into a shared
|
||||
vocabulary — a failed unit and an exited container mean different things, and
|
||||
flattening them would lose the distinction the operator needs.
|
||||
|
||||
Indexes: `{instance_id, server_id}` unique, plus multikey
|
||||
`{instance_id, "workloads.image"}` for the fleet-wide "which servers run image
|
||||
X" query.
|
||||
|
||||
### Why the OK/Error pairs exist
|
||||
|
||||
A host with no Docker installed and a host where Docker is installed and
|
||||
running nothing both produce an empty list. One should read "not in use here",
|
||||
the other "nothing running", and only the second deserves any alarm.
|
||||
|
||||
The error strings separate a third case the booleans alone cannot: Docker
|
||||
installed with the daemon down. "Not installed" and "installed but not
|
||||
responding" are different problems with different fixes, and collapsing them
|
||||
into one false boolean throws away the only thing that tells them apart.
|
||||
|
||||
### Why `Protected` is reported rather than derived
|
||||
|
||||
The agent already knows which unit and container it is. Sending that up lets
|
||||
the UI render the action disabled with a reason instead of offering a button
|
||||
whose refusal is already known.
|
||||
|
||||
The field is the courtesy; the agent's own check is the boundary. See the
|
||||
control section.
|
||||
|
||||
### No history
|
||||
|
||||
A workload list is state, not a record. Nobody asks what containers ran last
|
||||
Tuesday, and keeping it would grow a collection per server per minute in
|
||||
exchange for a question nobody has.
|
||||
|
||||
---
|
||||
|
||||
## Collectors
|
||||
|
||||
### Docker: two commands, no English parsing
|
||||
|
||||
```
|
||||
docker ps -aq
|
||||
docker inspect --format '{{json .}}' <ids…>
|
||||
```
|
||||
|
||||
Not `docker ps --format '{{json .}}'` alone. That reports health and uptime
|
||||
inside a human `Status` string — `"Up 2 hours (healthy)"` — and anything built
|
||||
on it is parsing English that is localised, reworded between releases, and
|
||||
silently different for a paused or restarting container. `inspect` returns
|
||||
`State.Health.Status`, `State.StartedAt` and `RestartCount` as typed fields.
|
||||
Two execs instead of one, and no parser to be wrong.
|
||||
|
||||
`RestartCount` justifies the second call by itself: a container cycling is the
|
||||
single thing this page most needs to show, and it is invisible in a list that
|
||||
only ever says "Up".
|
||||
|
||||
Compose stacks come from the `com.docker.compose.project` label. **No YAML is
|
||||
read from disk** — the label is what Docker itself treats as authoritative, and
|
||||
a compose file on disk may not be what is actually running.
|
||||
|
||||
Docker absent, or a socket that cannot be reached, sets `DockerOK: false`. It
|
||||
is not an error and produces no log line: most servers in a fleet built around
|
||||
SSH key management will not have Docker, and treating the normal case as a
|
||||
fault makes the feature look broken on the majority of the estate.
|
||||
|
||||
### systemd: filtered on purpose
|
||||
|
||||
```
|
||||
systemctl list-units --type=service --state=running,failed --no-legend --plain --no-pager
|
||||
systemctl list-unit-files --type=service --state=enabled --no-legend --plain --no-pager
|
||||
```
|
||||
|
||||
Two calls because "running or failed" and "enabled but stopped" are different
|
||||
questions, and an enabled unit that is not running is exactly the one worth
|
||||
seeing.
|
||||
|
||||
Excluded by prefix: `systemd-`, `user@`, `session-`, `init.scope`. A typical
|
||||
host carries 300+ units, the platform's own accounting for most of them.
|
||||
Listing all of them buries the ten anyone cares about — the same failure mode
|
||||
as an unfiltered vulnerability report, and the same fix.
|
||||
|
||||
Column output rather than `--output=json`: the JSON flag requires systemd 246+,
|
||||
and this fleet includes older stable distributions. The column format has been
|
||||
stable considerably longer than the JSON one has existed.
|
||||
|
||||
---
|
||||
|
||||
## Control actions
|
||||
|
||||
```
|
||||
container: docker {start|stop|restart} <id>
|
||||
unit: systemctl {start|stop|restart} <unit>
|
||||
```
|
||||
|
||||
Owner or admin only. Every action writes an audit event naming the actor, the
|
||||
server and the target.
|
||||
|
||||
### The protected set
|
||||
|
||||
Computed agent-side: `vantage-agent.service`, plus the container ID read from
|
||||
`/proc/self/cgroup` should the agent ever be run inside a container.
|
||||
|
||||
The agent refuses those before doing anything. As with the console relay
|
||||
hardcoding `127.0.0.1` agent-side, **the control plane may name a target, but
|
||||
the agent decides what it will do to itself**. A server-side denylist alone
|
||||
would be bypassed by the next dispatch path someone adds, and the failure is
|
||||
unrecoverable from the UI: a server that stops its own agent goes offline, and
|
||||
the way back is SSH or physical access — precisely what this feature exists to
|
||||
avoid needing.
|
||||
|
||||
### Timeouts
|
||||
|
||||
`docker stop` waits on a container that may ignore SIGTERM. `systemctl stop`
|
||||
on a unit with a long `TimeoutStopSec` blocks for exactly as long as that says.
|
||||
Both run under a 90-second context, and a timeout returns a real error rather
|
||||
than an ack implying success.
|
||||
|
||||
---
|
||||
|
||||
## Logs
|
||||
|
||||
```
|
||||
container: docker logs --tail 500 --timestamps <id>
|
||||
unit: journalctl -u <unit> -n 500 --no-pager --output=short-iso
|
||||
```
|
||||
|
||||
Capped at **500 lines and 256KB, whichever binds first**, with `truncated` set
|
||||
so the UI can say so. Two caps because 500 lines of a container emitting 4KB
|
||||
JSON blobs is 2MB, and a line count alone does not stop it — the same reasoning
|
||||
that gave workflow logs both a per-line and a per-run cap.
|
||||
|
||||
Live following is deliberately absent. The browser console already offers a
|
||||
real terminal on the same server, where `docker logs -f` works properly with
|
||||
its own scrollback and cancellation. Building a second streaming path — a
|
||||
relay listener, proxy bus keys, a WebSocket upgrade and a cancellation story
|
||||
for a follow nobody closed — to duplicate that would be a large amount of
|
||||
machinery aimed at a capability already shipped. A bounded snapshot answers
|
||||
"why did this restart", which is the question that sends people to the console
|
||||
in the first place.
|
||||
|
||||
### Log reads are owner or admin only, and audited
|
||||
|
||||
Unlike workflow logs, these cannot be masked. A workflow's logs can be masked
|
||||
because the run injected the secrets and therefore knows their values. A
|
||||
container's stdout is arbitrary and may contain credentials nobody declared —
|
||||
a connection string in a startup banner, a token in a stack trace.
|
||||
|
||||
So log reads sit behind the same role check as control actions and are audited.
|
||||
A member who can see the fleet cannot read its logs. This is a deliberate
|
||||
access decision, not an oversight, and it is why log reading is not simply
|
||||
folded in with the read-only snapshot endpoints.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
```
|
||||
GET /api/servers/:id/workloads # stored snapshot
|
||||
POST /api/servers/:id/workloads/refresh # dispatch, then refetch
|
||||
POST /api/servers/:id/workloads/:wid/action # {"action":"start|stop|restart"} (owner|admin)
|
||||
GET /api/servers/:id/workloads/:wid/logs?tail= # (owner|admin)
|
||||
GET /api/workloads?image=&stack=&state= # fleet-wide
|
||||
```
|
||||
|
||||
`:wid` is a container ID or a unit name, URL-encoded. Unit names carry dots and
|
||||
`@`, which are legal in a path segment but not worth relying on unencoded.
|
||||
|
||||
`tail` is clamped to the 500-line cap server-side; a client asking for more
|
||||
gets 500, not an error.
|
||||
|
||||
---
|
||||
|
||||
## UI
|
||||
|
||||
Server detail gains a **Workloads** tab, ordered compose stacks first — grouped
|
||||
under the stack name — then loose containers, then units.
|
||||
|
||||
That ordering is not cosmetic. A stack is one thing to an operator even when it
|
||||
is six containers, and a flat list turns one decision into six rows. It is the
|
||||
same argument that groups the vulnerabilities board by CVE rather than by
|
||||
finding.
|
||||
|
||||
A `/workloads` fleet view answers "which servers run image X", which is the
|
||||
reason the snapshot is stored at all rather than fetched on demand and
|
||||
discarded.
|
||||
|
||||
Three rules that follow directly from the model:
|
||||
|
||||
- **Protected rows render their actions disabled, with the reason**, rather
|
||||
than offering a button whose refusal is already known.
|
||||
- **`DockerOK: false` reads "Docker not in use on this server"**, never an
|
||||
empty list, and `DockerError` when present is shown as a distinct problem.
|
||||
- State never reads by colour alone: every pill carries a distinct shape and a
|
||||
text label, matching the existing monitor and severity pills.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
**No automated tests.** The repository has none today, and by explicit
|
||||
instruction this feature adds none — no `*_test.go`, no frontend test files.
|
||||
A deliberate decision by the repository owner, recorded so the absence reads as
|
||||
a choice rather than an omission.
|
||||
|
||||
The behaviours that would otherwise have been tested are the ones that fail
|
||||
quietly, and the implementation plan carries a manual check for each:
|
||||
|
||||
- **Parser output against real command output.** `docker inspect` must yield
|
||||
`RestartCount`, health and the compose label as `Stack`; the `systemctl`
|
||||
exclusion filter must drop `systemd-*` and `user@*` while keeping
|
||||
`nginx.service`. Both are verified against a live host rather than a fixture.
|
||||
- **Protected-set computation.** `vantage-agent.service` marked,
|
||||
`nginx.service` not. Getting this wrong in the permissive direction lets a
|
||||
server stop its own agent, which is unrecoverable from the UI.
|
||||
- **Hash order-independence.** An ordering-sensitive hash resends the full list
|
||||
every 60 seconds, which is invisible except as traffic.
|
||||
- **Log capping in both directions.** 600 lines in → 500 out with `truncated`;
|
||||
a 300KB blob of fewer than 500 lines → capped, `truncated`. The second is the
|
||||
case a line-count-only implementation silently fails, and it fails by sending
|
||||
megabytes rather than by erroring.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Behaviour |
|
||||
| ------- | --------- |
|
||||
| Docker not installed | `DockerOK: false`, no error, UI reads "not in use" |
|
||||
| Docker installed, daemon down | `DockerOK: false` **plus** `DockerError` — different message, different fix |
|
||||
| Agent offline | 503 from the existing dispatcher. No queueing: a command whose owner died must fail loudly |
|
||||
| Action on a protected workload | Agent refuses; API answers 409 naming the reason |
|
||||
| `stop` exceeds its timeout | Real error surfaced, never a hopeful ack. Snapshot refreshed afterwards |
|
||||
| Container removed between snapshot and action | Docker's "No such container" surfaced and a refresh dispatched — this is what on-demand refresh is for |
|
||||
| Log exceeds either cap | Truncated, flagged, and stated in the UI |
|
||||
| Instance deleted | **`server_workloads` must be added to the control plane's instance-deletion collection list**, alongside sub-project A's two collections |
|
||||
|
||||
---
|
||||
|
||||
## Deliberately out of scope
|
||||
|
||||
- **Live log following.** The console already does it. See the logs section.
|
||||
- **Creating, deleting or updating containers and units.** This is a control
|
||||
and visibility surface, not a deployment tool — workflows already exist for
|
||||
changing what a server runs, with snapshots, audit and rollback.
|
||||
- **`docker exec` into a container.** The console reaches the host; exec from
|
||||
the control plane is a second remote-execution path with its own audit and
|
||||
authorisation story, and it belongs in its own spec if anywhere.
|
||||
- **Kubernetes and containerd.** The Docker collector shells to the `docker`
|
||||
CLI, so a node whose runtime is containerd or CRI-O reports nothing from it —
|
||||
`DockerOK: false`, correctly, since Docker genuinely is not in use. Covering
|
||||
those runtimes means a `crictl`/`nerdctl` collector, and talking to a
|
||||
Kubernetes API server is a different subsystem again. Neither is v1.
|
||||
- **Podman as a supported runtime.** Its `docker`-compatible CLI means an
|
||||
aliased install will largely work, and that is a happy accident rather than a
|
||||
claim: nothing here is tested against Podman and its `RestartCount` and
|
||||
compose-label behaviour are not verified.
|
||||
- **Windows.** No systemd, and a different container story.
|
||||
- **Image vulnerability scanning.** Sub-project C, which needs this spec's
|
||||
image list and sub-project A's findings model.
|
||||
@@ -10,67 +10,54 @@ HQ portal.
|
||||
|
||||
## What a licence is
|
||||
|
||||
A signed file. It carries the instance UUID it belongs to, the tier, the server
|
||||
A signed file. It carries the instance ID it belongs to, the tier, the server
|
||||
allowance, feature toggles and an expiry. The control plane verifies the
|
||||
signature locally checking a licence never contacts HQ, and a running instance
|
||||
does not need HQ to be reachable.
|
||||
signature locally.
|
||||
|
||||
Signing happens in exactly one place, in HQ. The control plane can only verify.
|
||||
A running instance does not need HQ to be reachable.
|
||||
|
||||
## 1. Find your instance UUID
|
||||
:::info One Free per account, per deployment.
|
||||
The limit is enforced per account **and** deployment, so a Free cloud instance does not stop you claiming Free on a self-hosted install.
|
||||
:::
|
||||
|
||||
In the control plane, go to **Settings → Licence**. The instance UUID is shown
|
||||
there. It is the identity your licence binds to.
|
||||
## 1. Find your instance ID
|
||||
|
||||
## 2. Link the install to your HQ account
|
||||
In the control plane, go to **Settings → Licence**. The instance ID is shown there.
|
||||
|
||||
## 2. Create a free license
|
||||
|
||||
1. Sign in at [Vantage HQ](https://vantage-hq.hostxtra.co.uk). If you have no
|
||||
account, see [Accounts and signup](../hq/accounts-and-signup.md).
|
||||
2. Choose **Link an instance**.
|
||||
3. Paste the instance UUID and give it a name you will recognise.
|
||||
2. Click on the **Buy A Plan** button.
|
||||
3. Click on **Self Hosted** then click on the **Free** plan, then finally Paste the instance ID and give it a name you will recognise.
|
||||
|
||||
Linking claims the UUID for your account. A UUID already linked elsewhere is
|
||||
refused with a conflict rather than silently moved.
|
||||
You will then see the new instance on the **Overview** page.
|
||||
|
||||
## 3. Claim Free
|
||||
## 3. Downloading the free license
|
||||
|
||||
With the instance linked, choose **Claim Free** on it. HQ issues a Free licence
|
||||
bound to that UUID and hands it back.
|
||||
With the instance created go to the **Overview** page and expand the new instance.
|
||||
|
||||
:::info One Free per account, per deployment
|
||||
The limit is enforced per account **and** deployment, so a Free cloud instance
|
||||
does not stop you claiming Free on a self-hosted install. Both the friendly
|
||||
pre-check and the issuer apply the same rule deliberately, because a
|
||||
pre-check stricter than the issuer would refuse something that would actually
|
||||
have worked.
|
||||
:::
|
||||
Click on the **View Instance Settings** button. You can then click on the **Download License** or the **Copy to clipboard** button.
|
||||
|
||||
## 4. Install the licence
|
||||
|
||||
Download the licence from HQ and paste it in the control plane at
|
||||
**Settings → Licence**.
|
||||
|
||||
The instance validates the signature, checks the UUID matches its own, and
|
||||
The instance validates the signature, checks the ID matches its own, and
|
||||
starts reporting the tier, allowance and expiry.
|
||||
|
||||
:::warning Cloud instances cannot paste a licence
|
||||
On a cloud instance `POST /license` answers `409 cloud_managed`, and the UI
|
||||
hides the form entirely. A cloud licence is written directly by HQ. This is not
|
||||
a restriction the injection path has to work around it writes to the database,
|
||||
not through the endpoint.
|
||||
:::info Cloud instances do **not** require installing the license as this is done automatically.
|
||||
:::
|
||||
|
||||
## Renewing
|
||||
|
||||
Free licences are renewable from HQ within a renewal window near expiry;
|
||||
outside that window the renew call refuses. See [Free tier](../hq/free-tier.md).
|
||||
|
||||
Pasting a licence keeps working while the current one is expired that endpoint
|
||||
is exempt from the licence check, because it is the way out of degraded mode.
|
||||
outside that window you cannot renew early. See [Free tier](../hq/free-tier.md).
|
||||
|
||||
## Moving the install to new hardware
|
||||
|
||||
Rebuilding produces a new instance UUID, and a licence binds to a UUID. Use
|
||||
Rebuilding produces a new instance ID, and a licence binds to a ID. Use
|
||||
**Relink** in HQ to move the licence across. The number of relinks per term is
|
||||
capped; the portal shows how many you have left.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ title: First login
|
||||
sidebar_label: First login
|
||||
---
|
||||
|
||||
A fresh install has no users and no organisation. The first visit creates both.
|
||||
A fresh install has no users and no instance. The first visit creates both.
|
||||
|
||||
## 1. Bootstrap
|
||||
|
||||
@@ -13,48 +13,49 @@ Open the control plane in a browser. Because no user exists, you land on
|
||||
|
||||
Fill in:
|
||||
|
||||
| Field | Notes |
|
||||
| ----------------- | ----------------------------------------------------------------- |
|
||||
| Organisation name | Display name. Shown throughout the UI |
|
||||
| Slug | Lowercase, used in the hostname on cloud. Some names are reserved |
|
||||
| Your name | |
|
||||
| Email | Becomes your sign-in identity |
|
||||
| Password | Stored bcrypt-hashed |
|
||||
| Field | Notes |
|
||||
| ------------- | ------------------------------------- |
|
||||
| Instance name | Display name. Shown throughout the UI |
|
||||
| Email | Becomes your sign-in identity |
|
||||
| Password | Stored bcrypt-hashed |
|
||||
|
||||
Submitting creates the organisation and its **owner** you.
|
||||
Submitting creates the instance and its **owner** you.
|
||||
|
||||
:::warning Bootstrap works exactly once
|
||||
The endpoint is open only while the database has no users. As soon as the first
|
||||
one exists, `/setup` redirects to the login page and the bootstrap endpoint
|
||||
refuses. There is no second chance to create the first owner, so record the
|
||||
credentials before you close the tab.
|
||||
one exists, There is no second chance to create the first owner, so record the
|
||||
credentials before you continue.
|
||||
:::
|
||||
|
||||
## 2. Sign in
|
||||
## 2. Copy the Instance ID
|
||||
|
||||
You are taken to `/login`. Sign in with the email and password you just set.
|
||||
Once you have finished setup you will see the successfully created page.
|
||||
|
||||
Sessions are an opaque 32-byte token in the `km_session` cookie, with the body
|
||||
held in Redis for 24 hours. Restarting Redis signs everyone out and loses
|
||||
nothing else.
|
||||
This will show the Instance ID. You will need this ID when creating a license in the HQ.
|
||||
|
||||
## 3. Look around
|
||||
## 3. Sign in
|
||||
|
||||
You land on the fleet dashboard, which is empty. The sidebar is the whole
|
||||
Click the continue to sign in button on the successful setup page.
|
||||
|
||||
You will be taken to `/login`. Sign in with the email and password you just set.
|
||||
|
||||
## 4. Look around
|
||||
|
||||
You land on the servers dashboard, which is empty. The sidebar is the whole
|
||||
product:
|
||||
|
||||
| Section | What it does |
|
||||
| --------- | ----------------------------------------- |
|
||||
| Servers | The fleet enrol, inspect, console, update |
|
||||
| Keys | SSH public keys and their assignments |
|
||||
| Workflows | Compose and run scripted work |
|
||||
| Steps | The reusable step library |
|
||||
| Monitors | HTTP, TCP, ICMP and TLS checks |
|
||||
| Secrets | The encrypted vault |
|
||||
| Audit | Every mutating action |
|
||||
| Settings | Members, SSO, alerts, retention, licence |
|
||||
| Section | What it does |
|
||||
| --------- | ------------------------------------------ |
|
||||
| Servers | The server enrol, inspect, console, update |
|
||||
| Keys | SSH public keys and their assignments |
|
||||
| Workflows | Compose and run scripted work |
|
||||
| Steps | The reusable step library |
|
||||
| Monitors | HTTP, TCP, ICMP and TLS checks |
|
||||
| Secrets | The encrypted vault |
|
||||
| Audit | Every mutating action |
|
||||
| Settings | Members, SSO, alerts, retention, licence |
|
||||
|
||||
## 4. Add the rest of your team
|
||||
## 5. Add the rest of your team
|
||||
|
||||
Go to **Settings → Access**. Add members with a role:
|
||||
|
||||
@@ -66,12 +67,6 @@ Go to **Settings → Access**. Add members with a role:
|
||||
|
||||
Settings and organisation management require `owner` or `admin`.
|
||||
|
||||
If you would rather not manage passwords, configure single sign-on instead:
|
||||
see [Settings](../vantage/settings.md#single-sign-on). You can add more than
|
||||
one identity provider; each gets its own button on the login page, and no
|
||||
buttons appear at all until at least one provider is configured. Client
|
||||
secrets are stored encrypted.
|
||||
If you would rather not manage passwords, configure single sign-on instead: see [Settings](../vantage/settings.md#single-sign-on).
|
||||
|
||||
## Next
|
||||
|
||||
[Add your first server](./first-server.md).
|
||||
You can add more than one identity provider; each gets its own button on the login page, and no buttons appear at all until at least one provider is configured.
|
||||
|
||||
@@ -4,41 +4,39 @@ title: Add your first server
|
||||
sidebar_label: Add your first server
|
||||
---
|
||||
|
||||
Enrolling a machine means running one command on it. The control plane issues a
|
||||
Enrolling a server means running one command on it. The control plane issues a
|
||||
short-lived token, the install script fetches the agent and writes a config, and
|
||||
the machine registers itself.
|
||||
|
||||
## 1. Create the enrolment
|
||||
|
||||
In the UI, go to **Servers → Add server**. That calls `POST /api/servers/new`,
|
||||
which generates a server ID and a pre-registration token and hands back a ready
|
||||
one-liner.
|
||||
In the UI, go to **Servers → Add server** Then click the **Generate Install Command** button.
|
||||
This generates a server ID and a pre-registration token
|
||||
|
||||
:::warning The token is single-use and lives one hour
|
||||
It is the only credential in the flow, and it is spent the moment the agent
|
||||
calls `Register`. If you paste it somewhere and come back tomorrow, create a new
|
||||
enrolment instead nothing is lost by doing so.
|
||||
It is the only credential in the flow, and it is spent the moment the agent registers.
|
||||
:::
|
||||
|
||||
## 2. Run the one-liner
|
||||
|
||||
### Linux
|
||||
|
||||
Run the generated install script as root.
|
||||
|
||||
Here is an example of the install script:
|
||||
|
||||
```bash
|
||||
curl -fsSL "https://vantage.example.com/install?server_id=<id>&token=<token>" | bash
|
||||
```
|
||||
|
||||
Run it as root. The script:
|
||||
What the script does:
|
||||
|
||||
1. Detects architecture `x86_64` and `aarch64` only; anything else exits.
|
||||
2. Asks the Gitea API for the newest `agent/v*` release.
|
||||
3. Downloads the binary and `checksums.txt`, and **verifies the SHA-256**,
|
||||
aborting on a mismatch.
|
||||
4. Installs to `/usr/local/bin/vantage-agent`, mode `0755`.
|
||||
5. Writes `/etc/vantage/config.yaml` (directory `0700`, file `0600`) containing
|
||||
the server ID, the pre-registration token and the gRPC host.
|
||||
6. Writes `/etc/systemd/system/vantage-agent.service` with `Restart=always`, and
|
||||
runs `systemctl enable --now vantage-agent`.
|
||||
2. Downloads the binary and `checksums.txt`, and **verifies the SHA-256**, aborting on a mismatch.
|
||||
3. Installs to `/usr/local/bin/vantage-agent`, mode `0755`.
|
||||
4. Writes the config file at `/etc/vantage/config.yaml`
|
||||
1. This contains the server ID, the pre-registration token and the gRPC host.
|
||||
5. Writes the systemd service file `/etc/systemd/system/vantage-agent.service` and starts the agent.
|
||||
|
||||
### Windows
|
||||
|
||||
@@ -46,42 +44,30 @@ Run it as root. The script:
|
||||
irm "https://vantage.example.com/install.ps1?server_id=<id>&token=<token>" | iex
|
||||
```
|
||||
|
||||
Run from an elevated PowerShell. The agent is registered as a service through
|
||||
NSSM, with the config at `%ProgramData%\vantage\config.yaml`. There is also an
|
||||
MSI built by CI if you would rather deploy that.
|
||||
Run from an elevated PowerShell.
|
||||
|
||||
:::info Windows agents are second-class on purpose
|
||||
They register, heartbeat, run workflow steps and report inventory. They do
|
||||
**not** manage `authorized_keys` the key subsystem is Linux-only, and a
|
||||
Windows agent stops after the heartbeat portion of the poll.
|
||||
What the script does:
|
||||
|
||||
1. Creates the config at `%ProgramData%\vantage\config.yaml`.
|
||||
1. This contains the server ID, the pre-registration token and the gRPC host.
|
||||
2. Downloads the agent MSI from Gitea.
|
||||
3. Installs the MSI and creates the Windows service.
|
||||
4. Starts the agent.
|
||||
|
||||
:::info Windows agents do **not** manage `authorized_keys` as this is a Linux-only function.
|
||||
:::
|
||||
|
||||
## 3. Watch it come up
|
||||
|
||||
The server appears immediately as `pending`. Within one poll interval 30
|
||||
seconds it flips to `active`.
|
||||
The server appears immediately as `pending`. Within one poll interval, 30 seconds it becomes `active`.
|
||||
|
||||
On the machine:
|
||||
Check the systemd logs using the following commands:
|
||||
|
||||
```bash
|
||||
systemctl status vantage-agent
|
||||
journalctl -u vantage-agent -f
|
||||
```
|
||||
|
||||
What happens on that first run:
|
||||
|
||||
```
|
||||
1. Load /etc/vantage/config.yaml
|
||||
2. pre_reg_token present → register → save agent_token, clear pre_reg_token
|
||||
3. Reconnect with the permanent token
|
||||
4. Start: command stream · hourly update check · inventory · monitors
|
||||
5. Enter the key poll loop
|
||||
```
|
||||
|
||||
After registration the config no longer contains the pre-registration token; it
|
||||
contains a permanent agent token instead. The control plane stores only the
|
||||
SHA-256 of that token, never the token itself.
|
||||
|
||||
## 4. Confirm it works
|
||||
|
||||
Open the server's detail page. Within a minute or two you should see:
|
||||
@@ -104,7 +90,7 @@ Open the server's detail page. Within a minute or two you should see:
|
||||
A server is marked `offline` when its last-seen time passes the threshold; that
|
||||
sweep runs every two minutes, so allow for it before concluding anything.
|
||||
|
||||
## Next
|
||||
## Next Steps
|
||||
|
||||
- [Assign an SSH key](../vantage/ssh-keys.md)
|
||||
- [Run a workflow](../vantage/workflows.md)
|
||||
|
||||
@@ -4,8 +4,7 @@ title: Accounts and signup
|
||||
sidebar_label: Accounts and signup
|
||||
---
|
||||
|
||||
Vantage HQ, at `vantage-hq.hostxtra.co.uk`, is where you manage the **account**
|
||||
behind your instances: your team, your instances, their licences and billing.
|
||||
[Vantage HQ](https://vantage-hq.hostxtra.co.uk) is where you manage the **account**, your team, your instances, their licences and billing.
|
||||
|
||||
## An account is a team, not a person
|
||||
|
||||
@@ -31,26 +30,15 @@ Signup is **account-first**. Creating an account creates the account and you;
|
||||
it does not create a Vantage instance. Nothing exists in any control plane until
|
||||
you later create or link one.
|
||||
|
||||
1. Go to the signup form.
|
||||
1. Go to the [signup form](https://vantage.hostxtra.co.uk/start).
|
||||
2. Enter your name, email and a password.
|
||||
3. Check your email and click the verification link.
|
||||
|
||||
:::info Verify before you can sign in
|
||||
An unverified account gets a distinct "check your email" message rather than a
|
||||
generic authentication failure the address is already known to be yours, so
|
||||
there is nothing to protect by being vague.
|
||||
:::
|
||||
|
||||
Verification links are valid for **24 hours**. The token is 32 random bytes and
|
||||
only its SHA-256 hash is stored, so a leaked database yields no working links.
|
||||
|
||||
If the verification email cannot be sent, the signup is rolled back rather than
|
||||
left stranded retry rather than assuming a half-created account is in the way.
|
||||
Verification links are valid for **24 hours**.
|
||||
|
||||
## Signing in
|
||||
|
||||
Email and password. The session is a cookie, separate from the control plane's:
|
||||
signing in to HQ does not sign you in to an instance, and vice versa.
|
||||
Use the Email and password used in the signup form to login to the HQ, signing in to HQ does not sign you in to an instance, and vice versa.
|
||||
|
||||
## What comes next
|
||||
|
||||
@@ -65,14 +53,6 @@ signing in to HQ does not sign you in to an instance, and vice versa.
|
||||
|
||||
Three destinations: **Overview**, **People**, **Billing**.
|
||||
|
||||
Settings lives in the account menu rather than the nav, because it is your
|
||||
password rather than a place. The appearance toggle is there too.
|
||||
|
||||
Overview lists your instances. Each is one record, closed to a row and open to
|
||||
its licence contents, members and actions. It opens by default when it is your
|
||||
only instance or when it needs attention, and your manual choice is remembered.
|
||||
|
||||
There is deliberately no "your plan" card in the sidebar: tier, limits and
|
||||
expiry belong to a **licence**, and a licence belongs to one instance. An
|
||||
account with a Free cloud instance and a Professional self-hosted one has no
|
||||
single plan to show.
|
||||
- Overview lists your instances.
|
||||
- People shows all the account members and their roles.
|
||||
- Billing show the current subscriptions and subscription management.
|
||||
|
||||
+17
-50
@@ -8,56 +8,29 @@ Paid plans are billed through **Paddle**, which is the merchant of record. Your
|
||||
invoice, your card details and your tax handling are all Paddle's; HQ holds a
|
||||
customer reference and nothing sensitive.
|
||||
|
||||
Billing is **owner-only**.
|
||||
:::warning
|
||||
The Billing page requires the **owner-only** account role.
|
||||
:::
|
||||
|
||||
## Buying
|
||||
## Buying A Plan
|
||||
|
||||
Buying a plan license can be found in Vantage HQ by clicking on the **Buy a Plan** button on the **Overview** page.
|
||||
|
||||
### Cloud
|
||||
|
||||
Open the instance, change its configuration to what you want, and check out.
|
||||
Checkout runs in the browser.
|
||||
On the **Buy A Plan** page you will need to select the **Deployment** to **Cloud** then chose your **Billing** cycle (Monthly or Annually).
|
||||
|
||||
Then select your desired **Plan** and configure the features.
|
||||
|
||||
Finally specify the **Instance Name** and click the **Continue to payment** button.
|
||||
|
||||
### Self-hosted
|
||||
|
||||
**Buy self-hosted**, then bind the purchase to your install's UUID. See
|
||||
[Self-hosted instances](./self-hosted-instances.md).
|
||||
On the **Buy A Plan** page you will need to select the **Deployment** to **Self-Hosted** then chose your **Billing** cycle (Monthly or Annually).
|
||||
|
||||
## What you are buying
|
||||
Then select your desired **Plan** and configure the features.
|
||||
|
||||
A subscription's line items are the configuration: the plan base, the metered
|
||||
server count above the base, and any per-instance features. Changing the
|
||||
configuration changes the line items.
|
||||
|
||||
## Changing configuration
|
||||
|
||||
**Instance → Configuration**, adjust servers or features, and save.
|
||||
|
||||
- **Increases** take effect when the payment confirms.
|
||||
- **Reductions** are scheduled for the end of the term. The portal shows the
|
||||
date and the new value.
|
||||
|
||||
## The customer portal
|
||||
|
||||
**Billing → Manage** mints a Paddle customer-portal session where you can
|
||||
update your payment method, see invoices and cancel.
|
||||
|
||||
## How a licence follows a payment
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C["Checkout / change"] --> P["Paddle"]
|
||||
P -->|signed webhook| H["HQ"]
|
||||
H --> G["Entitlement: desired → granted"]
|
||||
G --> L["Licence signed from granted"]
|
||||
```
|
||||
|
||||
The webhook is the **only** issuing path for paid plans. It is signature
|
||||
verified, processed exactly once, and resolved from the subscription's _current_
|
||||
line items so a webhook that arrives out of order still produces the right
|
||||
answer rather than replaying a stale state.
|
||||
|
||||
A licence is signed from **granted** only. A checkout you abandon changes
|
||||
nothing.
|
||||
Finally specify the **Instance Name** and click the **Continue to payment** button.
|
||||
|
||||
## Cancelling and failed payments
|
||||
|
||||
@@ -66,18 +39,12 @@ Your licence runs to its grace-padded expiry and then lapses normally. There is
|
||||
no mid-term cut-off.
|
||||
|
||||
For a cloud Free instance, lapsing eventually leads to deletion see
|
||||
[Free tier](./free-tier.md). Paid instances are not reaped.
|
||||
[Free tier](./free-tier.md). Paid instances are not deleted.
|
||||
|
||||
## Renewals
|
||||
|
||||
At renewal the subscription bills again and the licence is reissued for the new
|
||||
term. It is also the only moment a scheduled **reduction** takes effect.
|
||||
|
||||
Self-hosted customers: download and paste the reissued licence. Cloud customers:
|
||||
nothing to do.
|
||||
|
||||
## Free is not in Paddle at all
|
||||
|
||||
Free has no subscription, no £0 line item and no Paddle record. It has its own
|
||||
renewal, in the portal. An account only acquires a Paddle customer reference
|
||||
with its first paid purchase.
|
||||
- Self-hosted customers: download and paste the reissued licence.
|
||||
- Cloud customers: the license is automatically linked to the instance.
|
||||
|
||||
@@ -25,11 +25,6 @@ features on a paid plan.
|
||||
The limit is enforced per account **and** deployment. A Free cloud instance does
|
||||
not prevent a Free self-hosted one they are separate slots.
|
||||
|
||||
## Free is outside Paddle
|
||||
|
||||
There is no subscription, no £0 line item and no invoice. Your account acquires
|
||||
a Paddle customer reference only with its first paid purchase.
|
||||
|
||||
## Renewing
|
||||
|
||||
Free licences have a term and must be renewed from the portal.
|
||||
|
||||
@@ -30,9 +30,9 @@ entitlement.
|
||||
|
||||
Two are per-instance toggles rather than tier bundles:
|
||||
|
||||
| Feature | What it enables |
|
||||
| --------- | ------------------------------------------------------------------------- |
|
||||
| `console` | The [browser console](../vantage/browser-console.md) |
|
||||
| Feature | What it enables |
|
||||
| --------- | -------------------------------------------------------------------- |
|
||||
| `console` | The [browser console](../vantage/browser-console.md) |
|
||||
| `oidc` | Per-instance [single sign-on](../vantage/settings.md#single-sign-on) |
|
||||
|
||||
No tier includes them by default; you enable them on the instances that need
|
||||
@@ -88,8 +88,3 @@ or let it be written for you (cloud).
|
||||
When you exceed your server allowance, enrolling another one is refused. The
|
||||
existing fleet is unaffected. Raise the allowance in the portal, or remove a
|
||||
server you are not using.
|
||||
|
||||
## Legacy tiers
|
||||
|
||||
An older `self_hosted` tier is mapped forward to self-hosted Professional
|
||||
wherever it appears. Nothing needs doing about it.
|
||||
|
||||
@@ -10,7 +10,6 @@ themselves on command.
|
||||
## Checking the current version
|
||||
|
||||
Each server's detail page shows the version it reported at its last sync.
|
||||
`GET /api/agent/latest-version` reports the newest release available.
|
||||
|
||||
## Updating from the UI
|
||||
|
||||
@@ -21,8 +20,6 @@ version. The agent then:
|
||||
2. Verifies the SHA-256 against `checksums.txt`.
|
||||
3. Stops itself, replaces the binary in place, and starts again.
|
||||
|
||||
`Restart=always` on the systemd unit is what makes the last step work.
|
||||
|
||||
The server briefly goes `offline` and comes back within a poll interval or two.
|
||||
|
||||
## Updating from the machine
|
||||
|
||||
@@ -52,16 +52,9 @@ cp /opt/vantage/.env /secure-location/vantage.env
|
||||
|
||||
Treat it as a credential in its own right it holds the encryption key.
|
||||
|
||||
## Run logs
|
||||
|
||||
Workflow run logs live in the `./data` bind mount, not in the database. They are
|
||||
swept on the retention schedule anyway, so most people do not back them up. If
|
||||
you keep them for compliance, set retention to `0` (forever) and include the
|
||||
directory.
|
||||
|
||||
## What a restore gives you
|
||||
|
||||
Everything: fleet, keys, assignments, workflows and their history, monitors and
|
||||
Everything: server, keys, assignments, workflows and their history, monitors and
|
||||
incidents, secrets, settings and the audit log.
|
||||
|
||||
What it does **not** do is reconcile the world. After a restore:
|
||||
|
||||
@@ -37,31 +37,9 @@ tls: true
|
||||
|
||||
:::danger This file is the credential
|
||||
`agent_token` is plaintext here and nowhere else the control plane holds only
|
||||
its SHA-256. Anyone who can read this file can act as this agent. That is why
|
||||
it is `0600` and the directory is `0700`.
|
||||
its SHA-256. Anyone who can read this file can act as this agent.
|
||||
:::
|
||||
|
||||
## Startup sequence
|
||||
|
||||
```
|
||||
1. Load the config
|
||||
2. pre_reg_token present → register → save agent_token,
|
||||
clear pre_reg_token, reconnect
|
||||
3. Start: command stream · hourly update check · inventory · monitors
|
||||
4. Enter the key poll loop
|
||||
```
|
||||
|
||||
## The poll loop
|
||||
|
||||
```
|
||||
1. Ask the control plane for the desired key state, reporting the
|
||||
agent version
|
||||
2. Non-Linux hosts stop here Windows agents register and heartbeat only
|
||||
3. Diff the desired keys against /root/.ssh/authorized_keys;
|
||||
unchanged → write nothing
|
||||
4. Changed → write a temp file, rename it over the real one, chmod 0600
|
||||
```
|
||||
|
||||
## Service management
|
||||
|
||||
### Linux
|
||||
@@ -77,21 +55,13 @@ journalctl -u vantage-agent -f
|
||||
|
||||
### Windows
|
||||
|
||||
A service registered through NSSM, or installed by the MSI that CI builds.
|
||||
A service registered through NSSM, or installed by the MSI.
|
||||
|
||||
```powershell
|
||||
Get-Service vantage-agent
|
||||
Restart-Service vantage-agent
|
||||
```
|
||||
|
||||
## Command-line flags
|
||||
|
||||
```
|
||||
vantage-agent -generate-key
|
||||
```
|
||||
|
||||
Generates a keypair locally. Normal operation takes no flags.
|
||||
|
||||
## Moving an agent to a new control plane
|
||||
|
||||
Change `server_url`, clear `agent_token`, set a fresh `pre_reg_token` from a new
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
id: vulnerabilities
|
||||
title: Vulnerabilities
|
||||
sidebar_label: Vulnerabilities
|
||||
---
|
||||
|
||||
Each Linux server reports the packages it has installed. Vantage matches them
|
||||
against the security advisories published by that server's own distribution and
|
||||
raises a finding for anything not yet patched.
|
||||
|
||||
Requires the **vulnerability scanning** feature on your licence. Without it,
|
||||
agents collect nothing at all — there is no inventory stored and no findings
|
||||
page to read.
|
||||
|
||||
## What gets scanned
|
||||
|
||||
Linux servers running `apt`, `dnf`/`yum`, `apk`, `zypper` or `pacman`. Agents
|
||||
report their package list hourly, and only when it has changed since the last
|
||||
report.
|
||||
|
||||
Windows servers are not scanned.
|
||||
|
||||
Some distributions publish no machine-readable advisory feed. Those servers
|
||||
show **unsupported** on their own page rather than appearing as having no
|
||||
vulnerabilities — the two are very different answers, and only one of them is
|
||||
good news.
|
||||
|
||||
## Why versions look "wrong"
|
||||
|
||||
A finding names the version your distribution ships, not the upstream release.
|
||||
Ubuntu's `openssl 3.0.2-0ubuntu1.15` carries security fixes backported into
|
||||
what still calls itself 3.0.2, so public CVE databases listing "3.0.2" as
|
||||
vulnerable are describing upstream, not your machine.
|
||||
|
||||
Vantage matches against your distribution's own advisories, which is why a
|
||||
server can be running a version some scanners flag while Vantage correctly
|
||||
reports it as patched.
|
||||
|
||||
For the same reason a severity here may be lower than the one you find on a CVE
|
||||
website. Debian and Red Hat routinely downgrade a rating when the vulnerable
|
||||
code path is not reachable in the way they build the package. Their rating is
|
||||
the accurate one for the package you are actually running.
|
||||
|
||||
## The board
|
||||
|
||||
`/vulnerabilities` groups findings by CVE. One row per CVE with the number of
|
||||
affected servers, expandable to the individual servers — the same CVE across
|
||||
forty machines is one decision, not forty.
|
||||
|
||||
Severity counts at the top filter the list when clicked. The state tabs switch
|
||||
between **open**, **accepted** and **fixed**.
|
||||
|
||||
The vulnerability database's age is shown above the board. If a pull has failed
|
||||
for long enough for the data to be stale, that becomes a warning: a low count
|
||||
against three-week-old data is not the same as a low count.
|
||||
|
||||
## Fixing something
|
||||
|
||||
A finding with a known fixed version gets an **Apply updates** button, which
|
||||
runs the same OS update the server page offers. There is no separate patching
|
||||
mechanism.
|
||||
|
||||
Vantage never patches automatically. An unattended upgrade triggered by a third
|
||||
party's data feed is a fleet-wide change nobody chose.
|
||||
|
||||
## Accepting a finding
|
||||
|
||||
Some findings cannot be fixed today: a kernel CVE waiting on a reboot window,
|
||||
or one with no vendor fix published at all.
|
||||
|
||||
**Accept** hides a finding from counts and alerts until a date you choose, with
|
||||
a reason that is recorded in the audit log along with your name. On that date it
|
||||
reopens by itself.
|
||||
|
||||
The expiry is required. A dismissal with no end date is how a finding gets
|
||||
forgotten, and it is exactly what an auditor will ask to see.
|
||||
|
||||
## Alerts
|
||||
|
||||
Alert rules live with your notification channels, under
|
||||
**Settings → Notification Channels**. A rule has a minimum severity, an optional
|
||||
server tag filter, and one or more channels.
|
||||
|
||||
A rule sends **one digest per scan** summarising what newly opened — never one
|
||||
message per finding. A database refresh can open several hundred findings at
|
||||
once, and a message each would flood the channel.
|
||||
|
||||
Findings that were already open do not re-alert.
|
||||
|
||||
## Fleet-wide package search
|
||||
|
||||
`GET /api/packages/search?name=openssl` answers which servers run a given
|
||||
package and at what version, across the whole fleet. Useful during an incident
|
||||
before a finding exists for it.
|
||||
@@ -147,6 +147,57 @@ A run shows the script that actually executed, not the current library version.
|
||||
|
||||
Targets run **in parallel**; steps within one server run **in order**.
|
||||
|
||||
## Schedules
|
||||
|
||||
A workflow can carry a schedule, and Vantage will start it the same way a person
|
||||
would — the same dispatch, the same snapshot, the same run page. A scheduled run
|
||||
is an ordinary run with `schedule` recorded as who triggered it.
|
||||
|
||||
Open a workflow, choose **Edit**, and tick **Run on a schedule**. The expression
|
||||
is standard five-field cron:
|
||||
|
||||
```
|
||||
minute hour day-of-month month day-of-week
|
||||
```
|
||||
|
||||
The presets write cron underneath, so you can start from one and adjust:
|
||||
|
||||
| Preset | Cron |
|
||||
| ------------------- | ----------- |
|
||||
| Hourly | `0 * * * *` |
|
||||
| Nightly, 02:00 | `0 2 * * *` |
|
||||
| Weekly, Sun 02:00 | `0 2 * * 0` |
|
||||
| Monthly, 1st 02:00 | `0 2 1 * *` |
|
||||
|
||||
There is no seconds field and no `@daily`-style shorthand. The next three
|
||||
occurrences are shown as you type, and they are computed by the server rather
|
||||
than the browser, so what you see is exactly what will fire.
|
||||
|
||||
### Timezones
|
||||
|
||||
A schedule stores an IANA timezone by name — `Europe/London`, not an offset.
|
||||
That is what makes a 02:00 job stay at 02:00 across a daylight-saving change
|
||||
instead of drifting an hour for half the year. An unknown zone is refused when
|
||||
you save it, not at 2am.
|
||||
|
||||
### Overlaps are skipped, not queued
|
||||
|
||||
If a run of the same workflow is still going when the next occurrence comes
|
||||
round, the occurrence is **skipped** and the reason recorded. It is not queued
|
||||
behind the running one. A patch workflow that takes longer than its interval
|
||||
should fall behind visibly rather than pile up.
|
||||
|
||||
### Missed occurrences
|
||||
|
||||
If the control plane was not running when an occurrence was due, it still fires
|
||||
when the control plane comes back — but only within **one hour** of the due
|
||||
time. Anything older is recorded as missed and dropped. A job missed by ten
|
||||
minutes during an upgrade should still run; one missed by two days should not
|
||||
suddenly fire at lunchtime.
|
||||
|
||||
Either kind of skip is shown on the workflow's schedule panel, with the time it
|
||||
was due and why it did not run.
|
||||
|
||||
## Watching a run
|
||||
|
||||
Step stdout and stderr stream back as chunks, are appended to a log file on the
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
id: workloads
|
||||
title: Workloads
|
||||
sidebar_label: Workloads
|
||||
---
|
||||
|
||||
A **workload** is one Docker container or one systemd service. Each Linux
|
||||
server reports what it runs, and you can start, stop and restart those
|
||||
workloads — and read a snapshot of their logs — without opening a console.
|
||||
|
||||
Available on every instance. No licence feature is required.
|
||||
|
||||
## What gets reported
|
||||
|
||||
Linux servers only. Agents report every 60 seconds, and an unchanged list costs
|
||||
a single small message rather than the whole thing again.
|
||||
|
||||
- **Containers** — every container, running or not, with its image, published
|
||||
ports, health, restart count and the compose stack it belongs to.
|
||||
- **Services** — systemd units that are running or failed, plus units that are
|
||||
enabled but currently stopped. The platform's own units (`systemd-*`,
|
||||
`user@*`, `session-*`) are filtered out; a typical host has 300 of them and
|
||||
they bury the ten you care about.
|
||||
|
||||
Windows servers report no workloads at all.
|
||||
|
||||
## Docker not in use is not an error
|
||||
|
||||
Three different things look identical if you are careless, and only one of them
|
||||
is a problem:
|
||||
|
||||
| What you see | What it means |
|
||||
| ------------ | ------------- |
|
||||
| "Docker is not in use on this server" | Docker is not installed. Normal, and not a fault |
|
||||
| "Docker is installed but not responding" | The daemon is down or the socket is unreachable |
|
||||
| An empty container list | Docker is running and there are no containers |
|
||||
|
||||
## Stacks are grouped
|
||||
|
||||
Compose stacks appear first, grouped under the stack name, then loose
|
||||
containers, then services. A stack is one thing even when it is six containers,
|
||||
and a flat list turns one decision into six rows.
|
||||
|
||||
The stack name comes from Docker's own `com.docker.compose.project` label. No
|
||||
compose file is read from disk — a file on disk may not be what is running.
|
||||
|
||||
## Controlling a workload
|
||||
|
||||
Start, stop and restart are **owner or admin only**, and every action is
|
||||
written to the audit log naming you, the server and the target.
|
||||
|
||||
The agent refuses to act on itself. `vantage-agent.service` is shown with its
|
||||
buttons disabled: a server that stops its own agent goes offline, and the only
|
||||
way back is SSH or physical access — which is exactly what this page exists to
|
||||
avoid needing.
|
||||
|
||||
A stop that never finishes is not reported as success. Both `docker stop` and
|
||||
`systemctl stop` run under a 90-second limit, and a timeout comes back as a
|
||||
real error.
|
||||
|
||||
## Reading logs
|
||||
|
||||
Logs are **owner or admin only** and every read is audited. Unlike workflow
|
||||
logs, a container's output cannot be masked: it is arbitrary, and a startup
|
||||
banner or a stack trace may contain credentials nobody declared.
|
||||
|
||||
A log read returns a snapshot of at most **500 lines or 256KB**, whichever
|
||||
limit is reached first, with the most recent output kept. When either limit
|
||||
binds, the dialog says so — a truncated log must never be read as a complete
|
||||
one.
|
||||
|
||||
There is no live following. The [browser console](./browser-console.md) already
|
||||
gives you a real terminal on the same server, where `docker logs -f` works
|
||||
properly with its own scrollback.
|
||||
|
||||
## Refreshing
|
||||
|
||||
Opening a server's Workloads panel asks its agent to report immediately, so
|
||||
what is on screen is current rather than up to a minute old. That matters
|
||||
because the panel has a Restart button on it: a stale row is not just a wrong
|
||||
impression, it is a wrong action aimed at something that already died.
|
||||
|
||||
If the agent is offline the refresh fails visibly rather than queueing. A
|
||||
command whose target cannot be reached must say so.
|
||||
|
||||
## Fleet view
|
||||
|
||||
**Workloads** in the sidebar searches the whole fleet by image, stack or state
|
||||
— "which of these servers is still on the old image" — and links each result
|
||||
back to its server.
|
||||
@@ -26,6 +26,8 @@ const sidebars: SidebarsConfig = {
|
||||
"vantage/ssh-keys",
|
||||
"vantage/workflows",
|
||||
"vantage/monitors",
|
||||
"vantage/vulnerabilities",
|
||||
"vantage/workloads",
|
||||
"vantage/notification-channels",
|
||||
"vantage/secrets",
|
||||
"vantage/browser-console",
|
||||
|
||||
+43
@@ -1,22 +1,62 @@
|
||||
cloud.google.com/go/compute v1.25.1/go.mod h1:oopOIR53ly6viBYxaDhBfJwzUAxf1zE//uf3IB011ls=
|
||||
cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=
|
||||
github.com/Intevation/gval v1.3.0/go.mod h1:xmGyGpP5be12EL0P12h+dqiYG8qn2j3PJxIgkoOHO5o=
|
||||
github.com/Intevation/jsonpath v0.2.1/go.mod h1:WnZ8weMmwAx/fAO3SutjYFU+v7DFreNYnibV7CiaYIw=
|
||||
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
|
||||
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
|
||||
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
|
||||
github.com/aquasecurity/go-gem-version v0.0.0-20201115065557-8eed6fe000ce/go.mod h1:HXgVzOPvXhVGLJs4ZKO817idqr/xhwsTcj17CLYY74s=
|
||||
github.com/aquasecurity/go-npm-version v0.0.1/go.mod h1:hxbJZtKlO4P8sZ9nztizR6XLoE33O+BkPmuYQ4ACyz0=
|
||||
github.com/aquasecurity/go-pep440-version v0.0.1/go.mod h1:3naPe+Bp6wi3n4l5iBFCZgS0JG8vY6FT0H4NGhFJ+i4=
|
||||
github.com/aquasecurity/go-version v0.0.1/go.mod h1:s1UU6/v2hctXcOa3OLwfj5d9yoXHa3ahf+ipSwEvGT0=
|
||||
github.com/briandowns/spinner v1.23.0/go.mod h1:rPG4gmXeN3wQV/TsAY4w8lPdIM6RX3yqeBQJSrbXjuE=
|
||||
github.com/bytedance/sonic/loader v0.3.0/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI=
|
||||
github.com/census-instrumentation/opencensus-proto v0.4.1/go.mod h1:4T9NM4+4Vw91VeyqjLS6ao50K5bOcLKN6Q42XnYaRYw=
|
||||
github.com/cheggaaa/pb/v3 v3.1.7/go.mod h1:/Ji89zfVPeC/u5j8ukD0MBPHt2bzTYp74lQ7KlgFWTQ=
|
||||
github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50/go.mod h1:5e1+Vvlzido69INQaVO6d87Qn543Xr6nooe9Kz7oBFM=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
|
||||
github.com/envoyproxy/go-control-plane v0.12.0/go.mod h1:ZBTaoJ23lqITozF0M6G4/IragXCQKCnYbmlmtHvwRG0=
|
||||
github.com/envoyproxy/protoc-gen-validate v1.0.4/go.mod h1:qys6tmnRsYrQqIhm2bvKZH4Blx/1gTIZ2UKVY1M+Yew=
|
||||
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
|
||||
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
|
||||
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
|
||||
github.com/gocsaf/csaf/v3 v3.1.1/go.mod h1:EpUCrQg69i+Y66MphmQvVbcj333GFLjXOYHg1zoXVso=
|
||||
github.com/golang/glog v1.2.0/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
|
||||
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
|
||||
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
|
||||
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||
github.com/josephburnett/jd/v2 v2.3.0/go.mod h1:0I5+gbo7y8diuajJjm79AF44eqTheSJy1K7DSbIUFAQ=
|
||||
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
|
||||
github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
|
||||
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
|
||||
github.com/masahiro331/go-mvn-version v0.0.0-20250131095131-f4974fa13b8a/go.mod h1:jZ3F25l7DbD7l7DcA8aj7eo1EZ84nbzcQHBB4lCSrI8=
|
||||
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
|
||||
github.com/package-url/packageurl-go v0.1.3/go.mod h1:nKAWB8E6uk1MHqiS/lQb9pYBGH2+mdJ2PJc2s50dQY0=
|
||||
github.com/pandatix/go-cvss v0.6.2/go.mod h1:jDXYlQBZrc8nvrMUVVvTG8PhmuShOnKrxP53nOFkt8Q=
|
||||
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
||||
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1/go.mod h1:uToXkOrWAZ6/Oc07xWQrPOhJotwFIyu2bBVN41fcDUY=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
|
||||
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/urfave/cli v1.22.16/go.mod h1:EeJR6BKodywf4zciqrdw6hpCPk68JO9z5LazXZMn5Po=
|
||||
go.etcd.io/gofail v0.2.0/go.mod h1:nL3ILMGfkXTekKI3clMBNazKnjUZjYLKmBHzsVAnC1o=
|
||||
go.mongodb.org/mongo-driver/v2 v2.5.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
|
||||
go.opentelemetry.io/otel/metric v1.34.0/go.mod h1:CEDrp0fy2D0MvkXE+dPV7cMi8tWZwX3dmaIhwPOaqHE=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.3/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
|
||||
golang.org/x/crypto v0.33.0/go.mod h1:bVdXmD7IV/4GdElGPozy6U7lWdRXA4qyRVGJV57uQ5M=
|
||||
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
||||
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
|
||||
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
|
||||
golang.org/x/exp v0.0.0-20251125195548-87e1e737ad39/go.mod h1:46edojNIoXTNOhySWIWdix628clX9ODXwPsQuG6hsK0=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
|
||||
golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
@@ -32,10 +72,13 @@ golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
|
||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
|
||||
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
|
||||
golang.org/x/time v0.8.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90=
|
||||
google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237/go.mod h1:Z5Iiy3jtmioajWHDGFk7CeugTyHtPvMHA4UTmUkyalE=
|
||||
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
|
||||
google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
|
||||
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
|
||||
k8s.io/utils v0.0.0-20201110183641-67b214c5f920/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
|
||||
|
||||
@@ -9,6 +9,8 @@ service Vantage {
|
||||
rpc SyncKeys(SyncRequest) returns (SyncResponse);
|
||||
rpc UploadGeneratedKey(UploadKeyRequest) returns (UploadKeyResponse);
|
||||
rpc ReportUpdates(ReportUpdatesRequest) returns (ReportUpdatesResponse);
|
||||
rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse);
|
||||
rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse);
|
||||
rpc ReportInventory(InventoryReport) returns (InventoryReportResponse);
|
||||
rpc SyncMonitors(SyncMonitorsRequest) returns (SyncMonitorsResponse);
|
||||
rpc ReportChecks(ReportChecksRequest) returns (ReportChecksResponse);
|
||||
@@ -37,6 +39,51 @@ message SyncRequest {
|
||||
|
||||
message SyncResponse {
|
||||
repeated string public_keys = 1;
|
||||
|
||||
// collect_packages tells the agent whether this instance's licence grants
|
||||
// vulnerability scanning. False means do not collect at all: no gRPC body,
|
||||
// no document, no storage. The server re-checks on ReportPackages — this
|
||||
// flag is the optimisation, the server check is the boundary.
|
||||
//
|
||||
// Absent reads as false, which is the safe direction: an old server that
|
||||
// does not send it leaves agents collecting nothing.
|
||||
bool collect_packages = 2;
|
||||
}
|
||||
|
||||
// ReportPackages carries a server's installed package set.
|
||||
//
|
||||
// The agent calls twice at most. The first call sends only the hash; if the
|
||||
// server already holds that hash it answers need_full = false and the ~150KB
|
||||
// body is never sent. A machine's package set changes rarely, so almost every
|
||||
// hour costs one small message.
|
||||
message ReportPackagesRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3;
|
||||
OSRelease os = 4;
|
||||
repeated InstalledPackage packages = 5; // empty on the offer call
|
||||
}
|
||||
|
||||
message ReportPackagesResponse {
|
||||
bool need_full = 1;
|
||||
}
|
||||
|
||||
message OSRelease {
|
||||
string family = 1;
|
||||
// version_id is not optional: Ubuntu 22.04 and 24.04 publish different fixed
|
||||
// versions for the same CVE, so a scan without it is guesswork.
|
||||
string version_id = 2;
|
||||
string arch = 3;
|
||||
}
|
||||
|
||||
message InstalledPackage {
|
||||
string name = 1;
|
||||
string version = 2;
|
||||
int32 epoch = 3;
|
||||
string arch = 4;
|
||||
// source_name is what the Debian and Ubuntu feeds are keyed on: one advisory
|
||||
// against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
string source_name = 5;
|
||||
}
|
||||
|
||||
message UploadKeyRequest {
|
||||
@@ -61,6 +108,7 @@ message AgentMessage {
|
||||
CommandResult result = 4;
|
||||
StepResult step_result = 5;
|
||||
StepOutputChunk step_output = 6;
|
||||
WorkloadLogsResult workload_logs_result = 7;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -183,6 +231,9 @@ message ServerCommand {
|
||||
CleanupWorkspaceCmd cleanup_workspace = 7;
|
||||
OpenProxyCmd open_proxy = 8;
|
||||
PingCmd ping = 9;
|
||||
RefreshWorkloadsCmd refresh_workloads = 10;
|
||||
ControlWorkloadCmd control_workload = 11;
|
||||
WorkloadLogsCmd workload_logs = 12;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -273,3 +324,66 @@ message ProxyServerMsg {
|
||||
ProxyClose close = 2;
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Workload registry
|
||||
|
||||
// ReportWorkloads carries what a server is running.
|
||||
//
|
||||
// Offer-then-send, the same handshake as ReportPackages: the agent calls once
|
||||
// with workloads empty, and resends with the body only if need_full is set.
|
||||
message ReportWorkloadsRequest {
|
||||
string server_id = 1;
|
||||
string agent_token = 2;
|
||||
string hash = 3;
|
||||
bool docker_ok = 4;
|
||||
string docker_error = 5;
|
||||
bool systemd_ok = 6;
|
||||
string systemd_error = 7;
|
||||
repeated Workload workloads = 8; // empty on the offer call
|
||||
// full marks the second call. It is not inferred from an empty workloads
|
||||
// list: a host running nothing sends an empty list as its full report.
|
||||
bool full = 9;
|
||||
}
|
||||
|
||||
message ReportWorkloadsResponse {
|
||||
bool need_full = 1;
|
||||
}
|
||||
|
||||
message Workload {
|
||||
string kind = 1; // "container" | "unit"
|
||||
string id = 2;
|
||||
string name = 3;
|
||||
string state = 4;
|
||||
string health = 5;
|
||||
string image = 6;
|
||||
string stack = 7;
|
||||
repeated string ports = 8;
|
||||
int32 restarts = 9;
|
||||
string started_at = 10; // RFC3339, empty when not running
|
||||
bool protected = 11;
|
||||
}
|
||||
|
||||
// RefreshWorkloadsCmd carries no payload back. It makes the agent report
|
||||
// immediately through ReportWorkloads, so there is exactly one writer for the
|
||||
// server_workloads collection rather than two arriving by different routes.
|
||||
message RefreshWorkloadsCmd {}
|
||||
|
||||
message ControlWorkloadCmd {
|
||||
string kind = 1;
|
||||
string id = 2;
|
||||
string action = 3; // "start" | "stop" | "restart"
|
||||
}
|
||||
|
||||
message WorkloadLogsCmd {
|
||||
string kind = 1;
|
||||
string id = 2;
|
||||
int32 tail = 3;
|
||||
}
|
||||
|
||||
message WorkloadLogsResult {
|
||||
string command_id = 1;
|
||||
string text = 2;
|
||||
bool truncated = 3;
|
||||
string error = 4;
|
||||
}
|
||||
|
||||
@@ -18,10 +18,19 @@ ARG VERSION=dev
|
||||
RUN cd server && CGO_ENABLED=0 GOOS=linux go build \
|
||||
-ldflags="-s -w -X main.Version=${VERSION}" -o /vantage-server ./cmd
|
||||
|
||||
# Staged so the scratch image below can have a /tmp. It cannot mkdir one
|
||||
# itself — scratch has no shell — and os.MkdirTemp fails outright without it.
|
||||
RUN mkdir -p /staging/tmp && chmod 1777 /staging/tmp
|
||||
|
||||
# Runtime stage
|
||||
FROM scratch
|
||||
|
||||
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
|
||||
|
||||
# vulndb unpacks the ~50MB trivy-db here. Without it the scheduler stops at
|
||||
# "temp dir: stat /tmp: no such file or directory" and no scanning happens,
|
||||
# while everything else in the process runs perfectly well.
|
||||
COPY --from=builder /staging/tmp /tmp
|
||||
COPY --from=builder /vantage-server /vantage-server
|
||||
|
||||
COPY default_steps/ /opt/default-steps/
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
grpcserver "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/monitorsched"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulnsched"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
@@ -131,6 +132,14 @@ func runSchemaSetup() {
|
||||
log.Printf("warning: failed to ensure workflow indexes: %v", err)
|
||||
}
|
||||
|
||||
if err := services.EnsureVulnIndexes(); err != nil {
|
||||
log.Printf("warning: failed to ensure vuln indexes: %v", err)
|
||||
}
|
||||
|
||||
if err := services.EnsureWorkloadIndexes(); err != nil {
|
||||
log.Printf("warning: failed to ensure workload indexes: %v", err)
|
||||
}
|
||||
|
||||
if instanceIDs, err := services.ListInstanceIDs(); err != nil {
|
||||
log.Printf("warning: failed to list instances for default step seeding: %v", err)
|
||||
} else {
|
||||
@@ -191,6 +200,12 @@ func serve() {
|
||||
LogEvent: services.LogEvent,
|
||||
})
|
||||
|
||||
vulnsched.Start(jobCtx, vulnsched.Deps{
|
||||
LogEvent: services.LogEvent,
|
||||
SendDigest: services.SendVulnDigest,
|
||||
})
|
||||
services.StartVulnSweeper(jobCtx)
|
||||
|
||||
ticker := time.NewTicker(2 * time.Minute)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
|
||||
+18
-1
@@ -3,20 +3,37 @@ module gitea.hostxtra.co.uk/mrhid6/vantage/server
|
||||
go 1.26
|
||||
|
||||
require (
|
||||
github.com/aquasecurity/trivy-db v0.0.0-20260713131703-4be526083c54
|
||||
github.com/coreos/go-oidc/v3 v3.18.0
|
||||
github.com/gin-gonic/gin v1.10.0
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f
|
||||
github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23
|
||||
github.com/knqyf263/go-rpm-version v0.0.0-20240918084003-2afd7dc6a38f
|
||||
github.com/opencontainers/image-spec v1.1.1
|
||||
github.com/redis/go-redis/v9 v9.20.1
|
||||
github.com/robfig/cron/v3 v3.0.1
|
||||
github.com/wwt/guac v1.3.2
|
||||
go.mongodb.org/mongo-driver/v2 v2.8.0
|
||||
golang.org/x/crypto v0.54.0
|
||||
golang.org/x/oauth2 v0.36.0
|
||||
google.golang.org/grpc v1.64.0
|
||||
oras.land/oras-go/v2 v2.6.2
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/hyperboloide/lk v0.0.0-20251220053519-b291812e3216 // indirect
|
||||
github.com/robfig/cron/v3 v3.0.1 // indirect
|
||||
github.com/oklog/ulid/v2 v2.1.1 // indirect
|
||||
github.com/opencontainers/go-digest v1.0.0 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/samber/lo v1.50.0 // indirect
|
||||
github.com/samber/oops v1.18.1 // indirect
|
||||
github.com/stretchr/objx v0.5.2 // indirect
|
||||
github.com/stretchr/testify v1.11.1 // indirect
|
||||
go.etcd.io/bbolt v1.4.3 // indirect
|
||||
go.opentelemetry.io/otel v1.34.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.34.0 // indirect
|
||||
)
|
||||
|
||||
require (
|
||||
|
||||
+40
-1
@@ -1,3 +1,7 @@
|
||||
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986 h1:2a30xLN2sUZcMXl50hg+PJCIDdJgIvIbVcKqLJ/ZrtM=
|
||||
github.com/aquasecurity/bolt-fixtures v0.0.0-20200903104109-d34e7f983986/go.mod h1:NT+jyeCzXk6vXR5MTkdn4z64TgGfE5HMLC8qfj5unl8=
|
||||
github.com/aquasecurity/trivy-db v0.0.0-20260713131703-4be526083c54 h1:4CZNoDkNfcuACevZeDraACGmP1+L0nKkRY52+jV8k1M=
|
||||
github.com/aquasecurity/trivy-db v0.0.0-20260713131703-4be526083c54/go.mod h1:iIEV2oGuZScvfyX2SMIn78iVMNnepgo0QuJJh/srgVI=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
@@ -35,6 +39,8 @@ github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBEx
|
||||
github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
|
||||
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
|
||||
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
|
||||
github.com/goccy/go-yaml v1.19.0 h1:EmkZ9RIsX+Uq4DYFowegAuJo8+xdX3T/2dwNPXbxEYE=
|
||||
github.com/goccy/go-yaml v1.19.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||
@@ -52,9 +58,17 @@ github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6K
|
||||
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f h1:GvCU5GXhHq+7LeOzx/haG7HSIZokl3/0GkoUFzsRJjg=
|
||||
github.com/knqyf263/go-apk-version v0.0.0-20200609155635-041fdbb8563f/go.mod h1:q59u9px8b7UTj0nIjEjvmTWekazka6xIt6Uogz5Dm+8=
|
||||
github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23 h1:dWzdsqjh1p2gNtRKqNwuBvKqMNwnLOPLzVZT1n6DK7s=
|
||||
github.com/knqyf263/go-deb-version v0.0.0-20241115132648-6f4aee6ccd23/go.mod h1:lUaIXCWzf7BRKTY5iEcrYy1TfgbYLYVIS/B2vPkJzOc=
|
||||
github.com/knqyf263/go-rpm-version v0.0.0-20240918084003-2afd7dc6a38f h1:xt29M2T6STgldg+WEP51gGePQCsQvklmP2eIhPIBK3g=
|
||||
github.com/knqyf263/go-rpm-version v0.0.0-20240918084003-2afd7dc6a38f/go.mod h1:i4sF0l1fFnY1aiw08QQSwVAFxHEm311Me3WsU/X7nL0=
|
||||
github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
|
||||
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
@@ -64,6 +78,15 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
|
||||
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
|
||||
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
|
||||
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
|
||||
github.com/oklog/ulid/v2 v2.1.1 h1:suPZ4ARWLOJLegGFiZZ1dFAkqzhMjL3J1TzI+5wHz8s=
|
||||
github.com/oklog/ulid/v2 v2.1.1/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ=
|
||||
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
|
||||
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
|
||||
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
|
||||
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
|
||||
github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
|
||||
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
|
||||
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
@@ -72,15 +95,21 @@ github.com/redis/go-redis/v9 v9.20.1 h1:sfCU6A8P3dXbKyWes02uxA2baehGux9dZHfEKtsT
|
||||
github.com/redis/go-redis/v9 v9.20.1/go.mod h1:v/M13XI1PVCDcm01VtPFOADfZtHf8YW3baQf57KlIkA=
|
||||
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
|
||||
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
|
||||
github.com/samber/lo v1.50.0 h1:XrG0xOeHs+4FQ8gJR97zDz5uOFMW7OwFWiFVzqopKgY=
|
||||
github.com/samber/lo v1.50.0/go.mod h1:RjZyNk6WSnUFRKK6EyOhsRJMqft3G+pg7dCWHQCWvsc=
|
||||
github.com/samber/oops v1.18.1 h1:qjhZbqbdyhWBKntkY8sxrDNKA8b4c5VHlmI1rli7X7M=
|
||||
github.com/samber/oops v1.18.1/go.mod h1:xYqvimigkKV70HyLXiBZJFpIWi2CGcc6Xx7eV+2HycI=
|
||||
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
|
||||
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
|
||||
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
|
||||
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
@@ -106,8 +135,14 @@ github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfS
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
|
||||
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
|
||||
go.mongodb.org/mongo-driver/v2 v2.8.0 h1:CxWDGQYY8QQwNjAl/aq2sfWakdnWZynnqJ9F4DhHbP8=
|
||||
go.mongodb.org/mongo-driver/v2 v2.8.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0=
|
||||
go.opentelemetry.io/otel v1.34.0 h1:zRLXxLCgL1WyKsPVrgbSdMN4c0FMkDAskSTQP+0hdUY=
|
||||
go.opentelemetry.io/otel v1.34.0/go.mod h1:OWFPOQ+h4G8xpyjgqo4SxJYdDQ/qmRH+wivy7zzx9oI=
|
||||
go.opentelemetry.io/otel/trace v1.34.0 h1:+ouXS2V8Rd4hp4580a8q23bg0azF2nI8cqLYnC8mh/k=
|
||||
go.opentelemetry.io/otel/trace v1.34.0/go.mod h1:Svm7lSjQD7kG7KJ/MUHPVXSDGz2OX4h0M2jHBhmSfRE=
|
||||
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
|
||||
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
|
||||
golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
|
||||
@@ -156,10 +191,14 @@ google.golang.org/grpc v1.64.0 h1:KH3VH9y/MgNQg1dE7b3XfVK0GsPSIzJwdF617gUSbvY=
|
||||
google.golang.org/grpc v1.64.0/go.mod h1:oxjF8E3FBnjp+/gVFYdWacaLDx9na1aqy9oovLpxQYg=
|
||||
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
|
||||
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
|
||||
oras.land/oras-go/v2 v2.6.2 h1:N04RXngAp1LJKTG6ifz3xHPipasEkWr+hFmInja5YKo=
|
||||
oras.land/oras-go/v2 v2.6.2/go.mod h1:PlTtg4JTDJkDe8yVHpM2wz7/YDc00GVas+i4jAW2TZ4=
|
||||
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
|
||||
|
||||
@@ -119,6 +119,28 @@ func RegisterRoutes(r *gin.Engine) {
|
||||
providers.POST("/:id/ack-notice", ackAuthProviderNotice)
|
||||
}
|
||||
apiGroup.GET("/auth/presets", auth.RequireRole("owner", "admin"), listAuthPresets)
|
||||
|
||||
apiGroup.GET("/vulnerabilities", listVulnerabilities)
|
||||
apiGroup.GET("/vulnerabilities/summary", vulnerabilitySummary)
|
||||
apiGroup.POST("/vulnerabilities/rescan", auth.RequireRole("owner", "admin"), rescanVulnerabilities)
|
||||
apiGroup.POST("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), acceptFinding)
|
||||
apiGroup.DELETE("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), unacceptFinding)
|
||||
apiGroup.GET("/servers/:id/vulnerabilities", listServerVulnerabilities)
|
||||
apiGroup.GET("/servers/:id/packages", getServerPackages)
|
||||
apiGroup.GET("/packages/search", searchPackages)
|
||||
apiGroup.GET("/vuln-rules", listVulnRules)
|
||||
apiGroup.POST("/vuln-rules", auth.RequireRole("owner", "admin"), createVulnRule)
|
||||
apiGroup.PUT("/vuln-rules/:id", auth.RequireRole("owner", "admin"), updateVulnRule)
|
||||
apiGroup.DELETE("/vuln-rules/:id", auth.RequireRole("owner", "admin"), deleteVulnRule)
|
||||
|
||||
// Control actions and log reads are owner|admin: container output is
|
||||
// arbitrary and cannot be masked, so a member who can see the fleet
|
||||
// still cannot read its logs.
|
||||
apiGroup.GET("/workloads", listWorkloads)
|
||||
apiGroup.GET("/servers/:id/workloads", getServerWorkloads)
|
||||
apiGroup.POST("/servers/:id/workloads/refresh", refreshServerWorkloads)
|
||||
apiGroup.POST("/servers/:id/workloads/:wid/action", auth.RequireRole("owner", "admin"), controlWorkload)
|
||||
apiGroup.GET("/servers/:id/workloads/:wid/logs", auth.RequireRole("owner", "admin"), getWorkloadLogs)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// vulnGroup is one CVE across every server it affects.
|
||||
//
|
||||
// The board groups by CVE rather than listing findings flat: the same CVE on
|
||||
// forty servers is one decision, and a flat list makes it look like forty.
|
||||
type vulnGroup struct {
|
||||
CVEID string `json:"cve_id"`
|
||||
Severity string `json:"severity"`
|
||||
Title string `json:"title,omitempty"`
|
||||
ServerCount int `json:"server_count"`
|
||||
Findings []models.VulnFinding `json:"findings"`
|
||||
}
|
||||
|
||||
func listVulnerabilities(c *gin.Context) {
|
||||
findings, err := services.ListInstanceFindings(auth.InstanceID(c), services.FindingFilter{
|
||||
Severity: c.Query("severity"),
|
||||
State: c.DefaultQuery("state", models.FindingOpen),
|
||||
ServerID: c.Query("server"),
|
||||
Tags: tagsFromQuery(c),
|
||||
})
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, groupByCVE(findings))
|
||||
}
|
||||
|
||||
// groupByCVE collapses findings into one row per CVE, most severe first.
|
||||
func groupByCVE(findings []models.VulnFinding) []vulnGroup {
|
||||
index := map[string]*vulnGroup{}
|
||||
order := []string{}
|
||||
|
||||
for _, f := range findings {
|
||||
g, ok := index[f.CVEID]
|
||||
if !ok {
|
||||
g = &vulnGroup{CVEID: f.CVEID, Severity: f.Severity, Title: f.Title}
|
||||
index[f.CVEID] = g
|
||||
order = append(order, f.CVEID)
|
||||
}
|
||||
// Several servers can disagree on severity when their distributions
|
||||
// rate the same CVE differently. The highest is shown, because that is
|
||||
// the one deciding whether anyone acts.
|
||||
if models.SeverityRank(f.Severity) > models.SeverityRank(g.Severity) {
|
||||
g.Severity = f.Severity
|
||||
}
|
||||
g.Findings = append(g.Findings, f)
|
||||
}
|
||||
|
||||
out := make([]vulnGroup, 0, len(order))
|
||||
for _, id := range order {
|
||||
g := index[id]
|
||||
servers := map[string]bool{}
|
||||
for _, f := range g.Findings {
|
||||
servers[f.ServerID] = true
|
||||
}
|
||||
g.ServerCount = len(servers)
|
||||
out = append(out, *g)
|
||||
}
|
||||
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
ri, rj := models.SeverityRank(out[i].Severity), models.SeverityRank(out[j].Severity)
|
||||
if ri != rj {
|
||||
return ri > rj
|
||||
}
|
||||
return out[i].ServerCount > out[j].ServerCount
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// tagsFromQuery reads repeated tag=key:value parameters.
|
||||
func tagsFromQuery(c *gin.Context) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, raw := range c.QueryArray("tag") {
|
||||
k, v, ok := strings.Cut(raw, ":")
|
||||
if !ok || k == "" {
|
||||
continue
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func vulnerabilitySummary(c *gin.Context) {
|
||||
counts, err := services.CountOpenFindingsBySeverity(auth.InstanceID(c))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
resp := gin.H{"counts": counts}
|
||||
|
||||
// Database freshness travels with the counts rather than living in
|
||||
// settings: a fleet scanned against a three-week-old database must say so
|
||||
// wherever its findings are read, not somewhere the reader has to go and
|
||||
// look for it.
|
||||
if meta, err := services.GetVulnDBMeta(); err == nil && meta != nil {
|
||||
resp["db_version"] = meta.DBVersion
|
||||
resp["pulled_at"] = meta.PulledAt
|
||||
resp["last_full_scan_at"] = meta.LastFullScanAt
|
||||
if meta.LastError != "" {
|
||||
resp["last_error"] = meta.LastError
|
||||
}
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, resp)
|
||||
}
|
||||
|
||||
func rescanVulnerabilities(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
|
||||
n, err := services.MarkInstanceForRescan(instanceID)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rescan", actorFromCtx(c), "", "",
|
||||
"queued "+strconv.FormatInt(n, 10)+" server(s) for rescan")
|
||||
c.JSON(http.StatusOK, gin.H{"queued": n})
|
||||
}
|
||||
|
||||
type acceptFindingRequest struct {
|
||||
Reason string `json:"reason"`
|
||||
Until time.Time `json:"until"`
|
||||
}
|
||||
|
||||
func acceptFinding(c *gin.Context) {
|
||||
var req acceptFindingRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
|
||||
// Both rejected deliberately. An acceptance with no reason is a dismissal
|
||||
// nobody can audit, and one already expired is a permanent dismissal
|
||||
// wearing an expiry — the graveyard the expiry exists to prevent.
|
||||
if strings.TrimSpace(req.Reason) == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "a reason is required"})
|
||||
return
|
||||
}
|
||||
if req.Until.IsZero() || !req.Until.After(time.Now()) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "until must be a future date"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
actor := actorFromCtx(c)
|
||||
|
||||
f, err := services.AcceptFinding(instanceID, c.Param("id"), actor, req.Reason, req.Until)
|
||||
if err != nil {
|
||||
writeFindingError(c, err)
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.accepted", actor, f.ServerID, "",
|
||||
f.CVEID+" on "+f.PackageName+" accepted until "+req.Until.Format(time.RFC3339)+": "+req.Reason)
|
||||
c.JSON(http.StatusOK, f)
|
||||
}
|
||||
|
||||
func unacceptFinding(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
actor := actorFromCtx(c)
|
||||
|
||||
f, err := services.UnacceptFinding(instanceID, c.Param("id"))
|
||||
if err != nil {
|
||||
writeFindingError(c, err)
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.unaccepted", actor, f.ServerID, "",
|
||||
f.CVEID+" on "+f.PackageName+" returned to open")
|
||||
c.JSON(http.StatusOK, f)
|
||||
}
|
||||
|
||||
func writeFindingError(c *gin.Context, err error) {
|
||||
if errors.Is(err, services.ErrFindingNotFound) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "finding not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
}
|
||||
|
||||
func listServerVulnerabilities(c *gin.Context) {
|
||||
findings, err := services.ListFindings(c.Request.Context(), auth.InstanceID(c), c.Param("id"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if findings == nil {
|
||||
findings = []models.VulnFinding{}
|
||||
}
|
||||
c.JSON(http.StatusOK, findings)
|
||||
}
|
||||
|
||||
func getServerPackages(c *gin.Context) {
|
||||
sp, err := services.ListPackages(auth.InstanceID(c), c.Param("id"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if sp == nil {
|
||||
// Not a 404: an agent that has not reported yet is the normal state for
|
||||
// the first hour after install, and is a different thing from a bad
|
||||
// server id.
|
||||
c.JSON(http.StatusOK, gin.H{"reported": false})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, sp)
|
||||
}
|
||||
|
||||
func searchPackages(c *gin.Context) {
|
||||
name := c.Query("name")
|
||||
if name == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "name is required"})
|
||||
return
|
||||
}
|
||||
hits, err := services.SearchPackages(auth.InstanceID(c), name)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, hits)
|
||||
}
|
||||
|
||||
func listVulnRules(c *gin.Context) {
|
||||
rules, err := services.ListVulnRules(auth.InstanceID(c))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, rules)
|
||||
}
|
||||
|
||||
func createVulnRule(c *gin.Context) {
|
||||
var r models.VulnAlertRule
|
||||
if err := c.ShouldBindJSON(&r); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
created, err := services.CreateVulnRule(instanceID, &r)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rule_created", actorFromCtx(c), "", "", "rule "+created.Name)
|
||||
c.JSON(http.StatusCreated, created)
|
||||
}
|
||||
|
||||
func updateVulnRule(c *gin.Context) {
|
||||
var r models.VulnAlertRule
|
||||
if err := c.ShouldBindJSON(&r); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
|
||||
instanceID := auth.InstanceID(c)
|
||||
if err := services.UpdateVulnRule(instanceID, c.Param("id"), &r); err != nil {
|
||||
if errors.Is(err, services.ErrVulnRuleNotFound) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "rule not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rule_updated", actorFromCtx(c), "", "", "rule "+r.Name)
|
||||
c.JSON(http.StatusOK, gin.H{"status": "updated"})
|
||||
}
|
||||
|
||||
func deleteVulnRule(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
if err := services.DeleteVulnRule(instanceID, c.Param("id")); err != nil {
|
||||
if errors.Is(err, services.ErrVulnRuleNotFound) {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "rule not found"})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "vuln.rule_deleted", actorFromCtx(c), "", "", "rule "+c.Param("id"))
|
||||
c.JSON(http.StatusOK, gin.H{"status": "deleted"})
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// getServerWorkloads returns the stored snapshot.
|
||||
//
|
||||
// A server that has never reported answers an empty list rather than 404: the
|
||||
// agent may simply not have got there yet, and 404 reads as "no such server".
|
||||
func getServerWorkloads(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
id := c.Param("id")
|
||||
|
||||
if _, err := services.GetServer(instanceID, id); err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
|
||||
return
|
||||
}
|
||||
|
||||
sw, err := services.GetWorkloads(instanceID, id)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if sw == nil {
|
||||
c.JSON(http.StatusOK, models.ServerWorkloads{
|
||||
ServerID: id,
|
||||
Workloads: []models.Workload{},
|
||||
})
|
||||
return
|
||||
}
|
||||
if sw.Workloads == nil {
|
||||
sw.Workloads = []models.Workload{}
|
||||
}
|
||||
c.JSON(http.StatusOK, sw)
|
||||
}
|
||||
|
||||
// refreshServerWorkloads nudges the agent to report now. It returns no data:
|
||||
// the client refetches the stored document once the agent has written it.
|
||||
func refreshServerWorkloads(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
id := c.Param("id")
|
||||
|
||||
s, err := services.GetServer(instanceID, id)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
|
||||
return
|
||||
}
|
||||
|
||||
if err := services.DispatchRefreshWorkloads(s.ServerID); err != nil {
|
||||
// Not queued: a command whose owner died must fail loudly, so the
|
||||
// client can show the stored snapshot as stale rather than pretend.
|
||||
c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusAccepted, gin.H{"message": "refresh requested"})
|
||||
}
|
||||
|
||||
func controlWorkload(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
id := c.Param("id")
|
||||
|
||||
wid, ok := workloadIDParam(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
var body struct {
|
||||
Action string `json:"action"`
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
if err := c.ShouldBindJSON(&body); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
|
||||
return
|
||||
}
|
||||
switch body.Action {
|
||||
case models.WorkloadStart, models.WorkloadStop, models.WorkloadRestart:
|
||||
default:
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "action must be start, stop or restart"})
|
||||
return
|
||||
}
|
||||
if body.Kind != models.WorkloadContainer && body.Kind != models.WorkloadUnit {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "kind must be container or unit"})
|
||||
return
|
||||
}
|
||||
|
||||
s, err := services.GetServer(instanceID, id)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
|
||||
return
|
||||
}
|
||||
|
||||
err = services.DispatchControlWorkload(s.ServerID, body.Kind, wid, body.Action)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, services.ErrAgentNotConnected):
|
||||
c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error()})
|
||||
case services.IsWorkloadProtected(err):
|
||||
// Nothing failed — the agent refused, which is the design. 409, not
|
||||
// 500, and the reason is carried through.
|
||||
c.JSON(http.StatusConflict, gin.H{"error": err.Error()})
|
||||
default:
|
||||
c.JSON(http.StatusBadGateway, gin.H{"error": err.Error()})
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
services.LogEvent(instanceID, "workload."+body.Action, actorFromCtx(c), s.ServerID, "",
|
||||
fmt.Sprintf("%s %s %s on %s", body.Action, body.Kind, wid, s.Hostname))
|
||||
c.JSON(http.StatusOK, gin.H{"message": body.Action + " ok"})
|
||||
}
|
||||
|
||||
func getWorkloadLogs(c *gin.Context) {
|
||||
instanceID := auth.InstanceID(c)
|
||||
id := c.Param("id")
|
||||
|
||||
wid, ok := workloadIDParam(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
kind := c.DefaultQuery("kind", models.WorkloadContainer)
|
||||
if kind != models.WorkloadContainer && kind != models.WorkloadUnit {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "kind must be container or unit"})
|
||||
return
|
||||
}
|
||||
|
||||
// Clamped rather than refused: a client asking for more than the cap gets
|
||||
// the cap, which is what it would have got anyway.
|
||||
tail, _ := strconv.Atoi(c.Query("tail"))
|
||||
if tail <= 0 || tail > services.MaxWorkloadLogLines {
|
||||
tail = services.MaxWorkloadLogLines
|
||||
}
|
||||
|
||||
s, err := services.GetServer(instanceID, id)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "server not found"})
|
||||
return
|
||||
}
|
||||
|
||||
text, truncated, err := services.DispatchWorkloadLogs(s.ServerID, kind, wid, tail)
|
||||
if err != nil {
|
||||
if errors.Is(err, services.ErrAgentNotConnected) {
|
||||
c.JSON(http.StatusServiceUnavailable, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusBadGateway, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// Audited because container output is arbitrary and cannot be masked: a
|
||||
// startup banner or a stack trace may carry credentials nobody declared.
|
||||
services.LogEvent(instanceID, "workload.logs_read", actorFromCtx(c), s.ServerID, "",
|
||||
fmt.Sprintf("read %s logs for %s on %s", kind, wid, s.Hostname))
|
||||
|
||||
c.JSON(http.StatusOK, gin.H{"text": text, "truncated": truncated})
|
||||
}
|
||||
|
||||
// listWorkloads answers the fleet-wide question, which is the reason the
|
||||
// snapshot is stored rather than fetched on demand and discarded.
|
||||
func listWorkloads(c *gin.Context) {
|
||||
hits, err := services.SearchWorkloads(auth.InstanceID(c),
|
||||
c.Query("image"), c.Query("stack"), c.Query("state"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, hits)
|
||||
}
|
||||
|
||||
// workloadIDParam decodes :wid. Unit names carry dots and '@', so the client
|
||||
// encodes it and this is where it comes back.
|
||||
func workloadIDParam(c *gin.Context) (string, bool) {
|
||||
raw := c.Param("wid")
|
||||
decoded, err := url.PathUnescape(raw)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid workload id"})
|
||||
return "", false
|
||||
}
|
||||
decoded = strings.TrimSpace(decoded)
|
||||
if decoded == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "workload id is required"})
|
||||
return "", false
|
||||
}
|
||||
return decoded, true
|
||||
}
|
||||
@@ -28,6 +28,45 @@ type SyncRequest struct {
|
||||
|
||||
type SyncResponse struct {
|
||||
PublicKeys []string `json:"public_keys"`
|
||||
// CollectPackages tells the agent whether this instance's licence grants
|
||||
// vulnerability scanning. Absent decodes as false, which is the safe
|
||||
// direction: an older server leaves agents collecting nothing.
|
||||
CollectPackages bool `json:"collect_packages,omitempty"`
|
||||
}
|
||||
|
||||
type OSRelease struct {
|
||||
Family string `json:"family"`
|
||||
// VersionId is not optional: Ubuntu 22.04 and 24.04 publish different fixed
|
||||
// versions for the same CVE, so a scan without it is guesswork.
|
||||
VersionId string `json:"version_id"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Epoch int32 `json:"epoch,omitempty"`
|
||||
Arch string `json:"arch,omitempty"`
|
||||
// SourceName is what the Debian and Ubuntu feeds are keyed on: one advisory
|
||||
// against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
SourceName string `json:"source_name,omitempty"`
|
||||
}
|
||||
|
||||
// ReportPackagesRequest carries a server's installed package set.
|
||||
//
|
||||
// The agent calls twice at most: first with Packages empty, offering only the
|
||||
// hash. If the server already holds it, NeedFull is false and the ~150KB body
|
||||
// is never sent.
|
||||
type ReportPackagesRequest struct {
|
||||
ServerId string `json:"server_id"`
|
||||
AgentToken string `json:"agent_token"`
|
||||
Hash string `json:"hash"`
|
||||
Os OSRelease `json:"os"`
|
||||
Packages []InstalledPackage `json:"packages,omitempty"`
|
||||
}
|
||||
|
||||
type ReportPackagesResponse struct {
|
||||
NeedFull bool `json:"need_full"`
|
||||
}
|
||||
|
||||
type UploadKeyRequest struct {
|
||||
@@ -159,6 +198,10 @@ type ServerCommand struct {
|
||||
CleanupWorkspace *CleanupWorkspaceCmd `json:"cleanup_workspace,omitempty"`
|
||||
OpenProxy *OpenProxyCmd `json:"open_proxy,omitempty"`
|
||||
Ping *PingCmd `json:"ping,omitempty"`
|
||||
|
||||
RefreshWorkloads *RefreshWorkloadsCmd `json:"refresh_workloads,omitempty"`
|
||||
ControlWorkload *ControlWorkloadCmd `json:"control_workload,omitempty"`
|
||||
WorkloadLogs *WorkloadLogsCmd `json:"workload_logs,omitempty"`
|
||||
}
|
||||
|
||||
// PingCmd is a server-originated liveness beat. It carries nothing and expects
|
||||
@@ -194,6 +237,8 @@ type AgentMessage struct {
|
||||
Result *CommandResult `json:"result,omitempty"`
|
||||
StepResult *StepResult `json:"step_result,omitempty"`
|
||||
StepOutput *StepOutputChunk `json:"step_output,omitempty"`
|
||||
|
||||
WorkloadLogsResult *WorkloadLogsResult `json:"workload_logs_result,omitempty"`
|
||||
}
|
||||
|
||||
type AgentReady struct{}
|
||||
@@ -326,6 +371,8 @@ type VantageServer interface {
|
||||
SyncKeys(context.Context, *SyncRequest) (*SyncResponse, error)
|
||||
UploadGeneratedKey(context.Context, *UploadKeyRequest) (*UploadKeyResponse, error)
|
||||
ReportUpdates(context.Context, *ReportUpdatesRequest) (*ReportUpdatesResponse, error)
|
||||
ReportPackages(context.Context, *ReportPackagesRequest) (*ReportPackagesResponse, error)
|
||||
ReportWorkloads(context.Context, *ReportWorkloadsRequest) (*ReportWorkloadsResponse, error)
|
||||
ReportInventory(context.Context, *InventoryReport) (*InventoryReportResponse, error)
|
||||
SyncMonitors(context.Context, *SyncMonitorsRequest) (*SyncMonitorsResponse, error)
|
||||
ReportChecks(context.Context, *ReportChecksRequest) (*ReportChecksResponse, error)
|
||||
@@ -351,6 +398,10 @@ func (UnimplementedVantageServer) ReportUpdates(context.Context, *ReportUpdatesR
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportUpdates not implemented")
|
||||
}
|
||||
|
||||
func (UnimplementedVantageServer) ReportPackages(context.Context, *ReportPackagesRequest) (*ReportPackagesResponse, error) {
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportPackages not implemented")
|
||||
}
|
||||
|
||||
func (UnimplementedVantageServer) ReportInventory(context.Context, *InventoryReport) (*InventoryReportResponse, error) {
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportInventory not implemented")
|
||||
}
|
||||
@@ -376,6 +427,8 @@ type VantageClient interface {
|
||||
SyncKeys(ctx context.Context, in *SyncRequest, opts ...grpc.CallOption) (*SyncResponse, error)
|
||||
UploadGeneratedKey(ctx context.Context, in *UploadKeyRequest, opts ...grpc.CallOption) (*UploadKeyResponse, error)
|
||||
ReportUpdates(ctx context.Context, in *ReportUpdatesRequest, opts ...grpc.CallOption) (*ReportUpdatesResponse, error)
|
||||
ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error)
|
||||
ReportWorkloads(ctx context.Context, in *ReportWorkloadsRequest, opts ...grpc.CallOption) (*ReportWorkloadsResponse, error)
|
||||
ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error)
|
||||
SyncMonitors(ctx context.Context, in *SyncMonitorsRequest, opts ...grpc.CallOption) (*SyncMonitorsResponse, error)
|
||||
ReportChecks(ctx context.Context, in *ReportChecksRequest, opts ...grpc.CallOption) (*ReportChecksResponse, error)
|
||||
@@ -423,6 +476,14 @@ func (c *keyManagerClient) ReportUpdates(ctx context.Context, in *ReportUpdatesR
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportPackages(ctx context.Context, in *ReportPackagesRequest, opts ...grpc.CallOption) (*ReportPackagesResponse, error) {
|
||||
out := new(ReportPackagesResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportPackages", in, out, opts...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportInventory(ctx context.Context, in *InventoryReport, opts ...grpc.CallOption) (*InventoryReportResponse, error) {
|
||||
out := new(InventoryReportResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportInventory", in, out, opts...); err != nil {
|
||||
@@ -475,6 +536,8 @@ var Vantage_ServiceDesc = grpc.ServiceDesc{
|
||||
{MethodName: "SyncKeys", Handler: _Vantage_SyncKeys_Handler},
|
||||
{MethodName: "UploadGeneratedKey", Handler: _Vantage_UploadGeneratedKey_Handler},
|
||||
{MethodName: "ReportUpdates", Handler: _Vantage_ReportUpdates_Handler},
|
||||
{MethodName: "ReportPackages", Handler: _Vantage_ReportPackages_Handler},
|
||||
{MethodName: "ReportWorkloads", Handler: _Vantage_ReportWorkloads_Handler},
|
||||
{MethodName: "ReportInventory", Handler: _Vantage_ReportInventory_Handler},
|
||||
{MethodName: "SyncMonitors", Handler: _Vantage_SyncMonitors_Handler},
|
||||
{MethodName: "ReportChecks", Handler: _Vantage_ReportChecks_Handler},
|
||||
@@ -556,6 +619,21 @@ func _Vantage_ReportUpdates_Handler(srv interface{}, ctx context.Context, dec fu
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _Vantage_ReportPackages_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(ReportPackagesRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(VantageServer).ReportPackages(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{Server: srv, FullMethod: "/vantage.v1.Vantage/ReportPackages"}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(VantageServer).ReportPackages(ctx, req.(*ReportPackagesRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _Vantage_ReportInventory_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(InventoryReport)
|
||||
if err := dec(in); err != nil {
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
package pb
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
// Workload registry messages. Hand-written like the rest of this package: the
|
||||
// .proto is the contract, this file is the Go side of it, and the two must be
|
||||
// changed together. The agent module carries the same declarations.
|
||||
|
||||
// Workload is one container or one systemd unit.
|
||||
type Workload struct {
|
||||
Kind string `json:"kind"`
|
||||
Id string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
State string `json:"state"`
|
||||
Health string `json:"health,omitempty"`
|
||||
Image string `json:"image,omitempty"`
|
||||
Stack string `json:"stack,omitempty"`
|
||||
Ports []string `json:"ports,omitempty"`
|
||||
Restarts int32 `json:"restarts,omitempty"`
|
||||
StartedAt string `json:"started_at,omitempty"` // RFC3339, empty when not running
|
||||
Protected bool `json:"protected,omitempty"`
|
||||
}
|
||||
|
||||
// ReportWorkloadsRequest carries what a server is running.
|
||||
//
|
||||
// Offer-then-send, the same handshake as ReportPackages: the agent calls once
|
||||
// with Workloads empty, and resends with the body only if NeedFull is set.
|
||||
type ReportWorkloadsRequest struct {
|
||||
ServerId string `json:"server_id"`
|
||||
AgentToken string `json:"agent_token"`
|
||||
Hash string `json:"hash"`
|
||||
DockerOk bool `json:"docker_ok"`
|
||||
DockerError string `json:"docker_error,omitempty"`
|
||||
SystemdOk bool `json:"systemd_ok"`
|
||||
SystemdError string `json:"systemd_error,omitempty"`
|
||||
Workloads []Workload `json:"workloads,omitempty"` // empty on the offer call
|
||||
// Full marks the second call. It is not inferred from an empty Workloads
|
||||
// slice: a host running nothing sends an empty list as its full report.
|
||||
Full bool `json:"full,omitempty"`
|
||||
}
|
||||
|
||||
type ReportWorkloadsResponse struct {
|
||||
NeedFull bool `json:"need_full"`
|
||||
}
|
||||
|
||||
// RefreshWorkloadsCmd carries no payload back. It makes the agent report
|
||||
// immediately through ReportWorkloads, so there is exactly one writer for the
|
||||
// server_workloads collection rather than two arriving by different routes.
|
||||
type RefreshWorkloadsCmd struct{}
|
||||
|
||||
type ControlWorkloadCmd struct {
|
||||
Kind string `json:"kind"`
|
||||
Id string `json:"id"`
|
||||
Action string `json:"action"` // start | stop | restart
|
||||
}
|
||||
|
||||
type WorkloadLogsCmd struct {
|
||||
Kind string `json:"kind"`
|
||||
Id string `json:"id"`
|
||||
Tail int32 `json:"tail,omitempty"`
|
||||
}
|
||||
|
||||
type WorkloadLogsResult struct {
|
||||
CommandId string `json:"command_id"`
|
||||
Text string `json:"text,omitempty"`
|
||||
Truncated bool `json:"truncated,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
func (UnimplementedVantageServer) ReportWorkloads(context.Context, *ReportWorkloadsRequest) (*ReportWorkloadsResponse, error) {
|
||||
return nil, status.Errorf(codes.Unimplemented, "method ReportWorkloads not implemented")
|
||||
}
|
||||
|
||||
func (c *keyManagerClient) ReportWorkloads(ctx context.Context, in *ReportWorkloadsRequest, opts ...grpc.CallOption) (*ReportWorkloadsResponse, error) {
|
||||
out := new(ReportWorkloadsResponse)
|
||||
if err := c.cc.Invoke(ctx, "/vantage.v1.Vantage/ReportWorkloads", in, out, opts...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func _Vantage_ReportWorkloads_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(ReportWorkloadsRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(VantageServer).ReportWorkloads(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{Server: srv, FullMethod: "/vantage.v1.Vantage/ReportWorkloads"}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(VantageServer).ReportWorkloads(ctx, req.(*ReportWorkloadsRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
@@ -63,7 +63,13 @@ func (s *vantageServer) SyncKeys(ctx context.Context, req *pb.SyncRequest) (*pb.
|
||||
return nil, status.Errorf(codes.Internal, "failed to build authorized keys: %v", err)
|
||||
}
|
||||
|
||||
return &pb.SyncResponse{PublicKeys: keys}, nil
|
||||
// Carried on the 30s key poll rather than its own RPC: the agent needs it
|
||||
// before its hourly package report, and this is the only message it already
|
||||
// receives that often.
|
||||
return &pb.SyncResponse{
|
||||
PublicKeys: keys,
|
||||
CollectPackages: services.VulnScanningEnabled(srv.InstanceID),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *vantageServer) UploadGeneratedKey(ctx context.Context, req *pb.UploadKeyRequest) (*pb.UploadKeyResponse, error) {
|
||||
@@ -104,6 +110,123 @@ func (s *vantageServer) ReportUpdates(ctx context.Context, req *pb.ReportUpdates
|
||||
return &pb.ReportUpdatesResponse{}, nil
|
||||
}
|
||||
|
||||
// ReportPackages stores a server's installed package set.
|
||||
//
|
||||
// It does NOT match against the vulnerability database. Matching happens in
|
||||
// vulnsched, on the leader: every replica would otherwise need the ~50MB
|
||||
// database resident, and a database refresh would have N replicas racing to
|
||||
// rescan the same fleet and sending N digests to the customer.
|
||||
func (s *vantageServer) ReportPackages(ctx context.Context, req *pb.ReportPackagesRequest) (*pb.ReportPackagesResponse, error) {
|
||||
srv, err := services.ValidateAgentToken(req.ServerId, req.AgentToken)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Unauthenticated, "invalid agent token")
|
||||
}
|
||||
|
||||
// The agent's collect_packages flag is an optimisation; this is the
|
||||
// boundary. An agent that ignores the flag still stores nothing.
|
||||
if !services.VulnScanningEnabled(srv.InstanceID) {
|
||||
return &pb.ReportPackagesResponse{NeedFull: false}, nil
|
||||
}
|
||||
|
||||
// The offer call: a hash and no packages. Answering NeedFull=false here is
|
||||
// what saves the ~150KB body on the overwhelming majority of reports.
|
||||
if len(req.Packages) == 0 {
|
||||
known, err := services.HasPackageHash(srv.InstanceID, srv.ServerID, req.Hash)
|
||||
if err != nil {
|
||||
log.Printf("package hash lookup for %s: %v", srv.ServerID, err)
|
||||
return nil, status.Errorf(codes.Internal, "package hash lookup failed")
|
||||
}
|
||||
return &pb.ReportPackagesResponse{NeedFull: !known}, nil
|
||||
}
|
||||
|
||||
pkgs := make([]models.InstalledPackage, len(req.Packages))
|
||||
for i, p := range req.Packages {
|
||||
pkgs[i] = models.InstalledPackage{
|
||||
Name: p.Name,
|
||||
Version: p.Version,
|
||||
Epoch: int(p.Epoch),
|
||||
Arch: p.Arch,
|
||||
SourceName: p.SourceName,
|
||||
}
|
||||
}
|
||||
|
||||
os := models.OSRelease{
|
||||
Family: req.Os.Family,
|
||||
VersionID: req.Os.VersionId,
|
||||
Arch: req.Os.Arch,
|
||||
}
|
||||
|
||||
if err := services.StorePackages(srv.InstanceID, srv.ServerID, os, req.Hash, pkgs); err != nil {
|
||||
log.Printf("store packages for %s: %v", srv.ServerID, err)
|
||||
return nil, status.Errorf(codes.Internal, "failed to store packages")
|
||||
}
|
||||
return &pb.ReportPackagesResponse{NeedFull: false}, nil
|
||||
}
|
||||
|
||||
// ReportWorkloads stores what a server is running.
|
||||
//
|
||||
// It is not gated by licence: the workload registry reads as core fleet
|
||||
// management rather than a premium add-on. If that ever changes, the check
|
||||
// belongs here — gating collection, not display — for the same reason it does
|
||||
// in ReportPackages.
|
||||
func (s *vantageServer) ReportWorkloads(ctx context.Context, req *pb.ReportWorkloadsRequest) (*pb.ReportWorkloadsResponse, error) {
|
||||
srv, err := services.ValidateAgentToken(req.ServerId, req.AgentToken)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Unauthenticated, "invalid agent token")
|
||||
}
|
||||
|
||||
// The offer call: a hash and no body. Answering NeedFull=false here is what
|
||||
// keeps an unchanged 60-second report to one small message.
|
||||
//
|
||||
// The offer is identified by Full, not by an empty Workloads slice: a host
|
||||
// genuinely running nothing sends an empty list as its FULL report, and
|
||||
// inferring the offer from emptiness would leave that host answering
|
||||
// NeedFull=true forever and never storing anything.
|
||||
if !req.Full {
|
||||
known, err := services.HasWorkloadHash(srv.InstanceID, srv.ServerID, req.Hash)
|
||||
if err != nil {
|
||||
log.Printf("workload hash lookup for %s: %v", srv.ServerID, err)
|
||||
return nil, status.Errorf(codes.Internal, "workload hash lookup failed")
|
||||
}
|
||||
return &pb.ReportWorkloadsResponse{NeedFull: !known}, nil
|
||||
}
|
||||
|
||||
wls := make([]models.Workload, len(req.Workloads))
|
||||
for i, w := range req.Workloads {
|
||||
wls[i] = models.Workload{
|
||||
Kind: w.Kind,
|
||||
ID: w.Id,
|
||||
Name: w.Name,
|
||||
State: w.State,
|
||||
Health: w.Health,
|
||||
Image: w.Image,
|
||||
Stack: w.Stack,
|
||||
Ports: w.Ports,
|
||||
Restarts: int(w.Restarts),
|
||||
Protected: w.Protected,
|
||||
}
|
||||
if w.StartedAt != "" {
|
||||
if t, err := time.Parse(time.RFC3339, w.StartedAt); err == nil {
|
||||
wls[i].StartedAt = t
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := storeWorkloadReport(srv.InstanceID, srv.ServerID, req, wls); err != nil {
|
||||
log.Printf("store workloads for %s: %v", srv.ServerID, err)
|
||||
return nil, status.Errorf(codes.Internal, "failed to store workloads")
|
||||
}
|
||||
return &pb.ReportWorkloadsResponse{NeedFull: false}, nil
|
||||
}
|
||||
|
||||
func storeWorkloadReport(instanceID, serverID string, req *pb.ReportWorkloadsRequest, wls []models.Workload) error {
|
||||
if wls == nil {
|
||||
wls = []models.Workload{}
|
||||
}
|
||||
return services.StoreWorkloads(instanceID, serverID, req.Hash, wls,
|
||||
req.DockerOk, req.DockerError, req.SystemdOk, req.SystemdError)
|
||||
}
|
||||
|
||||
func (s *vantageServer) ReportInventory(ctx context.Context, req *pb.InventoryReport) (*pb.InventoryReportResponse, error) {
|
||||
srv, err := services.ValidateAgentToken(req.ServerId, req.AgentToken)
|
||||
if err != nil {
|
||||
@@ -198,6 +321,13 @@ func (s *vantageServer) CommandStream(stream pb.Vantage_CommandStreamServer) err
|
||||
if m.Result != nil {
|
||||
r := m.Result
|
||||
log.Printf("agent %s cmd %s: success=%v %s", srv.ServerID, r.CommandId, r.Success, r.Message)
|
||||
// Republished so a control action waiting on another pod sees
|
||||
// it. Publishing with no subscriber is a no-op, so this is safe
|
||||
// for every command result rather than only the awaited ones.
|
||||
services.WorkloadResults.DeliverCommand(r)
|
||||
}
|
||||
if m.WorkloadLogsResult != nil {
|
||||
services.WorkloadResults.Deliver(m.WorkloadLogsResult)
|
||||
}
|
||||
if m.StepResult != nil {
|
||||
services.StepResults.Deliver(m.StepResult)
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
)
|
||||
|
||||
// Scan status values for ServerPackages.
|
||||
const (
|
||||
ScanStatusOK = "ok"
|
||||
ScanStatusUnsupported = "unsupported"
|
||||
)
|
||||
|
||||
type OSRelease struct {
|
||||
Family string `bson:"family" json:"family"`
|
||||
VersionID string `bson:"version_id" json:"version_id"`
|
||||
Arch string `bson:"arch" json:"arch"`
|
||||
}
|
||||
|
||||
type InstalledPackage struct {
|
||||
Name string `bson:"name" json:"name"`
|
||||
Version string `bson:"version" json:"version"`
|
||||
Epoch int `bson:"epoch,omitempty" json:"epoch,omitempty"`
|
||||
Arch string `bson:"arch" json:"arch"`
|
||||
// SourceName is what the Debian and Ubuntu feeds are keyed on. One advisory
|
||||
// against "openssl" covers the binaries libssl3, openssl and libssl-dev;
|
||||
// matching on binary name alone finds one of the three.
|
||||
SourceName string `bson:"source_name,omitempty" json:"source_name,omitempty"`
|
||||
}
|
||||
|
||||
// ServerPackages holds one server's whole package set in ONE document rather
|
||||
// than one document per package. The hash has already established that
|
||||
// something changed, so a report is a single atomic upsert with no delta logic
|
||||
// to get wrong. A typical Linux host is ~2000 packages and ~150KB, comfortably
|
||||
// inside the 16MB document limit.
|
||||
type ServerPackages struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
OS OSRelease `bson:"os" json:"os"`
|
||||
Hash string `bson:"hash" json:"hash"`
|
||||
Packages []InstalledPackage `bson:"packages" json:"packages"`
|
||||
CollectedAt time.Time `bson:"collected_at" json:"collected_at"`
|
||||
ScanPending bool `bson:"scan_pending" json:"scan_pending"`
|
||||
ScannedAt time.Time `bson:"scanned_at,omitempty" json:"scanned_at,omitempty"`
|
||||
// Status distinguishes a scanned host from one whose distribution we hold
|
||||
// no feed for. Reporting zero findings for an unsupported distribution is
|
||||
// indistinguishable from reporting a clean host, and one of those is a lie.
|
||||
Status string `bson:"status" json:"status"`
|
||||
DBVersion int `bson:"db_version" json:"db_version"`
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
)
|
||||
|
||||
// Finding states.
|
||||
const (
|
||||
FindingOpen = "open"
|
||||
FindingFixed = "fixed"
|
||||
FindingAccepted = "accepted"
|
||||
)
|
||||
|
||||
// Severities. Lowercase and fixed; SeverityRank orders them.
|
||||
const (
|
||||
SeverityUnknown = "unknown"
|
||||
SeverityLow = "low"
|
||||
SeverityMedium = "medium"
|
||||
SeverityHigh = "high"
|
||||
SeverityCritical = "critical"
|
||||
)
|
||||
|
||||
// SeverityRank orders severities for threshold comparisons. An unrecognised
|
||||
// value ranks lowest rather than panicking: severity comes from a third-party
|
||||
// feed, and an unexpected string must not stop a scan.
|
||||
func SeverityRank(s string) int {
|
||||
switch s {
|
||||
case SeverityCritical:
|
||||
return 4
|
||||
case SeverityHigh:
|
||||
return 3
|
||||
case SeverityMedium:
|
||||
return 2
|
||||
case SeverityLow:
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// Acceptance records a decision someone will be asked to justify, so who, why
|
||||
// and until when all live on the document as well as in the audit log.
|
||||
type Acceptance struct {
|
||||
By string `bson:"by" json:"by"`
|
||||
Reason string `bson:"reason" json:"reason"`
|
||||
Until time.Time `bson:"until" json:"until"`
|
||||
At time.Time `bson:"at" json:"at"`
|
||||
}
|
||||
|
||||
// VulnFinding is one vulnerable package on one server.
|
||||
//
|
||||
// Findings are never deleted when a package is patched: the state moves to
|
||||
// "fixed" with FixedAt stamped, which is what keeps "what did we remediate last
|
||||
// quarter" answerable.
|
||||
type VulnFinding struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"id"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
|
||||
CVEID string `bson:"cve_id" json:"cve_id"`
|
||||
PackageName string `bson:"package_name" json:"package_name"`
|
||||
Installed string `bson:"installed_version" json:"installed_version"`
|
||||
// FixedIn empty means no vendor fix has been published. That is a real and
|
||||
// common state and must never be conflated with "not vulnerable" — it is
|
||||
// the finding most in need of acceptance, since there is nothing to patch.
|
||||
FixedIn string `bson:"fixed_in,omitempty" json:"fixed_in,omitempty"`
|
||||
Severity string `bson:"severity" json:"severity"`
|
||||
CVSSScore float64 `bson:"cvss_score,omitempty" json:"cvss_score,omitempty"`
|
||||
Title string `bson:"title,omitempty" json:"title,omitempty"`
|
||||
References []string `bson:"references,omitempty" json:"references,omitempty"`
|
||||
|
||||
State string `bson:"state" json:"state"`
|
||||
FirstSeen time.Time `bson:"first_seen" json:"first_seen"`
|
||||
LastSeen time.Time `bson:"last_seen" json:"last_seen"`
|
||||
FixedAt *time.Time `bson:"fixed_at,omitempty" json:"fixed_at,omitempty"`
|
||||
Accepted *Acceptance `bson:"accepted,omitempty" json:"accepted,omitempty"`
|
||||
}
|
||||
|
||||
// VulnDBMeta is a singleton and deliberately carries no instance_id: the
|
||||
// vulnerability database is a property of the deployment, not of a tenant.
|
||||
// Same reasoning as the migrations collection, and the reason vulndb_meta is
|
||||
// absent from ScopedCollections.
|
||||
type VulnDBMeta struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
DBVersion int `bson:"db_version" json:"db_version"`
|
||||
PulledAt time.Time `bson:"pulled_at" json:"pulled_at"`
|
||||
LastFullScanAt time.Time `bson:"last_full_scan_at,omitempty" json:"last_full_scan_at,omitempty"`
|
||||
LastError string `bson:"last_error,omitempty" json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// VulnAlertRule routes newly opened findings to notification channels.
|
||||
//
|
||||
// Tags resolve through services.ResolveTargets rather than a second matcher:
|
||||
// that function is already the single answer to which servers a selector
|
||||
// touches, and a rule disagreeing with a workflow about what env:prod means
|
||||
// would be worse than having no filter.
|
||||
type VulnAlertRule struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"id"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
Name string `bson:"name" json:"name"`
|
||||
Enabled bool `bson:"enabled" json:"enabled"`
|
||||
MinSeverity string `bson:"min_severity" json:"min_severity"`
|
||||
Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"`
|
||||
ChannelIDs []string `bson:"channel_ids" json:"channel_ids"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
)
|
||||
|
||||
// Workload kinds.
|
||||
const (
|
||||
WorkloadContainer = "container"
|
||||
WorkloadUnit = "unit"
|
||||
)
|
||||
|
||||
// Control actions.
|
||||
const (
|
||||
WorkloadStart = "start"
|
||||
WorkloadStop = "stop"
|
||||
WorkloadRestart = "restart"
|
||||
)
|
||||
|
||||
// Workload is one container or one systemd unit.
|
||||
type Workload struct {
|
||||
Kind string `bson:"kind" json:"kind"` // container | unit
|
||||
ID string `bson:"id" json:"id"` // container id, or unit name
|
||||
Name string `bson:"name" json:"name"`
|
||||
|
||||
// State is deliberately NOT collapsed into a shared vocabulary across the
|
||||
// two kinds. Containers report running/exited/paused/restarting/created;
|
||||
// units report active/inactive/failed/activating. A failed unit and an
|
||||
// exited container mean different things, and flattening them loses the
|
||||
// distinction the operator needs.
|
||||
State string `bson:"state" json:"state"`
|
||||
Health string `bson:"health,omitempty" json:"health,omitempty"`
|
||||
|
||||
Image string `bson:"image,omitempty" json:"image,omitempty"`
|
||||
Stack string `bson:"stack,omitempty" json:"stack,omitempty"` // compose project label
|
||||
Ports []string `bson:"ports,omitempty" json:"ports,omitempty"`
|
||||
|
||||
Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"`
|
||||
StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"`
|
||||
|
||||
// Protected is computed agent-side and reported so the UI can render the
|
||||
// action disabled with a reason rather than offering a button whose refusal
|
||||
// is already known. The field is the courtesy; the agent's own check is the
|
||||
// boundary.
|
||||
Protected bool `bson:"protected" json:"protected"`
|
||||
}
|
||||
|
||||
// ServerWorkloads holds one server's whole workload list in ONE document.
|
||||
type ServerWorkloads struct {
|
||||
ID bson.ObjectID `bson:"_id,omitempty" json:"-"`
|
||||
InstanceID string `bson:"instance_id" json:"-"`
|
||||
ServerID string `bson:"server_id" json:"server_id"`
|
||||
Hash string `bson:"hash" json:"hash"`
|
||||
Workloads []Workload `bson:"workloads" json:"workloads"`
|
||||
CollectedAt time.Time `bson:"collected_at" json:"collected_at"`
|
||||
|
||||
// A host with no Docker and a host with Docker running nothing both produce
|
||||
// an empty list. One should read "not in use here", the other "nothing
|
||||
// running", and only the second deserves any alarm.
|
||||
//
|
||||
// The error strings separate a third case the booleans cannot: installed
|
||||
// with the daemon down. "Not installed" and "installed but not responding"
|
||||
// are different problems with different fixes.
|
||||
DockerOK bool `bson:"docker_ok" json:"docker_ok"`
|
||||
DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"`
|
||||
SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"`
|
||||
SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"`
|
||||
}
|
||||
@@ -26,7 +26,11 @@ func (e Event) title() string {
|
||||
verb = "is DOWN"
|
||||
}
|
||||
var s string
|
||||
if e.Type == TypeServer {
|
||||
if e.Type == TypeVuln {
|
||||
// A digest is not a transition. MonitorName already carries the whole
|
||||
// headline ("12 new critical across 4 servers"), so no verb applies.
|
||||
s = fmt.Sprintf("[Vantage] %s", e.MonitorName)
|
||||
} else if e.Type == TypeServer {
|
||||
if e.NewStatus == models.StatusDown {
|
||||
verb = "went offline"
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/shared/mail"
|
||||
)
|
||||
|
||||
// TypeVuln marks an event whose subject is a batch of new vulnerability
|
||||
// findings rather than a state transition. It exists so title() does not
|
||||
// describe a digest as something going "DOWN".
|
||||
const TypeVuln = "vulnerability"
|
||||
|
||||
// VulnDigest is one batch of newly opened findings, ready to send.
|
||||
//
|
||||
// One per rule per scan, never one per finding: a database refresh can open
|
||||
// several hundred at once, and a message each would rate-limit the webhook or
|
||||
// get the channel muted — either way the alerts stop being read.
|
||||
type VulnDigest struct {
|
||||
InstanceName string
|
||||
RuleName string
|
||||
Summary string
|
||||
TopSeverity string
|
||||
Count int
|
||||
Rows []mail.VulnDigestRow
|
||||
More int
|
||||
DBAge string
|
||||
}
|
||||
|
||||
// DispatchVulnDigest delivers a digest over one channel.
|
||||
//
|
||||
// SMTP gets its own template so a vulnerability digest does not arrive dressed
|
||||
// as a monitor alert. The other four transports carry short text, so they reuse
|
||||
// the existing Event path rather than growing a second payload shape per
|
||||
// transport.
|
||||
func DispatchVulnDigest(ch models.NotificationChannel, d VulnDigest) error {
|
||||
if ch.Type == models.ChannelSMTP {
|
||||
to := ch.Config["to"]
|
||||
sender := mail.Sender{
|
||||
Host: ch.Config["host"],
|
||||
Port: ch.Config["port"],
|
||||
From: ch.Config["from"],
|
||||
Username: ch.Config["username"],
|
||||
Password: ch.Config["password"],
|
||||
}
|
||||
if !sender.Enabled() || sender.Port == "" || to == "" {
|
||||
return fmt.Errorf("smtp: missing host/port/from/to")
|
||||
}
|
||||
return sender.SendVulnDigest(to, mail.VulnDigest{
|
||||
InstanceName: d.InstanceName,
|
||||
Count: d.Count,
|
||||
TopSeverity: d.TopSeverity,
|
||||
Summary: d.Summary,
|
||||
Rows: d.Rows,
|
||||
More: d.More,
|
||||
DBAge: d.DBAge,
|
||||
})
|
||||
}
|
||||
|
||||
return Dispatch(ch, Event{
|
||||
MonitorName: d.Summary,
|
||||
Type: TypeVuln,
|
||||
Message: vulnLines(d),
|
||||
})
|
||||
}
|
||||
|
||||
// vulnLines renders the finding list for the text-only transports, capped by
|
||||
// whatever the caller already put in Rows.
|
||||
func vulnLines(d VulnDigest) string {
|
||||
var s string
|
||||
for _, r := range d.Rows {
|
||||
fix := "no fix published"
|
||||
if r.FixedIn != "" {
|
||||
fix = "fixed in " + r.FixedIn
|
||||
}
|
||||
s += fmt.Sprintf("\n• %s (%s) — %s on %s, %s", r.CVEID, r.Severity, r.PackageName, r.ServerName, fix)
|
||||
}
|
||||
if d.More > 0 {
|
||||
s += fmt.Sprintf("\n…and %d more.", d.More)
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,420 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// FindingDiff is what one server's scan changes.
|
||||
type FindingDiff struct {
|
||||
Upserts []models.VulnFinding
|
||||
FixedIDs []bson.ObjectID
|
||||
ReopenIDs []bson.ObjectID
|
||||
// NewlyOpened is what the digest reports: findings that were not open
|
||||
// before this scan. A finding that was already open must not re-alert every
|
||||
// tick, or the digest becomes noise and stops being read.
|
||||
NewlyOpened []models.VulnFinding
|
||||
}
|
||||
|
||||
func findingKey(cveID, pkg string) string { return cveID + "\x00" + pkg }
|
||||
|
||||
// DiffFindings computes the state changes for one server's scan.
|
||||
//
|
||||
// Pure by design: no database, no clock of its own. The ordering below is
|
||||
// load-bearing — see the comment above the second loop.
|
||||
func DiffFindings(existing []models.VulnFinding, results []vulndb.Result, now time.Time) FindingDiff {
|
||||
var d FindingDiff
|
||||
|
||||
byKey := make(map[string]models.VulnFinding, len(existing))
|
||||
for _, f := range existing {
|
||||
byKey[findingKey(f.CVEID, f.PackageName)] = f
|
||||
}
|
||||
|
||||
seen := make(map[string]bool, len(results))
|
||||
for _, r := range results {
|
||||
key := findingKey(r.CVEID, r.PackageName)
|
||||
seen[key] = true
|
||||
|
||||
prev, had := byKey[key]
|
||||
|
||||
f := models.VulnFinding{
|
||||
CVEID: r.CVEID,
|
||||
PackageName: r.PackageName,
|
||||
Installed: r.Installed,
|
||||
FixedIn: r.FixedIn,
|
||||
Severity: r.Severity,
|
||||
State: models.FindingOpen,
|
||||
FirstSeen: now,
|
||||
LastSeen: now,
|
||||
}
|
||||
|
||||
if had {
|
||||
f.ID = prev.ID
|
||||
// Preserved, never overwritten: an upsert that moves first_seen
|
||||
// forward makes every finding look discovered today.
|
||||
f.FirstSeen = prev.FirstSeen
|
||||
|
||||
// A live acceptance survives the scan untouched: it is suppressed
|
||||
// from counts and alerts until its expiry, then reopens on its own.
|
||||
if prev.State == models.FindingAccepted && prev.Accepted != nil {
|
||||
if now.Before(prev.Accepted.Until) {
|
||||
continue
|
||||
}
|
||||
d.ReopenIDs = append(d.ReopenIDs, prev.ID)
|
||||
continue
|
||||
}
|
||||
|
||||
if prev.State != models.FindingOpen {
|
||||
d.NewlyOpened = append(d.NewlyOpened, f)
|
||||
}
|
||||
} else {
|
||||
d.NewlyOpened = append(d.NewlyOpened, f)
|
||||
}
|
||||
|
||||
d.Upserts = append(d.Upserts, f)
|
||||
}
|
||||
|
||||
// Anything we hold that this scan did not produce is fixed. This runs AFTER
|
||||
// the loop above, and the ordering matters: a finding that is both absent
|
||||
// and past its acceptance expiry must settle as fixed rather than reopening
|
||||
// on a package that no longer carries it.
|
||||
for _, f := range existing {
|
||||
if seen[findingKey(f.CVEID, f.PackageName)] {
|
||||
continue
|
||||
}
|
||||
if f.State == models.FindingFixed {
|
||||
continue
|
||||
}
|
||||
d.FixedIDs = append(d.FixedIDs, f.ID)
|
||||
}
|
||||
|
||||
return d
|
||||
}
|
||||
|
||||
// ErrFindingNotFound is returned for a finding that does not exist in this
|
||||
// instance. Callers turn it into a 404 — never a 403, which would confirm the
|
||||
// finding exists in someone else's instance.
|
||||
var ErrFindingNotFound = errors.New("finding not found")
|
||||
|
||||
// FindingFilter narrows a fleet-wide finding query. An empty field is no
|
||||
// filter.
|
||||
type FindingFilter struct {
|
||||
Severity string
|
||||
State string
|
||||
ServerID string
|
||||
Tags map[string]string
|
||||
}
|
||||
|
||||
// ListInstanceFindings returns findings across the whole fleet.
|
||||
func ListInstanceFindings(instanceID string, f FindingFilter) ([]models.VulnFinding, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
filter := bson.M{"instance_id": instanceID}
|
||||
if f.State != "" {
|
||||
filter["state"] = f.State
|
||||
}
|
||||
if f.Severity != "" {
|
||||
filter["severity"] = f.Severity
|
||||
}
|
||||
if f.ServerID != "" {
|
||||
filter["server_id"] = f.ServerID
|
||||
}
|
||||
|
||||
// The tag selector resolves through ResolveTargets, the single answer to
|
||||
// which servers a selector touches. A second matcher here could disagree
|
||||
// with what a workflow means by env:prod.
|
||||
if len(f.Tags) > 0 {
|
||||
servers, err := ResolveTargets(instanceID, nil, f.Tags)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids := make([]string, 0, len(servers))
|
||||
for _, s := range servers {
|
||||
ids = append(ids, s.ServerID)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return []models.VulnFinding{}, nil
|
||||
}
|
||||
filter["server_id"] = bson.M{"$in": ids}
|
||||
}
|
||||
|
||||
cur, err := db.Col("vuln_findings").Find(ctx, filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
out := []models.VulnFinding{}
|
||||
if err := cur.All(ctx, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// CountOpenFindingsBySeverity powers the summary tiles. Accepted findings are
|
||||
// excluded: they are suppressed from counts until their expiry, which is the
|
||||
// whole point of accepting one.
|
||||
func CountOpenFindingsBySeverity(instanceID string) (map[string]int, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cur, err := db.Col("vuln_findings").Aggregate(ctx, []bson.M{
|
||||
{"$match": bson.M{"instance_id": instanceID, "state": models.FindingOpen}},
|
||||
{"$group": bson.M{"_id": "$severity", "n": bson.M{"$sum": 1}}},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var rows []struct {
|
||||
Severity string `bson:"_id"`
|
||||
N int `bson:"n"`
|
||||
}
|
||||
if err := cur.All(ctx, &rows); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
counts := map[string]int{}
|
||||
for _, r := range rows {
|
||||
counts[r.Severity] = r.N
|
||||
}
|
||||
return counts, nil
|
||||
}
|
||||
|
||||
// MarkInstanceForRescan flags every server in an instance for rescanning and
|
||||
// returns how many were flagged.
|
||||
//
|
||||
// It does not scan. vulnsched picks the flags up on its next tick, which keeps
|
||||
// matching on the leader and means this endpoint cannot become a second
|
||||
// scanning path.
|
||||
func MarkInstanceForRescan(instanceID string) (int64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("server_packages").UpdateMany(ctx,
|
||||
bson.M{"instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{"scan_pending": true}},
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return res.ModifiedCount, nil
|
||||
}
|
||||
|
||||
// AcceptFinding suppresses a finding until a date, with a reason.
|
||||
//
|
||||
// The expiry is mandatory at the API layer. A finding reopens on its own when
|
||||
// it passes, which is what stops the accepted list becoming where risk goes to
|
||||
// be forgotten.
|
||||
func AcceptFinding(instanceID, findingID, actor, reason string, until time.Time) (*models.VulnFinding, error) {
|
||||
id, err := bson.ObjectIDFromHex(findingID)
|
||||
if err != nil {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var f models.VulnFinding
|
||||
err = db.Col("vuln_findings").FindOneAndUpdate(ctx,
|
||||
bson.M{"_id": id, "instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{
|
||||
"state": models.FindingAccepted,
|
||||
"accepted": models.Acceptance{
|
||||
By: actor,
|
||||
Reason: reason,
|
||||
Until: until,
|
||||
At: time.Now(),
|
||||
},
|
||||
}},
|
||||
options.FindOneAndUpdate().SetReturnDocument(options.After),
|
||||
).Decode(&f)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &f, nil
|
||||
}
|
||||
|
||||
// UnacceptFinding returns an accepted finding to open before its expiry.
|
||||
func UnacceptFinding(instanceID, findingID string) (*models.VulnFinding, error) {
|
||||
id, err := bson.ObjectIDFromHex(findingID)
|
||||
if err != nil {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var f models.VulnFinding
|
||||
err = db.Col("vuln_findings").FindOneAndUpdate(ctx,
|
||||
bson.M{"_id": id, "instance_id": instanceID},
|
||||
bson.M{
|
||||
"$set": bson.M{"state": models.FindingOpen},
|
||||
"$unset": bson.M{"accepted": ""},
|
||||
},
|
||||
options.FindOneAndUpdate().SetReturnDocument(options.After),
|
||||
).Decode(&f)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, ErrFindingNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &f, nil
|
||||
}
|
||||
|
||||
// ListFindings returns every finding held for one server.
|
||||
func ListFindings(ctx context.Context, instanceID, serverID string) ([]models.VulnFinding, error) {
|
||||
cur, err := db.Col("vuln_findings").Find(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var out []models.VulnFinding
|
||||
if err := cur.All(ctx, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ApplyFindingDiff writes a diff. Thin on purpose — the logic worth reading
|
||||
// twice is all in DiffFindings.
|
||||
func ApplyFindingDiff(ctx context.Context, instanceID, serverID string, d FindingDiff, now time.Time) error {
|
||||
col := db.Col("vuln_findings")
|
||||
|
||||
for _, f := range d.Upserts {
|
||||
_, err := col.UpdateOne(ctx,
|
||||
bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"cve_id": f.CVEID,
|
||||
"package_name": f.PackageName,
|
||||
},
|
||||
bson.M{
|
||||
"$set": bson.M{
|
||||
"installed_version": f.Installed,
|
||||
"fixed_in": f.FixedIn,
|
||||
"severity": f.Severity,
|
||||
"state": models.FindingOpen,
|
||||
"last_seen": now,
|
||||
},
|
||||
// first_seen is written only on insert, so a rescan cannot move
|
||||
// it forward.
|
||||
"$setOnInsert": bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"cve_id": f.CVEID,
|
||||
"package_name": f.PackageName,
|
||||
"first_seen": f.FirstSeen,
|
||||
},
|
||||
"$unset": bson.M{"fixed_at": "", "accepted": ""},
|
||||
},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if len(d.FixedIDs) > 0 {
|
||||
if _, err := col.UpdateMany(ctx,
|
||||
bson.M{"_id": bson.M{"$in": d.FixedIDs}},
|
||||
bson.M{"$set": bson.M{"state": models.FindingFixed, "fixed_at": now}},
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if len(d.ReopenIDs) > 0 {
|
||||
if _, err := col.UpdateMany(ctx,
|
||||
bson.M{"_id": bson.M{"$in": d.ReopenIDs}},
|
||||
bson.M{"$set": bson.M{"state": models.FindingOpen, "last_seen": now}, "$unset": bson.M{"accepted": ""}},
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// StartVulnSweeper deletes old FIXED findings. Open and accepted findings are
|
||||
// never swept at any setting: retention is about history, and an unresolved
|
||||
// vulnerability is not history.
|
||||
func StartVulnSweeper(ctx context.Context) {
|
||||
go func() {
|
||||
ticker := time.NewTicker(6 * time.Hour)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
sweepFixedFindings(ctx)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// defaultVulnRetentionDays is what an unset setting means. A pointer field and
|
||||
// this constant together give absent-means-90 and 0-means-forever, the same
|
||||
// shape as workflow log retention.
|
||||
const defaultVulnRetentionDays = 90
|
||||
|
||||
// sweepFixedFindings deletes fixed findings past each instance's retention.
|
||||
//
|
||||
// Only "fixed" is ever swept. An open or accepted finding is not history, it is
|
||||
// an outstanding decision, and deleting one on a timer would quietly shrink the
|
||||
// fleet's risk picture.
|
||||
func sweepFixedFindings(ctx context.Context) {
|
||||
instanceIDs, err := ListInstanceIDs()
|
||||
if err != nil {
|
||||
log.Printf("vuln sweeper: list instances: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, instanceID := range instanceIDs {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
|
||||
days := defaultVulnRetentionDays
|
||||
if s, err := GetSettings(instanceID); err == nil && s != nil && s.VulnFindingRetentionDays != nil {
|
||||
days = *s.VulnFindingRetentionDays
|
||||
}
|
||||
if days <= 0 {
|
||||
continue // 0 means keep forever
|
||||
}
|
||||
|
||||
cutoff := time.Now().AddDate(0, 0, -days)
|
||||
res, err := db.Col("vuln_findings").DeleteMany(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"state": models.FindingFixed,
|
||||
"fixed_at": bson.M{"$lt": cutoff},
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("vuln sweeper: delete for %s: %v", instanceID, err)
|
||||
continue
|
||||
}
|
||||
if res.DeletedCount > 0 {
|
||||
log.Printf("vuln sweeper: removed %d fixed findings for %s", res.DeletedCount, instanceID)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,10 @@ var ScopedCollections = []string{
|
||||
"console_sessions",
|
||||
"audit_logs",
|
||||
"auth_providers",
|
||||
"server_packages",
|
||||
"vuln_findings",
|
||||
"vuln_alert_rules",
|
||||
"server_workloads",
|
||||
}
|
||||
|
||||
// collectionRenames maps the two collections whose names change. Ordered so the
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/shared/license"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// VulnScanningEnabled reports whether this instance may collect packages.
|
||||
//
|
||||
// It reads the feature by name from shared/license and never switches on tier,
|
||||
// so changing what a tier includes needs no server release. A lapsed licence
|
||||
// collects nothing: there is no point accumulating inventory an instance cannot
|
||||
// act on.
|
||||
func VulnScanningEnabled(instanceID string) bool {
|
||||
st := GetLicenseState(instanceID)
|
||||
return st.Active() && st.Feature(license.FeatureVulnScanning)
|
||||
}
|
||||
|
||||
// HasPackageHash reports whether we already hold this exact package set, which
|
||||
// is what lets the agent skip sending ~150KB it has already sent.
|
||||
func HasPackageHash(instanceID, serverID, hash string) (bool, error) {
|
||||
err := db.Col("server_packages").FindOne(context.Background(), bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"hash": hash,
|
||||
}, options.FindOne().SetProjection(bson.M{"_id": 1})).Err()
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// StorePackages replaces a server's package set and marks it for scanning.
|
||||
//
|
||||
// It deliberately does NOT match against the vulnerability database. Matching
|
||||
// happens in vulnsched, on the leader, for two reasons: every replica would
|
||||
// otherwise need the ~50MB database resident, and a database refresh would have
|
||||
// N replicas racing to rescan the same fleet and sending N digests.
|
||||
func StorePackages(instanceID, serverID string, os models.OSRelease, hash string, pkgs []models.InstalledPackage) error {
|
||||
now := time.Now()
|
||||
_, err := db.Col("server_packages").UpdateOne(context.Background(),
|
||||
bson.M{"instance_id": instanceID, "server_id": serverID},
|
||||
bson.M{"$set": bson.M{
|
||||
"os": os,
|
||||
"hash": hash,
|
||||
"packages": pkgs,
|
||||
"collected_at": now,
|
||||
"scan_pending": true,
|
||||
}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
// ListPackages returns a server's stored package set, or nil when the agent has
|
||||
// not reported yet. A missing document is not an error: an agent that has never
|
||||
// reported is the normal state for the first hour after install.
|
||||
func ListPackages(instanceID, serverID string) (*models.ServerPackages, error) {
|
||||
var sp models.ServerPackages
|
||||
err := db.Col("server_packages").FindOne(context.Background(), bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
}).Decode(&sp)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sp, nil
|
||||
}
|
||||
|
||||
// PackageHit is one server running one package.
|
||||
type PackageHit struct {
|
||||
ServerID string `json:"server_id"`
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
|
||||
// SearchPackages answers "which servers run package X" across the fleet — the
|
||||
// question people actually ask during an incident.
|
||||
//
|
||||
// The Mongo filter narrows to documents containing the name; the second pass is
|
||||
// needed because a multikey match returns the whole document, not the matching
|
||||
// array element.
|
||||
func SearchPackages(instanceID, name string) ([]PackageHit, error) {
|
||||
ctx := context.Background()
|
||||
cur, err := db.Col("server_packages").Find(ctx, bson.M{
|
||||
"instance_id": instanceID,
|
||||
"packages.name": name,
|
||||
}, options.Find().SetProjection(bson.M{"server_id": 1, "packages": 1}))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var docs []models.ServerPackages
|
||||
if err := cur.All(ctx, &docs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
hits := []PackageHit{}
|
||||
for _, d := range docs {
|
||||
for _, p := range d.Packages {
|
||||
if p.Name == name {
|
||||
hits = append(hits, PackageHit{ServerID: d.ServerID, Name: p.Name, Version: p.Version})
|
||||
}
|
||||
}
|
||||
}
|
||||
return hits, nil
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// EnsureVulnIndexes declares the indexes for package inventory and findings.
|
||||
//
|
||||
// It warns rather than being fatal, matching EnsureSecretIndexes and
|
||||
// EnsureWorkflowIndexes: a missing index degrades these queries to a collection
|
||||
// scan, which is no reason to refuse to serve the fleet.
|
||||
func EnsureVulnIndexes() error {
|
||||
ctx := context.Background()
|
||||
|
||||
pkgIdx := []mongo.IndexModel{
|
||||
{
|
||||
Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "server_id", Value: 1}},
|
||||
Options: options.Index().SetUnique(true),
|
||||
},
|
||||
// Multikey, for fleet-wide package search: "who runs openssl 3.0.2?"
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "packages.name", Value: 1}}},
|
||||
// The scheduler's only query. Deliberately unscoped: it sweeps the whole
|
||||
// deployment on the leader, not one tenant.
|
||||
{Keys: bson.D{{Key: "scan_pending", Value: 1}}},
|
||||
}
|
||||
if _, err := db.Col("server_packages").Indexes().CreateMany(ctx, pkgIdx); err != nil {
|
||||
log.Printf("warning: server_packages indexes: %v", err)
|
||||
}
|
||||
|
||||
findingIdx := []mongo.IndexModel{
|
||||
{
|
||||
// This key is what makes a rescan an idempotent upsert rather than a
|
||||
// duplicate factory, and what lets first_seen survive a rescan.
|
||||
Keys: bson.D{
|
||||
{Key: "instance_id", Value: 1},
|
||||
{Key: "server_id", Value: 1},
|
||||
{Key: "cve_id", Value: 1},
|
||||
{Key: "package_name", Value: 1},
|
||||
},
|
||||
Options: options.Index().SetUnique(true),
|
||||
},
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "state", Value: 1}, {Key: "severity", Value: 1}}},
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "cve_id", Value: 1}}},
|
||||
}
|
||||
if _, err := db.Col("vuln_findings").Indexes().CreateMany(ctx, findingIdx); err != nil {
|
||||
log.Printf("warning: vuln_findings indexes: %v", err)
|
||||
}
|
||||
|
||||
if _, err := db.Col("vuln_alert_rules").Indexes().CreateOne(ctx, mongo.IndexModel{
|
||||
Keys: bson.D{{Key: "instance_id", Value: 1}},
|
||||
}); err != nil {
|
||||
log.Printf("warning: vuln_alert_rules indexes: %v", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/notify"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/shared/mail"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
)
|
||||
|
||||
// digestRowLimit caps how many findings a single digest lists by name. The
|
||||
// remainder is summarised as a count: a webhook payload holding six hundred
|
||||
// rows is not a notification, it is a report nobody reads in a chat client.
|
||||
const digestRowLimit = 20
|
||||
|
||||
// ErrVulnRuleNotFound is returned for a rule that does not exist in this
|
||||
// instance. Callers turn it into a 404.
|
||||
var ErrVulnRuleNotFound = fmt.Errorf("vulnerability alert rule not found")
|
||||
|
||||
func ListVulnRules(instanceID string) ([]models.VulnAlertRule, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cur, err := db.Col("vuln_alert_rules").Find(ctx, bson.M{"instance_id": instanceID})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
rules := []models.VulnAlertRule{}
|
||||
if err := cur.All(ctx, &rules); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return rules, nil
|
||||
}
|
||||
|
||||
func CreateVulnRule(instanceID string, r *models.VulnAlertRule) (*models.VulnAlertRule, error) {
|
||||
if err := validateVulnRule(instanceID, r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
r.ID = bson.NewObjectID()
|
||||
r.InstanceID = instanceID
|
||||
r.CreatedAt = time.Now()
|
||||
r.UpdatedAt = r.CreatedAt
|
||||
|
||||
if _, err := db.Col("vuln_alert_rules").InsertOne(ctx, r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func UpdateVulnRule(instanceID, ruleID string, r *models.VulnAlertRule) error {
|
||||
if err := validateVulnRule(instanceID, r); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
id, err := bson.ObjectIDFromHex(ruleID)
|
||||
if err != nil {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("vuln_alert_rules").UpdateOne(ctx,
|
||||
bson.M{"_id": id, "instance_id": instanceID},
|
||||
bson.M{"$set": bson.M{
|
||||
"name": r.Name,
|
||||
"enabled": r.Enabled,
|
||||
"min_severity": r.MinSeverity,
|
||||
"tags": r.Tags,
|
||||
"channel_ids": r.ChannelIDs,
|
||||
"updated_at": time.Now(),
|
||||
}},
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if res.MatchedCount == 0 {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func DeleteVulnRule(instanceID, ruleID string) error {
|
||||
id, err := bson.ObjectIDFromHex(ruleID)
|
||||
if err != nil {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
res, err := db.Col("vuln_alert_rules").DeleteOne(ctx, bson.M{"_id": id, "instance_id": instanceID})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if res.DeletedCount == 0 {
|
||||
return ErrVulnRuleNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateVulnRule rejects a rule that could never fire, and one naming a
|
||||
// channel from another instance. The channel check reuses validateChannelIDs so
|
||||
// there is one answer to "is this channel mine".
|
||||
func validateVulnRule(instanceID string, r *models.VulnAlertRule) error {
|
||||
if r.Name == "" {
|
||||
return fmt.Errorf("name is required")
|
||||
}
|
||||
switch r.MinSeverity {
|
||||
case models.SeverityUnknown, models.SeverityLow, models.SeverityMedium,
|
||||
models.SeverityHigh, models.SeverityCritical:
|
||||
default:
|
||||
return fmt.Errorf("min_severity %q is not a severity", r.MinSeverity)
|
||||
}
|
||||
if len(r.ChannelIDs) == 0 {
|
||||
return fmt.Errorf("at least one channel is required")
|
||||
}
|
||||
return validateChannelIDs(instanceID, r.ChannelIDs)
|
||||
}
|
||||
|
||||
// SendVulnDigest delivers one message per rule per tick — never one per
|
||||
// finding. See vulnsched for why the tick is the batch boundary.
|
||||
func SendVulnDigest(instanceID string, newly []models.VulnFinding) {
|
||||
rules, err := ListVulnRules(instanceID)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: list rules: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, rule := range rules {
|
||||
if !rule.Enabled {
|
||||
continue
|
||||
}
|
||||
|
||||
matched := filterBySeverity(newly, rule.MinSeverity)
|
||||
if len(matched) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
if len(rule.Tags) > 0 {
|
||||
// ResolveTargets is already the single answer to which servers a
|
||||
// selector touches. A rule that disagreed with a workflow about
|
||||
// what env:prod means would be worse than no filter at all.
|
||||
allowed, err := ResolveTargets(instanceID, nil, rule.Tags)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: resolve targets: %v", err)
|
||||
continue
|
||||
}
|
||||
matched = filterByServers(matched, allowed)
|
||||
if len(matched) == 0 {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
dispatchVulnDigest(instanceID, rule, matched)
|
||||
}
|
||||
}
|
||||
|
||||
func filterBySeverity(findings []models.VulnFinding, min string) []models.VulnFinding {
|
||||
floor := models.SeverityRank(min)
|
||||
out := make([]models.VulnFinding, 0, len(findings))
|
||||
for _, f := range findings {
|
||||
if models.SeverityRank(f.Severity) >= floor {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// filterByServers keeps findings on servers the rule's tag selector matched.
|
||||
// ResolveTargets answers in whole server documents, so the IDs are lifted here.
|
||||
func filterByServers(findings []models.VulnFinding, allowed []models.Server) []models.VulnFinding {
|
||||
set := make(map[string]bool, len(allowed))
|
||||
for _, s := range allowed {
|
||||
set[s.ServerID] = true
|
||||
}
|
||||
out := make([]models.VulnFinding, 0, len(findings))
|
||||
for _, f := range findings {
|
||||
if set[f.ServerID] {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// dispatchVulnDigest builds one digest and sends it over each of the rule's
|
||||
// channels.
|
||||
func dispatchVulnDigest(instanceID string, rule models.VulnAlertRule, findings []models.VulnFinding) {
|
||||
channels, err := GetChannels(instanceID, rule.ChannelIDs)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: load channels for rule %s: %v", rule.Name, err)
|
||||
return
|
||||
}
|
||||
|
||||
digest := buildVulnDigest(instanceID, rule, findings)
|
||||
|
||||
for _, ch := range channels {
|
||||
if !ch.Enabled {
|
||||
continue
|
||||
}
|
||||
go func(c models.NotificationChannel) {
|
||||
if err := notify.DispatchVulnDigest(c, digest); err != nil {
|
||||
log.Printf("vuln digest: dispatch to %s (%s): %v", c.Name, c.Type, err)
|
||||
}
|
||||
}(ch)
|
||||
}
|
||||
}
|
||||
|
||||
// buildVulnDigest turns a batch of findings into one message.
|
||||
//
|
||||
// Findings are ordered most severe first so the capped list shows the ones that
|
||||
// matter rather than whichever the scan happened to produce first.
|
||||
func buildVulnDigest(instanceID string, rule models.VulnAlertRule, findings []models.VulnFinding) notify.VulnDigest {
|
||||
sorted := make([]models.VulnFinding, len(findings))
|
||||
copy(sorted, findings)
|
||||
sort.SliceStable(sorted, func(i, j int) bool {
|
||||
return models.SeverityRank(sorted[i].Severity) > models.SeverityRank(sorted[j].Severity)
|
||||
})
|
||||
|
||||
counts := map[string]int{}
|
||||
servers := map[string]bool{}
|
||||
for _, f := range sorted {
|
||||
counts[f.Severity]++
|
||||
servers[f.ServerID] = true
|
||||
}
|
||||
|
||||
names := serverNames(instanceID)
|
||||
|
||||
shown := sorted
|
||||
more := 0
|
||||
if len(shown) > digestRowLimit {
|
||||
more = len(shown) - digestRowLimit
|
||||
shown = shown[:digestRowLimit]
|
||||
}
|
||||
|
||||
rows := make([]mail.VulnDigestRow, 0, len(shown))
|
||||
for _, f := range shown {
|
||||
name := names[f.ServerID]
|
||||
if name == "" {
|
||||
name = f.ServerID
|
||||
}
|
||||
rows = append(rows, mail.VulnDigestRow{
|
||||
CVEID: f.CVEID,
|
||||
Severity: f.Severity,
|
||||
PackageName: f.PackageName,
|
||||
ServerName: name,
|
||||
FixedIn: f.FixedIn,
|
||||
})
|
||||
}
|
||||
|
||||
top := models.SeverityUnknown
|
||||
if len(sorted) > 0 {
|
||||
top = sorted[0].Severity
|
||||
}
|
||||
|
||||
instanceName := instanceID
|
||||
if inst, err := GetInstance(instanceID); err == nil && inst != nil && inst.Name != "" {
|
||||
instanceName = inst.Name
|
||||
}
|
||||
|
||||
return notify.VulnDigest{
|
||||
InstanceName: instanceName,
|
||||
RuleName: rule.Name,
|
||||
Summary: summariseCounts(counts, len(servers)),
|
||||
TopSeverity: top,
|
||||
Count: len(sorted),
|
||||
Rows: rows,
|
||||
More: more,
|
||||
DBAge: vulnDBAge(),
|
||||
}
|
||||
}
|
||||
|
||||
// summariseCounts renders "12 new critical, 4 new high across 6 servers".
|
||||
func summariseCounts(counts map[string]int, serverCount int) string {
|
||||
order := []string{
|
||||
models.SeverityCritical, models.SeverityHigh,
|
||||
models.SeverityMedium, models.SeverityLow, models.SeverityUnknown,
|
||||
}
|
||||
parts := ""
|
||||
for _, sev := range order {
|
||||
if counts[sev] == 0 {
|
||||
continue
|
||||
}
|
||||
if parts != "" {
|
||||
parts += ", "
|
||||
}
|
||||
parts += fmt.Sprintf("%d new %s", counts[sev], sev)
|
||||
}
|
||||
if parts == "" {
|
||||
parts = "new findings"
|
||||
}
|
||||
plural := "servers"
|
||||
if serverCount == 1 {
|
||||
plural = "server"
|
||||
}
|
||||
return fmt.Sprintf("%s across %d %s", parts, serverCount, plural)
|
||||
}
|
||||
|
||||
// serverNames maps server IDs to display names for one instance. A digest that
|
||||
// named raw UUIDs would be unreadable in a chat client.
|
||||
func serverNames(instanceID string) map[string]string {
|
||||
out := map[string]string{}
|
||||
servers, err := ListServers(instanceID)
|
||||
if err != nil {
|
||||
log.Printf("vuln digest: list servers: %v", err)
|
||||
return out
|
||||
}
|
||||
for _, s := range servers {
|
||||
out[s.ServerID] = s.Hostname
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// vulnDBAge renders how long ago the vulnerability database was pulled.
|
||||
//
|
||||
// It is on every digest deliberately: a fleet scanned against a three-week-old
|
||||
// database must say so rather than let the reader assume freshness.
|
||||
func vulnDBAge() string {
|
||||
meta, err := GetVulnDBMeta()
|
||||
if err != nil || meta == nil || meta.PulledAt.IsZero() {
|
||||
return "an unknown time"
|
||||
}
|
||||
d := time.Since(meta.PulledAt)
|
||||
switch {
|
||||
case d < time.Hour:
|
||||
return fmt.Sprintf("%d minutes", int(d.Minutes()))
|
||||
case d < 48*time.Hour:
|
||||
return fmt.Sprintf("%d hours", int(d.Hours()))
|
||||
default:
|
||||
return fmt.Sprintf("%d days", int(d.Hours()/24))
|
||||
}
|
||||
}
|
||||
|
||||
// GetVulnDBMeta reads the deployment-wide vulnerability database metadata.
|
||||
// It carries no instance_id: the database is a property of the deployment, not
|
||||
// of a tenant.
|
||||
func GetVulnDBMeta() (*models.VulnDBMeta, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var meta models.VulnDBMeta
|
||||
err := db.Col("vulndb_meta").FindOne(ctx, bson.M{}).Decode(&meta)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &meta, nil
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// EnsureWorkloadIndexes declares the indexes for the workload registry.
|
||||
//
|
||||
// It warns rather than being fatal, matching EnsureSecretIndexes and
|
||||
// EnsureVulnIndexes: a missing index degrades these queries to a collection
|
||||
// scan, which is no reason to refuse to serve the fleet.
|
||||
func EnsureWorkloadIndexes() error {
|
||||
ctx := context.Background()
|
||||
|
||||
idx := []mongo.IndexModel{
|
||||
{
|
||||
Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "server_id", Value: 1}},
|
||||
Options: options.Index().SetUnique(true),
|
||||
},
|
||||
// Multikey, for the fleet-wide "which servers run image X" query, which
|
||||
// is the reason the snapshot is stored rather than fetched and discarded.
|
||||
{Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "workloads.image", Value: 1}}},
|
||||
}
|
||||
if _, err := db.Col("server_workloads").Indexes().CreateMany(ctx, idx); err != nil {
|
||||
log.Printf("warning: server_workloads indexes: %v", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/bus"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc/pb"
|
||||
)
|
||||
|
||||
// Workload results travel back over the bus for the same reason commands travel
|
||||
// out over it: the pod serving the HTTP request and the pod holding the agent's
|
||||
// stream are two different processes, and a map in one cannot be read by the
|
||||
// other.
|
||||
//
|
||||
// Await MUST be called before the command is dispatched, or a fast agent
|
||||
// answers into a channel nobody is listening on yet. See stepresults.go.
|
||||
|
||||
type workloadResultRegistry struct{}
|
||||
|
||||
var WorkloadResults = &workloadResultRegistry{}
|
||||
|
||||
// Await subscribes to a command's result channel for a log snapshot.
|
||||
func (r *workloadResultRegistry) Await(commandID string) (<-chan *pb.WorkloadLogsResult, func()) {
|
||||
out := make(chan *pb.WorkloadLogsResult, 1)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
raw, unsub, err := bus.Subscribe(ctx, bus.ResultChannel+commandID)
|
||||
if err != nil {
|
||||
log.Printf("workload results: subscribe for %s: %v", commandID, err)
|
||||
cancel()
|
||||
close(out)
|
||||
return out, func() {}
|
||||
}
|
||||
|
||||
go func() {
|
||||
defer close(out)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case b, ok := <-raw:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var res pb.WorkloadLogsResult
|
||||
if err := json.Unmarshal(b, &res); err != nil {
|
||||
log.Printf("workload results: undecodable result for %s: %v", commandID, err)
|
||||
return
|
||||
}
|
||||
out <- &res
|
||||
}
|
||||
}()
|
||||
|
||||
return out, func() {
|
||||
cancel()
|
||||
unsub()
|
||||
}
|
||||
}
|
||||
|
||||
// AwaitCommand subscribes to a command's result channel for a plain
|
||||
// CommandResult, which is what a control action answers with.
|
||||
//
|
||||
// A control action reuses CommandResult rather than growing a message of its
|
||||
// own: start, stop and restart succeed or fail, and that is exactly what
|
||||
// CommandResult already says.
|
||||
func (r *workloadResultRegistry) AwaitCommand(commandID string) (<-chan *pb.CommandResult, func()) {
|
||||
out := make(chan *pb.CommandResult, 1)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
raw, unsub, err := bus.Subscribe(ctx, bus.ResultChannel+commandID)
|
||||
if err != nil {
|
||||
log.Printf("workload results: subscribe for %s: %v", commandID, err)
|
||||
cancel()
|
||||
close(out)
|
||||
return out, func() {}
|
||||
}
|
||||
|
||||
go func() {
|
||||
defer close(out)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case b, ok := <-raw:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var res pb.CommandResult
|
||||
if err := json.Unmarshal(b, &res); err != nil {
|
||||
log.Printf("workload results: undecodable command result for %s: %v", commandID, err)
|
||||
return
|
||||
}
|
||||
out <- &res
|
||||
}
|
||||
}()
|
||||
|
||||
return out, func() {
|
||||
cancel()
|
||||
unsub()
|
||||
}
|
||||
}
|
||||
|
||||
// Deliver publishes a log result received from an agent. Called on the pod
|
||||
// holding that agent's stream, which is not usually the pod waiting for it.
|
||||
func (r *workloadResultRegistry) Deliver(res *pb.WorkloadLogsResult) {
|
||||
if res == nil || res.CommandId == "" {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), dispatchAckTimeout)
|
||||
defer cancel()
|
||||
if _, err := bus.Publish(ctx, bus.ResultChannel+res.CommandId, res); err != nil {
|
||||
log.Printf("workload results: publish for %s: %v", res.CommandId, err)
|
||||
}
|
||||
}
|
||||
|
||||
// DeliverCommand republishes a CommandResult onto the bus so a waiting pod can
|
||||
// see it. Publishing with no subscriber is a no-op, so this is safe to call for
|
||||
// every CommandResult rather than only the ones somebody is waiting on.
|
||||
func (r *workloadResultRegistry) DeliverCommand(res *pb.CommandResult) {
|
||||
if res == nil || res.CommandId == "" {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), dispatchAckTimeout)
|
||||
defer cancel()
|
||||
if _, err := bus.Publish(ctx, bus.ResultChannel+res.CommandId, res); err != nil {
|
||||
log.Printf("workload results: publish command result for %s: %v", res.CommandId, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc/pb"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"github.com/google/uuid"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// workloadResultTimeout bounds how long an API request waits for an agent to
|
||||
// answer a control action or a log read. It is well above the agent's own
|
||||
// 90-second control timeout and 60-second log timeout, so a slow-but-working
|
||||
// agent reports its real error rather than being cut off by this side.
|
||||
const workloadResultTimeout = 120 * time.Second
|
||||
|
||||
// MaxWorkloadLogLines mirrors the agent's own cap. It is declared again here
|
||||
// rather than imported: agent/ is a separate module with an internal/ tree, so
|
||||
// the two cannot share a constant. Change one, change the other — the same
|
||||
// shape of hazard as the mirrored token blocks in the web apps.
|
||||
const MaxWorkloadLogLines = 500
|
||||
|
||||
// workloadProtectedMarker is the text the agent's ErrProtected carries. The
|
||||
// refusal crosses the wire as a string, so this is how the control plane knows
|
||||
// a 409 is owed rather than a 502.
|
||||
const workloadProtectedMarker = "workload is protected"
|
||||
|
||||
// IsWorkloadProtected reports whether an agent refused because the target is
|
||||
// protected — the agent's own guard, which is the boundary. Nothing failed, so
|
||||
// the API answers 409 rather than an error status.
|
||||
func IsWorkloadProtected(err error) bool {
|
||||
return err != nil && strings.Contains(err.Error(), workloadProtectedMarker)
|
||||
}
|
||||
|
||||
func HasWorkloadHash(instanceID, serverID, hash string) (bool, error) {
|
||||
err := db.Col("server_workloads").FindOne(context.Background(), bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
"hash": hash,
|
||||
}, options.FindOne().SetProjection(bson.M{"_id": 1})).Err()
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return false, nil
|
||||
}
|
||||
return err == nil, err
|
||||
}
|
||||
|
||||
// StoreWorkloads replaces a server's workload list.
|
||||
func StoreWorkloads(instanceID, serverID, hash string, wls []models.Workload,
|
||||
dockerOK bool, dockerErr string, systemdOK bool, systemdErr string) error {
|
||||
|
||||
_, err := db.Col("server_workloads").UpdateOne(context.Background(),
|
||||
bson.M{"instance_id": instanceID, "server_id": serverID},
|
||||
bson.M{"$set": bson.M{
|
||||
"hash": hash,
|
||||
"workloads": wls,
|
||||
"collected_at": time.Now(),
|
||||
"docker_ok": dockerOK,
|
||||
"docker_error": dockerErr,
|
||||
"systemd_ok": systemdOK,
|
||||
"systemd_error": systemdErr,
|
||||
}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
func GetWorkloads(instanceID, serverID string) (*models.ServerWorkloads, error) {
|
||||
var sw models.ServerWorkloads
|
||||
err := db.Col("server_workloads").FindOne(context.Background(), bson.M{
|
||||
"instance_id": instanceID,
|
||||
"server_id": serverID,
|
||||
}).Decode(&sw)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sw, nil
|
||||
}
|
||||
|
||||
type WorkloadHit struct {
|
||||
ServerID string `json:"server_id"`
|
||||
Workload models.Workload `json:"workload"`
|
||||
}
|
||||
|
||||
// SearchWorkloads answers "which servers run image X" — the reason the snapshot
|
||||
// is stored rather than fetched on demand and discarded.
|
||||
func SearchWorkloads(instanceID, image, stack, state string) ([]WorkloadHit, error) {
|
||||
ctx := context.Background()
|
||||
|
||||
filter := bson.M{"instance_id": instanceID}
|
||||
if image != "" {
|
||||
filter["workloads.image"] = image
|
||||
}
|
||||
|
||||
cur, err := db.Col("server_workloads").Find(ctx, filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var docs []models.ServerWorkloads
|
||||
if err := cur.All(ctx, &docs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
hits := []WorkloadHit{}
|
||||
for _, d := range docs {
|
||||
for _, w := range d.Workloads {
|
||||
if image != "" && w.Image != image {
|
||||
continue
|
||||
}
|
||||
if stack != "" && w.Stack != stack {
|
||||
continue
|
||||
}
|
||||
if state != "" && w.State != state {
|
||||
continue
|
||||
}
|
||||
hits = append(hits, WorkloadHit{ServerID: d.ServerID, Workload: w})
|
||||
}
|
||||
}
|
||||
return hits, nil
|
||||
}
|
||||
|
||||
// DispatchRefreshWorkloads asks an agent to report immediately. It returns as
|
||||
// soon as the owning pod acks; the caller refetches the stored document.
|
||||
//
|
||||
// The refresh carries nothing back on purpose: the agent answers through the
|
||||
// normal ReportWorkloads RPC, so server_workloads has exactly one writer.
|
||||
func DispatchRefreshWorkloads(serverID string) error {
|
||||
return Dispatcher.dispatch(serverID, &pb.ServerCommand{
|
||||
CommandId: uuid.New().String(),
|
||||
RefreshWorkloads: &pb.RefreshWorkloadsCmd{},
|
||||
})
|
||||
}
|
||||
|
||||
// DispatchControlWorkload runs a control action and waits for the agent's
|
||||
// CommandResult.
|
||||
//
|
||||
// Await is called BEFORE dispatch. Reversing those two lines introduces a race
|
||||
// that only shows under load, on a fast agent answering into a channel nobody
|
||||
// has joined yet.
|
||||
func DispatchControlWorkload(serverID, kind, id, action string) error {
|
||||
commandID := uuid.New().String()
|
||||
|
||||
results, done := WorkloadResults.AwaitCommand(commandID)
|
||||
defer done()
|
||||
|
||||
if err := Dispatcher.dispatch(serverID, &pb.ServerCommand{
|
||||
CommandId: commandID,
|
||||
ControlWorkload: &pb.ControlWorkloadCmd{Kind: kind, Id: id, Action: action},
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
select {
|
||||
case res, ok := <-results:
|
||||
if !ok || res == nil {
|
||||
return fmt.Errorf("no result from agent for %s %s", action, id)
|
||||
}
|
||||
if !res.Success {
|
||||
return fmt.Errorf("%s", res.Message)
|
||||
}
|
||||
return nil
|
||||
case <-time.After(workloadResultTimeout):
|
||||
return fmt.Errorf("timed out waiting for the agent to %s %s", action, id)
|
||||
}
|
||||
}
|
||||
|
||||
// DispatchWorkloadLogs fetches a bounded log snapshot.
|
||||
//
|
||||
// Await is called BEFORE dispatch, for the same reason as above.
|
||||
func DispatchWorkloadLogs(serverID, kind, id string, tail int) (string, bool, error) {
|
||||
commandID := uuid.New().String()
|
||||
|
||||
results, done := WorkloadResults.Await(commandID)
|
||||
defer done()
|
||||
|
||||
if err := Dispatcher.dispatch(serverID, &pb.ServerCommand{
|
||||
CommandId: commandID,
|
||||
WorkloadLogs: &pb.WorkloadLogsCmd{Kind: kind, Id: id, Tail: int32(tail)},
|
||||
}); err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
|
||||
select {
|
||||
case res, ok := <-results:
|
||||
if !ok || res == nil {
|
||||
return "", false, fmt.Errorf("no log result from agent for %s", id)
|
||||
}
|
||||
if res.Error != "" {
|
||||
return "", false, fmt.Errorf("%s", res.Error)
|
||||
}
|
||||
return res.Text, res.Truncated, nil
|
||||
case <-time.After(workloadResultTimeout):
|
||||
return "", false, fmt.Errorf("timed out waiting for logs for %s", id)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
trivydb "github.com/aquasecurity/trivy-db/pkg/db"
|
||||
trivytypes "github.com/aquasecurity/trivy-db/pkg/types"
|
||||
)
|
||||
|
||||
// Advisory is one fixed-version statement for one source package.
|
||||
type Advisory struct {
|
||||
CVEID string
|
||||
// FixedVersion empty means no vendor fix has been published. It is a real
|
||||
// state, not an absence of data, and callers must treat it as vulnerable.
|
||||
FixedVersion string
|
||||
Severity string
|
||||
}
|
||||
|
||||
// VulnInfo is the CVE's own metadata, shared across every server it affects.
|
||||
type VulnInfo struct {
|
||||
Title string
|
||||
Severity string
|
||||
CVSSScore float64
|
||||
References []string
|
||||
}
|
||||
|
||||
// Store reads a pulled trivy-db.
|
||||
type Store struct {
|
||||
cfg trivydb.Config
|
||||
}
|
||||
|
||||
// Open opens the database in dir. trivy-db expects the directory, not the file:
|
||||
// it appends "trivy.db" itself.
|
||||
func Open(dir string) (*Store, error) {
|
||||
if err := trivydb.Init(dir); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Store{cfg: trivydb.Config{}}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Close() error { return trivydb.Close() }
|
||||
|
||||
// Advisories returns every advisory for a source package in a bucket.
|
||||
func (s *Store) Advisories(bucket, srcName string) ([]Advisory, error) {
|
||||
raw, err := s.cfg.GetAdvisories(bucket, srcName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]Advisory, 0, len(raw))
|
||||
for _, a := range raw {
|
||||
out = append(out, Advisory{
|
||||
CVEID: a.VulnerabilityID,
|
||||
FixedVersion: a.FixedVersion,
|
||||
// Advisory.Severity is trivy's numeric Severity type, unlike
|
||||
// Vulnerability.Severity which is a string. They are genuinely
|
||||
// different types in trivy-db, not an inconsistency here.
|
||||
Severity: severityFromLevel(a.Severity),
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Vulnerability returns a CVE's shared metadata.
|
||||
func (s *Store) Vulnerability(cveID string) (VulnInfo, error) {
|
||||
v, err := s.cfg.GetVulnerability(cveID)
|
||||
if err != nil {
|
||||
return VulnInfo{}, err
|
||||
}
|
||||
return VulnInfo{
|
||||
Title: v.Title,
|
||||
Severity: resolveSeverity(v),
|
||||
CVSSScore: topCVSS(v),
|
||||
References: v.References,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// resolveSeverity picks a CVE's severity: vendor, then NVD, then unknown.
|
||||
//
|
||||
// Never invented. This will surface as "why is this critical CVE marked low":
|
||||
// Debian and Red Hat routinely downgrade an NVD score because the vulnerable
|
||||
// path is not reachable in their build, and their rating is the accurate one
|
||||
// for that package. Where several vendors disagree the highest wins, because
|
||||
// under-reporting a vulnerability is the worse mistake.
|
||||
func resolveSeverity(v trivytypes.Vulnerability) string {
|
||||
best := 0
|
||||
for _, sev := range v.VendorSeverity {
|
||||
if int(sev) > best {
|
||||
best = int(sev)
|
||||
}
|
||||
}
|
||||
if best > 0 {
|
||||
return severityFromLevel(trivytypes.Severity(best))
|
||||
}
|
||||
|
||||
// Vulnerability.Severity is the deprecated NVD-derived string. Used only as
|
||||
// the fallback, which is exactly what it is good for.
|
||||
if s := strings.ToLower(strings.TrimSpace(v.Severity)); s != "" && s != "unknown" {
|
||||
return s
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
// topCVSS returns the highest V3 score any source published, or 0.
|
||||
func topCVSS(v trivytypes.Vulnerability) float64 {
|
||||
var top float64
|
||||
for _, c := range v.CVSS {
|
||||
if c.V3Score > top {
|
||||
top = c.V3Score
|
||||
}
|
||||
}
|
||||
return top
|
||||
}
|
||||
|
||||
// severityFromLevel maps trivy-db's numeric severity onto our lowercase
|
||||
// strings. The names are fixed by models.Severity* and must stay in step.
|
||||
func severityFromLevel(n trivytypes.Severity) string {
|
||||
switch int(n) {
|
||||
case 4:
|
||||
return "critical"
|
||||
case 3:
|
||||
return "high"
|
||||
case 2:
|
||||
return "medium"
|
||||
case 1:
|
||||
return "low"
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// rhelRebuilds share Red Hat's advisory feed rather than publishing their own.
|
||||
var rhelRebuilds = map[string]bool{
|
||||
"redhat": true, "centos": true, "rocky": true, "alma": true, "oracle": true,
|
||||
}
|
||||
|
||||
// Bucket maps an OS family and version onto the trivy-db bucket that holds its
|
||||
// advisories.
|
||||
//
|
||||
// It returns ErrUnsupportedFamily rather than a best guess when we have no
|
||||
// feed. A scan that cannot be performed must say so; reporting zero findings
|
||||
// for a distribution we do not cover is indistinguishable from reporting a
|
||||
// clean host, and one of those is a lie.
|
||||
func Bucket(family, versionID string) (string, error) {
|
||||
family = strings.ToLower(strings.TrimSpace(family))
|
||||
versionID = strings.TrimSpace(versionID)
|
||||
|
||||
switch {
|
||||
case family == "debian" || family == "ubuntu":
|
||||
if versionID == "" {
|
||||
return "", fmt.Errorf("%s requires a version id", family)
|
||||
}
|
||||
return family + " " + versionID, nil
|
||||
|
||||
case family == "alpine":
|
||||
if versionID == "" {
|
||||
return "", fmt.Errorf("alpine requires a version id")
|
||||
}
|
||||
return "alpine " + majorMinor(versionID), nil
|
||||
|
||||
case rhelRebuilds[family]:
|
||||
if versionID == "" {
|
||||
return "", fmt.Errorf("%s requires a version id", family)
|
||||
}
|
||||
return "redhat " + major(versionID), nil
|
||||
|
||||
default:
|
||||
return "", fmt.Errorf("%w: %s", ErrUnsupportedFamily, family)
|
||||
}
|
||||
}
|
||||
|
||||
func major(v string) string {
|
||||
if i := strings.Index(v, "."); i != -1 {
|
||||
return v[:i]
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func majorMinor(v string) string {
|
||||
parts := strings.Split(v, ".")
|
||||
if len(parts) >= 2 {
|
||||
return parts[0] + "." + parts[1]
|
||||
}
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
// AdvisorySource is the advisory lookup the matcher needs. *Store satisfies it.
|
||||
// The seam keeps the matching logic independent of how the database is opened.
|
||||
type AdvisorySource interface {
|
||||
Advisories(bucket, srcName string) ([]Advisory, error)
|
||||
}
|
||||
|
||||
// Result is one vulnerable package on one server, before it becomes a finding.
|
||||
type Result struct {
|
||||
CVEID string
|
||||
PackageName string // the BINARY package, which is what is installed
|
||||
Installed string
|
||||
FixedIn string
|
||||
Severity string
|
||||
}
|
||||
|
||||
// Match returns every advisory that the installed packages do not satisfy.
|
||||
//
|
||||
// Vulnerable means: no fix has been published, or the installed version sorts
|
||||
// strictly before the fixed version under the distribution's own ordering.
|
||||
// Equal is NOT vulnerable — that is the backported-fix case, where a
|
||||
// distribution patches in place without changing the upstream version, and
|
||||
// treating it as vulnerable reports a patched fleet as exposed.
|
||||
func Match(src AdvisorySource, os models.OSRelease, pkgs []models.InstalledPackage) ([]Result, error) {
|
||||
bucket, err := Bucket(os.Family, os.VersionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var out []Result
|
||||
for _, p := range pkgs {
|
||||
// Debian and Ubuntu advisories are keyed on the source package: one
|
||||
// advisory against "openssl" covers libssl3, openssl and libssl-dev.
|
||||
srcName := p.SourceName
|
||||
if srcName == "" {
|
||||
srcName = p.Name
|
||||
}
|
||||
|
||||
advs, err := src.Advisories(bucket, srcName)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("advisories for %s: %w", srcName, err)
|
||||
}
|
||||
|
||||
for _, a := range advs {
|
||||
// No published fix. Vulnerable, and the finding most in need of
|
||||
// acceptance, since there is nothing to patch.
|
||||
if a.FixedVersion == "" {
|
||||
out = append(out, Result{
|
||||
CVEID: a.CVEID, PackageName: p.Name,
|
||||
Installed: p.Version, Severity: a.Severity,
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
older, err := LessThan(os.Family, p.Version, a.FixedVersion)
|
||||
if err != nil {
|
||||
// Skip this one advisory rather than failing the whole server:
|
||||
// one unparseable version must not blind us to every other CVE
|
||||
// on the host. Log it — a silent skip is a silent false
|
||||
// negative, which is the direction that hurts.
|
||||
log.Printf("vulndb: compare %s %s vs %s: %v", p.Name, p.Version, a.FixedVersion, err)
|
||||
continue
|
||||
}
|
||||
if older {
|
||||
out = append(out, Result{
|
||||
CVEID: a.CVEID, PackageName: p.Name,
|
||||
Installed: p.Version, FixedIn: a.FixedVersion, Severity: a.Severity,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
|
||||
"oras.land/oras-go/v2"
|
||||
"oras.land/oras-go/v2/registry"
|
||||
"oras.land/oras-go/v2/registry/remote"
|
||||
)
|
||||
|
||||
// DefaultRef is the published trivy-db OCI artifact, rebuilt every six hours.
|
||||
const DefaultRef = "ghcr.io/aquasecurity/trivy-db:2"
|
||||
|
||||
// SupportedSchema is the trivy-db schema version this code understands.
|
||||
//
|
||||
// A different version is refused rather than parsed on the assumption it is
|
||||
// close enough. Mis-reading the schema would not fail loudly — it would return
|
||||
// no advisories, which is indistinguishable from a clean fleet.
|
||||
const SupportedSchema = 2
|
||||
|
||||
// dbFileName and metaFileName are the two files inside the artifact layer.
|
||||
const (
|
||||
dbFileName = "trivy.db"
|
||||
metaFileName = "metadata.json"
|
||||
)
|
||||
|
||||
// Ref returns the artifact reference, honouring VANTAGE_TRIVY_DB_REF so an
|
||||
// air-gapped deployment can mirror the artifact into its own registry, and so
|
||||
// a busy deployment can avoid the anonymous ghcr rate limit.
|
||||
func Ref() string {
|
||||
if v := os.Getenv("VANTAGE_TRIVY_DB_REF"); v != "" {
|
||||
return v
|
||||
}
|
||||
return DefaultRef
|
||||
}
|
||||
|
||||
// Disabled reports whether the puller and scheduler are switched off entirely.
|
||||
// Findings already written are still served, and still marked stale.
|
||||
func Disabled() bool {
|
||||
return strings.EqualFold(os.Getenv("VANTAGE_VULNDB_DISABLED"), "true")
|
||||
}
|
||||
|
||||
// dbMetadata is the subset of trivy-db's metadata.json we read.
|
||||
type dbMetadata struct {
|
||||
Version int `json:"Version"`
|
||||
}
|
||||
|
||||
// Pull fetches the trivy-db artifact into dir and returns its schema version.
|
||||
//
|
||||
// It extracts into a staging directory and only moves the files into place once
|
||||
// both are present and the schema has been accepted. A pull that fails partway
|
||||
// therefore leaves the previous database untouched rather than a half-written
|
||||
// one that Open would happily accept and scan against.
|
||||
func Pull(ctx context.Context, dir string) (int, error) {
|
||||
ref := Ref()
|
||||
|
||||
parsed, err := registry.ParseReference(ref)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("parse reference %q: %w", ref, err)
|
||||
}
|
||||
|
||||
repo, err := remote.NewRepository(ref)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("open repository %q: %w", ref, err)
|
||||
}
|
||||
|
||||
// The tag or digest half of the reference; the repository already knows the
|
||||
// registry and path.
|
||||
target := parsed.Reference
|
||||
if target == "" {
|
||||
target = "latest"
|
||||
}
|
||||
|
||||
_, manifestBytes, err := oras.FetchBytes(ctx, repo, target, oras.DefaultFetchBytesOptions)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("fetch manifest %s: %w", ref, err)
|
||||
}
|
||||
|
||||
var man ocispec.Manifest
|
||||
if err := json.Unmarshal(manifestBytes, &man); err != nil {
|
||||
return 0, fmt.Errorf("decode manifest %s: %w", ref, err)
|
||||
}
|
||||
if len(man.Layers) == 0 {
|
||||
return 0, fmt.Errorf("artifact %s has no layers", ref)
|
||||
}
|
||||
|
||||
// Streamed rather than buffered: the layer is ~50MB and there is no reason
|
||||
// to hold it in memory on the way to disk.
|
||||
rc, err := repo.Blobs().Fetch(ctx, man.Layers[0])
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("fetch layer: %w", err)
|
||||
}
|
||||
defer rc.Close()
|
||||
|
||||
staging, err := os.MkdirTemp(dir, ".staging-")
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("staging dir: %w", err)
|
||||
}
|
||||
defer os.RemoveAll(staging)
|
||||
|
||||
if err := extractTarGz(rc, staging); err != nil {
|
||||
return 0, fmt.Errorf("extract layer: %w", err)
|
||||
}
|
||||
|
||||
metaBytes, err := os.ReadFile(filepath.Join(staging, metaFileName))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read %s: %w", metaFileName, err)
|
||||
}
|
||||
var meta dbMetadata
|
||||
if err := json.Unmarshal(metaBytes, &meta); err != nil {
|
||||
return 0, fmt.Errorf("decode %s: %w", metaFileName, err)
|
||||
}
|
||||
if meta.Version != SupportedSchema {
|
||||
return 0, fmt.Errorf("trivy-db schema %d is not supported (want %d)", meta.Version, SupportedSchema)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(filepath.Join(staging, dbFileName)); err != nil {
|
||||
return 0, fmt.Errorf("artifact has no %s: %w", dbFileName, err)
|
||||
}
|
||||
|
||||
// Both files present and the schema accepted, so it is safe to replace.
|
||||
for _, name := range []string{dbFileName, metaFileName} {
|
||||
src := filepath.Join(staging, name)
|
||||
dst := filepath.Join(dir, name)
|
||||
if err := os.Remove(dst); err != nil && !os.IsNotExist(err) {
|
||||
return 0, fmt.Errorf("remove old %s: %w", name, err)
|
||||
}
|
||||
if err := os.Rename(src, dst); err != nil {
|
||||
return 0, fmt.Errorf("install %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
return meta.Version, nil
|
||||
}
|
||||
|
||||
// extractTarGz writes the artifact layer into dir. Paths are flattened and
|
||||
// checked so a crafted archive cannot write outside dir.
|
||||
func extractTarGz(r io.Reader, dir string) error {
|
||||
gz, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer gz.Close()
|
||||
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if hdr.Typeflag != tar.TypeReg {
|
||||
continue
|
||||
}
|
||||
name := filepath.Base(hdr.Name) // flatten; the archive is two files
|
||||
if name == "." || name == ".." || name == "" {
|
||||
continue
|
||||
}
|
||||
dst := filepath.Join(dir, name)
|
||||
if !strings.HasPrefix(dst, filepath.Clean(dir)+string(os.PathSeparator)) {
|
||||
return fmt.Errorf("archive entry escapes destination: %q", hdr.Name)
|
||||
}
|
||||
f, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(f, tr); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
// Package vulndb matches installed packages against distribution security
|
||||
// advisories.
|
||||
//
|
||||
// Version comparison is bought rather than written. Distribution version
|
||||
// ordering is subtle in ways that are invisible until they are wrong: dpkg has
|
||||
// epochs and sorts "~" before the empty string, rpmvercmp has its own segment
|
||||
// rules and treats "~" and "^" differently again, and any ordering that falls
|
||||
// back on string comparison puts 1.10 before 1.9. Every one of those mistakes
|
||||
// produces a false negative — a vulnerable host reported clean — which is the
|
||||
// failure nobody notices.
|
||||
package vulndb
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
apk "github.com/knqyf263/go-apk-version"
|
||||
deb "github.com/knqyf263/go-deb-version"
|
||||
rpm "github.com/knqyf263/go-rpm-version"
|
||||
)
|
||||
|
||||
// ErrUnsupportedFamily means we hold no comparator for this distribution, and
|
||||
// therefore cannot answer whether it is vulnerable. Callers must surface this
|
||||
// as "unsupported" and must never treat it as "not vulnerable".
|
||||
var ErrUnsupportedFamily = errors.New("unsupported OS family")
|
||||
|
||||
// LessThan reports whether version a sorts before version b under the ordering
|
||||
// rules of the given OS family.
|
||||
//
|
||||
// An unparseable or empty version is an error, never a quiet false. False here
|
||||
// means "not vulnerable", which is the dangerous direction to guess in.
|
||||
func LessThan(family, a, b string) (bool, error) {
|
||||
switch family {
|
||||
case "debian", "ubuntu":
|
||||
if a == "" || b == "" {
|
||||
return false, fmt.Errorf("empty deb version (a=%q b=%q)", a, b)
|
||||
}
|
||||
va, err := deb.NewVersion(a)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse deb version %q: %w", a, err)
|
||||
}
|
||||
vb, err := deb.NewVersion(b)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse deb version %q: %w", b, err)
|
||||
}
|
||||
return va.LessThan(vb), nil
|
||||
|
||||
case "redhat", "centos", "rocky", "alma", "amazon", "oracle", "suse", "opensuse", "sles":
|
||||
// go-rpm-version does not error; rpmvercmp is defined over arbitrary
|
||||
// strings. Guard empties so a missing version cannot read as equal.
|
||||
if a == "" || b == "" {
|
||||
return false, fmt.Errorf("empty rpm version (a=%q b=%q)", a, b)
|
||||
}
|
||||
return rpm.NewVersion(a).LessThan(rpm.NewVersion(b)), nil
|
||||
|
||||
case "alpine":
|
||||
if a == "" || b == "" {
|
||||
return false, fmt.Errorf("empty apk version (a=%q b=%q)", a, b)
|
||||
}
|
||||
va, err := apk.NewVersion(a)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse apk version %q: %w", a, err)
|
||||
}
|
||||
vb, err := apk.NewVersion(b)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("parse apk version %q: %w", b, err)
|
||||
}
|
||||
return va.LessThan(vb), nil
|
||||
|
||||
default:
|
||||
return false, fmt.Errorf("%w: %s", ErrUnsupportedFamily, family)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
// Package vulnsched owns the vulnerability scan loop.
|
||||
//
|
||||
// It runs inside bus.RunAsLeader("housekeeping", …) alongside monitorsched,
|
||||
// workflowsched and the sweepers: one role, one lock. N replicas each running
|
||||
// this loop would mean N copies of the ~50MB database resident, N rescans of
|
||||
// the same fleet on every database refresh, and N digests reaching the
|
||||
// customer for one set of findings.
|
||||
package vulnsched
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services"
|
||||
"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
const (
|
||||
tickInterval = 60 * time.Second
|
||||
// trivy-db is rebuilt every six hours; pulling more often buys nothing.
|
||||
dbMaxAge = 6 * time.Hour
|
||||
)
|
||||
|
||||
// Deps are injected from main.go rather than imported, following
|
||||
// workflowsched. It keeps this package's reach explicit and reviewable.
|
||||
type Deps struct {
|
||||
LogEvent func(instanceID, eventType, actor, serverID, keyID, details string)
|
||||
SendDigest func(instanceID string, newly []models.VulnFinding)
|
||||
}
|
||||
|
||||
type scheduler struct {
|
||||
deps Deps
|
||||
dir string
|
||||
store *vulndb.Store
|
||||
version int
|
||||
pulled time.Time
|
||||
}
|
||||
|
||||
func Start(ctx context.Context, deps Deps) {
|
||||
if vulndb.Disabled() {
|
||||
log.Println("vulnsched: disabled by VANTAGE_VULNDB_DISABLED")
|
||||
return
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "vantage-vulndb-")
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: temp dir: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
s := &scheduler{deps: deps, dir: dir}
|
||||
|
||||
go func() {
|
||||
defer os.RemoveAll(dir)
|
||||
defer s.closeStore()
|
||||
|
||||
ticker := time.NewTicker(tickInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.tick(ctx)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (s *scheduler) tick(ctx context.Context) {
|
||||
if err := s.ensureDB(ctx); err != nil {
|
||||
// Keep the last good database and carry on scanning against it. A
|
||||
// network blip must never clear findings or read as "all fixed".
|
||||
log.Printf("vulnsched: database unavailable: %v", err)
|
||||
s.recordDBError(ctx, err)
|
||||
if s.store == nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
s.scanPending(ctx)
|
||||
}
|
||||
|
||||
// ensureDB pulls a fresh database when the local copy is stale, and marks the
|
||||
// whole fleet for rescanning when the version changes — which is what makes a
|
||||
// newly published CVE flag existing servers within a minute rather than at the
|
||||
// next agent report.
|
||||
func (s *scheduler) ensureDB(ctx context.Context) error {
|
||||
if s.store != nil && time.Since(s.pulled) < dbMaxAge {
|
||||
return nil
|
||||
}
|
||||
|
||||
version, err := vulndb.Pull(ctx, s.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.closeStore()
|
||||
store, err := vulndb.Open(s.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.store = store
|
||||
s.pulled = time.Now()
|
||||
|
||||
changed := version != s.version
|
||||
s.version = version
|
||||
|
||||
_, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{},
|
||||
bson.M{"$set": bson.M{"db_version": version, "pulled_at": s.pulled}, "$unset": bson.M{"last_error": ""}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
|
||||
if changed {
|
||||
res, err := db.Col("server_packages").UpdateMany(ctx,
|
||||
bson.M{"status": bson.M{"$ne": models.ScanStatusUnsupported}},
|
||||
bson.M{"$set": bson.M{"scan_pending": true}},
|
||||
)
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: mark fleet pending: %v", err)
|
||||
} else {
|
||||
log.Printf("vulnsched: database version %d, %d servers marked for rescan", version, res.ModifiedCount)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *scheduler) recordDBError(ctx context.Context, err error) {
|
||||
_, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{},
|
||||
bson.M{"$set": bson.M{"last_error": err.Error()}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
}
|
||||
|
||||
func (s *scheduler) scanPending(ctx context.Context) {
|
||||
cur, err := db.Col("server_packages").Find(ctx, bson.M{"scan_pending": true})
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: find pending: %v", err)
|
||||
return
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
|
||||
var pending []models.ServerPackages
|
||||
if err := cur.All(ctx, &pending); err != nil {
|
||||
log.Printf("vulnsched: decode pending: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Newly opened findings are collected across the whole tick and sent as one
|
||||
// digest per instance. A database refresh can open several hundred findings
|
||||
// at once; one message per finding would rate-limit the webhook or get the
|
||||
// channel muted, and either way the alerts stop being read.
|
||||
newly := map[string][]models.VulnFinding{}
|
||||
|
||||
for _, sp := range pending {
|
||||
if ctx.Err() != nil {
|
||||
// Leadership lost. scan_pending is still set, so the next leader
|
||||
// picks these up — which is why it lives on the document.
|
||||
return
|
||||
}
|
||||
opened := s.scanOne(ctx, sp)
|
||||
newly[sp.InstanceID] = append(newly[sp.InstanceID], opened...)
|
||||
}
|
||||
|
||||
for instanceID, findings := range newly {
|
||||
if len(findings) > 0 && s.deps.SendDigest != nil {
|
||||
s.deps.SendDigest(instanceID, findings)
|
||||
}
|
||||
}
|
||||
|
||||
_, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{},
|
||||
bson.M{"$set": bson.M{"last_full_scan_at": time.Now()}},
|
||||
options.UpdateOne().SetUpsert(true),
|
||||
)
|
||||
}
|
||||
|
||||
func (s *scheduler) scanOne(ctx context.Context, sp models.ServerPackages) []models.VulnFinding {
|
||||
now := time.Now()
|
||||
|
||||
results, err := vulndb.Match(s.store, sp.OS, sp.Packages)
|
||||
if err != nil {
|
||||
// We hold no feed for this distribution, so we cannot answer whether it
|
||||
// is vulnerable. Say "unsupported" — reporting zero findings here would
|
||||
// be indistinguishable from reporting a clean host, and one of those is
|
||||
// a lie.
|
||||
status := models.ScanStatusUnsupported
|
||||
if !errors.Is(err, vulndb.ErrUnsupportedFamily) {
|
||||
log.Printf("vulnsched: scan %s: %v", sp.ServerID, err)
|
||||
status = sp.Status
|
||||
}
|
||||
s.clearPending(ctx, sp.ID, status, now)
|
||||
return nil
|
||||
}
|
||||
|
||||
existing, err := services.ListFindings(ctx, sp.InstanceID, sp.ServerID)
|
||||
if err != nil {
|
||||
log.Printf("vulnsched: list findings %s: %v", sp.ServerID, err)
|
||||
return nil
|
||||
}
|
||||
|
||||
diff := services.DiffFindings(existing, results, now)
|
||||
if err := services.ApplyFindingDiff(ctx, sp.InstanceID, sp.ServerID, diff, now); err != nil {
|
||||
log.Printf("vulnsched: apply diff %s: %v", sp.ServerID, err)
|
||||
return nil
|
||||
}
|
||||
|
||||
s.clearPending(ctx, sp.ID, models.ScanStatusOK, now)
|
||||
|
||||
for i := range diff.NewlyOpened {
|
||||
diff.NewlyOpened[i].ServerID = sp.ServerID
|
||||
}
|
||||
return diff.NewlyOpened
|
||||
}
|
||||
|
||||
func (s *scheduler) clearPending(ctx context.Context, id bson.ObjectID, status string, now time.Time) {
|
||||
_, _ = db.Col("server_packages").UpdateOne(ctx,
|
||||
bson.M{"_id": id},
|
||||
bson.M{"$set": bson.M{
|
||||
"scan_pending": false,
|
||||
"status": status,
|
||||
"scanned_at": now,
|
||||
"db_version": s.version,
|
||||
}},
|
||||
)
|
||||
}
|
||||
|
||||
func (s *scheduler) closeStore() {
|
||||
if s.store != nil {
|
||||
_ = s.store.Close()
|
||||
s.store = nil
|
||||
}
|
||||
}
|
||||
@@ -31,6 +31,10 @@ const (
|
||||
|
||||
FeatureConsole = "console" // browser SSH/RDP/VNC
|
||||
FeatureOIDC = "oidc" // per-instance single sign-on
|
||||
// FeatureVulnScanning gates package inventory collection as well as the
|
||||
// findings themselves. The gate is at collection, not display: an ungated
|
||||
// instance stores no inventory, and storage is the expensive half.
|
||||
FeatureVulnScanning = "vuln_scanning"
|
||||
)
|
||||
|
||||
// Support levels. Carried for display and enforced by nothing — there is no code
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
{{define "title"}}New vulnerabilities detected{{end}}
|
||||
{{define "pill"}}{{template "chip" (dict "label" (upper .TopSeverity) "tone" "down")}}{{end}}
|
||||
{{define "body"}}
|
||||
{{template "lead" .Summary}}
|
||||
{{template "rows" (list
|
||||
(dict "k" "Instance" "v" .InstanceName)
|
||||
(dict "k" "New findings" "v" .Count))}}
|
||||
{{range .Rows}}
|
||||
{{if .FixedIn}}{{template "well" (printf "%s (%s) — %s on %s, fixed in %s" .CVEID .Severity .PackageName .ServerName .FixedIn)}}{{else}}{{template "well" (printf "%s (%s) — %s on %s, no fix published" .CVEID .Severity .PackageName .ServerName)}}{{end}}
|
||||
{{end}}
|
||||
{{if .More}}{{template "p" (printf "…and %d more." .More)}}{{end}}
|
||||
{{template "note" (printf "Scanned against a vulnerability database pulled %s ago." .DBAge)}}
|
||||
{{end}}
|
||||
@@ -0,0 +1,12 @@
|
||||
{{define "subject"}}{{.Count}} new {{if eq .Count 1}}vulnerability{{else}}vulnerabilities{{end}} on {{.InstanceName}}{{end}}
|
||||
{{define "title"}}New vulnerabilities detected{{end}}
|
||||
{{define "pill"}}{{.TopSeverity}}{{end}}
|
||||
{{define "body"}}
|
||||
{{template "lead" .Summary}}
|
||||
|
||||
{{range .Rows}}- {{.CVEID}} ({{.Severity}}) — {{.PackageName}} on {{.ServerName}}{{if .FixedIn}}, fixed in {{.FixedIn}}{{else}}, no fix published{{end}}
|
||||
{{end}}
|
||||
{{if .More}}...and {{.More}} more.{{end}}
|
||||
|
||||
Scanned against vulnerability database pulled {{.DBAge}} ago.
|
||||
{{end}}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user