From 6adee810dc9fb451652029d1d13d0b16c23f31b0 Mon Sep 17 00:00:00 2001 From: mrhid6 Date: Thu, 13 Aug 2026 09:41:32 +0000 Subject: [PATCH] feat: Cleanup old docs --- cloud-instance-creation-process.md | 30 - .../plans/2026-08-03-multi-auth-providers.md | 3047 ----------------- .../plans/2026-08-04-scheduled-workflows.md | 1071 ------ .../plans/2026-08-04-server-tags.md | 1308 ------- ...8-06-package-inventory-and-cve-findings.md | 2588 -------------- .../plans/2026-08-06-workload-registry.md | 1489 -------- .../plans/2026-08-12-api-tokens-openapi.md | 2065 ----------- .../plans/2026-08-12-instance-rename.md | 938 ----- .../2026-08-03-multi-auth-providers-design.md | 307 -- ...ver-tags-and-scheduled-workflows-design.md | 235 -- ...ckage-inventory-and-cve-findings-design.md | 538 --- .../2026-08-06-workload-registry-design.md | 441 --- .../2026-08-12-api-tokens-openapi-design.md | 297 -- .../2026-08-12-instance-rename-design.md | 236 -- 14 files changed, 14590 deletions(-) delete mode 100644 cloud-instance-creation-process.md delete mode 100644 docs/superpowers/plans/2026-08-03-multi-auth-providers.md delete mode 100644 docs/superpowers/plans/2026-08-04-scheduled-workflows.md delete mode 100644 docs/superpowers/plans/2026-08-04-server-tags.md delete mode 100644 docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md delete mode 100644 docs/superpowers/plans/2026-08-06-workload-registry.md delete mode 100644 docs/superpowers/plans/2026-08-12-api-tokens-openapi.md delete mode 100644 docs/superpowers/plans/2026-08-12-instance-rename.md delete mode 100644 docs/superpowers/specs/2026-08-03-multi-auth-providers-design.md delete mode 100644 docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md delete mode 100644 docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md delete mode 100644 docs/superpowers/specs/2026-08-06-workload-registry-design.md delete mode 100644 docs/superpowers/specs/2026-08-12-api-tokens-openapi-design.md delete mode 100644 docs/superpowers/specs/2026-08-12-instance-rename-design.md diff --git a/cloud-instance-creation-process.md b/cloud-instance-creation-process.md deleted file mode 100644 index f6cdc72..0000000 --- a/cloud-instance-creation-process.md +++ /dev/null @@ -1,30 +0,0 @@ -# Current cloud instance process - -The current processs for creating cloud instances is incorrect. - -At the moment the cloud instance process is the following: - -- Customer goes to `https://vantage.hostxtra.co.uk/start` then fills in the form. -- Customer is then sent and email to verify -- Customer clicks the link and the instance is created in the DB. -- Customer can then access the instance. - -As the `/start` process is auto creating a new instance this should default to the free tier instance. - -The issue is that this doesn't create an `account` and `admin_instance` on the admin side. - -The Cloud instance creation / account creation needs to be restructured. - -for context when I say `hq` I mean `admin` - -- customer goes to `https://vantage.hostxtra.co.uk/start` and fills in the form. - - This is where the HQ account is created. -- The customer is then sent and email to verify their email address. -- The customer can then access the HQ customer portal. -- The customer can then create a free new instance in the HQ portal. -- The cloud instance is created in the DB. -- The HQ `account` and `admin_instance` is created and populated in the DB. -- A `Free` License is created and attached to the instance. -- The customer is then sent an email letting them know the instance has been created and when the license expires. -- The customer will need to renew the license after expiry, if they are on a Free license. - - This is so that unused instances can be cleaned up if no renew after a length of time has passed. diff --git a/docs/superpowers/plans/2026-08-03-multi-auth-providers.md b/docs/superpowers/plans/2026-08-03-multi-auth-providers.md deleted file mode 100644 index 54e9493..0000000 --- a/docs/superpowers/plans/2026-08-03-multi-auth-providers.md +++ /dev/null @@ -1,3047 +0,0 @@ -# Multiple Auth Providers Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let an instance configure any number of named auth providers, show one login button per enabled provider, and allow local password login to be turned off. - -**Architecture:** A new `auth_providers` collection replaces the one-per-instance `instance_oidc` document. Provider identity moves into the URL (`/auth/oidc/:providerId/start|callback`) and into the Redis state token, which now carries `{instance_id, provider_id}` as JSON instead of a bare instance ID. Presets are a Go lookup table that expands to a real issuer URL on save, so nothing downstream knows a preset existed. GitHub is OAuth2 rather than OIDC and takes a second branch in the callback that converges on one shared `completeSSOLogin`. - -**Tech Stack:** Go 1.x, gin, `github.com/coreos/go-oidc/v3`, `golang.org/x/oauth2`, MongoDB (`mongo-driver/v2`), Redis (`go-redis`), Next.js 16 App Router, React, TanStack Query, Tailwind 3. - -**Spec:** `docs/superpowers/specs/2026-08-03-multi-auth-providers-design.md` - -## Global Constraints - -- Module path is `gitea.hostxtra.co.uk/mrhid6/vantage`. Server imports are `.../server/internal/`. -- Every document in the control plane carries `instance_id`, and **every service query is scoped by it**. No unscoped lookup by email, ever. -- `web/` carries **no hex colour values**. Use the existing Tailwind tokens only (`text-text-primary`, `bg-surface-2`, `border-border`, `text-danger`, `text-warning`, `text-success`, `bg-well`, `accent-accent`). -- `web/` is dark-only. Do not add a light theme or a `data-theme` selector. -- Client secrets are AES-256-GCM via the existing `encryptString`/`decryptString` in `server/internal/services`. A secret is never serialised to JSON — the struct tag is `json:"-"`. -- Every mutating API path writes an audit event via `services.LogEvent(instanceID, eventType, actor, serverID, keyID, details)`. Pass `""` for `serverID` and `keyID`. The actor is `actorFromCtx(c)`. -- The licence feature gate for all of this is `services.GetLicenseState(instanceID).Feature("oidc")`, unchanged. -- Go: run `cd server && go build ./... && go vet ./...` before every commit. Tests: `cd server && go test ./...`. -- Web: run `cd web && npx tsc --noEmit` before every commit that touches `web/`. -- Commit messages use the repo's existing conventional prefixes (`feat:`, `fix:`, `docs:`, `refactor:`). - -## Testing Reality - -This repo has three `*_test.go` files total and **no MongoDB or gin test harness**. Do not invent one — standing up `mongodb-memory-server`-style infrastructure is out of scope for this work and would be a larger change than the feature. - -Consequently: - -- Logic that is **pure** gets real Go tests: preset expansion, the lockout guard predicate, GitHub email selection, the state token codec. These tasks are genuinely TDD. -- Logic that requires **Mongo, Redis or a browser** gets explicit manual verification steps with exact commands and expected output. Perform them; do not skip them and do not claim a step passed without running it. - -## File Structure - -**Server — created:** - -| File | Responsibility | -| --- | --- | -| `server/internal/models/auth_provider.go` | The `AuthProvider` document. | -| `server/internal/auth/presets.go` | Preset table, issuer expansion, default scopes. Pure; no I/O. | -| `server/internal/auth/presets_test.go` | Tests for the above. | -| `server/internal/auth/state.go` | `oidcState` struct plus its Redis save/consume. | -| `server/internal/auth/state_test.go` | Codec round-trip tests. | -| `server/internal/auth/github.go` | The OAuth2 branch: token exchange to verified email. | -| `server/internal/auth/github_test.go` | Email-selection tests against `httptest`. | -| `server/internal/services/auth_provider.go` | CRUD, guards, index. All Mongo access for providers. | -| `server/internal/services/auth_provider_guard_test.go` | Tests for the pure guard predicate. | -| `server/internal/services/migrate_auth_providers.go` | Migration `0005_auth_providers`. | -| `server/internal/api/auth_providers.go` | The five `/api/auth/providers` handlers. | - -**Server — modified:** `server/internal/auth/oidc.go`, `server/internal/auth/local.go`, `server/internal/auth/session.go`, `server/internal/api/handlers.go`, `server/internal/api/instance.go`, `server/internal/services/coreindexes.go`, `server/internal/services/settings.go`, `shared/models/settings.go`, `server/cmd/main.go`. - -**Server — deleted:** `server/internal/models/instance_oidc.go`, `server/internal/services/instance_oidc.go`, and the two handlers in `instance.go`. - -**Web — created:** `web/components/settings/AuthProvidersCard.tsx`, `web/components/settings/ProviderIcon.tsx`. - -**Web — modified:** `web/lib/api.ts`, `web/app/login/page.tsx`, `web/app/(app)/settings/page.tsx`. - -**Web — deleted:** `web/components/settings/OIDCCard.tsx`. - ---- - -### Task 1: The AuthProvider model and preset table - -**Files:** -- Create: `server/internal/models/auth_provider.go` -- Create: `server/internal/auth/presets.go` -- Test: `server/internal/auth/presets_test.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `models.AuthProvider` struct (fields as written below). - - `auth.Preset` struct and `auth.Presets() []Preset`. - - `auth.PresetByID(id string) (Preset, bool)`. - - `auth.ExpandIssuer(presetID, input string) (string, error)`. - - `auth.DefaultScopes(presetID string) []string`. - - `auth.KindFor(presetID string) string` returning `"oidc"` or `"oauth2"`. - -- [ ] **Step 1: Write the failing test** - -Create `server/internal/auth/presets_test.go`: - -```go -package auth - -import "testing" - -func TestExpandIssuerEntra(t *testing.T) { - got, err := ExpandIssuer("entra", "c0ffee00-1111-2222-3333-444455556666") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - want := "https://login.microsoftonline.com/c0ffee00-1111-2222-3333-444455556666/v2.0" - if got != want { - t.Errorf("got %q, want %q", got, want) - } -} - -func TestExpandIssuerGoogleIgnoresInput(t *testing.T) { - got, err := ExpandIssuer("google", "anything") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if got != "https://accounts.google.com" { - t.Errorf("got %q", got) - } -} - -func TestExpandIssuerOkta(t *testing.T) { - got, err := ExpandIssuer("okta", "acme.okta.com") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if got != "https://acme.okta.com/oauth2/default" { - t.Errorf("got %q", got) - } -} - -// A custom provider is stored exactly as typed. Rewriting it would break an -// issuer whose discovery document lives on a non-obvious path. -func TestExpandIssuerCustomIsVerbatim(t *testing.T) { - got, err := ExpandIssuer("", "https://id.example.com/realms/main") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if got != "https://id.example.com/realms/main" { - t.Errorf("got %q", got) - } -} - -// GitHub is OAuth2 and has no issuer at all. Returning an empty string rather -// than an error keeps the save path branch-free. -func TestExpandIssuerGitHubIsEmpty(t *testing.T) { - got, err := ExpandIssuer("github", "") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if got != "" { - t.Errorf("got %q, want empty", got) - } -} - -func TestExpandIssuerRequiresInputWhenTemplated(t *testing.T) { - if _, err := ExpandIssuer("entra", ""); err == nil { - t.Fatal("expected an error for an empty tenant ID") - } -} - -func TestExpandIssuerRejectsUnknownPreset(t *testing.T) { - if _, err := ExpandIssuer("nope", "x"); err == nil { - t.Fatal("expected an error for an unknown preset") - } -} - -func TestKindFor(t *testing.T) { - if KindFor("github") != "oauth2" { - t.Error("github should be oauth2") - } - if KindFor("entra") != "oidc" { - t.Error("entra should be oidc") - } - if KindFor("") != "oidc" { - t.Error("custom should be oidc") - } -} - -func TestDefaultScopes(t *testing.T) { - got := DefaultScopes("github") - want := []string{"read:user", "user:email"} - if len(got) != len(want) { - t.Fatalf("got %v", got) - } - for i := range want { - if got[i] != want[i] { - t.Fatalf("got %v, want %v", got, want) - } - } -} - -// The returned slice must not alias the table, or one caller mutating its -// scopes rewrites the defaults for every provider created afterwards. -func TestDefaultScopesReturnsACopy(t *testing.T) { - a := DefaultScopes("google") - a[0] = "mutated" - b := DefaultScopes("google") - if b[0] == "mutated" { - t.Fatal("DefaultScopes returned an aliased slice") - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd server && go test ./internal/auth/ -run TestExpand -v` -Expected: FAIL to build, `undefined: ExpandIssuer`. - -- [ ] **Step 3: Write the model** - -Create `server/internal/models/auth_provider.go`: - -```go -package models - -import ( - "time" - - "go.mongodb.org/mongo-driver/v2/bson" -) - -// AuthProvider is one configured identity provider for one instance. -// -// ProviderID is a short random identifier rather than the Mongo _id: it appears -// in the callback URL a customer pastes into their identity provider, and an -// _id there would publish a database key. -type AuthProvider struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"_id,omitempty"` - InstanceID string `bson:"instance_id" json:"instance_id"` - ProviderID string `bson:"provider_id" json:"provider_id"` - Name string `bson:"name" json:"name"` - Kind string `bson:"kind" json:"kind"` // "oidc" | "oauth2" - Preset string `bson:"preset" json:"preset"` // "" for custom - Issuer string `bson:"issuer" json:"issuer"` - ClientID string `bson:"client_id" json:"client_id"` - ClientSecretEnc string `bson:"client_secret_enc,omitempty" json:"-"` - Scopes []string `bson:"scopes" json:"scopes"` - Enabled bool `bson:"enabled" json:"enabled"` - // CallbackNotice marks a provider whose redirect URI changed at the upgrade - // to per-provider callbacks. Set only by migration 0005; cleared when an - // administrator acknowledges it in settings. - CallbackNotice bool `bson:"callback_notice" json:"callback_notice"` - Order int `bson:"order" json:"order"` - CreatedAt time.Time `bson:"created_at" json:"created_at"` - UpdatedAt time.Time `bson:"updated_at" json:"updated_at"` -} - -const ( - KindOIDC = "oidc" - KindOAuth2 = "oauth2" -) -``` - -- [ ] **Step 4: Write the preset table** - -Create `server/internal/auth/presets.go`: - -```go -package auth - -import ( - "fmt" - "strings" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" -) - -// Preset describes one well-known identity provider. -// -// This is a Go table rather than a collection on purpose: adding a preset is a -// commit and a review, not a row somebody typed into production. -type Preset struct { - ID string `json:"id"` - Label string `json:"label"` // shown in the add-provider picker - Kind string `json:"kind"` // models.KindOIDC | models.KindOAuth2 - IssuerFormat string `json:"-"` // %s is replaced by InputValue; empty means no issuer - InputLabel string `json:"input_label"` // empty means the preset asks for nothing - InputHint string `json:"input_hint"` - Scopes []string `json:"-"` -} - -var presets = []Preset{ - { - ID: "entra", - Label: "Microsoft Entra ID", - Kind: models.KindOIDC, - IssuerFormat: "https://login.microsoftonline.com/%s/v2.0", - InputLabel: "Directory (tenant) ID", - InputHint: "Found in Entra under Overview. A UUID, not your domain name.", - Scopes: []string{"openid", "profile", "email"}, - }, - { - ID: "google", - Label: "Google Workspace", - Kind: models.KindOIDC, - IssuerFormat: "https://accounts.google.com", - Scopes: []string{"openid", "profile", "email"}, - }, - { - ID: "okta", - Label: "Okta", - Kind: models.KindOIDC, - IssuerFormat: "https://%s/oauth2/default", - InputLabel: "Okta org domain", - InputHint: "e.g. acme.okta.com — no scheme, no trailing slash.", - Scopes: []string{"openid", "profile", "email"}, - }, - { - ID: "github", - Label: "GitHub", - Kind: models.KindOAuth2, - Scopes: []string{"read:user", "user:email"}, - }, - { - ID: "", - Label: "Other (OpenID Connect)", - Kind: models.KindOIDC, - IssuerFormat: "%s", - InputLabel: "Issuer URL", - InputHint: "The discovery base, e.g. https://id.example.com/realms/main", - Scopes: []string{"openid", "profile", "email"}, - }, -} - -// Presets returns the table for the settings UI to render a picker from. -func Presets() []Preset { - out := make([]Preset, len(presets)) - copy(out, presets) - return out -} - -func PresetByID(id string) (Preset, bool) { - for _, p := range presets { - if p.ID == id { - return p, true - } - } - return Preset{}, false -} - -// ExpandIssuer turns what the customer typed into the issuer URL that gets -// stored. The stored value is always fully resolved, so nothing downstream has -// to know a preset was involved. -func ExpandIssuer(presetID, input string) (string, error) { - p, ok := PresetByID(presetID) - if !ok { - return "", fmt.Errorf("unknown provider preset %q", presetID) - } - if p.IssuerFormat == "" { - return "", nil // OAuth2 providers have no issuer - } - if !strings.Contains(p.IssuerFormat, "%s") { - return p.IssuerFormat, nil // fixed issuer, input ignored - } - input = strings.TrimSpace(strings.TrimSuffix(input, "/")) - if input == "" { - return "", fmt.Errorf("%s is required", p.InputLabel) - } - return fmt.Sprintf(p.IssuerFormat, input), nil -} - -func DefaultScopes(presetID string) []string { - p, ok := PresetByID(presetID) - if !ok { - return []string{"openid", "profile", "email"} - } - out := make([]string, len(p.Scopes)) - copy(out, p.Scopes) - return out -} - -func KindFor(presetID string) string { - if p, ok := PresetByID(presetID); ok { - return p.Kind - } - return models.KindOIDC -} -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cd server && go test ./internal/auth/ -v` -Expected: PASS, all nine tests. - -- [ ] **Step 6: Build and vet** - -Run: `cd server && go build ./... && go vet ./...` -Expected: no output. - -- [ ] **Step 7: Commit** - -```bash -git add server/internal/models/auth_provider.go server/internal/auth/presets.go server/internal/auth/presets_test.go -git commit -m "feat: add AuthProvider model and identity provider presets" -``` - ---- - -### Task 2: Provider service layer, guards and index - -**Files:** -- Create: `server/internal/services/auth_provider.go` -- Test: `server/internal/services/auth_provider_guard_test.go` -- Modify: `server/internal/services/coreindexes.go:33-39` (replace the `instance_oidc` index) - -**Interfaces:** -- Consumes: `models.AuthProvider`, `models.KindOIDC`, `auth` package NOT imported here (services must not import auth — auth imports services, and the reverse would be an import cycle). Preset expansion happens in the **API layer**, which may import both. -- Produces: - - `services.ListAuthProviders(instanceID string) ([]models.AuthProvider, error)` — all, any enabled state, sorted by `order` then `created_at`. - - `services.ListEnabledAuthProviders(instanceID string) ([]models.AuthProvider, error)` - - `services.GetAuthProvider(instanceID, providerID string) (*models.AuthProvider, error)` - - `services.GetAuthProviderSecret(instanceID, providerID string) (string, error)` - - `services.CreateAuthProvider(p *models.AuthProvider, clientSecret string) (*models.AuthProvider, error)` — assigns `ProviderID`, `CreatedAt`, `UpdatedAt`, `Order`. - - `services.UpdateAuthProvider(instanceID, providerID string, in AuthProviderUpdate) error` - - `services.DeleteAuthProvider(instanceID, providerID string) error` - - `services.AckAuthProviderNotice(instanceID, providerID string) error` - - `services.AuthProviderUpdate` struct. - - `services.ErrLastProvider`, `services.ErrLocalLoginRequired` sentinel errors. - - `services.CheckLockout(localEnabled bool, enabledProviders int) error` — the pure predicate. - - `services.EnsureAuthProviderIndexes() error` - -- [ ] **Step 1: Write the failing guard test** - -Create `server/internal/services/auth_provider_guard_test.go`: - -```go -package services - -import ( - "errors" - "testing" -) - -// CheckLockout answers one question: would this end state leave nobody able to -// sign in? It is pure so both endpoints that can reach the condition share it -// and cannot disagree. -func TestCheckLockoutAllowsLocalOnly(t *testing.T) { - if err := CheckLockout(true, 0); err != nil { - t.Errorf("local login with no providers must be allowed: %v", err) - } -} - -func TestCheckLockoutAllowsProvidersOnly(t *testing.T) { - if err := CheckLockout(false, 1); err != nil { - t.Errorf("providers with no local login must be allowed: %v", err) - } -} - -func TestCheckLockoutAllowsBoth(t *testing.T) { - if err := CheckLockout(true, 2); err != nil { - t.Errorf("both must be allowed: %v", err) - } -} - -func TestCheckLockoutRefusesNeither(t *testing.T) { - err := CheckLockout(false, 0) - if !errors.Is(err, ErrLockout) { - t.Fatalf("got %v, want ErrLockout", err) - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd server && go test ./internal/services/ -run TestCheckLockout -v` -Expected: FAIL to build, `undefined: CheckLockout`. - -- [ ] **Step 3: Write the service layer** - -Create `server/internal/services/auth_provider.go`: - -```go -package services - -import ( - "context" - "errors" - "fmt" - "strings" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -// ErrLockout is returned when a change would leave an instance with neither -// local password login nor an enabled provider — nobody could sign in, and no -// endpoint exists to undo it without database access. -var ErrLockout = errors.New("that would leave nobody able to sign in") - -const authProviderCol = "auth_providers" - -func authProviderCtx() (context.Context, context.CancelFunc) { - return context.WithTimeout(context.Background(), 5*time.Second) -} - -// CheckLockout is pure so the two endpoints that can reach this condition — -// saving settings and changing a provider — share one answer. -func CheckLockout(localEnabled bool, enabledProviders int) error { - if localEnabled || enabledProviders > 0 { - return nil - } - return ErrLockout -} - -func EnsureAuthProviderIndexes() error { - ctx, cancel := authProviderCtx() - defer cancel() - _, err := db.Col(authProviderCol).Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "provider_id", Value: 1}}, - Options: options.Index().SetUnique(true), - }) - return err -} - -func ListAuthProviders(instanceID string) ([]models.AuthProvider, error) { - ctx, cancel := authProviderCtx() - defer cancel() - cur, err := db.Col(authProviderCol).Find(ctx, - bson.M{"instance_id": instanceID}, - options.Find().SetSort(bson.D{{Key: "order", Value: 1}, {Key: "created_at", Value: 1}})) - if err != nil { - return nil, err - } - out := []models.AuthProvider{} - if err := cur.All(ctx, &out); err != nil { - return nil, err - } - return out, nil -} - -func ListEnabledAuthProviders(instanceID string) ([]models.AuthProvider, error) { - all, err := ListAuthProviders(instanceID) - if err != nil { - return nil, err - } - out := []models.AuthProvider{} - for _, p := range all { - if p.Enabled { - out = append(out, p) - } - } - return out, nil -} - -func CountEnabledAuthProviders(instanceID string) (int, error) { - enabled, err := ListEnabledAuthProviders(instanceID) - if err != nil { - return 0, err - } - return len(enabled), nil -} - -// GetAuthProvider is scoped by instance. There is deliberately no lookup by -// provider_id alone: a provider ID travels in a URL, and an unscoped lookup -// would let one instance's callback resolve another instance's provider. -func GetAuthProvider(instanceID, providerID string) (*models.AuthProvider, error) { - ctx, cancel := authProviderCtx() - defer cancel() - var p models.AuthProvider - err := db.Col(authProviderCol).FindOne(ctx, - bson.M{"instance_id": instanceID, "provider_id": providerID}).Decode(&p) - if err != nil { - return nil, err - } - return &p, nil -} - -func GetAuthProviderSecret(instanceID, providerID string) (string, error) { - p, err := GetAuthProvider(instanceID, providerID) - if err != nil { - return "", err - } - if p.ClientSecretEnc == "" { - return "", fmt.Errorf("provider %q has no client secret configured", p.Name) - } - return decryptString(p.ClientSecretEnc) -} - -func CreateAuthProvider(p *models.AuthProvider, clientSecret string) (*models.AuthProvider, error) { - if strings.TrimSpace(p.Name) == "" { - return nil, errors.New("name is required") - } - if strings.TrimSpace(p.ClientID) == "" { - return nil, errors.New("client ID is required") - } - if clientSecret == "" { - return nil, errors.New("client secret is required") - } - enc, err := encryptString(clientSecret) - if err != nil { - return nil, err - } - id, err := randomProviderID() - if err != nil { - return nil, err - } - - existing, err := ListAuthProviders(p.InstanceID) - if err != nil { - return nil, err - } - - now := time.Now() - p.ProviderID = id - p.ClientSecretEnc = enc - p.CallbackNotice = false - p.Order = len(existing) - p.CreatedAt = now - p.UpdatedAt = now - - ctx, cancel := authProviderCtx() - defer cancel() - if _, err := db.Col(authProviderCol).InsertOne(ctx, p); err != nil { - return nil, err - } - return p, nil -} - -// AuthProviderUpdate carries only what an edit may change. Pointer fields are -// "leave alone when nil", which is what lets an empty client secret mean "keep -// the stored one" rather than "erase it". -type AuthProviderUpdate struct { - Name *string - Issuer *string - ClientID *string - ClientSecret *string - Scopes *[]string - Enabled *bool - Order *int -} - -func UpdateAuthProvider(instanceID, providerID string, in AuthProviderUpdate) error { - set := bson.M{"updated_at": time.Now()} - if in.Name != nil { - if strings.TrimSpace(*in.Name) == "" { - return errors.New("name is required") - } - set["name"] = *in.Name - } - if in.Issuer != nil { - set["issuer"] = *in.Issuer - } - if in.ClientID != nil { - set["client_id"] = *in.ClientID - } - if in.Scopes != nil { - set["scopes"] = *in.Scopes - } - if in.Order != nil { - set["order"] = *in.Order - } - if in.Enabled != nil { - set["enabled"] = *in.Enabled - } - // An empty secret means "keep what is stored". Only a non-empty one writes. - if in.ClientSecret != nil && *in.ClientSecret != "" { - enc, err := encryptString(*in.ClientSecret) - if err != nil { - return err - } - set["client_secret_enc"] = enc - } - - ctx, cancel := authProviderCtx() - defer cancel() - res, err := db.Col(authProviderCol).UpdateOne(ctx, - bson.M{"instance_id": instanceID, "provider_id": providerID}, - bson.M{"$set": set}) - if err != nil { - return err - } - if res.MatchedCount == 0 { - return mongo.ErrNoDocuments - } - return nil -} - -func DeleteAuthProvider(instanceID, providerID string) error { - ctx, cancel := authProviderCtx() - defer cancel() - res, err := db.Col(authProviderCol).DeleteOne(ctx, - bson.M{"instance_id": instanceID, "provider_id": providerID}) - if err != nil { - return err - } - if res.DeletedCount == 0 { - return mongo.ErrNoDocuments - } - return nil -} - -func AckAuthProviderNotice(instanceID, providerID string) error { - ctx, cancel := authProviderCtx() - defer cancel() - _, err := db.Col(authProviderCol).UpdateOne(ctx, - bson.M{"instance_id": instanceID, "provider_id": providerID}, - bson.M{"$set": bson.M{"callback_notice": false}}) - return err -} -``` - -- [ ] **Step 4: Add the provider ID generator** - -Append to `server/internal/services/auth_provider.go`: - -```go -// randomProviderID is 8 bytes hex: short enough to read in a URL, wide enough -// that guessing one is not a way to enumerate an instance's providers. -func randomProviderID() (string, error) { - b := make([]byte, 8) - if _, err := rand.Read(b); err != nil { - return "", err - } - return hex.EncodeToString(b), nil -} -``` - -Add `"crypto/rand"` and `"encoding/hex"` to the import block. - -- [ ] **Step 5: Run the guard test to verify it passes** - -Run: `cd server && go test ./internal/services/ -run TestCheckLockout -v` -Expected: PASS, four tests. - -- [ ] **Step 6: Swap the index builder** - -In `server/internal/services/coreindexes.go`, replace lines 33-39 (the `instance_oidc` index block) with: - -```go - // auth_providers is control-plane only, so its index stays here. The - // (instance_id, provider_id) pair is unique because a duplicate provider_id - // inside one instance would make the callback ambiguous. - if _, err := db.Col("auth_providers").Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "provider_id", Value: 1}}, - Options: options.Index().SetUnique(true), - }); err != nil { - return err - } - return nil -``` - -Then delete `EnsureAuthProviderIndexes` from `auth_provider.go` — it is now redundant with `EnsureAuthIndexes`, and two builders for one index is how they drift. Remove `mongo.IndexModel` and `options` from `auth_provider.go`'s imports if nothing else there uses them (`options.Find` does, so keep `options`). - -- [ ] **Step 7: Build, vet, test** - -Run: `cd server && go build ./... && go vet ./... && go test ./...` -Expected: no build or vet output; tests PASS. - -- [ ] **Step 8: Commit** - -```bash -git add server/internal/services/auth_provider.go server/internal/services/auth_provider_guard_test.go server/internal/services/coreindexes.go -git commit -m "feat: add auth provider service layer and lockout guard" -``` - ---- - -### Task 3: `local_login_enabled` setting - -**Files:** -- Modify: `shared/models/settings.go:32-40` -- Modify: `server/internal/services/settings.go` -- Test: `server/internal/services/settings_local_login_test.go` (create) - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `shared.Settings.LocalLoginEnabled *bool` - - `models.LocalLoginEnabled(s *Settings) bool` — nil-safe reader, in `shared/models/settings.go`. - - `services.IsLocalLoginEnabled(instanceID string) bool` - -- [ ] **Step 1: Write the failing test** - -Create `server/internal/services/settings_local_login_test.go`: - -```go -package services - -import ( - "testing" - - shared "gitea.hostxtra.co.uk/mrhid6/vantage/shared/models" -) - -// Absent on every existing settings document. Go's zero value for bool is -// false, so a plain bool field would silently disable password login on every -// instance in the fleet at upgrade. The pointer is what makes absent mean on. -func TestLocalLoginEnabledDefaultsTrueWhenAbsent(t *testing.T) { - s := &shared.Settings{} - if !shared.LocalLoginEnabled(s) { - t.Fatal("absent local_login_enabled must read as enabled") - } -} - -func TestLocalLoginEnabledRespectsExplicitFalse(t *testing.T) { - no := false - s := &shared.Settings{LocalLoginEnabled: &no} - if shared.LocalLoginEnabled(s) { - t.Fatal("explicit false must read as disabled") - } -} - -func TestLocalLoginEnabledNilSettingsIsEnabled(t *testing.T) { - if !shared.LocalLoginEnabled(nil) { - t.Fatal("nil settings must read as enabled") - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd server && go test ./internal/services/ -run TestLocalLogin -v` -Expected: FAIL to build, `undefined: shared.LocalLoginEnabled`. - -- [ ] **Step 3: Add the field and reader** - -In `shared/models/settings.go`, add to the `Settings` struct after `WorkflowLogRetentionDays`: - -```go - // LocalLoginEnabled is a pointer because it is absent on every settings - // document written before this feature existed, and a plain bool would read - // absent as disabled — turning off password login for the entire fleet at - // upgrade. Nil means enabled. - LocalLoginEnabled *bool `bson:"local_login_enabled,omitempty" json:"local_login_enabled,omitempty"` -``` - -And append to the same file: - -```go -// LocalLoginEnabled reads the setting with its absent-means-on default. Every -// caller must go through this rather than dereferencing the field. -func LocalLoginEnabled(s *Settings) bool { - if s == nil || s.LocalLoginEnabled == nil { - return true - } - return *s.LocalLoginEnabled -} -``` - -- [ ] **Step 4: Add the service reader** - -Append to `server/internal/services/auth_provider.go`: - -```go -// IsLocalLoginEnabled fails open. A settings read error must not lock an -// instance out of its own login page, and the safe direction here is the one -// that still asks for a password. -func IsLocalLoginEnabled(instanceID string) bool { - s, err := GetSettings(instanceID) - if err != nil { - return true - } - return shared.LocalLoginEnabled(s) -} -``` - -Add `shared "gitea.hostxtra.co.uk/mrhid6/vantage/shared/models"` to the imports. If `GetSettings` has a different name or signature in `server/internal/services/settings.go`, use the real one — check it with `grep -n "func GetSettings" server/internal/services/settings.go` and adapt. - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cd server && go test ./internal/services/ -v` -Expected: PASS. - -- [ ] **Step 6: Enforce the guard when settings are saved** - -In `server/internal/services/settings.go`, find the save function (`SaveSettings` or equivalent). Before it writes, add: - -```go - // The guard lives here rather than in the handler so the settings path and - // the provider path cannot disagree about what a lockout is. - if in.LocalLoginEnabled != nil && !*in.LocalLoginEnabled { - n, err := CountEnabledAuthProviders(instanceID) - if err != nil { - return err - } - if err := CheckLockout(false, n); err != nil { - return err - } - } -``` - -Adapt the receiver name (`in`, `s`, `settings`) to whatever the existing signature uses. - -- [ ] **Step 7: Map the guard to a 409 in the handler** - -`ErrLockout` is a refusal about the resource's state, not a server fault, and a -500 would tell the operator Vantage broke rather than that they were stopped. - -In `server/internal/api/handlers.go`, find `saveSettings` (it is in whichever -file defines it — `grep -rn "func saveSettings" server/internal/api/`). Where it -handles the error from the service call, add before the existing 500: - -```go - if errors.Is(err, services.ErrLockout) { - c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "local_login_required"}) - return - } -``` - -Add `"errors"` to that file's imports if absent. - -- [ ] **Step 8: Build, vet, test** - -Run: `cd server && go build ./... && go vet ./... && go test ./...` -Expected: clean. - -- [ ] **Step 9: Commit** - -```bash -git add shared/models/settings.go server/internal/services/settings.go server/internal/services/auth_provider.go server/internal/services/settings_local_login_test.go server/internal/api -git commit -m "feat: add local_login_enabled setting with absent-means-on default" -``` - ---- - -### Task 4: Migration 0005 - -**Files:** -- Create: `server/internal/services/migrate_auth_providers.go` -- Modify: `server/cmd/main.go:79-91` (call it inside `runSchemaSetup`) - -**Interfaces:** -- Consumes: `models.AuthProvider`, `models.KindOIDC`. -- Produces: `services.MigrateAuthProviders() error`. - -- [ ] **Step 1: Write the migration** - -Create `server/internal/services/migrate_auth_providers.go`: - -```go -package services - -import ( - "context" - "log" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "go.mongodb.org/mongo-driver/v2/bson" -) - -// legacyInstanceOIDC is the pre-0005 shape: one document per instance, in a -// collection this migration is the last thing to read. -type legacyInstanceOIDC struct { - InstanceID string `bson:"instance_id"` - Issuer string `bson:"issuer"` - ClientID string `bson:"client_id"` - ClientSecretEnc string `bson:"client_secret_enc"` - Enabled bool `bson:"enabled"` - UpdatedAt time.Time `bson:"updated_at"` -} - -// MigrateAuthProviders copies each instance_oidc document into auth_providers. -// -// The ciphertext is copied verbatim rather than decrypted and re-encrypted: a -// migration that needs KEY_ENCRYPTION_KEY fails on an instance that has none -// and strands the SSO configuration it was supposed to preserve. -// -// instance_oidc is left in place and no longer read. Nothing deletes it — a -// migration that drops the only copy of a client secret has no undo. -func MigrateAuthProviders() error { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) - defer cancel() - - const marker = "0005_auth_providers" - if n, _ := db.Col("migrations").CountDocuments(ctx, bson.M{"_id": marker}); n > 0 { - return nil - } - - cur, err := db.Col("instance_oidc").Find(ctx, bson.M{}) - if err != nil { - return err - } - var legacy []legacyInstanceOIDC - if err := cur.All(ctx, &legacy); err != nil { - return err - } - - migrated := 0 - for _, l := range legacy { - if l.InstanceID == "" { - continue - } - // Idempotent by skipping an instance that already has a provider, so a - // re-run after a partial failure completes rather than duplicating. - n, err := db.Col(authProviderCol).CountDocuments(ctx, bson.M{"instance_id": l.InstanceID}) - if err != nil { - return err - } - if n > 0 { - continue - } - providerID, err := randomProviderID() - if err != nil { - return err - } - now := time.Now() - p := models.AuthProvider{ - InstanceID: l.InstanceID, - ProviderID: providerID, - Name: "Single sign-on", - Kind: models.KindOIDC, - Preset: "", - Issuer: l.Issuer, - ClientID: l.ClientID, - ClientSecretEnc: l.ClientSecretEnc, - Scopes: []string{"openid", "profile", "email"}, - Enabled: l.Enabled, - // This provider's redirect URI has changed and nobody has been told - // yet. The settings card raises it until an administrator dismisses. - CallbackNotice: true, - Order: 0, - CreatedAt: now, - UpdatedAt: now, - } - if _, err := db.Col(authProviderCol).InsertOne(ctx, p); err != nil { - return err - } - migrated++ - } - if migrated > 0 { - log.Printf("0005: migrated %d OIDC configuration(s) to auth_providers", migrated) - } - - _, err = db.Col("migrations").InsertOne(ctx, bson.M{"_id": marker, "applied_at": time.Now()}) - return err -} -``` - -- [ ] **Step 2: Wire it into boot** - -In `server/cmd/main.go`, inside `runSchemaSetup`, after the `EnsureAuthIndexes` block (currently lines 100-102) and before `EnsureSecretIndexes`, add: - -```go - // 0005 runs AFTER EnsureAuthIndexes: the unique (instance_id, provider_id) - // index must exist before anything inserts providers, or a concurrent - // re-run could double-insert before the index is there to refuse it. - if err := services.MigrateAuthProviders(); err != nil { - log.Fatalf("auth provider migration failed: %v", err) - } -``` - -- [ ] **Step 3: Build and vet** - -Run: `cd server && go build ./... && go vet ./...` -Expected: no output. - -- [ ] **Step 4: Verify against a real database** - -This is the manual step — there is no Mongo test harness. Against a development database that has at least one `instance_oidc` document: - -```bash -# before -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.instance_oidc.countDocuments({})' -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.auth_providers.countDocuments({})' -``` - -Boot the server once (`cd server && go run ./cmd`), then: - -```bash -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.auth_providers.find({}, {name:1, enabled:1, callback_notice:1, provider_id:1, client_secret_enc:1}).pretty()' -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.migrations.findOne({_id:"0005_auth_providers"})' -``` - -Expected: one `auth_providers` document per `instance_oidc` document, `name: "Single sign-on"`, `callback_notice: true`, `enabled` matching the source, `client_secret_enc` **byte-identical** to the source, and the marker present. - -- [ ] **Step 5: Verify idempotency** - -Boot the server a second time, then re-run the count: - -```bash -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.auth_providers.countDocuments({})' -``` - -Expected: the same number as after the first boot. Then delete the marker and boot a third time to exercise the skip-if-present branch: - -```bash -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.migrations.deleteOne({_id:"0005_auth_providers"})' -``` - -Expected after boot: still the same count — the per-instance skip caught it. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/services/migrate_auth_providers.go server/cmd/main.go -git commit -m "feat: migrate instance_oidc into auth_providers (0005)" -``` - ---- - -### Task 5: State token carries the provider - -**Files:** -- Create: `server/internal/auth/state.go` -- Test: `server/internal/auth/state_test.go` -- Modify: `server/internal/auth/session.go:93-103` (delete the two old functions) - -**Interfaces:** -- Consumes: the `rdb`, `statePrefix` and `randomHex` already in `server/internal/auth/session.go`. -- Produces: - - `auth.oidcState` struct with `InstanceID` and `ProviderID`. - - `auth.saveState(ctx context.Context, state string, s oidcState) error` - - `auth.consumeState(ctx context.Context, state string) (oidcState, bool)` - - `auth.encodeState(s oidcState) (string, error)` / `auth.decodeState(raw string) (oidcState, bool)` — exported to the package only, tested directly. - -- [ ] **Step 1: Write the failing test** - -Create `server/internal/auth/state_test.go`: - -```go -package auth - -import "testing" - -func TestStateRoundTrip(t *testing.T) { - in := oidcState{InstanceID: "inst-1", ProviderID: "ab12cd34"} - raw, err := encodeState(in) - if err != nil { - t.Fatalf("encode: %v", err) - } - out, ok := decodeState(raw) - if !ok { - t.Fatal("decode returned not-ok") - } - if out != in { - t.Errorf("got %+v, want %+v", out, in) - } -} - -func TestDecodeStateRejectsGarbage(t *testing.T) { - if _, ok := decodeState("not json"); ok { - t.Fatal("garbage must not decode") - } -} - -func TestDecodeStateRejectsEmpty(t *testing.T) { - if _, ok := decodeState(""); ok { - t.Fatal("empty must not decode") - } -} - -// A state whose instance is missing is not a usable state: the callback would -// have nothing to scope its user lookup by. -func TestDecodeStateRejectsMissingInstance(t *testing.T) { - if _, ok := decodeState(`{"provider_id":"ab12cd34"}`); ok { - t.Fatal("a state without an instance must not decode") - } -} - -func TestDecodeStateRejectsMissingProvider(t *testing.T) { - if _, ok := decodeState(`{"instance_id":"inst-1"}`); ok { - t.Fatal("a state without a provider must not decode") - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd server && go test ./internal/auth/ -run TestState -v` -Expected: FAIL to build, `undefined: oidcState`. - -- [ ] **Step 3: Write the state codec** - -Create `server/internal/auth/state.go`: - -```go -package auth - -import ( - "context" - "encoding/json" - "time" -) - -// oidcState is what a login flow parks in Redis between the start redirect and -// the callback. It carries the provider as well as the instance: the callback's -// :providerId path segment is attacker-controlled, and this is the half that -// was issued by the start handler. -type oidcState struct { - InstanceID string `json:"instance_id"` - ProviderID string `json:"provider_id"` -} - -func encodeState(s oidcState) (string, error) { - b, err := json.Marshal(s) - if err != nil { - return "", err - } - return string(b), nil -} - -func decodeState(raw string) (oidcState, bool) { - var s oidcState - if raw == "" { - return oidcState{}, false - } - if err := json.Unmarshal([]byte(raw), &s); err != nil { - return oidcState{}, false - } - if s.InstanceID == "" || s.ProviderID == "" { - return oidcState{}, false - } - return s, true -} - -func saveState(ctx context.Context, state string, s oidcState) error { - raw, err := encodeState(s) - if err != nil { - return err - } - return rdb.Set(ctx, statePrefix+state, raw, 10*time.Minute).Err() -} - -// consumeState is GetDel: a state is single-use, so a replayed callback finds -// nothing and is refused. -func consumeState(ctx context.Context, state string) (oidcState, bool) { - raw, err := rdb.GetDel(ctx, statePrefix+state).Result() - if err != nil { - return oidcState{}, false - } - return decodeState(raw) -} -``` - -- [ ] **Step 4: Delete the old state functions** - -In `server/internal/auth/session.go`, delete `SaveStateInstance` and `ConsumeStateInstance` (lines 93-103). Remove `"time"` from its imports only if nothing else in the file uses it — `sessionTTL` likely does, so check with `go build` rather than assuming. - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cd server && go test ./internal/auth/ -v` -Expected: PASS. The package will not compile until Task 6 updates `oidc.go`'s callers — if `go test` fails to build for that reason, that is expected here and Task 6 resolves it. To keep this task independently green, do Step 6 before committing. - -- [ ] **Step 6: Stub the callers so the package builds** - -In `server/internal/auth/oidc.go`, change `SaveStateInstance(ctx, state, inst.InstanceID)` to `saveState(ctx, state, oidcState{InstanceID: inst.InstanceID, ProviderID: ""})` and `ConsumeStateInstance(ctx, c.Query("state"))` to: - -```go - st, ok := consumeState(ctx, c.Query("state")) - if !ok { - c.JSON(http.StatusBadRequest, gin.H{"error": "invalid state"}) - return - } - instanceID := st.InstanceID -``` - -This is temporary scaffolding that Task 6 replaces wholesale; it exists so this task's tests can run. - -- [ ] **Step 7: Build, vet, test** - -Run: `cd server && go build ./... && go vet ./... && go test ./internal/auth/ -v` -Expected: clean build; state tests PASS. - -- [ ] **Step 8: Commit** - -```bash -git add server/internal/auth/state.go server/internal/auth/state_test.go server/internal/auth/session.go server/internal/auth/oidc.go -git commit -m "refactor: carry provider id in the OIDC state token" -``` - ---- - -### Task 6: Per-provider OIDC flow - -**Files:** -- Modify: `server/internal/auth/oidc.go` (rewrite most of it) -- Modify: `server/internal/api/handlers.go:38-39` (route change) - -**Interfaces:** -- Consumes: `services.GetAuthProvider`, `services.GetAuthProviderSecret`, `auth.saveState`, `auth.consumeState`, `models.KindOIDC`, `models.KindOAuth2`. -- Produces: - - `auth.HandleSSOStart(c *gin.Context)` — replaces `HandleOIDCStart`. - - `auth.HandleSSOCallback(c *gin.Context)` — replaces `HandleOIDCCallback`. - - `auth.EvictProvider(providerID string)` — replaces `EvictOIDCProvider`. - - `auth.completeSSOLogin(c *gin.Context, instanceID, email, name string)` — shared tail, used by Task 7's GitHub branch. - - `auth.CallbackURL(c *gin.Context, providerID string) string` — exported so the settings API can report it. - -- [ ] **Step 1: Rewrite `oidc.go`** - -Replace the whole file with: - -```go -package auth - -import ( - "context" - "fmt" - "net/http" - "strings" - "sync" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "github.com/coreos/go-oidc/v3/oidc" - "github.com/gin-gonic/gin" - "golang.org/x/oauth2" -) - -var ( - provMu sync.Mutex - provCache = map[string]*oidc.Provider{} -) - -// EvictProvider drops a cached discovery document. Keyed on provider, not -// instance: an instance now has several, and evicting all of them because one -// changed would re-fetch discovery for providers nobody touched. -func EvictProvider(providerID string) { - provMu.Lock() - delete(provCache, providerID) - provMu.Unlock() -} - -func requestScheme(c *gin.Context) string { - if c.Request.TLS == nil && c.GetHeader("X-Forwarded-Proto") != "https" { - return "http" - } - return "https" -} - -// CallbackURL must return the same string in the start and callback halves of -// one flow, or the identity provider rejects the token exchange. -func CallbackURL(c *gin.Context, providerID string) string { - return fmt.Sprintf("%s://%s/auth/oidc/%s/callback", requestScheme(c), c.Request.Host, providerID) -} - -func oauthConfigFor(ctx context.Context, c *gin.Context, p *models.AuthProvider, secret string) (*oidc.Provider, *oauth2.Config, error) { - if p.Kind == models.KindOAuth2 { - return nil, githubOAuthConfig(p, secret, CallbackURL(c, p.ProviderID)), nil - } - provMu.Lock() - prov := provCache[p.ProviderID] - provMu.Unlock() - if prov == nil { - var err error - prov, err = oidc.NewProvider(ctx, p.Issuer) - if err != nil { - return nil, nil, fmt.Errorf("provider discovery failed: %w", err) - } - provMu.Lock() - provCache[p.ProviderID] = prov - provMu.Unlock() - } - return prov, &oauth2.Config{ - ClientID: p.ClientID, - ClientSecret: secret, - RedirectURL: CallbackURL(c, p.ProviderID), - Endpoint: prov.Endpoint(), - Scopes: p.Scopes, - }, nil -} - -// loadProvider resolves a provider strictly within one instance. There is no -// unscoped lookup: a provider ID travels in a URL, and an unscoped one would -// let a request against one instance's host drive another instance's provider. -func loadProvider(instanceID, providerID string) (*models.AuthProvider, string, error) { - p, err := services.GetAuthProvider(instanceID, providerID) - if err != nil { - return nil, "", fmt.Errorf("unknown provider") - } - if !p.Enabled { - return nil, "", fmt.Errorf("provider is disabled") - } - secret, err := services.GetAuthProviderSecret(instanceID, providerID) - if err != nil { - return nil, "", err - } - return p, secret, nil -} - -func HandleSSOStart(c *gin.Context) { - inst, ok := InstanceFromHost(c) - if !ok { - c.Redirect(http.StatusFound, "/login?error=unknown_host") - return - } - // Losing the feature stops new SSO logins. It deliberately does not touch - // session validation, so nobody is evicted mid-session. - if !services.GetLicenseState(inst.InstanceID).Feature("oidc") { - c.Redirect(http.StatusFound, "/login?error=oidc_unavailable") - return - } - providerID := c.Param("providerId") - p, secret, err := loadProvider(inst.InstanceID, providerID) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=provider_unavailable") - return - } - ctx := c.Request.Context() - _, oauthCfg, err := oauthConfigFor(ctx, c, p, secret) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=provider_unreachable") - return - } - state, err := randomHex(16) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=state_failed") - return - } - if err := saveState(ctx, state, oidcState{InstanceID: inst.InstanceID, ProviderID: p.ProviderID}); err != nil { - c.Redirect(http.StatusFound, "/login?error=state_failed") - return - } - c.Redirect(http.StatusFound, oauthCfg.AuthCodeURL(state)) -} - -func HandleSSOCallback(c *gin.Context) { - ctx := c.Request.Context() - st, ok := consumeState(ctx, c.Query("state")) - if !ok { - c.Redirect(http.StatusFound, "/login?error=invalid_state") - return - } - - // The path segment is attacker-controlled; the state was issued by the start - // handler. A mismatch means the two halves of this flow disagree about which - // provider is signing somebody in, and that is not a thing to resolve by - // picking one. The state has already been consumed, so this is not replayable. - if c.Param("providerId") != st.ProviderID { - c.Redirect(http.StatusFound, "/login?error=invalid_state") - return - } - - // The start handler checks this too, but an ungated callback is the half - // that matters: a start that refuses is a dead end, while a callback that - // completes signs somebody in. A licence that lapsed mid-flow stops the - // exchange here rather than after it. - // - // Resolved from the consumed state rather than from the host, because on - // this route the instance is whatever the state said and nobody is signed - // in yet. - if !services.GetLicenseState(st.InstanceID).Feature("oidc") { - c.Redirect(http.StatusFound, "/login?error=oidc_unavailable") - return - } - - p, secret, err := loadProvider(st.InstanceID, st.ProviderID) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=provider_unavailable") - return - } - provider, oauthCfg, err := oauthConfigFor(ctx, c, p, secret) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=provider_unreachable") - return - } - token, err := oauthCfg.Exchange(ctx, c.Query("code")) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=exchange_failed") - return - } - - var email, name string - if p.Kind == models.KindOAuth2 { - email, name, err = githubIdentity(ctx, oauthCfg, token) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=identity_failed") - return - } - } else { - rawIDToken, ok := token.Extra("id_token").(string) - if !ok { - c.Redirect(http.StatusFound, "/login?error=missing_id_token") - return - } - idToken, err := provider.Verifier(&oidc.Config{ClientID: oauthCfg.ClientID}).Verify(ctx, rawIDToken) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=verification_failed") - return - } - var claims struct { - Email string `json:"email"` - Name string `json:"name"` - } - if err := idToken.Claims(&claims); err != nil || claims.Email == "" { - c.Redirect(http.StatusFound, "/login?error=missing_email") - return - } - email, name = claims.Email, claims.Name - } - - completeSSOLogin(c, st.InstanceID, email, name) -} - -// completeSSOLogin is the tail both provider kinds share: resolve the user -// within the instance, provision on first sign-in, mint the session. -func completeSSOLogin(c *gin.Context, instanceID, email, name string) { - email = strings.ToLower(strings.TrimSpace(email)) - if email == "" { - c.Redirect(http.StatusFound, "/login?error=missing_email") - return - } - - // Scoped to the instance the callback state names, so an address that also - // exists in another instance is invisible here. That scoping replaces the - // cross-instance guard this code used to need: there is no longer a way for - // the lookup to return a user belonging to somebody else. - u, err := services.GetUserInInstanceByEmail(instanceID, email) - if err != nil { - u, err = services.CreateUser(instanceID, email, "", models.RoleMember, models.AuthOIDC) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=provisioning_failed") - return - } - } - - sessionID, err := SaveSession(c.Request.Context(), &Session{ - UserID: u.UserID, InstanceID: u.InstanceID, Role: u.Role, Email: u.Email, Name: name, - }) - if err != nil { - c.Redirect(http.StatusFound, "/login?error=session_failed") - return - } - _ = services.TouchLastLogin(u.UserID) - SetSessionCookie(c, sessionID) - c.Redirect(http.StatusFound, "/") -} - -func HandleLogout(c *gin.Context) { - if cookie, err := c.Request.Cookie(sessionCookieName); err == nil { - _ = DeleteSession(c.Request.Context(), cookie.Value) - } - http.SetCookie(c.Writer, &http.Cookie{ - Name: sessionCookieName, - Value: "", - Path: "/", - HttpOnly: true, - MaxAge: -1, - }) - c.Redirect(http.StatusFound, "/") -} -``` - -Note the failure mode change: this file used to answer JSON errors on the callback, which a browser mid-redirect renders as raw JSON. Every failure now redirects to `/login?error=` so the user lands somewhere they can act. - -- [ ] **Step 2: Swap the routes** - -In `server/internal/api/handlers.go`, replace lines 38-39 with: - -```go - r.GET("/auth/oidc/:providerId/start", auth.HandleSSOStart) - r.GET("/auth/oidc/:providerId/callback", auth.HandleSSOCallback) - r.GET("/auth/providers", auth.HandleListPublicProviders) -``` - -`HandleListPublicProviders` arrives in Task 8; if this task is being built alone, add the route then instead and keep the first two here. - -- [ ] **Step 3: Build** - -Run: `cd server && go build ./...` -Expected: failures naming `githubOAuthConfig` and `githubIdentity` — those are Task 7. If Task 7 is not yet done, add a temporary `server/internal/auth/github.go` containing only the two signatures returning `nil` and `("", "", errors.New("not implemented"))`, and remove the stub in Task 7. - -- [ ] **Step 4: Verify gin accepts the route shape** - -gin cannot register two different wildcard names at the same path position. `/auth/oidc/:providerId/start` and `/auth/oidc/:providerId/callback` share `:providerId`, so they are fine — but if any other route registers `/auth/oidc/:somethingElse`, gin panics at boot. - -Run: `cd server && go run ./cmd 2>&1 | head -20` -Expected: normal boot logs, **no** panic mentioning `wildcard segment`. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/auth/oidc.go server/internal/api/handlers.go -git commit -m "feat: per-provider SSO start and callback routes" -``` - ---- - -### Task 7: GitHub OAuth2 branch - -**Files:** -- Create: `server/internal/auth/github.go` -- Test: `server/internal/auth/github_test.go` - -**Interfaces:** -- Consumes: `models.AuthProvider`. -- Produces: - - `auth.githubOAuthConfig(p *models.AuthProvider, secret, redirectURL string) *oauth2.Config` - - `auth.githubIdentity(ctx context.Context, cfg *oauth2.Config, token *oauth2.Token) (email, name string, err error)` - - `auth.selectGitHubEmail(emails []githubEmail) (string, error)` — the pure part, tested directly. - - `auth.githubEmail` struct: `{Email string; Primary bool; Verified bool}`. - - `auth.githubAPIBase` package variable, so tests can point it at `httptest`. - -- [ ] **Step 1: Write the failing test** - -Create `server/internal/auth/github_test.go`: - -```go -package auth - -import ( - "context" - "encoding/json" - "net/http" - "net/http/httptest" - "testing" - - "golang.org/x/oauth2" -) - -func TestSelectGitHubEmailPrefersPrimaryVerified(t *testing.T) { - got, err := selectGitHubEmail([]githubEmail{ - {Email: "alt@example.com", Primary: false, Verified: true}, - {Email: "me@example.com", Primary: true, Verified: true}, - }) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if got != "me@example.com" { - t.Errorf("got %q", got) - } -} - -// An unverified address is not proof of control. Accepting one would let -// anyone holding a GitHub account claim any address in the instance, including -// an owner's. -func TestSelectGitHubEmailRefusesUnverifiedPrimary(t *testing.T) { - if _, err := selectGitHubEmail([]githubEmail{ - {Email: "me@example.com", Primary: true, Verified: false}, - }); err == nil { - t.Fatal("an unverified primary address must be refused") - } -} - -// Deliberately NOT falling back to a verified non-primary address: the spec -// requires both, and a silent fallback would sign somebody in as an identity -// they did not choose to present. -func TestSelectGitHubEmailRefusesVerifiedNonPrimaryOnly(t *testing.T) { - if _, err := selectGitHubEmail([]githubEmail{ - {Email: "alt@example.com", Primary: false, Verified: true}, - }); err == nil { - t.Fatal("a verified but non-primary address must be refused") - } -} - -func TestSelectGitHubEmailRefusesEmpty(t *testing.T) { - if _, err := selectGitHubEmail(nil); err == nil { - t.Fatal("no addresses must be refused") - } -} - -func TestGitHubIdentityReadsAPI(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - switch r.URL.Path { - case "/user/emails": - _ = json.NewEncoder(w).Encode([]githubEmail{ - {Email: "me@example.com", Primary: true, Verified: true}, - }) - case "/user": - _ = json.NewEncoder(w).Encode(map[string]string{"name": "Real Name"}) - default: - w.WriteHeader(http.StatusNotFound) - } - })) - defer srv.Close() - - old := githubAPIBase - githubAPIBase = srv.URL - defer func() { githubAPIBase = old }() - - email, name, err := githubIdentity(context.Background(), - &oauth2.Config{}, &oauth2.Token{AccessToken: "t"}) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if email != "me@example.com" { - t.Errorf("email: got %q", email) - } - if name != "Real Name" { - t.Errorf("name: got %q", name) - } -} - -// A name is cosmetic; an email is identity. A failing /user must not fail the -// sign-in. -func TestGitHubIdentityToleratesMissingName(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path == "/user/emails" { - w.Header().Set("Content-Type", "application/json") - _ = json.NewEncoder(w).Encode([]githubEmail{ - {Email: "me@example.com", Primary: true, Verified: true}, - }) - return - } - w.WriteHeader(http.StatusInternalServerError) - })) - defer srv.Close() - - old := githubAPIBase - githubAPIBase = srv.URL - defer func() { githubAPIBase = old }() - - email, name, err := githubIdentity(context.Background(), - &oauth2.Config{}, &oauth2.Token{AccessToken: "t"}) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if email != "me@example.com" { - t.Errorf("email: got %q", email) - } - if name != "" { - t.Errorf("name should be empty, got %q", name) - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd server && go test ./internal/auth/ -run TestGitHub -v` -Expected: FAIL to build, `undefined: selectGitHubEmail`. - -- [ ] **Step 3: Write the implementation** - -Create `server/internal/auth/github.go` (replacing any stub from Task 6): - -```go -package auth - -import ( - "context" - "encoding/json" - "errors" - "net/http" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "golang.org/x/oauth2" -) - -// githubAPIBase is a variable rather than a constant so tests can point it at -// an httptest server. Nothing in production reassigns it. -var githubAPIBase = "https://api.github.com" - -const ( - githubAuthURL = "https://github.com/login/oauth/authorize" - githubTokenURL = "https://github.com/login/oauth/access_token" -) - -type githubEmail struct { - Email string `json:"email"` - Primary bool `json:"primary"` - Verified bool `json:"verified"` -} - -// githubOAuthConfig builds the OAuth2 config for a GitHub provider. GitHub has -// no discovery document, so the endpoints are constants rather than fetched. -func githubOAuthConfig(p *models.AuthProvider, secret, redirectURL string) *oauth2.Config { - return &oauth2.Config{ - ClientID: p.ClientID, - ClientSecret: secret, - RedirectURL: redirectURL, - Scopes: p.Scopes, - Endpoint: oauth2.Endpoint{ - AuthURL: githubAuthURL, - TokenURL: githubTokenURL, - }, - } -} - -// selectGitHubEmail requires an address that is both primary and verified. -// -// Verified alone is not enough: a non-primary address is one the person happens -// to have proved, not the one they present as themselves. Primary alone is far -// worse — an unverified address is not proof of control at all, and accepting -// one would let anyone with a GitHub account claim any address in the instance. -func selectGitHubEmail(emails []githubEmail) (string, error) { - for _, e := range emails { - if e.Primary && e.Verified && e.Email != "" { - return e.Email, nil - } - } - return "", errors.New("no primary verified email address on the GitHub account") -} - -func githubGetJSON(ctx context.Context, client *http.Client, url string, out any) error { - req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) - if err != nil { - return err - } - req.Header.Set("Accept", "application/vnd.github+json") - resp, err := client.Do(req) - if err != nil { - return err - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - return errors.New("github api returned " + resp.Status) - } - return json.NewDecoder(resp.Body).Decode(out) -} - -// githubIdentity resolves the signed-in GitHub account to an email and a name. -// -// The name is best-effort: it is cosmetic, and a failing /user must not fail a -// sign-in whose identity is already established. -func githubIdentity(ctx context.Context, cfg *oauth2.Config, token *oauth2.Token) (string, string, error) { - client := cfg.Client(ctx, token) - - var emails []githubEmail - if err := githubGetJSON(ctx, client, githubAPIBase+"/user/emails", &emails); err != nil { - return "", "", err - } - email, err := selectGitHubEmail(emails) - if err != nil { - return "", "", err - } - - var user struct { - Name string `json:"name"` - } - if err := githubGetJSON(ctx, client, githubAPIBase+"/user", &user); err != nil { - return email, "", nil - } - return email, user.Name, nil -} -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `cd server && go test ./internal/auth/ -v` -Expected: PASS, all tests including the six GitHub ones. - -- [ ] **Step 5: Build and vet** - -Run: `cd server && go build ./... && go vet ./...` -Expected: no output. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/auth/github.go server/internal/auth/github_test.go -git commit -m "feat: add GitHub OAuth2 provider branch" -``` - ---- - -### Task 8: Provider REST API, public discovery, remove the old endpoints - -**Files:** -- Create: `server/internal/api/auth_providers.go` -- Modify: `server/internal/api/handlers.go` (routes) -- Modify: `server/internal/api/instance.go:122-157` (delete both handlers) -- Delete: `server/internal/models/instance_oidc.go`, `server/internal/services/instance_oidc.go` - -**Interfaces:** -- Consumes: everything from Tasks 1-7. -- Produces: - - `auth.HandleListPublicProviders(c *gin.Context)` — the unauthenticated `GET /auth/providers`. - - `api.listAuthProviders`, `api.createAuthProvider`, `api.updateAuthProvider`, `api.deleteAuthProvider`, `api.testAuthProvider`, `api.ackAuthProviderNotice`, `api.listAuthPresets`. - -- [ ] **Step 1: Write the public discovery handler** - -Append to `server/internal/auth/local.go`: - -```go -// HandleListPublicProviders is unauthenticated: it is what the login page reads -// to decide what to draw. It carries no issuer, no client ID and no secret — -// only what a button needs, because anyone who can reach the login page can -// read this. -func HandleListPublicProviders(c *gin.Context) { - type publicProvider struct { - ID string `json:"id"` - Name string `json:"name"` - Preset string `json:"preset"` - } - out := []publicProvider{} - - instanceID, err := resolveLoginInstance(c) - if err != nil { - // An unresolvable instance is not an error the login page can act on: - // it still has to render a password form. Answer the safe shape. - c.JSON(http.StatusOK, gin.H{"local_enabled": true, "providers": out}) - return - } - - // A lapsed licence stops SSO, so a button that cannot work is not offered. - if services.GetLicenseState(instanceID).Feature("oidc") { - providers, err := services.ListEnabledAuthProviders(instanceID) - if err == nil { - for _, p := range providers { - out = append(out, publicProvider{ID: p.ProviderID, Name: p.Name, Preset: p.Preset}) - } - } - } - - localEnabled := services.IsLocalLoginEnabled(instanceID) - // Belt and braces against a hand-edited database: a login page with neither - // a form nor a button is unrecoverable without database access. - if !localEnabled && len(out) == 0 { - localEnabled = true - } - c.JSON(http.StatusOK, gin.H{"local_enabled": localEnabled, "providers": out}) -} -``` - -- [ ] **Step 2: Refuse local login when it is disabled** - -In `server/internal/auth/local.go`, inside `HandleLocalLogin`, immediately after `instanceID, err := resolveLoginInstance(c)` succeeds, add: - -```go - // The login page hides the form, but the page is a courtesy and the API is - // the boundary. - if !services.IsLocalLoginEnabled(instanceID) { - c.JSON(http.StatusForbidden, gin.H{"error": "password sign-in is disabled for this instance"}) - return - } -``` - -- [ ] **Step 3: Write the management handlers** - -Create `server/internal/api/auth_providers.go`: - -```go -package api - -import ( - "errors" - "net/http" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "github.com/coreos/go-oidc/v3/oidc" - "github.com/gin-gonic/gin" - "go.mongodb.org/mongo-driver/v2/mongo" -) - -type authProviderView struct { - models.AuthProvider - ClientSecretSet bool `json:"client_secret_set"` - CallbackURL string `json:"callback_url"` -} - -func viewOf(c *gin.Context, p models.AuthProvider) authProviderView { - return authProviderView{ - AuthProvider: p, - ClientSecretSet: p.ClientSecretEnc != "", - CallbackURL: auth.CallbackURL(c, p.ProviderID), - } -} - -func listAuthPresets(c *gin.Context) { - c.JSON(http.StatusOK, auth.Presets()) -} - -func listAuthProviders(c *gin.Context) { - providers, err := services.ListAuthProviders(auth.InstanceID(c)) - if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - out := make([]authProviderView, 0, len(providers)) - for _, p := range providers { - out = append(out, viewOf(c, p)) - } - c.JSON(http.StatusOK, out) -} - -func createAuthProvider(c *gin.Context) { - var body struct { - Name string `json:"name"` - Preset string `json:"preset"` - IssuerInput string `json:"issuer_input"` - ClientID string `json:"client_id"` - ClientSecret string `json:"client_secret"` - Enabled bool `json:"enabled"` - } - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - issuer, err := auth.ExpandIssuer(body.Preset, body.IssuerInput) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - instanceID := auth.InstanceID(c) - p, err := services.CreateAuthProvider(&models.AuthProvider{ - InstanceID: instanceID, - Name: body.Name, - Kind: auth.KindFor(body.Preset), - Preset: body.Preset, - Issuer: issuer, - ClientID: body.ClientID, - Scopes: auth.DefaultScopes(body.Preset), - Enabled: body.Enabled, - }, body.ClientSecret) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - services.LogEvent(instanceID, "auth_provider.create", actorFromCtx(c), "", "", p.Name) - c.JSON(http.StatusCreated, viewOf(c, *p)) -} - -func updateAuthProvider(c *gin.Context) { - var body struct { - Name *string `json:"name"` - IssuerInput *string `json:"issuer_input"` - ClientID *string `json:"client_id"` - ClientSecret *string `json:"client_secret"` - Enabled *bool `json:"enabled"` - Order *int `json:"order"` - } - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - instanceID := auth.InstanceID(c) - providerID := c.Param("id") - - existing, err := services.GetAuthProvider(instanceID, providerID) - if err != nil { - c.JSON(http.StatusNotFound, gin.H{"error": "provider not found"}) - return - } - - if err := guardProviderChange(instanceID, existing, body.Enabled, false); err != nil { - c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "last_provider"}) - return - } - - in := services.AuthProviderUpdate{ - Name: body.Name, ClientID: body.ClientID, - ClientSecret: body.ClientSecret, Enabled: body.Enabled, Order: body.Order, - } - if body.IssuerInput != nil { - issuer, err := auth.ExpandIssuer(existing.Preset, *body.IssuerInput) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - in.Issuer = &issuer - } - if err := services.UpdateAuthProvider(instanceID, providerID, in); err != nil { - if errors.Is(err, mongo.ErrNoDocuments) { - c.JSON(http.StatusNotFound, gin.H{"error": "provider not found"}) - return - } - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - // Issuer, client ID or secret may have changed; the cached discovery - // document was built from the old ones. - auth.EvictProvider(providerID) - services.LogEvent(instanceID, "auth_provider.update", actorFromCtx(c), "", "", existing.Name) - c.JSON(http.StatusOK, gin.H{"saved": true}) -} - -func deleteAuthProvider(c *gin.Context) { - instanceID := auth.InstanceID(c) - providerID := c.Param("id") - - existing, err := services.GetAuthProvider(instanceID, providerID) - if err != nil { - c.JSON(http.StatusNotFound, gin.H{"error": "provider not found"}) - return - } - if err := guardProviderChange(instanceID, existing, nil, true); err != nil { - c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "last_provider"}) - return - } - if err := services.DeleteAuthProvider(instanceID, providerID); err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - auth.EvictProvider(providerID) - services.LogEvent(instanceID, "auth_provider.delete", actorFromCtx(c), "", "", existing.Name) - c.JSON(http.StatusOK, gin.H{"deleted": true}) -} - -// guardProviderChange asks whether the instance would still have a way in. -// Deleting and disabling reach the same condition, so they share one answer. -func guardProviderChange(instanceID string, existing *models.AuthProvider, enabled *bool, deleting bool) error { - losing := deleting || (enabled != nil && !*enabled) - if !losing || !existing.Enabled { - return nil - } - n, err := services.CountEnabledAuthProviders(instanceID) - if err != nil { - return err - } - return services.CheckLockout(services.IsLocalLoginEnabled(instanceID), n-1) -} - -func ackAuthProviderNotice(c *gin.Context) { - instanceID := auth.InstanceID(c) - if err := services.AckAuthProviderNotice(instanceID, c.Param("id")); err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - c.JSON(http.StatusOK, gin.H{"acknowledged": true}) -} - -// testAuthProvider proves the configuration is reachable. It signs nobody in. -func testAuthProvider(c *gin.Context) { - instanceID := auth.InstanceID(c) - p, err := services.GetAuthProvider(instanceID, c.Param("id")) - if err != nil { - c.JSON(http.StatusNotFound, gin.H{"error": "provider not found"}) - return - } - if p.Kind == models.KindOAuth2 { - // GitHub has no discovery document. The only meaningful check without - // a user token is that credentials are present. - if p.ClientID == "" || p.ClientSecretEnc == "" { - c.JSON(http.StatusOK, gin.H{"ok": false, "message": "client ID and secret are required"}) - return - } - c.JSON(http.StatusOK, gin.H{"ok": true, "message": "credentials are configured"}) - return - } - if _, err := oidc.NewProvider(c.Request.Context(), p.Issuer); err != nil { - c.JSON(http.StatusOK, gin.H{"ok": false, "message": err.Error()}) - return - } - c.JSON(http.StatusOK, gin.H{"ok": true, "message": "discovery document fetched"}) -} -``` - -- [ ] **Step 4: Register the routes and delete the old ones** - -In `server/internal/api/handlers.go`, inside the `instance` group, replace lines 103-104 with nothing, and add a new group after it: - -```go - providers := apiGroup.Group("/auth/providers") - providers.Use(auth.RequireRole("owner", "admin"), RequireFeature("oidc")) - { - providers.GET("", listAuthProviders) - providers.POST("", createAuthProvider) - providers.PUT("/:id", updateAuthProvider) - providers.DELETE("/:id", deleteAuthProvider) - providers.POST("/:id/test", testAuthProvider) - providers.POST("/:id/ack-notice", ackAuthProviderNotice) - } - apiGroup.GET("/auth/presets", auth.RequireRole("owner", "admin"), listAuthPresets) -``` - -Confirm `r.GET("/auth/providers", auth.HandleListPublicProviders)` is present in the unauthenticated block (added in Task 6 Step 2). - -- [ ] **Step 5: Delete the dead code** - -```bash -git rm server/internal/models/instance_oidc.go server/internal/services/instance_oidc.go -``` - -Then delete `getInstanceOIDC` and `putInstanceOIDC` from `server/internal/api/instance.go` (lines 122-157), and remove any import left unused. - -- [ ] **Step 6: Build, vet, test** - -Run: `cd server && go build ./... && go vet ./... && go test ./...` -Expected: clean. If the build names `EvictOIDCProvider`, a caller was missed — it was renamed to `EvictProvider` in Task 6. - -- [ ] **Step 7: Verify the endpoints by hand** - -With the server running and a session cookie in `$C` (copy `km_session` from a browser devtools cookie jar): - -```bash -# public discovery, no auth -curl -s localhost:8080/auth/providers | jq - -# must contain no issuer, client_id or secret -curl -s localhost:8080/auth/providers | jq 'tostring | test("issuer|client_id|secret") | not' - -# presets -curl -s -b "km_session=$C" localhost:8080/api/auth/presets | jq '.[].id' - -# create -curl -s -b "km_session=$C" -X POST localhost:8080/api/auth/providers \ - -H 'Content-Type: application/json' \ - -d '{"name":"Google","preset":"google","client_id":"x","client_secret":"y","enabled":true}' | jq - -# list -curl -s -b "km_session=$C" localhost:8080/api/auth/providers | jq '.[] | {name, provider_id, callback_url}' - -# the removed routes -curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/auth/oidc/start -curl -s -o /dev/null -w '%{http_code}\n' -b "km_session=$C" localhost:8080/api/instance/oidc -``` - -Expected: discovery returns `local_enabled` plus the created provider; the secret-leak check prints `true`; presets prints five IDs; create returns 201 with a `callback_url` containing the provider ID; both removed routes print `404`. - -- [ ] **Step 8: Verify the lockout guard** - -```bash -# turn local login off while one provider is enabled — should succeed -curl -s -b "km_session=$C" -X PUT localhost:8080/api/settings \ - -H 'Content-Type: application/json' -d '{"local_login_enabled":false}' | jq - -# now disable the only provider — should be refused -PID=$(curl -s -b "km_session=$C" localhost:8080/api/auth/providers | jq -r '.[0].provider_id') -curl -s -w '\n%{http_code}\n' -b "km_session=$C" -X PUT "localhost:8080/api/auth/providers/$PID" \ - -H 'Content-Type: application/json' -d '{"enabled":false}' -``` - -Expected: the second call returns `409` with `"code":"last_provider"`. Re-enable local login afterwards so the instance is usable. - -- [ ] **Step 9: Commit** - -```bash -git add -A server/internal/api server/internal/auth/local.go server/internal/models server/internal/services -git commit -m "feat: auth provider REST API and public provider discovery" -``` - ---- - -### Task 9: Web API client and login page - -**Files:** -- Modify: `web/lib/api.ts:352-360` (types), `:455-458` (redirect URL helper), `:482-487` (the two OIDC calls) -- Modify: `web/app/login/page.tsx` - -**Interfaces:** -- Consumes: the endpoints from Task 8. -- Produces: - - `PublicProvider`, `AuthProvider`, `AuthPreset`, `ProvidersResponse` TypeScript types. - - `auth.providers()`, `api.listAuthProviders()`, `api.createAuthProvider()`, `api.updateAuthProvider()`, `api.deleteAuthProvider()`, `api.testAuthProvider()`, `api.ackAuthProviderNotice()`, `api.listAuthPresets()`. - -- [ ] **Step 1: Replace the OIDC types** - -In `web/lib/api.ts`, replace the `OrgOIDCConfig` and `OrgOIDCInput` interfaces (lines 352-360) with: - -```ts -export interface PublicProvider { - id: string; - name: string; - preset: string; -} - -export interface ProvidersResponse { - local_enabled: boolean; - providers: PublicProvider[]; -} - -export interface AuthProvider { - provider_id: string; - instance_id: string; - name: string; - kind: "oidc" | "oauth2"; - preset: string; - issuer: string; - client_id: string; - scopes: string[]; - enabled: boolean; - callback_notice: boolean; - order: number; - created_at: string; - updated_at: string; - client_secret_set: boolean; - callback_url: string; -} - -export interface AuthPreset { - id: string; - label: string; - kind: "oidc" | "oauth2"; - input_label: string; - input_hint: string; -} - -export interface AuthProviderInput { - name: string; - preset: string; - issuer_input?: string; - client_id: string; - client_secret: string; - enabled: boolean; -} - -export interface AuthProviderUpdate { - name?: string; - issuer_input?: string; - client_id?: string; - client_secret?: string; - enabled?: boolean; - order?: number; -} -``` - -- [ ] **Step 2: Replace the client methods** - -In the `auth` object, replace `oidcRedirectUrl` (lines 455-458) with: - -```ts - /** Unauthenticated: what the login page draws itself from. */ - providers(): Promise { - return fetch(`${base()}/auth/providers`, { credentials: "include" }).then((r) => { - if (!r.ok) throw new Error("could not load sign-in options"); - return r.json(); - }); - }, - - /** Where a provider button sends the browser. */ - ssoStartUrl(providerId: string): string { - return `/auth/oidc/${providerId}/start`; - }, -``` - -Match `base()` to whatever the file's existing unauthenticated calls use — check how `bootstrapStatus` at line 429 builds its URL and follow it exactly rather than introducing a second convention. - -In the `api` object, replace `getInstanceOIDC` and `saveInstanceOIDC` (lines 482-487) with: - -```ts - listAuthPresets(): Promise { - return request("/auth/presets"); - }, - - listAuthProviders(): Promise { - return request("/auth/providers"); - }, - - createAuthProvider(input: AuthProviderInput): Promise { - return request("/auth/providers", { method: "POST", body: JSON.stringify(input) }); - }, - - updateAuthProvider(id: string, input: AuthProviderUpdate): Promise<{ saved: boolean }> { - return request<{ saved: boolean }>(`/auth/providers/${id}`, { method: "PUT", body: JSON.stringify(input) }); - }, - - deleteAuthProvider(id: string): Promise<{ deleted: boolean }> { - return request<{ deleted: boolean }>(`/auth/providers/${id}`, { method: "DELETE" }); - }, - - testAuthProvider(id: string): Promise<{ ok: boolean; message: string }> { - return request<{ ok: boolean; message: string }>(`/auth/providers/${id}/test`, { method: "POST" }); - }, - - ackAuthProviderNotice(id: string): Promise<{ acknowledged: boolean }> { - return request<{ acknowledged: boolean }>(`/auth/providers/${id}/ack-notice`, { method: "POST" }); - }, -``` - -- [ ] **Step 3: Add the provider icon component** - -Create `web/components/settings/ProviderIcon.tsx`: - -```tsx -/** - * One glyph per preset, plus a neutral key for anything custom. These are - * simplified marks drawn with currentColor rather than brand logos: a brand - * logo carries usage terms, and currentColor is what lets a button match the - * text beside it. - */ -export function ProviderIcon({ preset, className = "h-4 w-4" }: { preset: string; className?: string }) { - switch (preset) { - case "github": - return ( - - ); - case "google": - return ( - - ); - case "entra": - return ( - - ); - case "okta": - return ( - - ); - default: - return ( - - ); - } -} -``` - -- [ ] **Step 4: Rewrite the login page** - -Replace `web/app/login/page.tsx` with: - -```tsx -"use client"; - -import { useEffect, useState } from "react"; -import { useMutation } from "@tanstack/react-query"; -import { auth, type PublicProvider } from "@/lib/api"; -import { Button, Card } from "@/components/ui"; -import { Logo } from "@/components/Logo"; -import { NetworkBackground } from "@/components/NetworkBackground"; -import { ProviderIcon } from "@/components/settings/ProviderIcon"; - -const ERROR_MESSAGES: Record = { - oidc_unavailable: "Single sign-on is not available on this instance's plan.", - provider_unavailable: "That sign-in method is no longer available.", - provider_unreachable: "Could not reach the identity provider.", - invalid_state: "That sign-in attempt expired. Please try again.", - exchange_failed: "The identity provider rejected the sign-in.", - verification_failed: "The identity provider's response could not be verified.", - missing_id_token: "The identity provider returned no identity token.", - missing_email: "The identity provider returned no email address.", - identity_failed: "Could not read a verified email address from that account.", - provisioning_failed: "Could not create your account on this instance.", - session_failed: "Could not start your session. Please try again.", - state_failed: "Could not start sign-in. Please try again.", - unknown_host: "This address does not name a known instance.", -}; - -export default function LoginPage() { - const [email, setEmail] = useState(""); - const [password, setPassword] = useState(""); - const [instanceName, setInstanceName] = useState(""); - const [providers, setProviders] = useState([]); - // Default true so a slow or failing discovery call still renders something - // usable rather than an empty card. - const [localEnabled, setLocalEnabled] = useState(true); - const [ssoError, setSsoError] = useState(""); - - useEffect(() => { - const code = new URLSearchParams(window.location.search).get("error"); - if (code) setSsoError(ERROR_MESSAGES[code] ?? "Sign-in failed. Please try again."); - }, []); - - useEffect(() => { - (async () => { - try { - const s = await auth.bootstrapStatus(); - if (s.needs_setup) { - window.location.href = "/setup"; - return; - } - if (s.instance_name) setInstanceName(s.instance_name); - } catch {} - try { - const p = await auth.providers(); - setProviders(p.providers); - setLocalEnabled(p.local_enabled); - } catch {} - try { - await auth.me(); - window.location.href = "/"; - } catch {} - })(); - }, []); - - const { - mutate: signIn, - isPending, - error, - } = useMutation({ - mutationFn: () => auth.login(email, password), - onSuccess: () => { - window.location.href = "/"; - }, - }); - - function handleSubmit(e: React.FormEvent) { - e.preventDefault(); - signIn(); - } - - const showLocal = localEnabled || providers.length === 0; - const showDivider = showLocal && providers.length > 0; - - return ( -
- -
-
- -

Sign in to Vantage

- {/* A keyed label, not a heading: site/'s .tag treatment. */} -

{instanceName}

-
- - - {ssoError &&
{ssoError}
} - - {showLocal && ( -
-
- - setEmail(e.target.value)} - className="w-full rounded-lg border border-border bg-surface-2 px-3 py-2 text-sm text-text-primary focus:border-accent/50 focus:outline-none focus:ring-1 focus:ring-accent/30" - /> -
- -
- - setPassword(e.target.value)} - className="w-full rounded-lg border border-border bg-surface-2 px-3 py-2 text-sm text-text-primary focus:border-accent/50 focus:outline-none focus:ring-1 focus:ring-accent/30" - /> -
- - {error &&
{(error as Error).message}
} - - -
- )} - - {showDivider && ( -
-
- or -
-
- )} - - {providers.length > 0 && ( -
- {providers.map((p) => ( - - - - ))} -
- )} - -
-
- ); -} -``` - -- [ ] **Step 5: Typecheck** - -Run: `cd web && npx tsc --noEmit` -Expected: no errors. If `OrgOIDCConfig` is still referenced, `OIDCCard.tsx` has not been removed yet — that is Task 10; temporarily leave the old types in place and remove them there. - -- [ ] **Step 6: Verify the three rendering states in a browser** - -With the server and `web` running, and using `mongosh` to set state between checks: - -```bash -# state 1: no providers, local on -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.auth_providers.updateMany({},{$set:{enabled:false}})' -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.settings.updateMany({},{$unset:{local_login_enabled:""}})' -``` - -Load `/login`. Expected: password form only, **no divider and no buttons**. - -```bash -# state 2: providers on, local on -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.auth_providers.updateMany({},{$set:{enabled:true}})' -``` - -Reload. Expected: form, divider, one button per provider, each labelled `Continue with `. - -```bash -# state 3: providers on, local off -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.settings.updateMany({},{$set:{local_login_enabled:false}})' -``` - -Reload. Expected: buttons only, no form, no divider. - -```bash -# state 4: the impossible one — hand-edited, must not render an empty card -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.auth_providers.updateMany({},{$set:{enabled:false}})' -``` - -Reload. Expected: the password form is back. Then restore: - -```bash -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.settings.updateMany({},{$unset:{local_login_enabled:""}}); db.auth_providers.updateMany({},{$set:{enabled:true}})' -``` - -- [ ] **Step 7: Verify an error redirect renders** - -Load `/login?error=invalid_state` directly. -Expected: "That sign-in attempt expired. Please try again." in a danger-styled box above the form. - -- [ ] **Step 8: Commit** - -```bash -git add web/lib/api.ts web/app/login/page.tsx web/components/settings/ProviderIcon.tsx -git commit -m "feat: render one login button per configured auth provider" -``` - ---- - -### Task 10: Settings card - -**Files:** -- Create: `web/components/settings/AuthProvidersCard.tsx` -- Delete: `web/components/settings/OIDCCard.tsx` -- Modify: `web/app/(app)/settings/page.tsx:13` and `:194` - -**Interfaces:** -- Consumes: the `api` methods from Task 9, `Field`/`inputClass` from `./Field`, `SectionCard` from `./SectionCard`, `ProviderIcon`. -- Produces: `AuthProvidersCard`. - -- [ ] **Step 1: Write the card** - -Create `web/components/settings/AuthProvidersCard.tsx`: - -```tsx -"use client"; - -import { useState } from "react"; -import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; -import { api, type AuthProvider, type AuthPreset } from "@/lib/api"; -import { Button, Card } from "@/components/ui"; -import { Field, inputClass } from "./Field"; -import { SectionCard } from "./SectionCard"; -import { ProviderIcon } from "./ProviderIcon"; - -function IdentityIcon() { - return ( - - - - ); -} - -function CallbackRow({ url }: { url: string }) { - const [copied, setCopied] = useState(false); - return ( -
- {url} - -
- ); -} - -function AddProviderForm({ presets, onDone }: { presets: AuthPreset[]; onDone: () => void }) { - const [preset, setPreset] = useState("google"); - const [name, setName] = useState(""); - const [issuerInput, setIssuerInput] = useState(""); - const [clientId, setClientId] = useState(""); - const [clientSecret, setClientSecret] = useState(""); - - const chosen = presets.find((p) => p.id === preset); - - const { mutate: create, isPending, error } = useMutation({ - mutationFn: () => - api.createAuthProvider({ - name: name || chosen?.label || "Single sign-on", - preset, - issuer_input: issuerInput, - client_id: clientId, - client_secret: clientSecret, - enabled: true, - }), - onSuccess: onDone, - }); - - return ( -
{ - e.preventDefault(); - create(); - }} - className="space-y-4 rounded border border-border bg-surface-2 p-4" - > - - - - - - setName(e.target.value)} placeholder={chosen?.label ?? ""} className={inputClass} /> - - - {chosen?.input_label && ( - - setIssuerInput(e.target.value)} className={inputClass} /> - - )} - - - setClientId(e.target.value)} className={inputClass} /> - - - - setClientSecret(e.target.value)} className={inputClass} /> - - - {error &&
{(error as Error).message}
} - -
- - -
-

You will get a callback URL to register with your provider once it is added.

-
- ); -} - -function ProviderRow({ p }: { p: AuthProvider }) { - const queryClient = useQueryClient(); - const [secret, setSecret] = useState(""); - const [testResult, setTestResult] = useState<{ ok: boolean; message: string } | null>(null); - const invalidate = () => queryClient.invalidateQueries({ queryKey: ["auth-providers"] }); - - const { mutate: update, error: updateError } = useMutation({ - mutationFn: (patch: Parameters[1]) => api.updateAuthProvider(p.provider_id, patch), - onSuccess: () => { - setSecret(""); - invalidate(); - }, - }); - const { mutate: remove, error: deleteError } = useMutation({ - mutationFn: () => api.deleteAuthProvider(p.provider_id), - onSuccess: invalidate, - }); - const { mutate: test, isPending: testing } = useMutation({ - mutationFn: () => api.testAuthProvider(p.provider_id), - onSuccess: setTestResult, - }); - const { mutate: ack } = useMutation({ - mutationFn: () => api.ackAuthProviderNotice(p.provider_id), - onSuccess: invalidate, - }); - - const error = (updateError ?? deleteError) as Error | null; - - return ( -
-
- -
-

{p.name}

-

{p.issuer || "GitHub"}

-
- -
- - {p.callback_notice && ( -
-

This provider's callback URL has changed. Update it in your identity provider or sign-in will fail.

- -
- )} - -
-

Callback URL to register with this provider:

- -
- -
-
- - setSecret(e.target.value)} className={inputClass} /> - -
- -
- - {testResult && ( -
- {testResult.message} -
- )} - - {error &&
{error.message}
} - -
- - -
-
- ); -} - -export function AuthProvidersCard({ localLoginEnabled, onLocalLoginChange }: { localLoginEnabled: boolean; onLocalLoginChange: (v: boolean) => void }) { - const queryClient = useQueryClient(); - const [adding, setAdding] = useState(false); - const { data: providers, isLoading } = useQuery({ queryKey: ["auth-providers"], queryFn: api.listAuthProviders }); - const { data: presets } = useQuery({ queryKey: ["auth-presets"], queryFn: api.listAuthPresets }); - - if (isLoading) { - return ( - -
-
-
- - ); - } - - const list = providers ?? []; - const enabledCount = list.filter((p) => p.enabled).length; - - return ( - } - > -
- {list.length === 0 &&

No providers configured. Members sign in with a password.

} - {list.map((p) => ( - - ))} -
- -
- {adding ? ( - { - setAdding(false); - queryClient.invalidateQueries({ queryKey: ["auth-providers"] }); - }} - /> - ) : ( - - )} -
- -
- -

- {enabledCount === 0 - ? "Enable at least one provider before turning this off, or nobody could sign in." - : "Turn this off to require members to use a provider above."} -

-
- - ); -} -``` - -- [ ] **Step 2: Wire it into the settings page** - -In `web/app/(app)/settings/page.tsx`: - -- Line 13: replace `import { OIDCCard } from "@/components/settings/OIDCCard";` with `import { AuthProvidersCard } from "@/components/settings/AuthProvidersCard";` -- Line 194: replace `` with: - -```tsx - saveSettings({ ...settingsPayload, local_login_enabled: v })} - /> -``` - -Adapt `settingsPayload` to whatever the page already passes to its save mutation — read the surrounding code and match it rather than inventing a new shape. Add `local_login_enabled?: boolean` to `saveSettings`'s parameter type in `web/lib/api.ts:593`, and `local_login_enabled?: boolean` to the `Settings` interface. - -- [ ] **Step 3: Delete the old card** - -```bash -git rm web/components/settings/OIDCCard.tsx -``` - -- [ ] **Step 4: Typecheck** - -Run: `cd web && npx tsc --noEmit` -Expected: no errors. If `OrgOIDCConfig` or `OrgOIDCInput` are now unreferenced, delete them from `web/lib/api.ts`. - -- [ ] **Step 5: Verify in a browser** - -At `/settings`, in the Access group: - -1. The card lists existing providers, each with its callback URL and a working Copy button. -2. A migrated provider shows the amber callback-changed warning; clicking "I've updated it" makes it disappear and it stays gone after a reload. -3. "Add provider" with preset Google asks for a label, client ID and secret and **no issuer field**; with "Other" it asks for an issuer URL; with Entra it asks for a tenant ID. -4. "Test connection" on a Google provider reports the discovery document was fetched. -5. With one enabled provider, unticking "Allow email and password sign-in" saves. Then unticking that provider's Enabled reports the 409 message rather than a generic failure. -6. With zero enabled providers, the local-login checkbox is disabled and explains why. - -- [ ] **Step 6: Commit** - -```bash -git add -A web/components/settings web/app "web/lib/api.ts" -git commit -m "feat: manage multiple sign-in providers from settings" -``` - ---- - -### Task 11: End-to-end sign-in and documentation - -**Files:** -- Modify: `CLAUDE.md` (Auth and Orgs, REST API, MongoDB Collections, Migrations sections) -- Modify: `docsite/docs/vantage/settings.md` -- Modify: `docsite/docs/getting-started/first-login.md` - -**Interfaces:** -- Consumes: everything. -- Produces: no code. - -- [ ] **Step 1: Sign in end-to-end with a real provider** - -Configure one real OIDC provider (Google is the least setup) with the callback URL the settings card shows. Then: - -1. Sign out. -2. On `/login`, click the provider's button. -3. Complete the provider's consent screen. - -Expected: redirected to `/`, signed in, and a `users` document exists with `auth_source: "oidc"` scoped to this instance: - -```bash -mongosh "$MONGO_URI/vantage" --quiet --eval 'db.users.find({auth_source:"oidc"},{email:1,instance_id:1,role:1})' -``` - -- [ ] **Step 2: Verify two providers in one instance resolve independently** - -Add a second provider. Confirm each button lands on its own provider's consent screen, and that its callback URL contains its own provider ID. - -- [ ] **Step 3: Verify the state cross-check refuses a tampered callback** - -Start a sign-in, capture the `state` from the provider's redirect back, and replay it against a **different** provider's callback path: - -```bash -curl -s -o /dev/null -w '%{redirect_url}\n' "localhost:8080/auth/oidc//callback?state=&code=x" -``` - -Expected: redirects to `/login?error=invalid_state`. - -- [ ] **Step 4: Update CLAUDE.md** - -In the **Auth and Orgs** section, replace the OIDC bullet with: - -```markdown -- **Auth providers** — configured _per instance_ in `auth_providers`, any number of them, each named and independently enabled. Issuer, client ID and an encrypted client secret per provider. `/auth/oidc/:providerId/start` → `/auth/oidc/:providerId/callback`. Presets (Entra, Google, Okta, GitHub) are a Go table in `server/internal/auth/presets.go` and expand to a real issuer on save, so nothing downstream knows a preset existed. GitHub is OAuth2 rather than OIDC and takes its own branch, requiring an address that is both primary **and** verified — an unverified address is not proof of control. -- **Local login** — `settings.local_login_enabled`, a `*bool` because absent must mean enabled; a plain bool would disable password sign-in fleet-wide at upgrade. `services.CheckLockout` refuses any change leaving neither local login nor an enabled provider, and is enforced in the service layer so the settings path and the provider path cannot disagree. -``` - -In **REST API**, replace the `org` block's OIDC lines with: - -``` -providers GET,POST /auth/providers · PUT,DELETE /auth/providers/:id - POST /auth/providers/:id/{test,ack-notice} · GET /auth/presets (owner|admin) -``` - -and add `GET /auth/providers` to the unauthenticated list, noting it carries no issuer, client ID or secret. - -In **MongoDB Collections**, replace `org_oidc` with `auth_providers` and add this note: - -- `auth_providers.provider_id` is a short random identifier, not the Mongo `_id`: it appears in the callback URL a customer pastes into their IdP, and an `_id` there would publish a database key. `callback_notice` marks a provider migrated from the old single-provider shape, whose redirect URI therefore changed. - -In **Migrations**, add `0005_auth_providers` to the list with a note that it copies ciphertext verbatim rather than re-encrypting, so it does not need `KEY_ENCRYPTION_KEY`. - -- [ ] **Step 5: Update the docsite** - -In `docsite/docs/vantage/settings.md`, rewrite the single sign-on section to cover adding several providers, the per-provider callback URL, the presets and their inputs, and turning off password sign-in — including that at least one provider must be enabled first. - -Add an upgrade note stating plainly: **existing single sign-on stops working until the callback URL shown in settings is registered with the identity provider**, because callbacks are now per provider. Password sign-in is unaffected, so an administrator can always sign in to make the change. - -In `docsite/docs/getting-started/first-login.md`, update any text describing one SSO button to describe one button per configured provider, and note that no buttons appear when none are configured. - -- [ ] **Step 6: Verify the docs build** - -Run: `cd docsite && npm run build` -Expected: build succeeds, no broken-link warnings for the pages touched. - -- [ ] **Step 7: Full check** - -Run: `cd server && go build ./... && go vet ./... && go test ./...` then `cd web && npx tsc --noEmit` -Expected: all clean. - -- [ ] **Step 8: Commit** - -```bash -git add CLAUDE.md docsite/docs -git commit -m "docs: document multiple auth providers and the callback URL change" -``` - ---- - -## Self-Review Notes - -Checked against the spec: - -- Data model, presets, `local_login_enabled` pointer → Tasks 1, 3. -- Migration 0005 including verbatim ciphertext and idempotency → Task 4. -- Per-provider routes, removed legacy routes, state cross-check → Tasks 5, 6, 8. -- GitHub primary-and-verified → Task 7. -- Public discovery with no secrets, management API, lockout guards, audit → Task 8. -- Login page's four rendering states → Task 9 Step 6. -- Settings card with per-provider callback URL and dismissable notice → Task 10. -- Upgrade impact documented → Task 11. - -Two deviations from the spec, both deliberate: - -1. The spec puts `EnsureAuthProviderIndexes` in the provider service; Task 2 folds it into the existing `EnsureAuthIndexes` instead, because two builders for one index is how they drift, and `EnsureAuthIndexes` is already fatal-on-failure where this index needs to be. -2. The spec has the callback answer JSON errors as the current code does; Tasks 6 and 9 redirect to `/login?error=` throughout, because a browser mid-redirect renders a JSON body as raw text on a blank page. diff --git a/docs/superpowers/plans/2026-08-04-scheduled-workflows.md b/docs/superpowers/plans/2026-08-04-scheduled-workflows.md deleted file mode 100644 index fc0b4c4..0000000 --- a/docs/superpowers/plans/2026-08-04-scheduled-workflows.md +++ /dev/null @@ -1,1071 +0,0 @@ -# Scheduled Workflows Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let a workflow carry a cron schedule in a named timezone, and have one leader-elected scheduler fire it as an ordinary run. - -**Architecture:** `Workflow` gains a `schedule` block and a persisted `next_run_at`. A new `server/internal/workflowsched` package runs inside the existing `bus.RunAsLeader("housekeeping", …)` and ticks every 30 seconds, claiming each due workflow with an atomic `findOneAndUpdate` on its `next_run_at` before starting a run through the same `TriggerWorkflow` path a person uses. Cron arithmetic and the fire/skip decision are pure functions, tested without a database. - -**Tech Stack:** Go 1.26, `github.com/robfig/cron/v3` (parser only), mongo-driver v2, gin, Next.js 16 + TanStack Query. - -## Global Constraints - -- Cron expressions are standard **5-field** (minute hour dom month dow). No seconds field, no `@every`. -- Timezones are IANA names (`Europe/London`). Validated at save time; an unknown zone is a 400. -- Grace window for a missed occurrence: **1 hour**. Older misses are recorded and dropped. -- A scheduled run never starts while a run of the same workflow is active. -- A scheduled run uses the same `TriggerWorkflow` path as a manual one, with `triggered_by: "schedule"`. There must be no second dispatch path. -- Every skip writes both `last_skipped` on the workflow and an audit event via `services.LogEvent`. -- All background work runs inside the single `RunAsLeader("housekeeping", …)` and must return when its context is cancelled. -- Tests run under plain `go test ./...` with no database, no network, no build tags. -- No component in `web/` may carry a hex colour; `web/` is dark-only. - -**Depends on:** nothing in the server-tags plan. The two can land in either order. - ---- - -## File Structure - -**Create:** -- `server/internal/workflowsched/cron.go` — expression parsing, next-occurrence arithmetic, the fire/skip decision. Pure. -- `server/internal/workflowsched/cron_test.go` — unit tests for all of the above. -- `server/internal/workflowsched/sched.go` — the ticking loop, the atomic claim, the database side. -- `web/components/workflows/ScheduleCard.tsx` — the schedule editor. - -**Modify:** -- `server/go.mod` — add `github.com/robfig/cron/v3`. -- `server/cmd/main.go` — `import _ "time/tzdata"`; start the scheduler under the leader lock; index builder call. -- `server/internal/models/workflow.go` — `Schedule`, `Skip`, and the three timestamp fields. -- `server/internal/services/workflows.go` — `SetSchedule`, `EnsureWorkflowIndexes` gains `next_run_at`. -- `server/internal/api/workflows.go` — two routes and their handlers. -- `web/lib/api.ts` — types and client methods. -- `web/app/(app)/workflows/[id]/page.tsx` — mount the schedule card. -- `web/app/(app)/workflows/page.tsx` — schedule chip and next run. -- `docsite/docs/vantage/workflows.md`, `CLAUDE.md`. - ---- - -### Task 1: Cron arithmetic - -**Files:** -- Create: `server/internal/workflowsched/cron.go` -- Test: `server/internal/workflowsched/cron_test.go` -- Modify: `server/go.mod` - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `workflowsched.ParseSchedule(expr, tz string) (cron.Schedule, error)` - - `workflowsched.NextOccurrence(expr, tz string, from time.Time) (time.Time, error)` - - `workflowsched.ErrBadSchedule` (an `error` value) - -- [ ] **Step 1: Add the dependency** - -Run: - -```bash -cd server && go get github.com/robfig/cron/v3@v3.0.1 -``` - -Expected: `go.mod` and `go.sum` updated. Only the parser is used; the library's own scheduler and goroutines are not. - -- [ ] **Step 2: Write the failing test** - -Create `server/internal/workflowsched/cron_test.go`: - -```go -package workflowsched - -import ( - "testing" - "time" -) - -func mustTime(t *testing.T, layout, value, zone string) time.Time { - t.Helper() - loc, err := time.LoadLocation(zone) - if err != nil { - t.Fatalf("load %s: %v", zone, err) - } - parsed, err := time.ParseInLocation(layout, value, loc) - if err != nil { - t.Fatalf("parse %s: %v", value, err) - } - return parsed -} - -func TestParseScheduleRejectsBadInput(t *testing.T) { - cases := []struct{ expr, tz string }{ - {"", "UTC"}, - {"not a cron", "UTC"}, - {"0 2 * *", "UTC"}, // four fields - {"0 2 * * * *", "UTC"}, // six fields — no seconds field is supported - {"99 2 * * *", "UTC"}, // minute out of range - {"0 2 * * 0", "Mars/Olympus"}, // unknown zone - {"0 2 * * 0", ""}, // empty zone - } - for _, tc := range cases { - if _, err := ParseSchedule(tc.expr, tc.tz); err == nil { - t.Fatalf("expected %q / %q to be rejected", tc.expr, tc.tz) - } - } -} - -func TestNextOccurrenceUsesTheStoredZone(t *testing.T) { - // 02:00 every Sunday, London. From Friday, the next is Sunday 02:00 local. - from := mustTime(t, "2006-01-02 15:04", "2026-08-07 12:00", "Europe/London") - - got, err := NextOccurrence("0 2 * * 0", "Europe/London", from) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - - want := mustTime(t, "2006-01-02 15:04", "2026-08-09 02:00", "Europe/London") - if !got.Equal(want) { - t.Fatalf("got %s, want %s", got, want) - } -} - -func TestNextOccurrenceCrossesDST(t *testing.T) { - // London leaves BST at 02:00 on 2026-10-25. A 02:30 daily job on the 24th - // must next fire at 02:30 GMT on the 25th — an interval of 25 hours, not - // 24. This is the whole reason the zone is stored rather than a UTC offset. - from := mustTime(t, "2006-01-02 15:04", "2026-10-24 03:00", "Europe/London") - - got, err := NextOccurrence("30 2 * * *", "Europe/London", from) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - - loc, _ := time.LoadLocation("Europe/London") - local := got.In(loc) - if local.Day() != 25 || local.Hour() != 2 || local.Minute() != 30 { - t.Fatalf("got %s, want 2026-10-25 02:30 local", local) - } - if delta := got.Sub(from); delta != 24*time.Hour-30*time.Minute+time.Hour { - t.Fatalf("expected the DST hour to be added, got a delta of %s", delta) - } -} - -func TestNextOccurrenceIsStrictlyAfterFrom(t *testing.T) { - exact := mustTime(t, "2006-01-02 15:04", "2026-08-09 02:00", "Europe/London") - - got, err := NextOccurrence("0 2 * * 0", "Europe/London", exact) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if !got.After(exact) { - t.Fatalf("next occurrence %s must be strictly after %s, or a fired tick refires forever", got, exact) - } -} -``` - -- [ ] **Step 3: Run test to verify it fails** - -Run: `cd server && go test ./internal/workflowsched/ -v` -Expected: FAIL — `undefined: ParseSchedule` - -- [ ] **Step 4: Write minimal implementation** - -Create `server/internal/workflowsched/cron.go`: - -```go -// Package workflowsched fires workflow runs on a cron schedule. -// -// Only robfig/cron's parser is used — Parse and Next. Its own scheduler is -// not, because this work runs under the housekeeping leader lock and has to -// stop the moment leadership is lost. -package workflowsched - -import ( - "errors" - "fmt" - "time" - - "github.com/robfig/cron/v3" -) - -// ErrBadSchedule covers both a malformed expression and an unknown timezone. -// Handlers map it to 400 — both are the caller's mistake, and both are much -// cheaper to find at save time than at 2am. -var ErrBadSchedule = errors.New("invalid schedule") - -// Standard 5-field cron: minute hour dom month dow. Deliberately no seconds -// field and no descriptors — a schedule a person cannot read back is a -// schedule nobody can audit. -var cronParser = cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow) - -func ParseSchedule(expr, tz string) (cron.Schedule, error) { - if tz == "" { - return nil, fmt.Errorf("%w: a timezone is required", ErrBadSchedule) - } - if _, err := time.LoadLocation(tz); err != nil { - return nil, fmt.Errorf("%w: unknown timezone %q", ErrBadSchedule, tz) - } - sched, err := cronParser.Parse(expr) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrBadSchedule, err) - } - return sched, nil -} - -// NextOccurrence returns the first firing strictly after from, computed in the -// schedule's own zone so that a DST boundary moves the wall-clock time the way -// a person expects rather than drifting by an hour for half the year. -func NextOccurrence(expr, tz string, from time.Time) (time.Time, error) { - sched, err := ParseSchedule(expr, tz) - if err != nil { - return time.Time{}, err - } - loc, err := time.LoadLocation(tz) - if err != nil { - return time.Time{}, fmt.Errorf("%w: unknown timezone %q", ErrBadSchedule, tz) - } - return sched.Next(from.In(loc)), nil -} -``` - -- [ ] **Step 5: Run test to verify it passes** - -Run: `cd server && go test ./internal/workflowsched/ -v` -Expected: PASS, four tests. - -If `TestNextOccurrenceCrossesDST` fails with a zone-loading error rather than a wrong time, the tzdata import in Task 4 is the fix — but it must not be needed for `go test` on a developer machine, which has a system zone database. A failure here on a machine *with* zoneinfo is a real bug in the arithmetic. - -- [ ] **Step 6: Commit** - -```bash -git add server/go.mod server/go.sum server/internal/workflowsched/ -git commit -m "feat: cron parsing and next-occurrence arithmetic for workflow schedules" -``` - ---- - -### Task 2: The fire/skip decision - -**Files:** -- Modify: `server/internal/workflowsched/cron.go` -- Test: `server/internal/workflowsched/cron_test.go` - -**Interfaces:** -- Consumes: nothing from Task 1 at runtime. -- Produces: `workflowsched.Decision` (a string type with constants `Fire`, `SkipMissed`, `SkipRunning`) and `workflowsched.Decide(due, now time.Time, runActive bool) Decision`. - -- [ ] **Step 1: Write the failing test** - -Append to `server/internal/workflowsched/cron_test.go`: - -```go -func TestDecide(t *testing.T) { - now := time.Date(2026, 8, 9, 2, 0, 0, 0, time.UTC) - - cases := []struct { - name string - due time.Time - runActive bool - want Decision - }{ - {"on time", now, false, Fire}, - {"ten minutes late still fires", now.Add(-10 * time.Minute), false, Fire}, - {"fifty-nine minutes late still fires", now.Add(-59 * time.Minute), false, Fire}, - {"just past the grace window is missed", now.Add(-61 * time.Minute), false, SkipMissed}, - {"two days late is missed", now.Add(-48 * time.Hour), false, SkipMissed}, - {"an active run wins over on time", now, true, SkipRunning}, - {"an active run wins over a late one", now.Add(-10 * time.Minute), true, SkipRunning}, - {"a missed one is missed even with a run active", now.Add(-48 * time.Hour), true, SkipMissed}, - } - - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - if got := Decide(tc.due, now, tc.runActive); got != tc.want { - t.Fatalf("got %q, want %q", got, tc.want) - } - }) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd server && go test ./internal/workflowsched/ -run TestDecide -v` -Expected: FAIL — `undefined: Decide` - -- [ ] **Step 3: Write minimal implementation** - -Append to `server/internal/workflowsched/cron.go`: - -```go -type Decision string - -const ( - Fire Decision = "fire" - SkipMissed Decision = "missed" - SkipRunning Decision = "already_running" -) - -// GraceWindow is how late an occurrence may be and still run. A job missed by -// ten minutes during a deploy should still run; one missed by two days should -// not fire at lunchtime. -const GraceWindow = time.Hour - -// Decide is the whole fire/skip policy, kept pure so it can be tested without -// a database and read without following a loop. -// -// The missed check comes first: an occurrence that is already too old to run -// should be recorded as missed regardless of what is running now, or a slow -// run would relabel a stale occurrence as a fresh conflict. -func Decide(due, now time.Time, runActive bool) Decision { - if now.Sub(due) > GraceWindow { - return SkipMissed - } - if runActive { - return SkipRunning - } - return Fire -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd server && go test ./internal/workflowsched/ -v` -Expected: PASS, all tests including the eight `Decide` subtests. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/workflowsched/cron.go server/internal/workflowsched/cron_test.go -git commit -m "feat: fire, missed and already-running decision for scheduled workflows" -``` - ---- - -### Task 3: Schedule model and persistence - -**Files:** -- Modify: `server/internal/models/workflow.go:47-56` -- Modify: `server/internal/services/workflows.go:39` (`EnsureWorkflowIndexes`) and append `SetSchedule` - -**Interfaces:** -- Consumes: `workflowsched.NextOccurrence`, `workflowsched.ErrBadSchedule`. -- Produces: - - `models.Schedule{Enabled bool; Cron string; TZ string}` - - `models.Skip{Reason string; Due time.Time; At time.Time}` - - `models.Workflow.Schedule *Schedule`, `.NextRunAt *time.Time`, `.LastRunAt *time.Time`, `.LastSkipped *Skip` - - `services.SetSchedule(instanceID, workflowID string, s *models.Schedule) (*time.Time, error)` - -- [ ] **Step 1: Add the model types** - -In `server/internal/models/workflow.go`, above `type Workflow struct`: - -```go -type Schedule struct { - Enabled bool `bson:"enabled" json:"enabled"` - Cron string `bson:"cron" json:"cron"` // 5-field: minute hour dom month dow - TZ string `bson:"tz" json:"tz"` // IANA name, e.g. Europe/London -} - -// Skip records why an occurrence did not run. Recording a reason nobody reads -// is the same as not recording one, so this is surfaced in the UI. -type Skip struct { - Reason string `bson:"reason" json:"reason"` // "missed" | "already_running" - Due time.Time `bson:"due" json:"due"` - At time.Time `bson:"at" json:"at"` -} -``` - -Inside `type Workflow struct`, after `Steps`: - -```go - Schedule *Schedule `bson:"schedule,omitempty" json:"schedule,omitempty"` - NextRunAt *time.Time `bson:"next_run_at,omitempty" json:"next_run_at,omitempty"` - LastRunAt *time.Time `bson:"last_run_at,omitempty" json:"last_run_at,omitempty"` - LastSkipped *Skip `bson:"last_skipped,omitempty" json:"last_skipped,omitempty"` -``` - -- [ ] **Step 2: Add the index** - -Inside `EnsureWorkflowIndexes` in `server/internal/services/workflows.go`, alongside the existing `workflows` index: - -```go - if _, err := db.Col("workflows").Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "next_run_at", Value: 1}}, - }); err != nil { - return err - } -``` - -- [ ] **Step 3: Add SetSchedule** - -Append to `server/internal/services/workflows.go`: - -```go -// SetSchedule validates and stores a workflow's schedule, computing the first -// occurrence. next_run_at is persisted rather than held in memory: a leader -// handover between computing an occurrence and firing it would otherwise lose -// it or fire it twice. -// -// Passing s == nil, or a disabled schedule, clears next_run_at so the -// scheduler's query stops matching the document at all. -func SetSchedule(instanceID, workflowID string, s *models.Schedule) (*time.Time, error) { - ctx, cancel := wfCtx() - defer cancel() - - set := bson.M{"schedule": s, "updated_at": time.Now()} - unset := bson.M{} - - var next *time.Time - if s != nil && s.Enabled { - at, err := workflowsched.NextOccurrence(s.Cron, s.TZ, time.Now()) - if err != nil { - return nil, err - } - next = &at - set["next_run_at"] = at - } else { - unset["next_run_at"] = "" - } - - update := bson.M{"$set": set} - if len(unset) > 0 { - update["$unset"] = unset - } - - res, err := db.Col("workflows").UpdateOne(ctx, - bson.M{"workflow_id": workflowID, "instance_id": instanceID}, update) - if err != nil { - return nil, err - } - if res.MatchedCount == 0 { - return nil, mongo.ErrNoDocuments - } - return next, nil -} -``` - -Add `"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"` to that file's imports. - -- [ ] **Step 4: Verify it builds and tests pass** - -Run: `cd server && go build ./... && go test ./...` -Expected: build silent, tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/models/workflow.go server/internal/services/workflows.go -git commit -m "feat: persist a workflow schedule and its next occurrence" -``` - ---- - -### Task 4: The scheduler loop - -**Files:** -- Create: `server/internal/workflowsched/sched.go` -- Modify: `server/cmd/main.go` — imports and the `RunAsLeader` block at line 173 - -**Interfaces:** -- Consumes: `Decide`, `NextOccurrence`, `models.Workflow`, `services.TriggerWorkflow`, `services.LogEvent`. -- Produces: `workflowsched.Start(ctx context.Context)`. - -- [ ] **Step 1: Write the loop** - -Create `server/internal/workflowsched/sched.go`: - -```go -package workflowsched - -import ( - "context" - "log" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -const tickInterval = 30 * time.Second - -// Start runs the scheduler until ctx is cancelled. It is called inside -// bus.RunAsLeader("housekeeping", …) alongside monitorsched and the sweepers: -// one role, one lock. N replicas each running this loop would fire every -// scheduled workflow N times. -func Start(ctx context.Context) { - go func() { - ticker := time.NewTicker(tickInterval) - defer ticker.Stop() - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - tick(ctx) - } - } - }() -} - -func tick(ctx context.Context) { - now := time.Now() - - cur, err := db.Col("workflows").Find(ctx, bson.M{ - "schedule.enabled": true, - "next_run_at": bson.M{"$lte": now}, - }) - if err != nil { - log.Printf("workflowsched: find due: %v", err) - return - } - defer cur.Close(ctx) - - var due []models.Workflow - if err := cur.All(ctx, &due); err != nil { - log.Printf("workflowsched: decode due: %v", err) - return - } - - for _, wf := range due { - if ctx.Err() != nil { - return - } - process(ctx, wf, now) - } -} - -func process(ctx context.Context, wf models.Workflow, now time.Time) { - if wf.NextRunAt == nil || wf.Schedule == nil { - return - } - dueAt := *wf.NextRunAt - - next, err := NextOccurrence(wf.Schedule.Cron, wf.Schedule.TZ, now) - if err != nil { - // A schedule that no longer parses cannot be advanced, and leaving - // next_run_at in the past would spin this loop every 30 seconds - // forever. Disable it and say so. - log.Printf("workflowsched: workflow %s has an unusable schedule, disabling: %v", wf.WorkflowID, err) - disable(ctx, wf, err.Error()) - return - } - - // The claim. Matching on the current next_run_at as well as the id means a - // second process reaching this document after another has claimed it - // matches nothing and does nothing. This — not the leader lock — is what - // makes a double fire impossible; the lock only keeps it cheap. - res, err := db.Col("workflows").UpdateOne(ctx, - bson.M{"workflow_id": wf.WorkflowID, "next_run_at": dueAt}, - bson.M{"$set": bson.M{"next_run_at": next}}, - ) - if err != nil { - log.Printf("workflowsched: claim %s: %v", wf.WorkflowID, err) - return - } - if res.MatchedCount == 0 { - return // claimed elsewhere - } - - switch Decide(dueAt, now, hasActiveRun(ctx, wf.InstanceID, wf.WorkflowID)) { - case SkipMissed: - recordSkip(ctx, wf, string(SkipMissed), dueAt, now) - case SkipRunning: - recordSkip(ctx, wf, string(SkipRunning), dueAt, now) - case Fire: - if _, err := services.TriggerWorkflow(wf.InstanceID, wf.WorkflowID, "schedule"); err != nil { - log.Printf("workflowsched: trigger %s: %v", wf.WorkflowID, err) - recordSkip(ctx, wf, "error: "+err.Error(), dueAt, now) - return - } - _, _ = db.Col("workflows").UpdateOne(ctx, - bson.M{"workflow_id": wf.WorkflowID}, - bson.M{"$set": bson.M{"last_run_at": now}, "$unset": bson.M{"last_skipped": ""}}, - ) - services.LogEvent(wf.InstanceID, "workflow.scheduled_run", "schedule", "", "", - "workflow "+wf.Name+" started on schedule") - } -} - -func hasActiveRun(ctx context.Context, instanceID, workflowID string) bool { - err := db.Col("workflow_runs").FindOne(ctx, bson.M{ - "instance_id": instanceID, - "workflow_id": workflowID, - "status": "running", - }, options.FindOne().SetProjection(bson.M{"_id": 1})).Err() - return err == nil -} - -func recordSkip(ctx context.Context, wf models.Workflow, reason string, due, at time.Time) { - _, _ = db.Col("workflows").UpdateOne(ctx, - bson.M{"workflow_id": wf.WorkflowID}, - bson.M{"$set": bson.M{"last_skipped": models.Skip{Reason: reason, Due: due, At: at}}}, - ) - services.LogEvent(wf.InstanceID, "workflow.schedule_skipped", "schedule", "", "", - "workflow "+wf.Name+" skipped "+due.Format(time.RFC3339)+": "+reason) -} - -func disable(ctx context.Context, wf models.Workflow, reason string) { - _, _ = db.Col("workflows").UpdateOne(ctx, - bson.M{"workflow_id": wf.WorkflowID}, - bson.M{ - "$set": bson.M{"schedule.enabled": false}, - "$unset": bson.M{"next_run_at": ""}, - }, - ) - services.LogEvent(wf.InstanceID, "workflow.schedule_disabled", "schedule", "", "", - "workflow "+wf.Name+" schedule disabled: "+reason) -} -``` - -Note the import list above omits `mongo` — this file uses `bson` and `options` but no driver error values. - -- [ ] **Step 2: Embed the timezone database** - -In `server/cmd/main.go`, in the import block: - -```go - _ "time/tzdata" -``` - -**This is load-bearing.** `server/Dockerfile` builds on Alpine, which ships no zoneinfo, so without it `time.LoadLocation("Europe/London")` fails in production and every schedule silently falls back to UTC — an hour wrong for half the year, in the direction nobody notices until a maintenance window lands in business hours. It works on a developer machine either way, which is exactly why it gets forgotten. - -- [ ] **Step 3: Start it under the leader lock** - -In `server/cmd/main.go`, inside the existing `bus.RunAsLeader(ctx, "housekeeping", func(jobCtx context.Context) {` block, beside `monitorsched.Start(jobCtx)`: - -```go - workflowsched.Start(jobCtx) -``` - -Add the import: `"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/workflowsched"`. - -- [ ] **Step 4: Verify it builds and tests pass** - -Run: `cd server && go build ./... && go vet ./... && go test ./...` -Expected: build silent, tests PASS. - -- [ ] **Step 5: Manual smoke test** - -Set a workflow's schedule to `* * * * *` (every minute) in `UTC` directly in Mongo, along with `next_run_at` set to now, and watch a run start within 30 seconds. Then, while it is running, confirm the next occurrence records `last_skipped.reason: "already_running"` rather than starting a second run. Finally set `next_run_at` to two days ago and confirm `"missed"`. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/workflowsched/sched.go server/cmd/main.go -git commit -m "feat: fire scheduled workflow runs from the housekeeping leader" -``` - ---- - -### Task 5: Schedule API - -**Files:** -- Modify: `server/internal/api/workflows.go` - -**Interfaces:** -- Consumes: `services.SetSchedule`, `workflowsched.NextOccurrence`, `workflowsched.ErrBadSchedule`. -- Produces: `PUT /api/workflows/:id/schedule`, `GET /api/workflows/:id/schedule/preview?cron=…&tz=…`. - -- [ ] **Step 1: Register the routes** - -In `registerWorkflowRoutes`, beside the existing `/workflows/:id/run`: - -```go - g.PUT("/workflows/:id/schedule", putWorkflowSchedule) - g.GET("/workflows/:id/schedule/preview", previewWorkflowSchedule) -``` - -- [ ] **Step 2: Add the handlers** - -Append to `server/internal/api/workflows.go`: - -```go -func putWorkflowSchedule(c *gin.Context) { - var body models.Schedule - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": "invalid body"}) - return - } - - instanceID := auth.InstanceID(c) - next, err := services.SetSchedule(instanceID, c.Param("id"), &body) - if err != nil { - if errors.Is(err, workflowsched.ErrBadSchedule) { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - if errors.Is(err, mongo.ErrNoDocuments) { - c.JSON(http.StatusNotFound, gin.H{"error": "workflow not found"}) - return - } - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - - services.LogEvent(instanceID, "workflow.schedule_updated", auth.Email(c), "", "", - fmt.Sprintf("schedule %q %s enabled=%v", body.Cron, body.TZ, body.Enabled)) - c.JSON(http.StatusOK, gin.H{"schedule": body, "next_run_at": next}) -} - -// previewWorkflowSchedule exists so the browser and the scheduler agree on -// what a cron string means. A client-side cron parser that disagrees with the -// server by one field is a bug found in production, at night. -func previewWorkflowSchedule(c *gin.Context) { - expr := c.Query("cron") - tz := c.Query("tz") - - occurrences := make([]time.Time, 0, 3) - from := time.Now() - for i := 0; i < 3; i++ { - next, err := workflowsched.NextOccurrence(expr, tz, from) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - occurrences = append(occurrences, next) - from = next - } - c.JSON(http.StatusOK, gin.H{"occurrences": occurrences}) -} -``` - -Add whichever of `"errors"`, `"fmt"`, `"time"`, `"go.mongodb.org/mongo-driver/v2/mongo"`, the `models` package and the `workflowsched` package are not already imported. Use the same actor helper the neighbouring handlers in this file use for `LogEvent`. - -- [ ] **Step 3: Verify it builds** - -Run: `cd server && go build ./... && go vet ./internal/api/` -Expected: no output. - -- [ ] **Step 4: Manual smoke test** - -```bash -curl -b cookies.txt -X PUT localhost:8080/api/workflows//schedule \ - -H 'content-type: application/json' -d '{"enabled":true,"cron":"0 2 * * 0","tz":"Europe/London"}' -curl -b cookies.txt 'localhost:8080/api/workflows//schedule/preview?cron=0+2+*+*+0&tz=Europe/London' -curl -b cookies.txt -X PUT localhost:8080/api/workflows//schedule \ - -H 'content-type: application/json' -d '{"enabled":true,"cron":"nope","tz":"UTC"}' # expect 400 -``` - -Expected: the first returns a `next_run_at` matching the first preview occurrence, and the third is 400. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/api/workflows.go -git commit -m "feat: schedule and preview endpoints for workflows" -``` - ---- - -### Task 6: Frontend API client - -**Files:** -- Modify: `web/lib/api.ts` - -**Interfaces:** -- Consumes: the routes from Task 5. -- Produces: `Schedule`, `Skip`, `Workflow.schedule`/`.next_run_at`/`.last_run_at`/`.last_skipped`, `api.setWorkflowSchedule`, `api.previewSchedule`. - -- [ ] **Step 1: Add the types** - -```ts -export interface Schedule { - enabled: boolean; - cron: string; - tz: string; -} - -export interface Skip { - reason: string; - due: string; - at: string; -} -``` - -In the `Workflow` interface: - -```ts - schedule?: Schedule; - next_run_at?: string; - last_run_at?: string; - last_skipped?: Skip; -``` - -- [ ] **Step 2: Add the client methods** - -```ts - setWorkflowSchedule(workflowId: string, schedule: Schedule): Promise<{ schedule: Schedule; next_run_at: string | null }> { - return request(`/workflows/${workflowId}/schedule`, { method: "PUT", body: JSON.stringify(schedule) }); - }, - - previewSchedule(workflowId: string, cron: string, tz: string): Promise<{ occurrences: string[] }> { - return request(`/workflows/${workflowId}/schedule/preview?cron=${encodeURIComponent(cron)}&tz=${encodeURIComponent(tz)}`); - }, -``` - -- [ ] **Step 3: Verify it typechecks** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json` -Expected: no output. - -- [ ] **Step 4: Commit** - -```bash -git add web/lib/api.ts -git commit -m "feat: schedule methods on the web api client" -``` - ---- - -### Task 7: Schedule card - -**Files:** -- Create: `web/components/workflows/ScheduleCard.tsx` -- Modify: `web/app/(app)/workflows/[id]/page.tsx` - -**Interfaces:** -- Consumes: `api.setWorkflowSchedule`, `api.previewSchedule`, `Workflow.schedule`, `Workflow.last_skipped`. -- Produces: ``. - -- [ ] **Step 1: Write the component** - -Create `web/components/workflows/ScheduleCard.tsx`: - -```tsx -"use client"; - -import { useState } from "react"; -import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; -import { api, Workflow } from "@/lib/api"; -import { Button } from "@/components/ui"; - -/* - * Presets write cron underneath rather than being their own storage format: - * one representation, and the raw field is always the truth. The next three - * occurrences come from the server so the browser cannot disagree with the - * scheduler about what an expression means. - */ - -const PRESETS: { label: string; cron: string }[] = [ - { label: "Hourly", cron: "0 * * * *" }, - { label: "Nightly, 02:00", cron: "0 2 * * *" }, - { label: "Weekly, Sun 02:00", cron: "0 2 * * 0" }, - { label: "Monthly, 1st 02:00", cron: "0 2 1 * *" }, -]; - -const ZONES = ["UTC", "Europe/London", "Europe/Berlin", "America/New_York", "America/Los_Angeles", "Asia/Singapore", "Australia/Sydney"]; - -export function ScheduleCard({ workflow }: { workflow: Workflow }) { - const queryClient = useQueryClient(); - const [enabled, setEnabled] = useState(workflow.schedule?.enabled ?? false); - const [cron, setCron] = useState(workflow.schedule?.cron ?? "0 2 * * 0"); - const [tz, setTz] = useState(workflow.schedule?.tz ?? Intl.DateTimeFormat().resolvedOptions().timeZone ?? "UTC"); - const [error, setError] = useState(null); - - const { data: preview } = useQuery({ - queryKey: ["schedule-preview", workflow.workflow_id, cron, tz], - queryFn: () => api.previewSchedule(workflow.workflow_id, cron, tz), - retry: false, - }); - - const { mutate: save, isPending } = useMutation({ - mutationFn: () => api.setWorkflowSchedule(workflow.workflow_id, { enabled, cron, tz }), - onSuccess: () => { - queryClient.invalidateQueries({ queryKey: ["workflows"] }); - setError(null); - }, - onError: (e: Error) => setError(e.message), - }); - - return ( -
-
-

Schedule

- - {enabled ? "Active" : "Off"} - -
- -
- - -
- {PRESETS.map((p) => ( - - ))} -
- -
- - - -
- -
-

Next three runs

- {preview ? ( -
    - {preview.occurrences.map((o) => ( -
  • {new Date(o).toLocaleString()}
    • - ))} -
- ) : ( -

That expression is not valid.

- )} -
- - {workflow.last_skipped && ( -

- Skipped {new Date(workflow.last_skipped.due).toLocaleString()} —{" "} - {workflow.last_skipped.reason === "already_running" - ? "previous run still active" - : workflow.last_skipped.reason === "missed" - ? "the control plane was not running at the time" - : workflow.last_skipped.reason} -

- )} - - {error &&

{error}

} - -
- -
-
-
- ); -} -``` - -- [ ] **Step 2: Mount it** - -In `web/app/(app)/workflows/[id]/page.tsx`, render `` in the sidebar or below the step designer, wherever the page's existing panels sit. - -- [ ] **Step 3: Verify it builds** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json && npx next build` -Expected: `✓ Compiled successfully`. - -- [ ] **Step 4: Manual check** - -Enable a weekly schedule, confirm the three previewed occurrences are Sundays at 02:00 local, save, reload, and confirm it persisted. Type nonsense into the cron field and confirm the preview panel says the expression is not valid rather than showing stale occurrences. - -- [ ] **Step 5: Commit** - -```bash -git add web/components/workflows/ScheduleCard.tsx "web/app/(app)/workflows/[id]/page.tsx" -git commit -m "feat: schedule editor on the workflow page" -``` - ---- - -### Task 8: Schedule on the workflows list - -**Files:** -- Modify: `web/app/(app)/workflows/page.tsx` - -- [ ] **Step 1: Add the chip and next-run column** - -For each workflow row, when `w.schedule?.enabled`, render a mono chip carrying the cron expression and the next run as relative time: - -```tsx -{w.schedule?.enabled && ( - - {w.schedule.cron} - -)} -{w.next_run_at && ( - - next {new Date(w.next_run_at).toLocaleString()} - -)} -``` - -- [ ] **Step 2: Verify it builds** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json && npx next build` -Expected: `✓ Compiled successfully`. - -- [ ] **Step 3: Commit** - -```bash -git add "web/app/(app)/workflows/page.tsx" -git commit -m "feat: show workflow schedules in the list" -``` - ---- - -### Task 9: Documentation - -**Files:** -- Modify: `docsite/docs/vantage/workflows.md`, `CLAUDE.md` - -- [ ] **Step 1: Document schedules for users** - -Add a "Schedules" section to `docsite/docs/vantage/workflows.md`: the 5-field cron format with the four preset equivalents, that the timezone is stored by name so DST is handled, that an overlapping occurrence is skipped rather than queued, and that an occurrence missed by more than an hour is recorded and dropped rather than fired late. - -- [ ] **Step 2: Update the contributor map** - -In `CLAUDE.md`, under Workflows, add: schedules live on the workflow document with a persisted `next_run_at`; `workflowsched` runs inside the single `RunAsLeader("housekeeping", …)`; the atomic claim on `next_run_at` — not the lock — is what prevents a double fire; `import _ "time/tzdata"` is required because Alpine ships no zone database and its absence silently reverts every schedule to UTC. - -- [ ] **Step 3: Commit** - -```bash -git add docsite/docs/vantage/workflows.md CLAUDE.md -git commit -m "docs: scheduled workflows" -``` - ---- - -## Verification - -```bash -cd server && go build ./... && go vet ./... && go test ./... -cd ../web && npx tsc --noEmit -p tsconfig.json && npx next build -cd .. && graphify update . -``` - -All must pass. `go test ./internal/workflowsched/` must report the cron, DST and decision suites passing. - -Two things no unit test here covers, because both need a real database: - -- **The claim under contention.** The spec asks that two concurrent claims of the same due workflow start exactly one run. `Decide` is tested; the `findOneAndUpdate` guard is not. Verify it by hand with two server processes pointed at one Mongo and one workflow due immediately — exactly one run document should appear. -- **Persistence across a restart.** Set a schedule, restart the server, and confirm the workflow still fires at its stated time. `next_run_at` is persisted precisely so that a restart or a leader handover does not lose it, and a regression there is invisible until the night it matters. diff --git a/docs/superpowers/plans/2026-08-04-server-tags.md b/docs/superpowers/plans/2026-08-04-server-tags.md deleted file mode 100644 index 734ba6d..0000000 --- a/docs/superpowers/plans/2026-08-04-server-tags.md +++ /dev/null @@ -1,1308 +0,0 @@ -# Server Tags Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Give every server a `key:value` tag map, let workflows target servers by tag selector unioned with their explicit list, and surface both in the UI. - -**Architecture:** Tags are a `map[string]string` field on the existing `servers` documents — no new collection. Validation and matching live as pure functions in `server/internal/services` so they can be unit tested without a database; the Mongo-backed wrappers around them stay thin. Workflow targeting gains `target_tags` beside the untouched `target_server_ids`, and one `ResolveTargets` function produces the distinct union that `TriggerWorkflow` already expects as a list of server IDs. - -**Tech Stack:** Go 1.26, gin, mongo-driver v2, Next.js 16 + TanStack Query, Tailwind 3. - -## Global Constraints - -- Tag keys and values are lowercase `[a-z0-9_-]` only. Key max 32 chars, value max 64 chars, max 20 tags per server. -- Reserved key prefix `sys:` — rejected on user writes, unused otherwise. -- Every mutating API path writes an audit event via `services.LogEvent`. -- Every service query is scoped by `instance_id`. No exceptions, no unscoped lookups. -- No component in `web/` may carry a hex colour; use the Tailwind token names only. -- `web/` is dark-only. Do not add a light theme. -- This repository has **no Go tests today**. Tests added here are the first; they must run under plain `go test ./...` from the repo root with no database, no network, and no build tags. -- Commit messages use the `feat:` / `fix:` / `docs:` prefixes already in `git log`. - ---- - -## File Structure - -**Create:** -- `server/internal/services/tags.go` — tag validation, normalisation, matching. Pure functions plus the Mongo-backed read/write. -- `server/internal/services/tags_test.go` — unit tests for the pure half. -- `server/internal/services/targets.go` — `ResolveTargets` and its pure core. -- `server/internal/services/targets_test.go` — unit tests for the pure core. -- `web/components/servers/TagChips.tsx` — display and inline edit of one server's tags. -- `web/components/servers/TagFilterBar.tsx` — the fleet list filter. - -**Modify:** -- `server/internal/models/server.go` — add `Tags`. -- `server/internal/models/workflow.go` — add `TargetTags`. -- `server/internal/services/servers.go` — `ListServers` gains a tag filter; add `EnsureServerIndexes`. -- `server/internal/services/workflow_runner.go:20-34` — `TriggerWorkflow` resolves targets. -- `server/internal/services/workflows.go` — validate `target_tags` on create/update. -- `server/internal/api/handlers.go` — three routes and their handlers. -- `server/cmd/main.go:111` — call `EnsureServerIndexes`. -- `web/lib/api.ts` — types and client methods. -- `web/app/(app)/servers/page.tsx` — filter bar and tag column. -- `web/app/(app)/servers/[id]/page.tsx` — tag chips in the header. -- `web/app/(app)/workflows/[id]/page.tsx` — target section. -- `docsite/docs/vantage/servers.md`, `docsite/docs/vantage/workflows.md` — document tags. -- `CLAUDE.md` — one line in the servers/workflows subsystem notes. - ---- - -### Task 1: Tag validation - -**Files:** -- Create: `server/internal/services/tags.go` -- Test: `server/internal/services/tags_test.go` -- Modify: `server/internal/models/server.go:46-67` - -**Interfaces:** -- Consumes: nothing. -- Produces: `services.ValidateTags(tags map[string]string) error`, `services.ErrInvalidTag` (a `error` value), and `models.Server.Tags map[string]string`. - -- [ ] **Step 1: Write the failing test** - -Create `server/internal/services/tags_test.go`: - -```go -package services - -import ( - "strings" - "testing" -) - -func TestValidateTags(t *testing.T) { - long33 := strings.Repeat("a", 33) - long65 := strings.Repeat("b", 65) - - tooMany := map[string]string{} - for i := 0; i < 21; i++ { - tooMany[string(rune('a'+i))] = "x" - } - - cases := []struct { - name string - tags map[string]string - ok bool - }{ - {"empty is fine", map[string]string{}, true}, - {"nil is fine", nil, true}, - {"simple pair", map[string]string{"env": "prod"}, true}, - {"dash and underscore", map[string]string{"team_name": "core-infra"}, true}, - {"digits", map[string]string{"tier1": "web2"}, true}, - {"uppercase key", map[string]string{"Env": "prod"}, false}, - {"uppercase value", map[string]string{"env": "Prod"}, false}, - {"space in value", map[string]string{"env": "pro d"}, false}, - {"colon in key", map[string]string{"en:v": "prod"}, false}, - {"empty key", map[string]string{"": "prod"}, false}, - {"empty value", map[string]string{"env": ""}, false}, - {"key too long", map[string]string{long33: "prod"}, false}, - {"value too long", map[string]string{"env": long65}, false}, - {"reserved prefix", map[string]string{"sys:os": "linux"}, false}, - {"too many tags", tooMany, false}, - } - - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - err := ValidateTags(tc.tags) - if tc.ok && err != nil { - t.Fatalf("expected valid, got %v", err) - } - if !tc.ok && err == nil { - t.Fatal("expected an error, got nil") - } - }) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd server && go test ./internal/services/ -run TestValidateTags -v` -Expected: FAIL — `undefined: ValidateTags` - -- [ ] **Step 3: Write minimal implementation** - -Create `server/internal/services/tags.go`: - -```go -package services - -import ( - "errors" - "fmt" - "strings" -) - -// ErrInvalidTag is returned for any tag the rules below reject. Handlers map -// it to 400 — a malformed tag is the caller's mistake, not a server fault. -var ErrInvalidTag = errors.New("invalid tag") - -const ( - maxTagKeyLen = 32 - maxTagValueLen = 64 - maxTagsPerHost = 20 - // Reserved for tags the agent may derive from inventory later. Refusing - // it now means a user tag written today can never collide with a system - // tag invented tomorrow. - sysTagPrefix = "sys:" -) - -func validTagRunes(s string) bool { - for _, r := range s { - switch { - case r >= 'a' && r <= 'z': - case r >= '0' && r <= '9': - case r == '-' || r == '_': - default: - return false - } - } - return true -} - -// ValidateTags enforces the shape of a whole tag map. It lives in the service -// layer rather than a handler so that every write path — the tags endpoint, -// server create, anything added later — agrees on what a valid tag is. -func ValidateTags(tags map[string]string) error { - if len(tags) > maxTagsPerHost { - return fmt.Errorf("%w: at most %d tags per server", ErrInvalidTag, maxTagsPerHost) - } - for k, v := range tags { - if strings.HasPrefix(k, sysTagPrefix) { - return fmt.Errorf("%w: keys beginning %q are reserved", ErrInvalidTag, sysTagPrefix) - } - if k == "" || len(k) > maxTagKeyLen || !validTagRunes(k) { - return fmt.Errorf("%w: key %q must be 1-%d chars of a-z, 0-9, - or _", ErrInvalidTag, k, maxTagKeyLen) - } - if v == "" || len(v) > maxTagValueLen || !validTagRunes(v) { - return fmt.Errorf("%w: value for %q must be 1-%d chars of a-z, 0-9, - or _", ErrInvalidTag, k, maxTagValueLen) - } - } - return nil -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd server && go test ./internal/services/ -run TestValidateTags -v` -Expected: PASS, all 15 subtests. - -- [ ] **Step 5: Add the model field** - -In `server/internal/models/server.go`, inside `type Server struct`, after the `Inventory` line: - -```go - Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"` -``` - -- [ ] **Step 6: Verify it builds** - -Run: `cd server && go build ./...` -Expected: no output. - -- [ ] **Step 7: Commit** - -```bash -git add server/internal/services/tags.go server/internal/services/tags_test.go server/internal/models/server.go -git commit -m "feat: validate server tags and add the model field" -``` - ---- - -### Task 2: Tag parsing for query strings - -**Files:** -- Modify: `server/internal/services/tags.go` -- Test: `server/internal/services/tags_test.go` - -**Interfaces:** -- Consumes: `ValidateTags`, `ErrInvalidTag` from Task 1. -- Produces: `services.ParseTagFilters(raw []string) (map[string]string, error)`. - -- [ ] **Step 1: Write the failing test** - -Append to `server/internal/services/tags_test.go`: - -```go -func TestParseTagFilters(t *testing.T) { - got, err := ParseTagFilters([]string{"env:prod", "role:web"}) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if len(got) != 2 || got["env"] != "prod" || got["role"] != "web" { - t.Fatalf("unexpected map: %#v", got) - } - - empty, err := ParseTagFilters(nil) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if len(empty) != 0 { - t.Fatalf("expected empty map, got %#v", empty) - } - - for _, bad := range []string{"env", "env:", ":prod", "env:prod:extra", "ENV:prod", ""} { - if _, err := ParseTagFilters([]string{bad}); err == nil { - t.Fatalf("expected %q to be rejected", bad) - } - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd server && go test ./internal/services/ -run TestParseTagFilters -v` -Expected: FAIL — `undefined: ParseTagFilters` - -- [ ] **Step 3: Write minimal implementation** - -Append to `server/internal/services/tags.go`: - -```go -// ParseTagFilters turns repeated ?tag=key:value query values into a map. -// -// A malformed filter is an error rather than a silently ignored value: a -// filter that matches nothing and a filter that is nonsense look identical in -// a list, and only one of them is the caller's fault. -func ParseTagFilters(raw []string) (map[string]string, error) { - out := make(map[string]string, len(raw)) - for _, r := range raw { - k, v, found := strings.Cut(r, ":") - if !found { - return nil, fmt.Errorf("%w: filter %q must be key:value", ErrInvalidTag, r) - } - if strings.Contains(v, ":") { - return nil, fmt.Errorf("%w: filter %q has more than one colon", ErrInvalidTag, r) - } - out[k] = v - } - if err := ValidateTags(out); err != nil { - return nil, err - } - return out, nil -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd server && go test ./internal/services/ -run TestParseTagFilters -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/services/tags.go server/internal/services/tags_test.go -git commit -m "feat: parse repeated tag query filters" -``` - ---- - -### Task 3: Target resolution core - -**Files:** -- Create: `server/internal/services/targets.go` -- Test: `server/internal/services/targets_test.go` - -**Interfaces:** -- Consumes: `models.Server` with `Tags` from Task 1. -- Produces: `services.MatchesTags(srv models.Server, sel map[string]string) bool` and `services.UnionTargets(all []models.Server, ids []string, sel map[string]string) []models.Server`, plus `services.ErrNoTargets`. - -- [ ] **Step 1: Write the failing test** - -Create `server/internal/services/targets_test.go`: - -```go -package services - -import ( - "testing" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" -) - -func fixtureServers() []models.Server { - return []models.Server{ - {ServerID: "a", Hostname: "web-1", Tags: map[string]string{"env": "prod", "role": "web"}}, - {ServerID: "b", Hostname: "web-2", Tags: map[string]string{"env": "prod", "role": "web"}}, - {ServerID: "c", Hostname: "db-1", Tags: map[string]string{"env": "prod", "role": "db"}}, - {ServerID: "d", Hostname: "web-3", Tags: map[string]string{"env": "staging", "role": "web"}}, - {ServerID: "e", Hostname: "untagged-1"}, - } -} - -func ids(servers []models.Server) []string { - out := make([]string, 0, len(servers)) - for _, s := range servers { - out = append(out, s.ServerID) - } - return out -} - -func equalIDs(t *testing.T, got []models.Server, want ...string) { - t.Helper() - gotIDs := ids(got) - if len(gotIDs) != len(want) { - t.Fatalf("got %v, want %v", gotIDs, want) - } - for i := range want { - if gotIDs[i] != want[i] { - t.Fatalf("got %v, want %v", gotIDs, want) - } - } -} - -func TestMatchesTags(t *testing.T) { - web1 := fixtureServers()[0] - - if !MatchesTags(web1, map[string]string{"env": "prod"}) { - t.Fatal("expected single-key match") - } - if !MatchesTags(web1, map[string]string{"env": "prod", "role": "web"}) { - t.Fatal("expected AND across keys to match") - } - if MatchesTags(web1, map[string]string{"env": "prod", "role": "db"}) { - t.Fatal("AND across keys must not match on one key alone") - } - if MatchesTags(web1, map[string]string{"team": "core"}) { - t.Fatal("absent key must not match") - } - if MatchesTags(web1, map[string]string{}) { - t.Fatal("an empty selector must match nothing, not everything") - } -} - -func TestUnionTargets(t *testing.T) { - all := fixtureServers() - - t.Run("ids only", func(t *testing.T) { - equalIDs(t, UnionTargets(all, []string{"a", "c"}, nil), "a", "c") - }) - - t.Run("selector only", func(t *testing.T) { - equalIDs(t, UnionTargets(all, nil, map[string]string{"env": "prod", "role": "web"}), "a", "b") - }) - - t.Run("union deduplicates", func(t *testing.T) { - // "a" is named explicitly AND matched by the selector; it appears once. - equalIDs(t, UnionTargets(all, []string{"a", "c"}, map[string]string{"role": "web"}), "a", "b", "c", "d") - }) - - t.Run("unknown id is dropped", func(t *testing.T) { - equalIDs(t, UnionTargets(all, []string{"a", "does-not-exist"}, nil), "a") - }) - - t.Run("both empty yields nothing", func(t *testing.T) { - equalIDs(t, UnionTargets(all, nil, nil)) - }) - - t.Run("order follows the fleet, not the arguments", func(t *testing.T) { - equalIDs(t, UnionTargets(all, []string{"c", "a"}, nil), "a", "c") - }) -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd server && go test ./internal/services/ -run 'TestMatchesTags|TestUnionTargets' -v` -Expected: FAIL — `undefined: MatchesTags` - -- [ ] **Step 3: Write minimal implementation** - -Create `server/internal/services/targets.go`: - -```go -package services - -import ( - "errors" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" -) - -// ErrNoTargets means a workflow named no servers and matched none. Handlers -// map it to 400: a workflow that matches nothing must say so rather than -// report success over zero servers. -var ErrNoTargets = errors.New("workflow has no target servers") - -// MatchesTags reports whether srv carries every pair in sel — AND across keys. -// An empty selector matches nothing. That is deliberate: the alternative, -// "matches everything", turns a cleared field in the workflow designer into a -// fleet-wide run. -func MatchesTags(srv models.Server, sel map[string]string) bool { - if len(sel) == 0 { - return false - } - for k, v := range sel { - if srv.Tags[k] != v { - return false - } - } - return true -} - -// UnionTargets returns the distinct union of the servers named by ids and -// those matching sel, in the order they appear in all. -// -// Order comes from the fleet rather than the arguments so that two workflows -// naming the same servers in a different order still run them in the same -// order, which makes two runs comparable line by line. -// -// Offline servers are NOT filtered out. The dispatcher already answers 503 per -// server, and a patch run that silently omits an unreachable machine is worse -// than one that visibly fails on it. -func UnionTargets(all []models.Server, ids []string, sel map[string]string) []models.Server { - named := make(map[string]bool, len(ids)) - for _, id := range ids { - named[id] = true - } - - out := make([]models.Server, 0, len(ids)+len(all)) - for _, s := range all { - if named[s.ServerID] || MatchesTags(s, sel) { - out = append(out, s) - } - } - return out -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd server && go test ./internal/services/ -run 'TestMatchesTags|TestUnionTargets' -v` -Expected: PASS, all subtests. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/services/targets.go server/internal/services/targets_test.go -git commit -m "feat: resolve workflow targets as the union of ids and a tag selector" -``` - ---- - -### Task 4: Database-backed tag reads and writes - -**Files:** -- Modify: `server/internal/services/tags.go`, `server/internal/services/servers.go`, `server/internal/services/targets.go` -- Modify: `server/cmd/main.go:111` - -**Interfaces:** -- Consumes: `ValidateTags`, `UnionTargets`, `ErrNoTargets`. -- Produces: - - `services.SetServerTags(instanceID, serverID string, tags map[string]string) error` - - `services.KnownTags(instanceID string) (map[string][]string, error)` - - `services.ListServersFiltered(instanceID string, sel map[string]string) ([]models.Server, error)` - - `services.ResolveTargets(instanceID string, ids []string, sel map[string]string) ([]models.Server, error)` - - `services.EnsureServerIndexes() error` - -- [ ] **Step 1: Add the Mongo-backed tag functions** - -Append to `server/internal/services/tags.go`: - -```go -// SetServerTags replaces a server's whole tag map. -// -// Replace rather than patch: a tag set is small enough that sending all of it -// is free, and last-write-wins over a whole map is easier to reason about than -// merge semantics between two people editing the same server. -func SetServerTags(instanceID, serverID string, tags map[string]string) error { - if err := ValidateTags(tags); err != nil { - return err - } - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - res, err := db.Col("servers").UpdateOne(ctx, - bson.M{"server_id": serverID, "instance_id": instanceID}, - bson.M{"$set": bson.M{"tags": tags}}, - ) - if err != nil { - return err - } - if res.MatchedCount == 0 { - return mongo.ErrNoDocuments - } - return nil -} - -// KnownTags returns every key in use in this instance with its distinct -// values, for the UI's pickers. This is an aggregation rather than a -// maintained registry: a tag is a property of a server, not an entity, and a -// registry would need reference counting to know when a tag stopped existing. -func KnownTags(instanceID string) (map[string][]string, error) { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - cur, err := db.Col("servers").Find(ctx, - bson.M{"instance_id": instanceID, "tags": bson.M{"$exists": true}}, - options.Find().SetProjection(bson.M{"tags": 1}), - ) - if err != nil { - return nil, err - } - defer cur.Close(ctx) - - seen := map[string]map[string]bool{} - for cur.Next(ctx) { - var s models.Server - if err := cur.Decode(&s); err != nil { - return nil, err - } - for k, v := range s.Tags { - if seen[k] == nil { - seen[k] = map[string]bool{} - } - seen[k][v] = true - } - } - if err := cur.Err(); err != nil { - return nil, err - } - - out := make(map[string][]string, len(seen)) - for k, vals := range seen { - list := make([]string, 0, len(vals)) - for v := range vals { - list = append(list, v) - } - sort.Strings(list) - out[k] = list - } - return out, nil -} -``` - -Add to that file's imports: `"context"`, `"sort"`, `"time"`, `"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db"`, `"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"`, `"go.mongodb.org/mongo-driver/v2/bson"`, `"go.mongodb.org/mongo-driver/v2/mongo"`, `"go.mongodb.org/mongo-driver/v2/mongo/options"`. - -- [ ] **Step 2: Add the filtered list and the index builder** - -Append to `server/internal/services/servers.go`: - -```go -// ListServersFiltered is ListServers with an optional tag selector. An empty -// selector returns the whole fleet — unlike MatchesTags, where empty means -// "nothing", because here the caller is a list view whose default is -// "everything", not a run about to touch machines. -func ListServersFiltered(instanceID string, sel map[string]string) ([]models.Server, error) { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - filter := bson.M{"instance_id": instanceID} - for k, v := range sel { - filter["tags."+k] = v - } - - cur, err := db.Col("servers").Find(ctx, filter, options.Find().SetSort(bson.D{{Key: "hostname", Value: 1}})) - if err != nil { - return nil, err - } - defer cur.Close(ctx) - - servers := []models.Server{} - if err := cur.All(ctx, &servers); err != nil { - return nil, err - } - return servers, nil -} - -// EnsureServerIndexes declares the wildcard index over the tag subdocument. -// It is wildcard because the queried key is chosen by the user at request time -// and cannot be named in advance. -// -// Non-fatal, following EnsureSecretIndexes: a missing index degrades tag -// filtering to a collection scan over a small collection, which is slower. -// A fatal error here would refuse to boot the fleet list over it. -func EnsureServerIndexes() error { - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - - _, err := db.Col("servers").Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "tags.$**", Value: 1}}, - }) - return err -} -``` - -Add `"go.mongodb.org/mongo-driver/v2/mongo"` to that file's imports if it is not already there. - -- [ ] **Step 3: Add ResolveTargets** - -Append to `server/internal/services/targets.go`: - -```go -// ResolveTargets is the database-backed wrapper around UnionTargets. It is the -// single answer to "which servers does this workflow touch", used by the run -// path and by validation alike, so the two cannot disagree. -func ResolveTargets(instanceID string, ids []string, sel map[string]string) ([]models.Server, error) { - all, err := ListServers(instanceID) - if err != nil { - return nil, err - } - matched := UnionTargets(all, ids, sel) - if len(matched) == 0 { - return nil, ErrNoTargets - } - return matched, nil -} -``` - -Add `"gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models"` to that file's imports (already present from Task 3). - -- [ ] **Step 4: Wire the index builder into boot** - -In `server/cmd/main.go`, beside the existing `EnsureSecretIndexes` call around line 111: - -```go - if err := services.EnsureServerIndexes(); err != nil { - log.Printf("server index warning: %v", err) - } -``` - -- [ ] **Step 5: Verify it builds and existing tests still pass** - -Run: `cd server && go build ./... && go test ./...` -Expected: build silent, tests PASS. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/services/tags.go server/internal/services/servers.go server/internal/services/targets.go server/cmd/main.go -git commit -m "feat: read and write server tags, resolve targets from the database" -``` - ---- - -### Task 5: Tag API routes - -**Files:** -- Modify: `server/internal/api/handlers.go` — routes at line 53-61, handlers near `listServers` at line 121. - -**Interfaces:** -- Consumes: `SetServerTags`, `KnownTags`, `ListServersFiltered`, `ParseTagFilters`, `ErrInvalidTag`. -- Produces: `PUT /api/servers/:id/tags`, `GET /api/servers/tags`, and `?tag=` on `GET /api/servers`. - -- [ ] **Step 1: Register the routes** - -In `RegisterRoutes`, immediately after `apiGroup.GET("/servers", listServers)`: - -```go - apiGroup.GET("/servers/tags", listKnownTags) -``` - -and after `apiGroup.POST("/servers/:id/apply-updates", applyUpdates)`: - -```go - apiGroup.PUT("/servers/:id/tags", putServerTags) -``` - -`/servers/tags` must be registered **before** the `/servers/:id` routes are matched — gin resolves static segments ahead of wildcards, so this works, but keep it grouped with the other `/servers` routes for readability. - -- [ ] **Step 2: Replace the listServers handler** - -Replace `listServers` in `server/internal/api/handlers.go:121-128` with: - -```go -func listServers(c *gin.Context) { - sel, err := services.ParseTagFilters(c.QueryArray("tag")) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - servers, err := services.ListServersFiltered(auth.InstanceID(c), sel) - if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - c.JSON(http.StatusOK, servers) -} -``` - -- [ ] **Step 3: Add the two new handlers** - -Append near the other server handlers in the same file: - -```go -func listKnownTags(c *gin.Context) { - tags, err := services.KnownTags(auth.InstanceID(c)) - if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - c.JSON(http.StatusOK, tags) -} - -func putServerTags(c *gin.Context) { - var body struct { - Tags map[string]string `json:"tags"` - } - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": "invalid body"}) - return - } - - instanceID := auth.InstanceID(c) - serverID := c.Param("id") - - before, err := services.GetServer(instanceID, serverID) - if err != nil { - c.JSON(http.StatusNotFound, gin.H{"error": "server not found"}) - return - } - - if err := services.SetServerTags(instanceID, serverID, body.Tags); err != nil { - if errors.Is(err, services.ErrInvalidTag) { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - - services.LogEvent(instanceID, "server.tags_updated", auth.Email(c), serverID, "", - fmt.Sprintf("tags %v -> %v", before.Tags, body.Tags)) - c.JSON(http.StatusOK, gin.H{"tags": body.Tags}) -} -``` - -Add `"errors"` and `"fmt"` to the file's imports if absent. If the actor helper in this file is named something other than `auth.Email(c)`, use whatever the neighbouring handlers use — grep one existing `services.LogEvent(` call in this file and copy its actor argument exactly. - -- [ ] **Step 4: Verify it builds** - -Run: `cd server && go build ./... && go vet ./internal/api/` -Expected: no output. - -- [ ] **Step 5: Manual smoke test** - -Run the stack (`cd deploy && docker compose up -d`), sign in, then: - -```bash -curl -b cookies.txt -X PUT localhost:8080/api/servers//tags \ - -H 'content-type: application/json' -d '{"tags":{"env":"prod","role":"web"}}' -curl -b cookies.txt 'localhost:8080/api/servers?tag=env:prod' -curl -b cookies.txt localhost:8080/api/servers/tags -curl -b cookies.txt -X PUT localhost:8080/api/servers//tags \ - -H 'content-type: application/json' -d '{"tags":{"ENV":"prod"}}' # expect 400 -``` - -Expected: the first three succeed, the fourth is 400 with an `invalid tag` message, and `/api/audit` shows a `server.tags_updated` event. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/api/handlers.go -git commit -m "feat: tag endpoints for servers" -``` - ---- - -### Task 6: Workflow tag targeting - -**Files:** -- Modify: `server/internal/models/workflow.go:47-56` -- Modify: `server/internal/services/workflows.go:262-289` -- Modify: `server/internal/services/workflow_runner.go:20-69` - -**Interfaces:** -- Consumes: `ResolveTargets`, `ErrNoTargets`, `ValidateTags`. -- Produces: `models.Workflow.TargetTags map[string]string`. - -- [ ] **Step 1: Add the model field** - -In `server/internal/models/workflow.go`, inside `type Workflow struct`, after `TargetServerIDs`: - -```go - TargetTags map[string]string `bson:"target_tags,omitempty" json:"target_tags,omitempty"` -``` - -- [ ] **Step 2: Validate it on create and update** - -In `server/internal/services/workflows.go`, inside both `CreateWorkflow` and `UpdateWorkflow`, before the existing `validateTargetServers` call: - -```go - if err := ValidateTags(w.TargetTags); err != nil { - return err // CreateWorkflow returns (nil, err) - } -``` - -Make sure `UpdateWorkflow` also persists the field — if it builds an explicit `$set` document, add `"target_tags": w.TargetTags` to it. - -- [ ] **Step 3: Resolve targets in TriggerWorkflow** - -In `server/internal/services/workflow_runner.go`, replace lines 25-34 (the `len(wf.TargetServerIDs) == 0` guard and the `validateTargetServers` call) with: - -```go - targets, err := ResolveTargets(instanceID, wf.TargetServerIDs, wf.TargetTags) - if err != nil { - return "", err - } - if len(wf.Steps) == 0 { - return "", fmt.Errorf("workflow has no steps") - } -``` - -Then replace the `ServerRuns` construction at lines 57-69 with: - -```go - ServerRuns: make([]models.ServerRun, 0, len(targets)), - } - for _, srv := range targets { - sr := models.ServerRun{ServerID: srv.ServerID, Hostname: srv.Hostname, Status: "queued", RunEnv: map[string]string{}} - for _, rs := range resolved { - sr.Steps = append(sr.Steps, models.StepRun{Order: rs.Order, Name: rs.Name, Status: "queued", OutputEnv: map[string]string{}}) - } - run.ServerRuns = append(run.ServerRuns, sr) - } -``` - -This also removes the per-server `getServerByID` lookup — `ResolveTargets` already returned whole `models.Server` values, so the hostname is in hand. - -- [ ] **Step 4: Map ErrNoTargets to 400 in the API** - -In `server/internal/api/workflows.go`, in the handler for `POST /workflows/:id/run`, before the generic 500: - -```go - if errors.Is(err, services.ErrNoTargets) { - c.JSON(http.StatusBadRequest, gin.H{"error": "this workflow matches no servers"}) - return - } -``` - -Add `"errors"` to the imports if absent. - -- [ ] **Step 5: Verify it builds and tests pass** - -Run: `cd server && go build ./... && go test ./...` -Expected: build silent, tests PASS. If `validateTargetServers` is now unused, delete it rather than leaving dead code. - -- [ ] **Step 6: Manual smoke test** - -Tag two servers `env:test`, set a workflow's `target_tags` to `{"env":"test"}` with an empty server list, run it, and confirm the run's `server_runs` holds exactly those two. Then clear both the list and the tags and confirm the run endpoint answers 400. - -- [ ] **Step 7: Commit** - -```bash -git add server/internal/models/workflow.go server/internal/services/workflows.go server/internal/services/workflow_runner.go server/internal/api/workflows.go -git commit -m "feat: target workflow runs by tag selector" -``` - ---- - -### Task 7: Frontend API client - -**Files:** -- Modify: `web/lib/api.ts` — `Server` interface near line 24, `Workflow` interface, and the `api` object near line 542. - -**Interfaces:** -- Consumes: the routes from Tasks 5 and 6. -- Produces: `Server.tags`, `Workflow.target_tags`, `api.setServerTags`, `api.listKnownTags`, `api.listServers(tags?)`. - -- [ ] **Step 1: Add the types** - -In the `Server` interface add: - -```ts - tags?: Record; -``` - -In the `Workflow` and `WorkflowInput` interfaces add: - -```ts - target_tags?: Record; -``` - -- [ ] **Step 2: Add the client methods** - -Replace the existing `listServers` in the `api` object and add two methods beside it: - -```ts - listServers(tags?: Record): Promise { - const params = Object.entries(tags ?? {}).map(([k, v]) => `tag=${encodeURIComponent(`${k}:${v}`)}`); - return request(`/servers${params.length ? `?${params.join("&")}` : ""}`); - }, - - listKnownTags(): Promise> { - return request>("/servers/tags"); - }, - - setServerTags(serverId: string, tags: Record): Promise<{ tags: Record }> { - return request<{ tags: Record }>(`/servers/${serverId}/tags`, { - method: "PUT", - body: JSON.stringify({ tags }), - }); - }, -``` - -- [ ] **Step 3: Verify it typechecks** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json` -Expected: no output. Any call site of `api.listServers()` still compiles because the argument is optional. - -- [ ] **Step 4: Commit** - -```bash -git add web/lib/api.ts -git commit -m "feat: tag methods on the web api client" -``` - ---- - -### Task 8: Tag chips and editor - -**Files:** -- Create: `web/components/servers/TagChips.tsx` -- Modify: `web/app/(app)/servers/[id]/page.tsx` — header block. - -**Interfaces:** -- Consumes: `api.setServerTags`, `api.listKnownTags`, `Server.tags`. -- Produces: `} editable={boolean} />`. - -- [ ] **Step 1: Write the component** - -Create `web/components/servers/TagChips.tsx`: - -```tsx -"use client"; - -import { useState } from "react"; -import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; -import { api } from "@/lib/api"; -import { Button } from "@/components/ui"; - -/* - * A tag is key:value, so the chip shows both halves with the key dimmed — the - * value is the part people scan for, the key is what disambiguates it. - */ - -export function TagChips({ serverId, tags, editable = false }: { serverId: string; tags?: Record; editable?: boolean }) { - const queryClient = useQueryClient(); - const [editing, setEditing] = useState(false); - const [draft, setDraft] = useState<[string, string][]>(Object.entries(tags ?? {})); - const [error, setError] = useState(null); - - const { data: known } = useQuery({ - queryKey: ["server-tags"], - queryFn: () => api.listKnownTags(), - enabled: editing, - staleTime: 60_000, - }); - - const { mutate: save, isPending } = useMutation({ - mutationFn: () => api.setServerTags(serverId, Object.fromEntries(draft.filter(([k, v]) => k && v))), - onSuccess: () => { - queryClient.invalidateQueries({ queryKey: ["servers"] }); - queryClient.invalidateQueries({ queryKey: ["server-tags"] }); - setEditing(false); - setError(null); - }, - onError: (e: Error) => setError(e.message), - }); - - const entries = Object.entries(tags ?? {}); - - if (!editing) { - return ( -
- {entries.length === 0 && No tags} - {entries.map(([k, v]) => ( - - {k}: - {v} - - ))} - {editable && ( - - )} -
- ); - } - - return ( -
- {Object.keys(known ?? {}).map((k) => - -
- {draft.map(([k, v], i) => ( -
- setDraft((d) => d.map((row, j) => (j === i ? [e.target.value, row[1]] : row)))} - placeholder="env" - className="w-32 rounded-lg border border-border bg-surface-2 px-2 py-1 font-mono text-xs text-text-primary focus:border-accent/50 focus:outline-none" - /> - : - setDraft((d) => d.map((row, j) => (j === i ? [row[0], e.target.value] : row)))} - placeholder="prod" - className="w-40 rounded-lg border border-border bg-surface-2 px-2 py-1 font-mono text-xs text-text-primary focus:border-accent/50 focus:outline-none" - /> - -
- ))} -
- - {draft.length < 20 && ( - - )} - - {error &&

{error}

} - -
- - -
-
- ); -} -``` - -- [ ] **Step 2: Mount it on the server detail header** - -In `web/app/(app)/servers/[id]/page.tsx`, import it and render it directly under the hostname heading: - -```tsx - -``` - -- [ ] **Step 3: Verify it builds** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json && npx next build` -Expected: `✓ Compiled successfully`. - -- [ ] **Step 4: Manual check** - -Open a server, add `env:prod`, save, reload — the chip persists. Try `ENV:prod` and confirm the inline error names the rule rather than showing a raw 400. - -- [ ] **Step 5: Commit** - -```bash -git add web/components/servers/TagChips.tsx "web/app/(app)/servers/[id]/page.tsx" -git commit -m "feat: view and edit server tags" -``` - ---- - -### Task 9: Fleet filter bar - -**Files:** -- Create: `web/components/servers/TagFilterBar.tsx` -- Modify: `web/app/(app)/servers/page.tsx` - -**Interfaces:** -- Consumes: `api.listKnownTags`, `api.listServers(tags)`. -- Produces: `} onChange={(v) => void} />`. - -- [ ] **Step 1: Write the component** - -Create `web/components/servers/TagFilterBar.tsx`: - -```tsx -"use client"; - -import { useQuery } from "@tanstack/react-query"; -import { api } from "@/lib/api"; - -export function TagFilterBar({ value, onChange }: { value: Record; onChange: (v: Record) => void }) { - const { data: known } = useQuery({ queryKey: ["server-tags"], queryFn: () => api.listKnownTags(), staleTime: 60_000 }); - - const keys = Object.keys(known ?? {}).sort(); - if (keys.length === 0) return null; - - const active = Object.entries(value); - - return ( -
- {keys.map((k) => ( - - ))} - {active.length > 0 && ( - - )} -
- ); -} -``` - -- [ ] **Step 2: Drive the list from the URL** - -In `web/app/(app)/servers/page.tsx`, read the filter from `useSearchParams`, write it back with `router.replace`, and pass it to the query so a filtered fleet view is a shareable URL: - -```tsx -const searchParams = useSearchParams(); -const router = useRouter(); - -const selected = Object.fromEntries( - searchParams.getAll("tag").map((t) => t.split(":")).filter((p) => p.length === 2), -) as Record; - -function setSelected(next: Record) { - const qs = Object.entries(next).map(([k, v]) => `tag=${encodeURIComponent(`${k}:${v}`)}`).join("&"); - router.replace(qs ? `/servers?${qs}` : "/servers"); -} - -const { data: servers, isLoading } = useQuery({ - queryKey: ["servers", selected], - queryFn: () => api.listServers(selected), -}); -``` - -Render `` above the list, and add a tags cell to each row using the read-only form: ``. - -A page reading `useSearchParams` must be inside a `` boundary in the App Router. If the build complains, wrap the page body in one. - -- [ ] **Step 3: Verify it builds** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json && npx next build` -Expected: `✓ Compiled successfully`. - -- [ ] **Step 4: Manual check** - -Filter to `env: prod`, copy the URL, open it in a new tab, and confirm the filter is still applied. - -- [ ] **Step 5: Commit** - -```bash -git add web/components/servers/TagFilterBar.tsx "web/app/(app)/servers/page.tsx" -git commit -m "feat: filter the fleet list by tag" -``` - ---- - -### Task 10: Workflow target section - -**Files:** -- Modify: `web/app/(app)/workflows/[id]/page.tsx` - -**Interfaces:** -- Consumes: `Workflow.target_tags`, `api.listServers()`, `api.listKnownTags()`. -- Produces: no new exports. - -- [ ] **Step 1: Add the tag selector beside the server picker** - -In the workflow's target editor, add a key/value row editor writing to `target_tags`, using the same two-input pattern as `TagChips` (repeat it locally rather than extracting — the editing model differs: this one has no save button of its own, it feeds the workflow's own save). - -- [ ] **Step 2: Add the resolved readout** - -Compute the union client-side from the server list already fetched — no new endpoint, because the browser holds the whole fleet already: - -```tsx -const matched = (servers ?? []).filter( - (s) => - targetServerIds.includes(s.server_id) || - (Object.keys(targetTags).length > 0 && Object.entries(targetTags).every(([k, v]) => s.tags?.[k] === v)), -); -``` - -Render it as the readout the design promised: - -```tsx -

s.hostname).join("\n")}> - Runs on {matched.length} {matched.length === 1 ? "server" : "servers"} -

-{matched.length === 0 && ( -

This workflow matches no servers and cannot run.

-)} -``` - -The filter above must stay identical in meaning to `UnionTargets` and `MatchesTags` in Go — empty selector matches nothing, AND across keys. If one changes, change both. - -- [ ] **Step 3: Verify it builds** - -Run: `cd web && npx tsc --noEmit -p tsconfig.json && npx next build` -Expected: `✓ Compiled successfully`. - -- [ ] **Step 4: Manual check** - -Set a tag selector with no explicit servers, confirm the readout counts correctly and that saving then running dispatches to exactly those machines. - -- [ ] **Step 5: Commit** - -```bash -git add "web/app/(app)/workflows/[id]/page.tsx" -git commit -m "feat: target workflows by tag in the designer" -``` - ---- - -### Task 11: Documentation - -**Files:** -- Modify: `docsite/docs/vantage/servers.md`, `docsite/docs/vantage/workflows.md`, `CLAUDE.md` - -- [ ] **Step 1: Document tags for users** - -Add a "Tags" section to `docsite/docs/vantage/servers.md` covering: what a tag is (`key:value`), the character and count rules stated plainly, how to filter the fleet, and that the filter is in the URL. Add a "Targeting" section to `docsite/docs/vantage/workflows.md` covering the union rule, that an empty selector matches nothing, and that offline servers are still targeted and will fail visibly rather than being skipped. - -- [ ] **Step 2: Update the contributor map** - -In `CLAUDE.md`, under the SSH keys / Workflows subsystem notes, add a short paragraph: tags are a map on `servers` with no registry collection, `ResolveTargets` is the single answer to which servers a workflow touches, the union deduplicates, an empty selector matches nothing on purpose, and the wildcard index exists because the queried key is user-chosen. - -- [ ] **Step 3: Commit** - -```bash -git add docsite/docs/vantage/servers.md docsite/docs/vantage/workflows.md CLAUDE.md -git commit -m "docs: server tags and workflow tag targeting" -``` - ---- - -## Verification - -Full check before calling this done: - -```bash -cd server && go build ./... && go vet ./... && go test ./... -cd ../web && npx tsc --noEmit -p tsconfig.json && npx next build -cd .. && graphify update . -``` - -All must pass. `go test ./...` should report the tag and target suites passing, and no package should report a build failure. diff --git a/docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md b/docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md deleted file mode 100644 index b0c41e4..0000000 --- a/docs/superpowers/plans/2026-08-06-package-inventory-and-cve-findings.md +++ /dev/null @@ -1,2588 +0,0 @@ -# Package Inventory and CVE Findings Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Agents report the packages installed on each Linux server; the control plane matches them against distribution security feeds (`trivy-db`) and raises findings that link to the existing `ApplyUpdatesCmd` patching path. - -**Architecture:** The agent collects packages on its existing hourly loop and offers a hash first, sending the ~150KB list only when the server says it differs. `ReportPackages` does **not** match — it upserts the list and sets `scan_pending: true`. A new `vulnsched` loop, running inside the existing `bus.RunAsLeader("housekeeping", …)`, pulls `trivy-db` to an ephemeral directory, scans pending servers, diffs findings and emits one batched digest per tick. Matching logic lives in `server/internal/vulndb` as pure functions behind a small interface. - -**Tech Stack:** Go 1.26, gin, mongo-driver v2, gRPC, bbolt, Next.js 16 + TanStack Query, Tailwind 3. - -**Spec:** `docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md`. Read it before starting. Where this plan and the spec disagree, the spec wins and the plan is wrong. - -## Global Constraints - -- **Do not write tests.** No `*_test.go`, no npm test files, no test scaffolding. This is a deliberate instruction from the repository owner, not an oversight. Verification in this plan is `go build`, `go vet`, `npm run build` and manual checks. -- **Linux only.** Windows agents must not collect packages and must not appear in findings. Do not add an MSRC source, a `Get-HotFix` collector or KB supersedence logic — that is a separate spec. -- Every service query is scoped by `instance_id`. No unscoped lookups, ever. -- Every mutating API path writes an audit event via `services.LogEvent(instanceID, eventType, actor, serverID, keyID, details string)`. -- Licence features are read with `lic.HasFeature("vuln_scanning")`. **Never switch on `Tier`.** -- Module path prefix is `gitea.hostxtra.co.uk/mrhid6/vantage/`. -- Mongo driver is **v2**: `go.mongodb.org/mongo-driver/v2/bson`, and ObjectIDs are `bson.ObjectID`, not `primitive.ObjectID`. The spec's struct listings use `primitive.` — that is spec shorthand; **write `bson.ObjectID`**. -- Collections are reached via `db.Col("name")`. -- No component in `web/` may carry a hex colour; use Tailwind token names only. `web/` is dark-only — do not add a light theme. -- Commit messages use the `feat:` / `fix:` / `docs:` prefixes already in `git log`. -- Severity strings are lowercase and fixed: `critical`, `high`, `medium`, `low`, `unknown`. -- Finding states are exactly `open`, `fixed`, `accepted`. - -## Correctness risks carried without tests - -These are the places where a mistake produces a **wrong answer rather than a crash**, so nothing will surface them automatically. Read the code twice at each: - -1. **Version comparison (Task 1).** `dpkg` ordering has epochs and sorts `~` before the empty string, so `3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own segment rules. Any fallback to string comparison orders `1.10` before `1.9`. Every one of those mistakes reports a vulnerable fleet as clean. -2. **The backport case (Task 11).** Installed `1:3.0.2-0ubuntu1.15` against advisory fixed-in `1:3.0.2-0ubuntu1.15` must resolve to **not vulnerable**. Equal must not compare as less-than. -3. **`first_seen` preservation (Task 12).** An upsert that overwrites it makes every finding look discovered today. -4. **Fixed-before-reopen ordering (Task 12).** A finding both absent from the scan and past its acceptance expiry must settle `fixed`, not reopen. -5. **Log/line caps** do not apply here, but the **200k-row equivalent** does: `server_packages` is one document per server and must stay one, or the 16MB limit becomes reachable. - -Manual verification for 1 and 2 is described inline in those tasks. - -## Dependencies to add - -Run from the repo root; `go.work` covers all modules. - -```bash -cd server -go get github.com/aquasecurity/trivy-db@latest -go get github.com/knqyf263/go-deb-version@latest -go get github.com/knqyf263/go-rpm-version@latest -go get github.com/knqyf263/go-apk-version@latest -go get oras.land/oras-go/v2@latest -go get go.etcd.io/bbolt@latest -``` - -`oras-go` pulls the OCI artifact. `trivy-db` provides the BoltDB schema reader. The three version modules are small and each does one job. - ---- - -## File Structure - -**Create — agent:** -- `agent/internal/packages/packages.go` — `Collect()` dispatch by package manager -- `agent/internal/packages/parse.go` — pure output parsers, one per manager -- `agent/internal/packages/osrelease.go` — `/etc/os-release` parsing - -**Create — server:** -- `server/internal/models/packages.go` — `ServerPackages`, `InstalledPackage`, `OSRelease` -- `server/internal/models/vuln.go` — `VulnFinding`, `Acceptance`, `VulnDBMeta`, `VulnAlertRule` -- `server/internal/vulndb/version.go` — comparator dispatch by OS family -- `server/internal/vulndb/ecosystem.go` — OS family/version → trivy-db bucket, or unsupported -- `server/internal/vulndb/pull.go` — OCI fetch to temp dir -- `server/internal/vulndb/db.go` — advisory lookup over the BoltDB -- `server/internal/vulndb/match.go` — pure matching -- `server/internal/services/packages.go` — store, hash compare, package search -- `server/internal/services/findings.go` — finding diff/state machine + acceptance -- `server/internal/services/vulnrules.go` — alert rule CRUD, digest dispatch -- `server/internal/services/vulnindexes.go` — index builder -- `server/internal/vulnsched/sched.go` — the leader-owned loop -- `server/internal/api/vulnerabilities.go` — REST handlers -- `shared/mail/vuln.go` — `SendVulnDigest` -- `shared/mail/templates/vuln_digest.html.tmpl`, `vuln_digest.txt.tmpl` - -**Create — web:** -- `web/app/(app)/vulnerabilities/page.tsx` — fleet board grouped by CVE -- `web/components/vulnerabilities/FindingRow.tsx` — one CVE, expandable -- `web/components/vulnerabilities/AcceptDialog.tsx` — reason + until -- `web/components/vulnerabilities/DBFreshness.tsx` — `pulled_at` age banner -- `web/components/settings/VulnAlertRulesCard.tsx` - -**Modify:** -- `proto/vantage/v1/vantage.proto` — `ReportPackages` RPC and messages; `SyncResponse.collect_packages` -- `agent/internal/sync/sync.go:370-411` — `runUpdateCheck` also reports packages -- `server/internal/grpc/server.go` — `ReportPackages` handler; `SyncKeys` sets the flag -- `server/cmd/main.go:184-206` — start `vulnsched` and the sweeper inside `RunAsLeader` -- `server/cmd/main.go` (schema setup, near the other `Ensure*Indexes` calls) — `EnsureVulnIndexes` -- `server/internal/api/handlers.go` — register routes -- `shared/models/settings.go` — `VulnFindingRetentionDays *int` -- `admin/internal/models/entitlements.go` — `vuln_scanning` toggle -- `web/lib/api.ts` — types and client methods -- `web/components/Sidebar.tsx` — Vulnerabilities entry -- `web/app/(app)/servers/[id]/page.tsx` — Vulnerabilities and Packages tabs -- `web/app/(app)/settings/notifications/page.tsx` — alert rules card -- The control plane's instance-deletion collection list — add both new collections -- `CLAUDE.md`, `docsite/docs/vantage/` — document the subsystem - ---- - -### Task 1: Version comparators - -The most correctness-critical code in the feature. Its failure mode is silent: a wrong comparison reports a vulnerable fleet as clean. - -**Files:** -- Create: `server/internal/vulndb/version.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: `vulndb.LessThan(family, a, b string) (bool, error)` and `vulndb.ErrUnsupportedFamily`. - -- [ ] **Step 1: Add the dependencies** - -```bash -cd server -go get github.com/knqyf263/go-deb-version@latest -go get github.com/knqyf263/go-rpm-version@latest -go get github.com/knqyf263/go-apk-version@latest -``` - -- [ ] **Step 2: Write the implementation** - -Create `server/internal/vulndb/version.go`: - -```go -// Package vulndb matches installed packages against distribution security -// advisories. -// -// Version comparison is bought rather than written. Distribution version -// ordering is subtle in ways that are invisible until they are wrong: dpkg has -// epochs and sorts "~" before the empty string, rpmvercmp has its own segment -// rules and treats "~" and "^" differently again, and any ordering that falls -// back on string comparison puts 1.10 before 1.9. Every one of those mistakes -// produces a false negative — a vulnerable host reported clean — which is the -// failure nobody notices. -package vulndb - -import ( - "errors" - "fmt" - - apk "github.com/knqyf263/go-apk-version" - deb "github.com/knqyf263/go-deb-version" - rpm "github.com/knqyf263/go-rpm-version" -) - -// ErrUnsupportedFamily means we hold no comparator for this distribution, and -// therefore cannot answer whether it is vulnerable. Callers must surface this -// as "unsupported" and must never treat it as "not vulnerable". -var ErrUnsupportedFamily = errors.New("unsupported OS family") - -// LessThan reports whether version a sorts before version b under the ordering -// rules of the given OS family. -func LessThan(family, a, b string) (bool, error) { - switch family { - case "debian", "ubuntu": - if a == "" || b == "" { - return false, fmt.Errorf("empty deb version (a=%q b=%q)", a, b) - } - va, err := deb.NewVersion(a) - if err != nil { - return false, fmt.Errorf("parse deb version %q: %w", a, err) - } - vb, err := deb.NewVersion(b) - if err != nil { - return false, fmt.Errorf("parse deb version %q: %w", b, err) - } - return va.LessThan(vb), nil - - case "redhat", "centos", "rocky", "alma", "amazon", "oracle", "suse", "opensuse", "sles": - // go-rpm-version does not error; rpmvercmp is defined over arbitrary - // strings. Guard empties so a missing version cannot read as equal. - if a == "" || b == "" { - return false, fmt.Errorf("empty rpm version (a=%q b=%q)", a, b) - } - return rpm.NewVersion(a).LessThan(rpm.NewVersion(b)), nil - - case "alpine": - if a == "" || b == "" { - return false, fmt.Errorf("empty apk version (a=%q b=%q)", a, b) - } - va, err := apk.NewVersion(a) - if err != nil { - return false, fmt.Errorf("parse apk version %q: %w", a, err) - } - vb, err := apk.NewVersion(b) - if err != nil { - return false, fmt.Errorf("parse apk version %q: %w", b, err) - } - return va.LessThan(vb), nil - - default: - return false, fmt.Errorf("%w: %s", ErrUnsupportedFamily, family) - } -} -``` - -- [ ] **Step 3: Verify the ordering by hand** - -This replaces the test that would normally guard it. Write a scratch `main.go` outside the repo (or use `go run` on a temp file), call `LessThan` with each pair below, and confirm every result. **Do not skip this** — these are the exact cases that go wrong silently. - -| family | a | b | expected | -| ------ | - | - | -------- | -| ubuntu | `1:3.0.2-0ubuntu1.15` | `1:3.0.2-0ubuntu1.15` | `false` — the backport case; equal is not less-than | -| ubuntu | `1:3.0.2-0ubuntu1.14` | `1:3.0.2-0ubuntu1.15` | `true` | -| ubuntu | `1:3.0.2-0ubuntu1.16` | `1:3.0.2-0ubuntu1.15` | `false` | -| debian | `1.0~rc1` | `1.0` | `true` — tilde sorts before empty | -| debian | `1.0` | `1.0~rc1` | `false` | -| debian | `2.0` | `1:1.0` | `true` — epoch dominates | -| debian | `1.9` | `1.10` | `true` — numeric, not lexical | -| redhat | `1.2.3-4.el9` | `1.2.3-4.el9` | `false` | -| redhat | `1.2.3-3.el9` | `1.2.3-4.el9` | `true` | -| redhat | `2:1.0-1` | `1:9.0-1` | `false` | -| rocky | `1.9-1` | `1.10-1` | `true` | -| alpine | `1.2.3-r0` | `1.2.3-r0` | `false` | -| alpine | `1.2.3-r0` | `1.2.3-r1` | `true` | -| alpine | `1.9-r0` | `1.10-r0` | `true` | -| arch | `1.0` | `2.0` | error wrapping `ErrUnsupportedFamily` | - -Delete the scratch file afterwards. If any row disagrees, the bug is in this file or in the chosen library — resolve it before continuing, because everything downstream inherits it. - -- [ ] **Step 4: Build** - -Run: `cd server && go build ./... && go vet ./...` -Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/vulndb/version.go server/go.mod server/go.sum -git commit -m "feat: version comparators for distro package ordering" -``` - ---- - -### Task 2: Ecosystem mapping - -**Files:** -- Create: `server/internal/vulndb/ecosystem.go` - -**Interfaces:** -- Consumes: `vulndb.ErrUnsupportedFamily` from Task 1. -- Produces: `vulndb.Bucket(family, versionID string) (string, error)`. - -- [ ] **Step 1: Write the implementation** - -Create `server/internal/vulndb/ecosystem.go`: - -```go -package vulndb - -import ( - "fmt" - "strings" -) - -// rhelRebuilds share Red Hat's advisory feed rather than publishing their own. -var rhelRebuilds = map[string]bool{ - "redhat": true, "centos": true, "rocky": true, "alma": true, "oracle": true, -} - -// Bucket maps an OS family and version onto the trivy-db bucket that holds its -// advisories. -// -// It returns ErrUnsupportedFamily rather than a best guess when we have no -// feed. A scan that cannot be performed must say so; reporting zero findings -// for a distribution we do not cover is indistinguishable from reporting a -// clean host, and one of those is a lie. -func Bucket(family, versionID string) (string, error) { - family = strings.ToLower(strings.TrimSpace(family)) - versionID = strings.TrimSpace(versionID) - - switch { - case family == "debian" || family == "ubuntu": - if versionID == "" { - return "", fmt.Errorf("%s requires a version id", family) - } - return family + " " + versionID, nil - - case family == "alpine": - if versionID == "" { - return "", fmt.Errorf("alpine requires a version id") - } - return "alpine " + majorMinor(versionID), nil - - case rhelRebuilds[family]: - if versionID == "" { - return "", fmt.Errorf("%s requires a version id", family) - } - return "redhat " + major(versionID), nil - - default: - return "", fmt.Errorf("%w: %s", ErrUnsupportedFamily, family) - } -} - -func major(v string) string { - if i := strings.Index(v, "."); i != -1 { - return v[:i] - } - return v -} - -func majorMinor(v string) string { - parts := strings.Split(v, ".") - if len(parts) >= 2 { - return parts[0] + "." + parts[1] - } - return v -} -``` - -Expected behaviour, for reference while reading it back: `("ubuntu","22.04")` → `ubuntu 22.04`; `("debian","12")` → `debian 12`; `("alpine","3.19.1")` → `alpine 3.19`; `("rocky","9.3")` → `redhat 9`; `("arch","")` → `ErrUnsupportedFamily`; `("ubuntu","")` → error, because Ubuntu 22.04 and 24.04 publish different fixed versions for the same CVE. - -- [ ] **Step 2: Build** - -Run: `cd server && go build ./...` -Expected: exit 0. - -- [ ] **Step 3: Commit** - -```bash -git add server/internal/vulndb/ecosystem.go -git commit -m "feat: map OS family and version to trivy-db advisory buckets" -``` - ---- - -### Task 3: Agent-side OS release detection - -**Files:** -- Create: `agent/internal/packages/osrelease.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: `packages.OSRelease{Family, VersionID, Arch string}`, `packages.ParseOSRelease(r io.Reader) (OSRelease, error)`, `packages.DetectOS() (OSRelease, error)`. - -- [ ] **Step 1: Write the implementation** - -Create `agent/internal/packages/osrelease.go`: - -```go -package packages - -import ( - "bufio" - "errors" - "io" - "os" - "runtime" - "strings" -) - -// OSRelease identifies the distribution well enough to select an advisory -// feed. VersionID is not optional: Ubuntu 22.04 and 24.04 publish different -// fixed versions for the same CVE. -type OSRelease struct { - Family string - VersionID string - Arch string -} - -// ParseOSRelease reads the os-release format: KEY=value, one per line, with -// values optionally quoted, and # comments. -func ParseOSRelease(r io.Reader) (OSRelease, error) { - out := OSRelease{Arch: runtime.GOARCH} - sc := bufio.NewScanner(r) - for sc.Scan() { - line := strings.TrimSpace(sc.Text()) - if line == "" || strings.HasPrefix(line, "#") { - continue - } - key, val, ok := strings.Cut(line, "=") - if !ok { - continue - } - val = strings.Trim(strings.TrimSpace(val), `"'`) - switch strings.TrimSpace(key) { - case "ID": - out.Family = strings.ToLower(val) - case "VERSION_ID": - out.VersionID = val - } - } - if err := sc.Err(); err != nil { - return OSRelease{}, err - } - if out.Family == "" { - return OSRelease{}, errors.New("os-release has no ID") - } - return out, nil -} - -// DetectOS reads /etc/os-release. -func DetectOS() (OSRelease, error) { - f, err := os.Open("/etc/os-release") - if err != nil { - return OSRelease{}, err - } - defer f.Close() - return ParseOSRelease(f) -} -``` - -Note the quote stripping handles both `ID=ubuntu` and `ID="rocky"`, which real distributions both emit. - -- [ ] **Step 2: Build** - -Run: `cd agent && go build ./...` -Expected: exit 0. - -- [ ] **Step 3: Commit** - -```bash -git add agent/internal/packages/osrelease.go -git commit -m "feat: agent parses /etc/os-release for distro identification" -``` - ---- - -### Task 4: Agent-side package collection - -**Files:** -- Create: `agent/internal/packages/parse.go`, `agent/internal/packages/packages.go` - -**Interfaces:** -- Consumes: `packages.OSRelease` from Task 3. -- Produces: `packages.Package{Name, Version, Arch, SourceName string; Epoch int}`, `packages.Collect() (OSRelease, []Package, error)`, `packages.Hash([]Package) string`. - -**Why `SourceName` matters:** Debian and Ubuntu advisories are keyed on the **source** package. A CVE against `openssl` covers the binaries `libssl3`, `openssl` and `libssl-dev`; matching on binary name alone misses two of the three. - -- [ ] **Step 1: Write the parsers** - -Create `agent/internal/packages/parse.go`: - -```go -package packages - -import ( - "crypto/sha256" - "encoding/hex" - "sort" - "strconv" - "strings" -) - -// Package is one installed package as the distribution reports it. Version is -// the distribution's own version string, verbatim — never normalised, because -// the advisory feeds are keyed on exactly this form. -type Package struct { - Name string - Version string - Epoch int - Arch string - SourceName string -} - -// ParseDpkg reads tab-separated output of -// dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${source:Package}\n' -func ParseDpkg(out string) []Package { - var pkgs []Package - for _, line := range strings.Split(out, "\n") { - if strings.TrimSpace(line) == "" { - continue - } - f := strings.Split(line, "\t") - if len(f) < 3 { - continue - } - p := Package{Name: f[0], Version: f[1], Arch: f[2]} - if len(f) > 3 && f[3] != "" { - p.SourceName = f[3] - } else { - p.SourceName = p.Name - } - pkgs = append(pkgs, p) - } - return pkgs -} - -// ParseRPM reads tab-separated output of -// rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n' -func ParseRPM(out string) []Package { - var pkgs []Package - for _, line := range strings.Split(out, "\n") { - if strings.TrimSpace(line) == "" { - continue - } - f := strings.Split(line, "\t") - if len(f) < 4 { - continue - } - epoch := 0 - // rpm prints "(none)" rather than omitting the field when there is no - // epoch, and that must become 0 rather than failing the line. - if f[1] != "" && f[1] != "(none)" { - if n, err := strconv.Atoi(f[1]); err == nil { - epoch = n - } - } - p := Package{Name: f[0], Epoch: epoch, Version: f[2], Arch: f[3]} - if len(f) > 4 { - p.SourceName = srcRPMName(f[4]) - } - if p.SourceName == "" { - p.SourceName = p.Name - } - pkgs = append(pkgs, p) - } - return pkgs -} - -// srcRPMName reduces "openssl-3.0.7-24.el9.src.rpm" to "openssl" by dropping -// the trailing ".src.rpm" and then the version and release segments, which are -// the last two hyphen-separated fields. -func srcRPMName(s string) string { - s = strings.TrimSuffix(s, ".src.rpm") - parts := strings.Split(s, "-") - if len(parts) <= 2 { - return s - } - return strings.Join(parts[:len(parts)-2], "-") -} - -// ParseAPK reads "apk info -v" output: one "name-version-rREV" per line. Alpine -// has no separate source package, so SourceName mirrors Name. -func ParseAPK(out string) []Package { - var pkgs []Package - for _, line := range strings.Split(out, "\n") { - line = strings.TrimSpace(line) - if line == "" { - continue - } - name, version := splitAPK(line) - if name == "" { - continue - } - pkgs = append(pkgs, Package{Name: name, Version: version, SourceName: name}) - } - return pkgs -} - -// splitAPK finds the version boundary from the right. The version is always the -// last two hyphen-separated fields ("-r"), which is reliable where -// scanning from the left is not: names legitimately contain digits, so "musl" in -// "musl-1.2.4_git20230717-r4" cannot be found by looking for the first digit. -func splitAPK(s string) (name, version string) { - last := strings.LastIndex(s, "-") - if last <= 0 { - return "", "" - } - prev := strings.LastIndex(s[:last], "-") - if prev <= 0 { - return "", "" - } - return s[:prev], s[prev+1:] -} - -// Hash fingerprints a package set so an unchanged set never has to be sent. -// It sorts first: the ordering of dpkg or rpm output is not guaranteed stable, -// and an ordering-sensitive hash would resend the full list every hour. -func Hash(pkgs []Package) string { - lines := make([]string, 0, len(pkgs)) - for _, p := range pkgs { - lines = append(lines, p.Name+"\x00"+strconv.Itoa(p.Epoch)+"\x00"+p.Version+"\x00"+p.Arch) - } - sort.Strings(lines) - h := sha256.New() - for _, l := range lines { - h.Write([]byte(l)) - h.Write([]byte("\n")) - } - return hex.EncodeToString(h.Sum(nil)) -} -``` - -- [ ] **Step 2: Write the collector** - -Create `agent/internal/packages/packages.go`: - -```go -package packages - -import ( - "context" - "fmt" - "os/exec" - "runtime" - "time" -) - -const collectTimeout = 2 * time.Minute - -// Collect enumerates installed packages. Linux only: Windows agents are -// second-class by design and vulnerability scanning needs a different source, -// a different collector and a different matcher, all of which are out of scope. -func Collect() (OSRelease, []Package, error) { - if runtime.GOOS != "linux" { - return OSRelease{}, nil, fmt.Errorf("package collection is linux-only, got %s", runtime.GOOS) - } - - osrel, err := DetectOS() - if err != nil { - return OSRelease{}, nil, fmt.Errorf("detect os: %w", err) - } - - ctx, cancel := context.WithTimeout(context.Background(), collectTimeout) - defer cancel() - - switch { - case have("dpkg-query"): - out, err := run(ctx, "dpkg-query", "-W", "-f", - `${Package}\t${Version}\t${Architecture}\t${source:Package}\n`) - if err != nil { - return osrel, nil, err - } - return osrel, ParseDpkg(out), nil - - case have("rpm"): - out, err := run(ctx, "rpm", "-qa", "--qf", - `%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n`) - if err != nil { - return osrel, nil, err - } - return osrel, ParseRPM(out), nil - - case have("apk"): - out, err := run(ctx, "apk", "info", "-v") - if err != nil { - return osrel, nil, err - } - return osrel, ParseAPK(out), nil - - default: - return osrel, nil, fmt.Errorf("no supported package manager found") - } -} - -func have(bin string) bool { - _, err := exec.LookPath(bin) - return err == nil -} - -func run(ctx context.Context, name string, args ...string) (string, error) { - out, err := exec.CommandContext(ctx, name, args...).Output() - if err != nil { - return "", fmt.Errorf("%s: %w", name, err) - } - return string(out), nil -} -``` - -- [ ] **Step 3: Verify against a real host** - -On any Linux machine with Docker, confirm the command shapes produce what the parsers expect: - -```bash -docker run --rm ubuntu:22.04 dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${source:Package}\n' | head -5 -docker run --rm rockylinux:9 rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}-%{RELEASE}\t%{ARCH}\t%{SOURCERPM}\n' | head -5 -docker run --rm alpine:3.19 apk info -v | head -5 -``` - -Check specifically that the dpkg output's fourth column holds a source name, that rpm prints `(none)` for packages without an epoch, and that at least one Alpine line has a name containing a digit or underscore (e.g. `musl-1.2.4_git…-r4`) so `splitAPK`'s right-to-left approach is exercised. - -- [ ] **Step 4: Build** - -Run: `cd agent && go build ./... && go vet ./...` -Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add agent/internal/packages/ -git commit -m "feat: agent collects installed packages per package manager" -``` - ---- - -### Task 5: Models and indexes - -**Files:** -- Create: `server/internal/models/packages.go`, `server/internal/models/vuln.go`, `server/internal/services/vulnindexes.go` -- Modify: `server/cmd/main.go` (schema setup, beside the other `Ensure*Indexes` calls) - -**Interfaces:** -- Consumes: nothing. -- Produces: `models.ServerPackages`, `models.InstalledPackage`, `models.OSRelease`, `models.VulnFinding`, `models.Acceptance`, `models.VulnDBMeta`, `models.VulnAlertRule`, `services.EnsureVulnIndexes() error`. - -- [ ] **Step 1: Write the package models** - -Create `server/internal/models/packages.go`: - -```go -package models - -import ( - "time" - - "go.mongodb.org/mongo-driver/v2/bson" -) - -// Scan status values for ServerPackages. -const ( - ScanStatusOK = "ok" - ScanStatusUnsupported = "unsupported" -) - -type OSRelease struct { - Family string `bson:"family" json:"family"` - VersionID string `bson:"version_id" json:"version_id"` - Arch string `bson:"arch" json:"arch"` -} - -type InstalledPackage struct { - Name string `bson:"name" json:"name"` - Version string `bson:"version" json:"version"` - Epoch int `bson:"epoch,omitempty" json:"epoch,omitempty"` - Arch string `bson:"arch" json:"arch"` - SourceName string `bson:"source_name,omitempty" json:"source_name,omitempty"` -} - -// ServerPackages holds one server's whole package set in ONE document rather -// than one document per package. The hash has already established that -// something changed, so a report is a single atomic upsert with no delta logic -// to get wrong. A typical Linux host is ~2000 packages and ~150KB, comfortably -// inside the 16MB document limit. -type ServerPackages struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"-"` - InstanceID string `bson:"instance_id" json:"-"` - ServerID string `bson:"server_id" json:"server_id"` - OS OSRelease `bson:"os" json:"os"` - Hash string `bson:"hash" json:"hash"` - Packages []InstalledPackage `bson:"packages" json:"packages"` - CollectedAt time.Time `bson:"collected_at" json:"collected_at"` - ScanPending bool `bson:"scan_pending" json:"scan_pending"` - ScannedAt time.Time `bson:"scanned_at,omitempty" json:"scanned_at,omitempty"` - Status string `bson:"status" json:"status"` - DBVersion int `bson:"db_version" json:"db_version"` -} -``` - -- [ ] **Step 2: Write the vulnerability models** - -Create `server/internal/models/vuln.go`: - -```go -package models - -import ( - "time" - - "go.mongodb.org/mongo-driver/v2/bson" -) - -// Finding states. -const ( - FindingOpen = "open" - FindingFixed = "fixed" - FindingAccepted = "accepted" -) - -// Severities, lowest to highest. SeverityRank orders them. -const ( - SeverityUnknown = "unknown" - SeverityLow = "low" - SeverityMedium = "medium" - SeverityHigh = "high" - SeverityCritical = "critical" -) - -func SeverityRank(s string) int { - switch s { - case SeverityCritical: - return 4 - case SeverityHigh: - return 3 - case SeverityMedium: - return 2 - case SeverityLow: - return 1 - default: - return 0 - } -} - -// Acceptance records a decision someone will be asked to justify, so who, -// why and until when all live on the document as well as in the audit log. -type Acceptance struct { - By string `bson:"by" json:"by"` - Reason string `bson:"reason" json:"reason"` - Until time.Time `bson:"until" json:"until"` - At time.Time `bson:"at" json:"at"` -} - -// VulnFinding is one vulnerable package on one server. -// -// Findings are never deleted when a package is patched: the state moves to -// "fixed" with FixedAt stamped, which is what keeps "what did we remediate -// last quarter" answerable. -type VulnFinding struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"id"` - InstanceID string `bson:"instance_id" json:"-"` - ServerID string `bson:"server_id" json:"server_id"` - - CVEID string `bson:"cve_id" json:"cve_id"` - PackageName string `bson:"package_name" json:"package_name"` - Installed string `bson:"installed_version" json:"installed_version"` - // FixedIn empty means no vendor fix has been published. That is a real and - // common state and must never be conflated with "not vulnerable" — it is - // the finding most in need of acceptance, since there is nothing to patch. - FixedIn string `bson:"fixed_in,omitempty" json:"fixed_in,omitempty"` - Severity string `bson:"severity" json:"severity"` - CVSSScore float64 `bson:"cvss_score,omitempty" json:"cvss_score,omitempty"` - Title string `bson:"title,omitempty" json:"title,omitempty"` - References []string `bson:"references,omitempty" json:"references,omitempty"` - - State string `bson:"state" json:"state"` - FirstSeen time.Time `bson:"first_seen" json:"first_seen"` - LastSeen time.Time `bson:"last_seen" json:"last_seen"` - FixedAt *time.Time `bson:"fixed_at,omitempty" json:"fixed_at,omitempty"` - Accepted *Acceptance `bson:"accepted,omitempty" json:"accepted,omitempty"` -} - -// VulnDBMeta is a singleton and deliberately carries no instance_id: the -// vulnerability database is a property of the deployment, not of a tenant. -// Same reasoning as the migrations collection. -type VulnDBMeta struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"-"` - DBVersion int `bson:"db_version" json:"db_version"` - PulledAt time.Time `bson:"pulled_at" json:"pulled_at"` - LastFullScanAt time.Time `bson:"last_full_scan_at,omitempty" json:"last_full_scan_at,omitempty"` - LastError string `bson:"last_error,omitempty" json:"last_error,omitempty"` -} - -type VulnAlertRule struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"id"` - InstanceID string `bson:"instance_id" json:"-"` - Name string `bson:"name" json:"name"` - Enabled bool `bson:"enabled" json:"enabled"` - MinSeverity string `bson:"min_severity" json:"min_severity"` - Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"` - ChannelIDs []string `bson:"channel_ids" json:"channel_ids"` - CreatedAt time.Time `bson:"created_at" json:"created_at"` - UpdatedAt time.Time `bson:"updated_at" json:"updated_at"` -} -``` - -- [ ] **Step 3: Write the index builder** - -Create `server/internal/services/vulnindexes.go`: - -```go -package services - -import ( - "context" - "log" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -// EnsureVulnIndexes declares the indexes for package inventory and findings. -// -// It warns rather than being fatal, matching EnsureSecretIndexes and -// EnsureWorkflowIndexes: a missing index degrades these queries to a collection -// scan, which is no reason to refuse to serve the fleet. -func EnsureVulnIndexes() error { - ctx := context.Background() - - pkgIdx := []mongo.IndexModel{ - { - Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "server_id", Value: 1}}, - Options: options.Index().SetUnique(true), - }, - // Multikey, for fleet-wide package search: "who runs openssl 3.0.2?" - {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "packages.name", Value: 1}}}, - {Keys: bson.D{{Key: "scan_pending", Value: 1}}}, - } - if _, err := db.Col("server_packages").Indexes().CreateMany(ctx, pkgIdx); err != nil { - log.Printf("warning: server_packages indexes: %v", err) - } - - findingIdx := []mongo.IndexModel{ - { - // This key is what makes a rescan an idempotent upsert rather than - // a duplicate factory, and what lets first_seen survive a rescan. - Keys: bson.D{ - {Key: "instance_id", Value: 1}, - {Key: "server_id", Value: 1}, - {Key: "cve_id", Value: 1}, - {Key: "package_name", Value: 1}, - }, - Options: options.Index().SetUnique(true), - }, - {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "state", Value: 1}, {Key: "severity", Value: 1}}}, - {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "cve_id", Value: 1}}}, - } - if _, err := db.Col("vuln_findings").Indexes().CreateMany(ctx, findingIdx); err != nil { - log.Printf("warning: vuln_findings indexes: %v", err) - } - - if _, err := db.Col("vuln_alert_rules").Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "instance_id", Value: 1}}, - }); err != nil { - log.Printf("warning: vuln_alert_rules indexes: %v", err) - } - - return nil -} -``` - -- [ ] **Step 4: Wire it into schema setup** - -In `server/cmd/main.go`, find `runSchemaSetup` (near line 75) and the block calling the other `Ensure*Indexes` functions. Add, alongside the non-fatal ones: - -```go - if err := services.EnsureVulnIndexes(); err != nil { - log.Printf("warning: vuln indexes: %v", err) - } -``` - -- [ ] **Step 5: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/models/packages.go server/internal/models/vuln.go server/internal/services/vulnindexes.go server/cmd/main.go -git commit -m "feat: models and indexes for package inventory and CVE findings" -``` - ---- - -### Task 6: Proto and the hash handshake - -**Files:** -- Modify: `proto/vantage/v1/vantage.proto` -- Regenerate: `agent/internal/grpc/pb/`, `server/internal/grpc/pb/` - -**Interfaces:** -- Consumes: nothing. -- Produces: `pb.ReportPackagesRequest`, `pb.ReportPackagesResponse{NeedFull bool}`, `pb.InstalledPackage`, `pb.OSRelease`, `SyncResponse.CollectPackages bool`. - -- [ ] **Step 1: Add the RPC and messages** - -In `proto/vantage/v1/vantage.proto`, add to the `Vantage` service block: - -```protobuf - rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse); -``` - -And these messages at the end of the file: - -```protobuf -// ReportPackages carries a server's installed package set. -// -// The agent calls twice at most. The first call sends only the hash; if the -// server already holds that hash it answers need_full = false and the ~150KB -// body is never sent. A machine's package set changes rarely, so almost every -// hour costs one small message. -message ReportPackagesRequest { - string server_id = 1; - string agent_token = 2; - string hash = 3; - OSRelease os = 4; - repeated InstalledPackage packages = 5; // empty on the offer call -} - -message ReportPackagesResponse { - bool need_full = 1; -} - -message OSRelease { - string family = 1; - string version_id = 2; - string arch = 3; -} - -message InstalledPackage { - string name = 1; - string version = 2; - int32 epoch = 3; - string arch = 4; - string source_name = 5; -} -``` - -- [ ] **Step 2: Add the collect flag to SyncResponse** - -Find `message SyncResponse` and add a field using the next free number **in that message** — check the file, do not reuse a number: - -```protobuf - // collect_packages tells the agent whether this instance's licence grants - // vulnerability scanning. False means do not collect at all: no gRPC body, - // no document, no storage. The server re-checks on ReportPackages — this - // flag is the optimisation, the server check is the boundary. - bool collect_packages = ; -``` - -- [ ] **Step 3: Regenerate** - -Run the project's existing protoc generation command. Find it with: - -```bash -grep -rn "protoc" --include=Makefile --include="*.sh" --include="*.yml" . | head -``` - -Generated output must land in both `agent/internal/grpc/pb/` and `server/internal/grpc/pb/`. - -- [ ] **Step 4: Build both modules and commit** - -Run: `cd agent && go build ./... && cd ../server && go build ./...` - -```bash -git add proto/ agent/internal/grpc/pb/ server/internal/grpc/pb/ -git commit -m "feat: ReportPackages RPC with hash short-circuit" -``` - ---- - -### Task 7: Agent reports packages - -**Files:** -- Modify: `agent/internal/sync/sync.go:370-411` (`runUpdateCheck`), `agent/internal/grpc/client.go` - -**Interfaces:** -- Consumes: `packages.Collect`, `packages.Hash` (Task 4); `pb.ReportPackagesRequest` (Task 6). -- Produces: `grpcclient.Client.ReportPackages(req *pb.ReportPackagesRequest) (bool, error)` returning `need_full`. - -- [ ] **Step 1: Add the client method** - -In `agent/internal/grpc/client.go`, following the shape of the existing `ReportUpdates`: - -```go -// ReportPackages sends a package report and returns whether the server wants -// the full list. -func (c *Client) ReportPackages(req *pb.ReportPackagesRequest) (bool, error) { - ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) - defer cancel() - resp, err := c.client.ReportPackages(ctx, req) - if err != nil { - return false, err - } - return resp.GetNeedFull(), nil -} -``` - -Match the surrounding file's timeout and context conventions if they differ. - -- [ ] **Step 2: Extend the hourly loop** - -In `agent/internal/sync/sync.go`, inside `runUpdateCheck`'s `doCheck` closure, after the existing `client.ReportUpdates(...)` call succeeds: - -```go - reportPackages(client, cfg) -``` - -Then add to the same file: - -```go -// reportPackages offers a hash of the installed package set and sends the full -// list only if the server does not already hold it. It runs on the same hourly -// cadence as the update check because a package set changes on roughly the same -// schedule, and reusing the loop means one timer rather than two. -func reportPackages(client *grpcclient.Client, cfg *config.Config) { - if runtime.GOOS != "linux" { - return - } - if !collectPackagesEnabled() { - return - } - - osrel, pkgs, err := packages.Collect() - if err != nil { - log.Printf("package collection error: %v", err) - return - } - - pbOS := &pb.OSRelease{ - Family: osrel.Family, - VersionId: osrel.VersionID, - Arch: osrel.Arch, - } - hash := packages.Hash(pkgs) - - needFull, err := client.ReportPackages(&pb.ReportPackagesRequest{ - ServerId: cfg.ServerID, - AgentToken: cfg.AgentToken, - Hash: hash, - Os: pbOS, - }) - if err != nil { - log.Printf("ReportPackages offer error: %v", err) - return - } - if !needFull { - return - } - - pbPkgs := make([]*pb.InstalledPackage, 0, len(pkgs)) - for _, p := range pkgs { - pbPkgs = append(pbPkgs, &pb.InstalledPackage{ - Name: p.Name, - Version: p.Version, - Epoch: int32(p.Epoch), - Arch: p.Arch, - SourceName: p.SourceName, - }) - } - - if _, err := client.ReportPackages(&pb.ReportPackagesRequest{ - ServerId: cfg.ServerID, - AgentToken: cfg.AgentToken, - Hash: hash, - Os: pbOS, - Packages: pbPkgs, - }); err != nil { - log.Printf("ReportPackages full error: %v", err) - return - } - log.Printf("reported %d installed packages", len(pkgs)) -} -``` - -- [ ] **Step 3: Store the collect flag from SyncKeys** - -`SyncResponse.collect_packages` arrives on the 30s key poll, which is a different goroutine from the hourly loop — hence the atomic: - -```go -var collectPackagesFlag atomic.Bool - -func collectPackagesEnabled() bool { return collectPackagesFlag.Load() } -``` - -In `poll()` (around line 94), after a successful `SyncKeys` response: - -```go - collectPackagesFlag.Store(resp.GetCollectPackages()) -``` - -Add `"sync/atomic"`, `"runtime"` and the `packages` import to the file's import block. - -- [ ] **Step 4: Build and commit** - -Run: `cd agent && go build ./... && go vet ./...` - -```bash -git add agent/internal/sync/sync.go agent/internal/grpc/client.go -git commit -m "feat: agent reports installed packages on the hourly loop" -``` - ---- - -### Task 8: Server stores package reports - -**Files:** -- Create: `server/internal/services/packages.go` -- Modify: `server/internal/grpc/server.go` - -**Interfaces:** -- Consumes: `models.ServerPackages` (Task 5), `pb.ReportPackagesRequest` (Task 6). -- Produces: `services.HasPackageHash`, `services.StorePackages`, `services.ListPackages`, `services.SearchPackages`, `services.VulnScanningEnabled`. - -**Critical:** `StorePackages` must **not** match. It upserts and sets `scan_pending: true`. Matching inline would require every replica to hold the 50MB database and would have N replicas racing on a database refresh. - -- [ ] **Step 1: Write the service** - -Create `server/internal/services/packages.go`: - -```go -package services - -import ( - "context" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -// HasPackageHash reports whether we already hold this exact package set, which -// is what lets the agent skip sending ~150KB it has already sent. -func HasPackageHash(instanceID, serverID, hash string) (bool, error) { - err := db.Col("server_packages").FindOne(context.Background(), bson.M{ - "instance_id": instanceID, - "server_id": serverID, - "hash": hash, - }, options.FindOne().SetProjection(bson.M{"_id": 1})).Err() - if err == mongo.ErrNoDocuments { - return false, nil - } - return err == nil, err -} - -// StorePackages replaces a server's package set and marks it for scanning. -// -// It deliberately does NOT match against the vulnerability database. Matching -// happens in vulnsched, on the leader, for two reasons: every replica would -// otherwise need the ~50MB database resident, and a database refresh would have -// N replicas racing to rescan the same fleet and sending N digests. -func StorePackages(instanceID, serverID string, os models.OSRelease, hash string, pkgs []models.InstalledPackage) error { - now := time.Now() - _, err := db.Col("server_packages").UpdateOne(context.Background(), - bson.M{"instance_id": instanceID, "server_id": serverID}, - bson.M{"$set": bson.M{ - "os": os, - "hash": hash, - "packages": pkgs, - "collected_at": now, - "scan_pending": true, - }}, - options.UpdateOne().SetUpsert(true), - ) - return err -} - -func ListPackages(instanceID, serverID string) (*models.ServerPackages, error) { - var sp models.ServerPackages - err := db.Col("server_packages").FindOne(context.Background(), bson.M{ - "instance_id": instanceID, - "server_id": serverID, - }).Decode(&sp) - if err == mongo.ErrNoDocuments { - return nil, nil - } - if err != nil { - return nil, err - } - return &sp, nil -} - -type PackageHit struct { - ServerID string `json:"server_id"` - Name string `json:"name"` - Version string `json:"version"` -} - -// SearchPackages answers "which servers run package X" across the fleet — the -// question people actually ask during an incident. -func SearchPackages(instanceID, name string) ([]PackageHit, error) { - ctx := context.Background() - cur, err := db.Col("server_packages").Find(ctx, bson.M{ - "instance_id": instanceID, - "packages.name": name, - }, options.Find().SetProjection(bson.M{"server_id": 1, "packages": 1})) - if err != nil { - return nil, err - } - defer cur.Close(ctx) - - var docs []models.ServerPackages - if err := cur.All(ctx, &docs); err != nil { - return nil, err - } - - hits := []PackageHit{} - for _, d := range docs { - for _, p := range d.Packages { - if p.Name == name { - hits = append(hits, PackageHit{ServerID: d.ServerID, Name: p.Name, Version: p.Version}) - } - } - } - return hits, nil -} - -// VulnScanningEnabled reports whether this instance's licence grants -// vulnerability scanning. It reads the feature by name and never switches on -// tier, so changing what a tier includes needs no server release. -func VulnScanningEnabled(instanceID string) bool { - lic, err := GetLicense(instanceID) - if err != nil || lic == nil { - return false - } - return lic.HasFeature("vuln_scanning") -} -``` - -**Before building:** find the existing licence accessor in `server/internal/services/licence.go` — the one `RequireActiveLicense` in `server/internal/api/licence.go` uses — and call **that** rather than assuming `GetLicense` exists. Adjust the name and signature to match. - -- [ ] **Step 2: Add the gRPC handler** - -In `server/internal/grpc/server.go`, following the shape of the existing `ReportUpdates` (near line 87): - -```go -func (s *vantageServer) ReportPackages(ctx context.Context, req *pb.ReportPackagesRequest) (*pb.ReportPackagesResponse, error) { - srv, err := services.ValidateAgentToken(req.GetServerId(), req.GetAgentToken()) - if err != nil { - return nil, status.Error(codes.Unauthenticated, "invalid agent credentials") - } - - // The agent flag is an optimisation; this check is the boundary. - if !services.VulnScanningEnabled(srv.InstanceID) { - return &pb.ReportPackagesResponse{NeedFull: false}, nil - } - - // The offer call: no packages, just a hash. - if len(req.GetPackages()) == 0 { - known, err := services.HasPackageHash(srv.InstanceID, req.GetServerId(), req.GetHash()) - if err != nil { - return nil, status.Error(codes.Internal, "package hash lookup failed") - } - return &pb.ReportPackagesResponse{NeedFull: !known}, nil - } - - pkgs := make([]models.InstalledPackage, 0, len(req.GetPackages())) - for _, p := range req.GetPackages() { - pkgs = append(pkgs, models.InstalledPackage{ - Name: p.GetName(), - Version: p.GetVersion(), - Epoch: int(p.GetEpoch()), - Arch: p.GetArch(), - SourceName: p.GetSourceName(), - }) - } - - os := models.OSRelease{ - Family: req.GetOs().GetFamily(), - VersionID: req.GetOs().GetVersionId(), - Arch: req.GetOs().GetArch(), - } - - if err := services.StorePackages(srv.InstanceID, req.GetServerId(), os, req.GetHash(), pkgs); err != nil { - return nil, status.Error(codes.Internal, "failed to store packages") - } - return &pb.ReportPackagesResponse{NeedFull: false}, nil -} -``` - -Match the neighbouring handlers exactly on how `ValidateAgentToken` is called and what it returns in this file. - -- [ ] **Step 3: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/services/packages.go server/internal/grpc/server.go -git commit -m "feat: store agent package reports and mark them for scanning" -``` - ---- - -### Task 9: Serve the collect flag from SyncKeys - -**Files:** -- Modify: `server/internal/grpc/server.go` — `SyncKeys` (near line 47) - -- [ ] **Step 1: Set the flag** - -In `SyncKeys`, where the `SyncResponse` is constructed, add — using whatever the response variable is actually called: - -```go - resp.CollectPackages = services.VulnScanningEnabled(srv.InstanceID) -``` - -- [ ] **Step 2: Build and commit** - -Run: `cd server && go build ./...` - -```bash -git add server/internal/grpc/server.go -git commit -m "feat: tell agents whether to collect packages via SyncKeys" -``` - ---- - -### Task 10: trivy-db puller and advisory lookup - -**Files:** -- Create: `server/internal/vulndb/pull.go`, `server/internal/vulndb/db.go` - -**Interfaces:** -- Consumes: nothing from earlier tasks. -- Produces: `vulndb.Pull(ctx, dir string) (version int, err error)`, `vulndb.Open(dir string) (*Store, error)`, `(*Store).Advisories(bucket, srcName string) ([]Advisory, error)`, `(*Store).Vulnerability(cveID string) (VulnInfo, error)`, `(*Store).Close() error`, `vulndb.Ref()`, `vulndb.Disabled()`. - -- [ ] **Step 1: Write the puller skeleton** - -Create `server/internal/vulndb/pull.go`: - -```go -package vulndb - -import ( - "archive/tar" - "compress/gzip" - "context" - "fmt" - "io" - "os" - "path/filepath" - "strings" -) - -// DefaultRef is the published trivy-db OCI artifact, rebuilt every six hours. -const DefaultRef = "ghcr.io/aquasecurity/trivy-db:2" - -// SupportedSchema is the trivy-db schema version this code understands. -const SupportedSchema = 2 - -// Ref returns the artifact reference, honouring VANTAGE_TRIVY_DB_REF so an -// air-gapped deployment can mirror the artifact into its own registry, and so -// a busy deployment can avoid the anonymous ghcr rate limit. -func Ref() string { - if v := os.Getenv("VANTAGE_TRIVY_DB_REF"); v != "" { - return v - } - return DefaultRef -} - -// Disabled reports whether the puller and scheduler are switched off entirely. -// Findings already written are still served, and still marked stale. -func Disabled() bool { - return strings.EqualFold(os.Getenv("VANTAGE_VULNDB_DISABLED"), "true") -} - -// Pull fetches the trivy-db artifact into dir and returns the schema version -// recorded in its metadata. -// -// Implementation requirements: -// - Use oras.land/oras-go/v2 with a remote repository for Ref(). -// - The artifact has a single layer: a gzipped tar containing "db/trivy.db" -// and "db/metadata.json". -// - Extract both into dir via extractTarGz below, writing to a temp path and -// renaming into place, so a failed pull cannot leave a half-written -// database that Open would accept. -// - Read metadata.json for "Version" (an int) and return it. -// - REFUSE a version other than SupportedSchema rather than mis-parsing it. -func Pull(ctx context.Context, dir string) (int, error) { - return 0, fmt.Errorf("not implemented") -} - -// extractTarGz writes the artifact layer into dir. Paths are flattened and -// checked so a crafted archive cannot write outside dir. -func extractTarGz(r io.Reader, dir string) error { - gz, err := gzip.NewReader(r) - if err != nil { - return err - } - defer gz.Close() - - tr := tar.NewReader(gz) - for { - hdr, err := tr.Next() - if err == io.EOF { - return nil - } - if err != nil { - return err - } - if hdr.Typeflag != tar.TypeReg { - continue - } - name := filepath.Base(hdr.Name) // flatten; the archive is two files - if name == "." || name == ".." || name == "" { - continue - } - dst := filepath.Join(dir, name) - if !strings.HasPrefix(dst, filepath.Clean(dir)+string(os.PathSeparator)) { - return fmt.Errorf("archive entry escapes destination: %q", hdr.Name) - } - f, err := os.Create(dst) - if err != nil { - return err - } - if _, err := io.Copy(f, tr); err != nil { - f.Close() - return err - } - if err := f.Close(); err != nil { - return err - } - } -} -``` - -- [ ] **Step 2: Implement `Pull` with oras** - -Replace the `Pull` body following the requirements in its doc comment. The exact `oras-go` v2 call shape depends on the version `go get` resolved, so write it against the resolved API rather than from memory. - -- [ ] **Step 3: Verify the pull by hand** - -This is the one piece that cannot be verified by reading. From a scratch `main.go`: - -```go -dir, _ := os.MkdirTemp("", "trivydb-") -v, err := vulndb.Pull(context.Background(), dir) -fmt.Println(v, err) -// then: ls the dir — expect trivy.db and metadata.json -``` - -Confirm the returned version equals `SupportedSchema`, both files exist, and `trivy.db` is tens of MB rather than a few bytes. Delete the scratch file afterwards. - -- [ ] **Step 4: Write the advisory store** - -Create `server/internal/vulndb/db.go`: - -```go -package vulndb - -import ( - trivydb "github.com/aquasecurity/trivy-db/pkg/db" -) - -// Advisory is one fixed-version statement for one source package. -type Advisory struct { - CVEID string - // FixedVersion empty means no vendor fix has been published. It is a real - // state, not an absence of data, and callers must treat it as vulnerable. - FixedVersion string - Severity string -} - -// VulnInfo is the CVE's own metadata, shared across every server it affects. -type VulnInfo struct { - Title string - Severity string - CVSSScore float64 - References []string -} - -// Store reads a pulled trivy-db. -type Store struct { - cfg trivydb.Config -} - -// Open opens the database in dir read-only. -func Open(dir string) (*Store, error) { - if err := trivydb.Init(dir); err != nil { - return nil, err - } - return &Store{cfg: trivydb.Config{}}, nil -} - -func (s *Store) Close() error { return trivydb.Close() } - -// Advisories returns every advisory for a source package in a bucket. -func (s *Store) Advisories(bucket, srcName string) ([]Advisory, error) { - raw, err := s.cfg.GetAdvisories(bucket, srcName) - if err != nil { - return nil, err - } - out := make([]Advisory, 0, len(raw)) - for _, a := range raw { - out = append(out, Advisory{ - CVEID: a.VulnerabilityID, - FixedVersion: a.FixedVersion, - Severity: severityName(int(a.Severity)), - }) - } - return out, nil -} - -// Vulnerability returns a CVE's shared metadata. -func (s *Store) Vulnerability(cveID string) (VulnInfo, error) { - v, err := s.cfg.GetVulnerability(cveID) - if err != nil { - return VulnInfo{}, err - } - return VulnInfo{ - Title: v.Title, - Severity: severityName(int(v.Severity)), - References: v.References, - }, nil -} - -// severityName maps trivy-db's integer severity onto our lowercase strings. -// -// Severity resolves vendor → NVD → unknown and is never invented. This will -// surface as "why is this critical CVE marked low": Debian and Red Hat -// routinely downgrade an NVD score because the vulnerable path is not -// reachable in their build, and their rating is the accurate one for that -// package. -func severityName(n int) string { - switch n { - case 4: - return "critical" - case 3: - return "high" - case 2: - return "medium" - case 1: - return "low" - default: - return "unknown" - } -} -``` - -Verify `GetAdvisories`, `GetVulnerability`, `Init` and `Close` against the version `go get` actually resolved. Adjust the wrappers, **not** the `Advisory`/`VulnInfo` shapes, which later tasks depend on. - -- [ ] **Step 5: Build and commit** - -Run: `cd server && go build ./...` - -```bash -git add server/internal/vulndb/pull.go server/internal/vulndb/db.go -git commit -m "feat: pull trivy-db and read its advisories" -``` - ---- - -### Task 11: The matcher - -**Files:** -- Create: `server/internal/vulndb/match.go` - -**Interfaces:** -- Consumes: `vulndb.LessThan` (Task 1), `vulndb.Bucket` (Task 2), `vulndb.Advisory` (Task 10), `models.InstalledPackage` (Task 5). -- Produces: `vulndb.AdvisorySource` (interface), `vulndb.Match(src AdvisorySource, os models.OSRelease, pkgs []models.InstalledPackage) ([]Result, error)`, `vulndb.Result`. - -- [ ] **Step 1: Write the implementation** - -Create `server/internal/vulndb/match.go`: - -```go -package vulndb - -import ( - "fmt" - "log" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" -) - -// AdvisorySource is the advisory lookup the matcher needs. *Store satisfies it. -// The seam keeps the matching logic independent of how the database is opened. -type AdvisorySource interface { - Advisories(bucket, srcName string) ([]Advisory, error) -} - -// Result is one vulnerable package on one server, before it becomes a finding. -type Result struct { - CVEID string - PackageName string // the BINARY package, which is what is installed - Installed string - FixedIn string - Severity string -} - -// Match returns every advisory that the installed packages do not satisfy. -// -// Vulnerable means: no fix has been published, or the installed version sorts -// strictly before the fixed version under the distribution's own ordering. -// Equal is NOT vulnerable — that is the backported-fix case, where a -// distribution patches in place without changing the upstream version, and -// treating it as vulnerable reports a patched fleet as exposed. -func Match(src AdvisorySource, os models.OSRelease, pkgs []models.InstalledPackage) ([]Result, error) { - bucket, err := Bucket(os.Family, os.VersionID) - if err != nil { - return nil, err - } - - var out []Result - for _, p := range pkgs { - // Debian and Ubuntu advisories are keyed on the source package: one - // advisory against "openssl" covers libssl3, openssl and libssl-dev. - srcName := p.SourceName - if srcName == "" { - srcName = p.Name - } - - advs, err := src.Advisories(bucket, srcName) - if err != nil { - return nil, fmt.Errorf("advisories for %s: %w", srcName, err) - } - - for _, a := range advs { - // No published fix. Vulnerable, and the finding most in need of - // acceptance, since there is nothing to patch. - if a.FixedVersion == "" { - out = append(out, Result{ - CVEID: a.CVEID, PackageName: p.Name, - Installed: p.Version, Severity: a.Severity, - }) - continue - } - - older, err := LessThan(os.Family, p.Version, a.FixedVersion) - if err != nil { - // Skip this one advisory rather than failing the whole server: - // one unparseable version must not blind us to every other CVE - // on the host. Log it — a silent skip is a silent false - // negative, which is the direction that hurts. - log.Printf("vulndb: compare %s %s vs %s: %v", p.Name, p.Version, a.FixedVersion, err) - continue - } - if older { - out = append(out, Result{ - CVEID: a.CVEID, PackageName: p.Name, - Installed: p.Version, FixedIn: a.FixedVersion, Severity: a.Severity, - }) - } - } - } - return out, nil -} -``` - -- [ ] **Step 2: Verify the matching by hand** - -From a scratch `main.go`, define a small in-memory `AdvisorySource` and confirm each case. **These are the behaviours that silently produce wrong answers.** - -```go -type fake map[string][]vulndb.Advisory -func (f fake) Advisories(bucket, src string) ([]vulndb.Advisory, error) { - return f[bucket+"\x00"+src], nil -} -``` - -| Setup | Expected | -| ----- | -------- | -| advisory `openssl` fixed `1:3.0.2-0ubuntu1.15`; installed `libssl3` at `1:3.0.2-0ubuntu1.15`, source `openssl`, ubuntu 22.04 | **0 results** — the backport case | -| same advisory; installed at `1:3.0.2-0ubuntu1.14` | 1 result, `PackageName` = `libssl3` (the binary, not the source), `FixedIn` set | -| same advisory; three binaries `libssl3`, `openssl`, `libssl-dev` all at `…1.14`, all source `openssl` | **3 results** — one advisory covers every binary of the source | -| advisory with `FixedVersion: ""` on `bash` | 1 result with empty `FixedIn` | -| os family `arch` | error wrapping `ErrUnsupportedFamily` | -| alpine 3.19.1, package `openssl` at `3.1.4-r5`, **no** `SourceName` set, advisory fixed `3.1.4-r6` | 1 result — the fallback to `Name` works | - -Delete the scratch file afterwards. - -- [ ] **Step 3: Build and commit** - -Run: `cd server && go build ./...` - -```bash -git add server/internal/vulndb/match.go -git commit -m "feat: match installed packages against distro advisories" -``` - ---- - -### Task 12: Finding state machine - -**Files:** -- Create: `server/internal/services/findings.go` - -**Interfaces:** -- Consumes: `vulndb.Result` (Task 11), `models.VulnFinding` (Task 5). -- Produces: `services.DiffFindings(existing []models.VulnFinding, results []vulndb.Result, now time.Time) FindingDiff`, `services.FindingDiff`, `services.ApplyFindingDiff`, `services.ListFindings`. - -- [ ] **Step 1: Write the pure diff** - -Create `server/internal/services/findings.go`: - -```go -package services - -import ( - "context" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -// FindingDiff is what one server's scan changes. -type FindingDiff struct { - Upserts []models.VulnFinding - FixedIDs []bson.ObjectID - ReopenIDs []bson.ObjectID - // NewlyOpened is what the digest reports: findings that were not open - // before this scan. A finding that was already open must not re-alert every - // tick, or the digest becomes noise and stops being read. - NewlyOpened []models.VulnFinding -} - -func findingKey(cveID, pkg string) string { return cveID + "\x00" + pkg } - -// DiffFindings computes the state changes for one server's scan. -// -// Pure by design: no database, no clock of its own. The ordering below is -// load-bearing — see the comment above the second loop. -func DiffFindings(existing []models.VulnFinding, results []vulndb.Result, now time.Time) FindingDiff { - var d FindingDiff - - byKey := make(map[string]models.VulnFinding, len(existing)) - for _, f := range existing { - byKey[findingKey(f.CVEID, f.PackageName)] = f - } - - seen := make(map[string]bool, len(results)) - for _, r := range results { - key := findingKey(r.CVEID, r.PackageName) - seen[key] = true - - prev, had := byKey[key] - - f := models.VulnFinding{ - CVEID: r.CVEID, - PackageName: r.PackageName, - Installed: r.Installed, - FixedIn: r.FixedIn, - Severity: r.Severity, - State: models.FindingOpen, - FirstSeen: now, - LastSeen: now, - } - - if had { - f.ID = prev.ID - // Preserved, never overwritten: an upsert that moves first_seen - // forward makes every finding look discovered today. - f.FirstSeen = prev.FirstSeen - - // A live acceptance survives the scan untouched: it is suppressed - // from counts and alerts until its expiry, then reopens on its own. - if prev.State == models.FindingAccepted && prev.Accepted != nil { - if now.Before(prev.Accepted.Until) { - continue - } - d.ReopenIDs = append(d.ReopenIDs, prev.ID) - continue - } - - if prev.State != models.FindingOpen { - d.NewlyOpened = append(d.NewlyOpened, f) - } - } else { - d.NewlyOpened = append(d.NewlyOpened, f) - } - - d.Upserts = append(d.Upserts, f) - } - - // Anything we hold that this scan did not produce is fixed. This runs AFTER - // the loop above, and the ordering matters: a finding that is both absent - // and past its acceptance expiry must settle as fixed rather than reopening - // on a package that no longer carries it. - for _, f := range existing { - if seen[findingKey(f.CVEID, f.PackageName)] { - continue - } - if f.State == models.FindingFixed { - continue - } - d.FixedIDs = append(d.FixedIDs, f.ID) - } - - return d -} -``` - -- [ ] **Step 2: Add the persistence half** - -Append to the same file: - -```go -// ListFindings returns every finding held for one server. -func ListFindings(ctx context.Context, instanceID, serverID string) ([]models.VulnFinding, error) { - cur, err := db.Col("vuln_findings").Find(ctx, bson.M{ - "instance_id": instanceID, - "server_id": serverID, - }) - if err != nil { - return nil, err - } - defer cur.Close(ctx) - - var out []models.VulnFinding - if err := cur.All(ctx, &out); err != nil { - return nil, err - } - return out, nil -} - -// ApplyFindingDiff writes a diff. Thin on purpose — the logic worth reading -// twice is all in DiffFindings. -func ApplyFindingDiff(ctx context.Context, instanceID, serverID string, d FindingDiff, now time.Time) error { - col := db.Col("vuln_findings") - - for _, f := range d.Upserts { - _, err := col.UpdateOne(ctx, - bson.M{ - "instance_id": instanceID, - "server_id": serverID, - "cve_id": f.CVEID, - "package_name": f.PackageName, - }, - bson.M{ - "$set": bson.M{ - "installed_version": f.Installed, - "fixed_in": f.FixedIn, - "severity": f.Severity, - "state": models.FindingOpen, - "last_seen": now, - }, - // first_seen is written only on insert, so a rescan cannot move - // it forward. - "$setOnInsert": bson.M{ - "instance_id": instanceID, - "server_id": serverID, - "cve_id": f.CVEID, - "package_name": f.PackageName, - "first_seen": f.FirstSeen, - }, - "$unset": bson.M{"fixed_at": "", "accepted": ""}, - }, - options.UpdateOne().SetUpsert(true), - ) - if err != nil { - return err - } - } - - if len(d.FixedIDs) > 0 { - if _, err := col.UpdateMany(ctx, - bson.M{"_id": bson.M{"$in": d.FixedIDs}}, - bson.M{"$set": bson.M{"state": models.FindingFixed, "fixed_at": now}}, - ); err != nil { - return err - } - } - - if len(d.ReopenIDs) > 0 { - if _, err := col.UpdateMany(ctx, - bson.M{"_id": bson.M{"$in": d.ReopenIDs}}, - bson.M{"$set": bson.M{"state": models.FindingOpen, "last_seen": now}, "$unset": bson.M{"accepted": ""}}, - ); err != nil { - return err - } - } - - return nil -} -``` - -- [ ] **Step 3: Read the diff back against these cases** - -No test, so verify by reading. Walk the function with each and confirm the branch taken: - -| Input | Expected | -| ----- | -------- | -| no existing, one result | one upsert, state `open`, one `NewlyOpened` | -| existing open with `first_seen` 30 days ago, same result | one upsert with the **old** `first_seen`, `LastSeen` = now, **zero** `NewlyOpened` | -| existing open, **no** results | one `FixedIDs` entry, zero upserts | -| existing accepted with `until` in the past, still in results | one `ReopenIDs` entry | -| existing accepted with `until` in the future, still in results | zero `ReopenIDs`, zero `NewlyOpened` | -| existing accepted with `until` in the past, **not** in results | one `FixedIDs` entry, **zero** `ReopenIDs` | - -The last row is the ordering trap: it only holds because the fixed loop runs after the results loop. - -- [ ] **Step 4: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/services/findings.go -git commit -m "feat: finding state machine with acceptance expiry" -``` - ---- - -### Task 13: The scheduler - -**Files:** -- Create: `server/internal/vulnsched/sched.go` -- Modify: `server/cmd/main.go:184-206` - -**Interfaces:** -- Consumes: `vulndb.Pull`, `vulndb.Open`, `vulndb.Match`, `services.DiffFindings`, `services.ApplyFindingDiff`, `services.ListFindings`. -- Produces: `vulnsched.Start(ctx, deps Deps)`, `vulnsched.Deps`. - -**Read `server/internal/workflowsched/sched.go` first.** Same `Start`/`tick` shape, same `Deps` injection, same "return when the context is cancelled" contract. - -- [ ] **Step 1: Write the scheduler** - -Create `server/internal/vulnsched/sched.go`: - -```go -// Package vulnsched owns the vulnerability scan loop. -// -// It runs inside bus.RunAsLeader("housekeeping", …) alongside monitorsched, -// workflowsched and the sweepers: one role, one lock. N replicas each running -// this loop would mean N copies of the ~50MB database resident, N rescans of -// the same fleet on every database refresh, and N digests reaching the -// customer for one set of findings. -package vulnsched - -import ( - "context" - "errors" - "log" - "os" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/vulndb" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -const ( - tickInterval = 60 * time.Second - // trivy-db is rebuilt every six hours; pulling more often buys nothing. - dbMaxAge = 6 * time.Hour -) - -// Deps are injected from main.go rather than imported, following -// workflowsched. It keeps this package's reach explicit and reviewable. -type Deps struct { - LogEvent func(instanceID, eventType, actor, serverID, keyID, details string) - SendDigest func(instanceID string, newly []models.VulnFinding) -} - -type scheduler struct { - deps Deps - dir string - store *vulndb.Store - version int - pulled time.Time -} - -func Start(ctx context.Context, deps Deps) { - if vulndb.Disabled() { - log.Println("vulnsched: disabled by VANTAGE_VULNDB_DISABLED") - return - } - - dir, err := os.MkdirTemp("", "vantage-vulndb-") - if err != nil { - log.Printf("vulnsched: temp dir: %v", err) - return - } - - s := &scheduler{deps: deps, dir: dir} - - go func() { - defer os.RemoveAll(dir) - defer s.closeStore() - - ticker := time.NewTicker(tickInterval) - defer ticker.Stop() - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - s.tick(ctx) - } - } - }() -} - -func (s *scheduler) tick(ctx context.Context) { - if err := s.ensureDB(ctx); err != nil { - // Keep the last good database and carry on scanning against it. A - // network blip must never clear findings or read as "all fixed". - log.Printf("vulnsched: database unavailable: %v", err) - s.recordDBError(ctx, err) - if s.store == nil { - return - } - } - s.scanPending(ctx) -} - -// ensureDB pulls a fresh database when the local copy is stale, and marks the -// whole fleet for rescanning when the version changes — which is what makes a -// newly published CVE flag existing servers within a minute rather than at the -// next agent report. -func (s *scheduler) ensureDB(ctx context.Context) error { - if s.store != nil && time.Since(s.pulled) < dbMaxAge { - return nil - } - - version, err := vulndb.Pull(ctx, s.dir) - if err != nil { - return err - } - - s.closeStore() - store, err := vulndb.Open(s.dir) - if err != nil { - return err - } - s.store = store - s.pulled = time.Now() - - changed := version != s.version - s.version = version - - _, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{}, - bson.M{"$set": bson.M{"db_version": version, "pulled_at": s.pulled}, "$unset": bson.M{"last_error": ""}}, - options.UpdateOne().SetUpsert(true), - ) - - if changed { - res, err := db.Col("server_packages").UpdateMany(ctx, - bson.M{"status": bson.M{"$ne": models.ScanStatusUnsupported}}, - bson.M{"$set": bson.M{"scan_pending": true}}, - ) - if err != nil { - log.Printf("vulnsched: mark fleet pending: %v", err) - } else { - log.Printf("vulnsched: database version %d, %d servers marked for rescan", version, res.ModifiedCount) - } - } - return nil -} - -func (s *scheduler) recordDBError(ctx context.Context, err error) { - _, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{}, - bson.M{"$set": bson.M{"last_error": err.Error()}}, - options.UpdateOne().SetUpsert(true), - ) -} - -func (s *scheduler) scanPending(ctx context.Context) { - cur, err := db.Col("server_packages").Find(ctx, bson.M{"scan_pending": true}) - if err != nil { - log.Printf("vulnsched: find pending: %v", err) - return - } - defer cur.Close(ctx) - - var pending []models.ServerPackages - if err := cur.All(ctx, &pending); err != nil { - log.Printf("vulnsched: decode pending: %v", err) - return - } - - // Newly opened findings are collected across the whole tick and sent as one - // digest per instance. A database refresh can open several hundred findings - // at once; one message per finding would rate-limit the webhook or get the - // channel muted, and either way the alerts stop being read. - newly := map[string][]models.VulnFinding{} - - for _, sp := range pending { - if ctx.Err() != nil { - // Leadership lost. scan_pending is still set, so the next leader - // picks these up — which is why it lives on the document. - return - } - opened := s.scanOne(ctx, sp) - newly[sp.InstanceID] = append(newly[sp.InstanceID], opened...) - } - - for instanceID, findings := range newly { - if len(findings) > 0 && s.deps.SendDigest != nil { - s.deps.SendDigest(instanceID, findings) - } - } - - _, _ = db.Col("vulndb_meta").UpdateOne(ctx, bson.M{}, - bson.M{"$set": bson.M{"last_full_scan_at": time.Now()}}, - options.UpdateOne().SetUpsert(true), - ) -} - -func (s *scheduler) scanOne(ctx context.Context, sp models.ServerPackages) []models.VulnFinding { - now := time.Now() - - results, err := vulndb.Match(s.store, sp.OS, sp.Packages) - if err != nil { - // We hold no feed for this distribution, so we cannot answer whether it - // is vulnerable. Say "unsupported" — reporting zero findings here would - // be indistinguishable from reporting a clean host, and one of those is - // a lie. - status := models.ScanStatusUnsupported - if !errors.Is(err, vulndb.ErrUnsupportedFamily) { - log.Printf("vulnsched: scan %s: %v", sp.ServerID, err) - status = sp.Status - } - s.clearPending(ctx, sp.ID, status, now) - return nil - } - - existing, err := services.ListFindings(ctx, sp.InstanceID, sp.ServerID) - if err != nil { - log.Printf("vulnsched: list findings %s: %v", sp.ServerID, err) - return nil - } - - diff := services.DiffFindings(existing, results, now) - if err := services.ApplyFindingDiff(ctx, sp.InstanceID, sp.ServerID, diff, now); err != nil { - log.Printf("vulnsched: apply diff %s: %v", sp.ServerID, err) - return nil - } - - s.clearPending(ctx, sp.ID, models.ScanStatusOK, now) - - for i := range diff.NewlyOpened { - diff.NewlyOpened[i].ServerID = sp.ServerID - } - return diff.NewlyOpened -} - -func (s *scheduler) clearPending(ctx context.Context, id bson.ObjectID, status string, now time.Time) { - _, _ = db.Col("server_packages").UpdateOne(ctx, - bson.M{"_id": id}, - bson.M{"$set": bson.M{ - "scan_pending": false, - "status": status, - "scanned_at": now, - "db_version": s.version, - }}, - ) -} - -func (s *scheduler) closeStore() { - if s.store != nil { - _ = s.store.Close() - s.store = nil - } -} -``` - -- [ ] **Step 2: Wire it into main.go** - -In `server/cmd/main.go`, inside `bus.RunAsLeader(ctx, "housekeeping", …)` (line 184), after `workflowsched.Start(...)`: - -```go - vulnsched.Start(jobCtx, vulnsched.Deps{ - LogEvent: services.LogEvent, - SendDigest: services.SendVulnDigest, - }) - services.StartVulnSweeper(jobCtx) -``` - -Add the `vulnsched` import. `services.SendVulnDigest` and `services.StartVulnSweeper` arrive in Tasks 14 and 15 — stub them as no-ops in `server/internal/services/vulnrules.go` now and fill them in there. - -- [ ] **Step 3: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/vulnsched/ server/cmd/main.go -git commit -m "feat: leader-owned vulnerability scan loop" -``` - ---- - -### Task 14: Alert rules and the digest - -**Files:** -- Create: `server/internal/services/vulnrules.go`, `shared/mail/vuln.go`, `shared/mail/templates/vuln_digest.html.tmpl`, `shared/mail/templates/vuln_digest.txt.tmpl` - -**Interfaces:** -- Consumes: `models.VulnAlertRule`, `models.VulnFinding`, `services.ResolveTargets`. -- Produces: `services.SendVulnDigest`, `services.ListVulnRules/CreateVulnRule/UpdateVulnRule/DeleteVulnRule`, `mail.Sender.SendVulnDigest`. - -- [ ] **Step 1: Write the rule service and dispatch** - -Create `server/internal/services/vulnrules.go` with CRUD for `vuln_alert_rules` following the shape of the existing notification-channel service in `server/internal/services/channels.go`, plus: - -```go -// SendVulnDigest delivers one message per rule per tick — never one per -// finding. See vulnsched for why the tick is the batch boundary. -func SendVulnDigest(instanceID string, newly []models.VulnFinding) { - rules, err := ListVulnRules(instanceID) - if err != nil { - log.Printf("vuln digest: list rules: %v", err) - return - } - - for _, rule := range rules { - if !rule.Enabled { - continue - } - - matched := filterBySeverity(newly, rule.MinSeverity) - if len(matched) == 0 { - continue - } - - if len(rule.Tags) > 0 { - // ResolveTargets is already the single answer to which servers a - // selector touches. A rule that disagreed with a workflow about - // what env:prod means would be worse than no filter at all. - allowed, err := ResolveTargets(instanceID, nil, rule.Tags) - if err != nil { - log.Printf("vuln digest: resolve targets: %v", err) - continue - } - matched = filterByServers(matched, allowed) - if len(matched) == 0 { - continue - } - } - - for _, chID := range rule.ChannelIDs { - dispatchVulnDigest(instanceID, chID, rule.Name, matched) - } - } -} - -func filterBySeverity(findings []models.VulnFinding, min string) []models.VulnFinding { - floor := models.SeverityRank(min) - out := make([]models.VulnFinding, 0, len(findings)) - for _, f := range findings { - if models.SeverityRank(f.Severity) >= floor { - out = append(out, f) - } - } - return out -} - -func filterByServers(findings []models.VulnFinding, allowed []string) []models.VulnFinding { - set := make(map[string]bool, len(allowed)) - for _, id := range allowed { - set[id] = true - } - out := make([]models.VulnFinding, 0, len(findings)) - for _, f := range findings { - if set[f.ServerID] { - out = append(out, f) - } - } - return out -} -``` - -Implement `dispatchVulnDigest` following how `server/internal/notify` already dispatches a monitor alert to a channel by ID, with a summary line of the form `"12 new critical, 4 new high across 6 servers"`. **Verify `ResolveTargets`' actual signature** in `server/internal/services/targets.go` and adjust the call. - -Also add `StartVulnSweeper` here as a no-op stub if Task 13 has already referenced it; Task 15 replaces it. - -- [ ] **Step 2: Write the mail templates** - -Create `shared/mail/templates/vuln_digest.txt.tmpl`. **`subject` is defined in the txt file only** — `html/template` would escape an ampersand in an instance name, and mail clients show subjects verbatim: - -``` -{{define "subject"}}{{.Count}} new {{if eq .Count 1}}vulnerability{{else}}vulnerabilities{{end}} on {{.InstanceName}}{{end}} -{{define "title"}}New vulnerabilities detected{{end}} -{{define "pill"}}{{.TopSeverity}}{{end}} -{{define "body"}} -{{template "lead" .Summary}} - -{{range .Rows}}- {{.CVEID}} ({{.Severity}}) — {{.PackageName}} on {{.ServerName}}{{if .FixedIn}}, fixed in {{.FixedIn}}{{else}}, no fix published{{end}} -{{end}} -{{if .More}}...and {{.More}} more.{{end}} - -Scanned against vulnerability database pulled {{.DBAge}} ago. -{{end}} -``` - -Create `shared/mail/templates/vuln_digest.html.tmpl` defining `title`, `pill` and `body` only, composing the existing `p`, `lead`, `button`, `rows` and `chip` helpers from `layout.html.tmpl`. Read `shared/mail/monitor.go`'s template pair and match it exactly. **Carry no new hex colours** — every colour in the email system lives in `layout.html.tmpl`. - -Create `shared/mail/vuln.go` with `VulnDigestData` and `SendVulnDigest`, following `shared/mail/monitor.go`. - -- [ ] **Step 3: Verify the templates parse** - -`shared/mail` parses its templates in `init()`, so a mistyped field is a **boot-time panic**, not a runtime error. Confirm before committing: - -```bash -cd shared && go build ./mail/ -cd ../server && go build ./... && go run ./cmd --help 2>&1 | head -3 -``` - -If the binary starts far enough to print usage, the template set parsed. Then send one digest through a real channel from a dev instance and read the delivered message — the template can parse and still render nonsense, and nothing else will catch that. - -- [ ] **Step 4: Commit** - -```bash -git add server/internal/services/vulnrules.go shared/mail/ -git commit -m "feat: vulnerability alert rules and batched digest" -``` - ---- - -### Task 15: REST API, retention setting and sweeper - -**Files:** -- Create: `server/internal/api/vulnerabilities.go` -- Modify: `shared/models/settings.go`, `server/internal/api/handlers.go`, `server/internal/services/findings.go` - -- [ ] **Step 1: Add the retention setting** - -In `shared/models/settings.go`, beside `WorkflowLogRetentionDays`: - -```go - // VulnFindingRetentionDays is a pointer for the same reason - // WorkflowLogRetentionDays is: absent must mean the default, not zero. - // Nil is 90 days, 0 is forever. Only "fixed" findings are ever swept. - VulnFindingRetentionDays *int `bson:"vuln_finding_retention_days,omitempty" json:"vuln_finding_retention_days,omitempty"` -``` - -- [ ] **Step 2: Add the sweeper** - -In `server/internal/services/findings.go`, following `StartAuditSweeper` in `audit_retention.go`: - -```go -// StartVulnSweeper deletes old FIXED findings. Open and accepted findings are -// never swept at any setting: retention is about history, and an unresolved -// vulnerability is not history. -func StartVulnSweeper(ctx context.Context) { - go func() { - ticker := time.NewTicker(6 * time.Hour) - defer ticker.Stop() - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - sweepFixedFindings(ctx) - } - } - }() -} -``` - -Implement `sweepFixedFindings` to read each instance's setting, resolve nil to 90 and skip 0, then delete `{instance_id, state: "fixed", fixed_at: {$lt: cutoff}}`. Model it on `StartAuditSweeper`. Remove the Task 14 stub. - -- [ ] **Step 3: Write the handlers** - -Create `server/internal/api/vulnerabilities.go` following the conventions in `server/internal/api/handlers.go` — `auth.InstanceID(c)` for scoping, `actorFromCtx(c)` for the audit actor: - -- `listVulnerabilities` — `GET /api/vulnerabilities`, query params `severity`, `state`, `server`, plus tag filters. **Groups by CVE** in the response: `{cve_id, severity, title, server_count, servers: [...]}`. -- `vulnerabilitySummary` — `GET /api/vulnerabilities/summary`: severity counts for open findings, plus `{db_version, pulled_at, last_error}` from `vulndb_meta`. -- `rescanVulnerabilities` — `POST /api/vulnerabilities/rescan`: sets `scan_pending: true` on every `server_packages` document for the instance. Audit `vuln.rescan`. -- `acceptFinding` — `POST /api/vulnerabilities/:id/accept`, body `{reason string, until time}`. **Reject an empty reason and a past `until` with 400.** Sets state `accepted` with the `Acceptance` block. Audit `vuln.accepted` with reason and expiry in the details. -- `unacceptFinding` — `DELETE /api/vulnerabilities/:id/accept`: back to `open`. Audit `vuln.unaccepted`. -- `listServerVulnerabilities` — `GET /api/servers/:id/vulnerabilities`. -- `getServerPackages` — `GET /api/servers/:id/packages`. -- `searchPackages` — `GET /api/packages/search?name=`, calls `services.SearchPackages`. -- Alert rule CRUD — `GET,POST /api/vuln-rules`, `PUT,DELETE /api/vuln-rules/:id`. - -- [ ] **Step 4: Register the routes** - -In `server/internal/api/handlers.go`, inside the `apiGroup` block: - -```go - apiGroup.GET("/vulnerabilities", listVulnerabilities) - apiGroup.GET("/vulnerabilities/summary", vulnerabilitySummary) - apiGroup.POST("/vulnerabilities/rescan", auth.RequireRole("owner", "admin"), rescanVulnerabilities) - apiGroup.POST("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), acceptFinding) - apiGroup.DELETE("/vulnerabilities/:id/accept", auth.RequireRole("owner", "admin"), unacceptFinding) - apiGroup.GET("/servers/:id/vulnerabilities", listServerVulnerabilities) - apiGroup.GET("/servers/:id/packages", getServerPackages) - apiGroup.GET("/packages/search", searchPackages) - apiGroup.GET("/vuln-rules", listVulnRules) - apiGroup.POST("/vuln-rules", auth.RequireRole("owner", "admin"), createVulnRule) - apiGroup.PUT("/vuln-rules/:id", auth.RequireRole("owner", "admin"), updateVulnRule) - apiGroup.DELETE("/vuln-rules/:id", auth.RequireRole("owner", "admin"), deleteVulnRule) -``` - -**Check `auth.RequireRole`'s actual variadic signature** before using it with two arguments. - -- [ ] **Step 5: Verify by hand** - -With a dev instance running, confirm each: an accept with an empty reason returns 400; an accept with a past `until` returns 400; a member session gets 403 on accept and 200 on the list; a rescan sets `scan_pending` (check in Mongo). - -- [ ] **Step 6: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/api/ shared/models/settings.go server/internal/services/findings.go -git commit -m "feat: vulnerability REST API, retention setting and sweeper" -``` - ---- - -### Task 16: Web UI - -**Files:** -- Create: `web/app/(app)/vulnerabilities/page.tsx`, `web/components/vulnerabilities/{FindingRow,AcceptDialog,DBFreshness}.tsx`, `web/components/settings/VulnAlertRulesCard.tsx` -- Modify: `web/lib/api.ts`, `web/components/Sidebar.tsx`, `web/app/(app)/servers/[id]/page.tsx`, `web/app/(app)/settings/notifications/page.tsx` - -- [ ] **Step 1: Add API client types and methods** - -In `web/lib/api.ts`, add `VulnFinding`, `VulnGroup`, `VulnSummary`, `ServerPackages` and `VulnAlertRule` types mirroring the Go JSON tags exactly, plus one client method per Task 15 route. Follow the file's existing conventions. - -- [ ] **Step 2: Build the fleet board** - -Create `web/app/(app)/vulnerabilities/page.tsx`. **Grouped by CVE, one row per CVE with an affected-server count, expandable to the servers.** The same CVE across 40 servers is one decision; a flat list makes it look like forty. - -Requirements: -- Filter bar: severity, state (default `open`), tag filter following `TagFilterBar`'s pattern from the servers page. -- `DBFreshness` banner at the top, always visible, showing `pulled_at` age and `last_error` when set. A fleet scanning against a three-week-old database must say so rather than quietly report all-clear. -- A server whose `status` is `unsupported` renders as **"unsupported"**, never as zero findings. -- Every severity pill carries a distinct shape and a text label — state never reads by colour alone, matching the existing monitor pills. -- Findings with no `fixed_in` show "no fix published" rather than an empty cell. -- Tailwind token names only. No hex values. - -- [ ] **Step 3: Build the accept dialog** - -Create `web/components/vulnerabilities/AcceptDialog.tsx`: required reason textarea, required `until` date defaulting to 30 days out, and copy stating the finding reopens automatically on that date. - -- [ ] **Step 4: Add the remediation action** - -On a finding with `fixed_in`, render an **Apply updates** button calling the existing `POST /api/servers/:id/apply-updates`. No new endpoint and no new mechanism. - -- [ ] **Step 5: Add the tabs and nav** - -Add **Vulnerabilities** and **Packages** tabs to `web/app/(app)/servers/[id]/page.tsx`, following the existing tab pattern. Add a **Vulnerabilities** entry to `web/components/Sidebar.tsx`. Mount `VulnAlertRulesCard` on `web/app/(app)/settings/notifications/page.tsx` beside the channels. - -- [ ] **Step 6: Verify** - -Run: `cd web && npm run lint && npm run build` -Expected: exit 0, no type errors. - -Then load the pages against a dev instance and confirm: the CVE grouping expands, the freshness banner shows an age, an unsupported server says so, and the accept dialog rejects an empty reason. - -- [ ] **Step 7: Commit** - -```bash -git add web/ -git commit -m "feat: vulnerability findings UI" -``` - ---- - -### Task 17: Instance deletion, entitlement and documentation - -**Files:** -- Modify: the control plane's instance-deletion collection list, `admin/internal/models/entitlements.go`, `CLAUDE.md`, `docsite/docs/vantage/` - -- [ ] **Step 1: Add the new collections to instance deletion** - -Find the instance-deletion routine — the one place that knows which collections carry `instance_id`: - -```bash -cd server && grep -rn "instance_id" internal/services/ | grep -i "delete\|purge\|reap" -``` - -Add `server_packages` and `vuln_findings` to its collection list. **Missing this orphans a tenant's package data indefinitely**, and it is the easiest thing in this feature to forget. - -- [ ] **Step 2: Add the admin entitlement toggle** - -In `admin/internal/models/entitlements.go`, add `vuln_scanning` to the feature toggles, following the pattern the existing per-instance toggles use. It must be **off for both Free plans**. Confirm the licence issue path copies it into `License.Features`. - -- [ ] **Step 3: Verify the whole tree builds** - -From the repo root: - -```bash -go build ./... && go vet ./... -``` - -Expected: exit 0. - -- [ ] **Step 4: End-to-end check** - -Against a dev instance with one Linux agent: enable `vuln_scanning`, wait for the hourly report (or restart the agent to force one), confirm a `server_packages` document appears with `scan_pending: true`, and that within ~60s `vulnsched` clears the flag and writes findings. Confirm a second agent report with an unchanged package set sends no body. - -- [ ] **Step 5: Document it** - -Add a **Package inventory and CVE findings** subsystem section to `CLAUDE.md` covering: the backport trap and why matching uses distro feeds; why only the leader matches; that findings go `fixed` rather than being deleted; the two new environment variables. Add a user-facing page under `docsite/docs/vantage/`. - -While in `CLAUDE.md`, correct the `shared/mail/render_test.go` claim — no such file exists, and this plan does not create one. - -- [ ] **Step 6: Commit** - -```bash -git add -A -git commit -m "docs: document package inventory and CVE findings" -``` - ---- - -## Spec coverage - -- Backport trap → Tasks 1 and 11, with the equal-version case in both manual checks. -- trivy-db as the source, ephemeral storage, `VANTAGE_TRIVY_DB_REF` → Task 10. -- Leader-only matching, `scan_pending` persisted, tick-boundary batching → Tasks 8 and 13. -- Hash short-circuit wire path → Tasks 6 and 7. -- All four collections and their indexes → Task 5. -- `SourceName` covering every binary of a source package → Tasks 4 and 11. -- Unsupported distro never reads as clean → Tasks 2, 13 and 16. -- Severity vendor → NVD → unknown → Task 10 (`severityName`). -- Findings lifecycle with acceptance expiry, fixed-before-reopen ordering → Task 12. -- Alert rules through `ResolveTargets`, one digest per tick → Task 14. -- Entitlement gated at collection, `HasFeature` not tier → Tasks 8, 9 and 17. -- REST API, retention, sweeper → Task 15. -- CVE-grouped board, DB freshness on screen, remediation via the existing endpoint → Task 16. -- Instance-deletion collection list → Task 17. - -## Known gaps the executing engineer must close - -- **`vulndb.Pull` ships as a stub** with implementation requirements rather than working oras code (Task 10). It is the one piece that cannot be written blind: the exact `oras-go` v2 call shape depends on the resolved version. Verify it by hand (Task 10, Step 3) before Task 13. -- **Six call sites are written against signatures that must be confirmed first**, each flagged at the point of use: the licence accessor (Task 8), `ValidateAgentToken`'s return shape (Task 8), `ResolveTargets` (Task 14), `auth.RequireRole`'s variadic form (Task 15), the `trivy-db` API names (Task 10), and `shared/mail`'s render conventions (Task 14). -- **No automated tests exist for any of this**, by instruction. The manual verification tables in Tasks 1, 11 and 12 are the substitute and are the difference between shipping this and shipping something that reports a patched fleet as vulnerable, or worse, the reverse. Do not skip them. diff --git a/docs/superpowers/plans/2026-08-06-workload-registry.md b/docs/superpowers/plans/2026-08-06-workload-registry.md deleted file mode 100644 index 0785cec..0000000 --- a/docs/superpowers/plans/2026-08-06-workload-registry.md +++ /dev/null @@ -1,1489 +0,0 @@ -# Workload Registry Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Agents enumerate what each Linux server actually runs — Docker containers, the compose stacks grouping them, and systemd services — and report it to the control plane. Containers and units can be started, stopped and restarted from the UI, and a bounded snapshot of their logs read without opening a console. - -**Architecture:** A **workload** is one container or one systemd unit. The agent collects both on a 60-second ticker and reports through a `ReportWorkloads` RPC with a hash short-circuit. On-demand refresh does not return data — `RefreshWorkloadsCmd` makes the agent report immediately through that same RPC, so there is one writer for the collection. Control actions and log reads travel out through the existing `commandDispatcher` and answer back over the bus, mirroring `StepResults`. - -**Tech Stack:** Go 1.26, gin, mongo-driver v2, gRPC, Redis (bus), Next.js 16 + TanStack Query, Tailwind 3. - -**Spec:** `docs/superpowers/specs/2026-08-06-workload-registry-design.md`. Read it before starting. Where this plan and the spec disagree, the spec wins and the plan is wrong. - -## Orientation for a fresh session - -This is sub-project B of four. Sub-project A (package inventory and CVE findings) is independent and may or may not be built yet — **this plan shares no code with it** and can be executed first, second, or alone. If A exists, its `ServerPackages` model is the shape `ServerWorkloads` deliberately mirrors. - -Key existing machinery this plan builds on, with the reasoning that must not be broken: - -- `server/internal/services/dispatch.go` — `commandDispatcher.send()` publishes a command envelope addressed to the pod holding that agent's stream and waits for its ack. **Request/ack, not a queue:** a command whose owner died must fail loudly (503) rather than sit in a queue while the operator is told it worked. -- `server/internal/services/stepresults.go` — `StepResults.Await/Deliver`. `Await` subscribes to the result channel **before** the command is dispatched. Copy that ordering exactly; a fast agent otherwise answers into a channel nobody has joined. -- `agent/internal/sync/sync.go` — the command stream handler, with a `handleX` function per `ServerCommand` variant. -- `bus.RunAsLeader("housekeeping", …)` in `server/cmd/main.go` — nothing in this plan needs it. There is no scheduler here; the agent drives its own ticker. - -## Global Constraints - -- **Do not write tests.** No `*_test.go`, no npm test files, no test scaffolding. This is a deliberate instruction from the repository owner, not an oversight. Verification is `go build`, `go vet`, `npm run build` and the manual checks written into the tasks. -- **Linux only.** Windows agents must not collect workloads. systemd does not exist there and the container story differs; a separate spec covers it if ever. -- **Not gated by licence.** v1 ships to every instance. Do not add a `HasFeature` check. -- Every service query is scoped by `instance_id`. No unscoped lookups, ever. -- Every mutating API path writes an audit event via `services.LogEvent(instanceID, eventType, actor, serverID, keyID, details string)`. -- Module path prefix is `gitea.hostxtra.co.uk/mrhid6/vantage/`. -- Mongo driver is **v2**: `go.mongodb.org/mongo-driver/v2/bson`; ObjectIDs are `bson.ObjectID`, not `primitive.ObjectID`. -- Collections are reached via `db.Col("name")`. -- No component in `web/` may carry a hex colour; use Tailwind token names only. `web/` is dark-only — do not add a light theme. -- Commit messages use the `feat:` / `fix:` / `docs:` prefixes already in `git log`. -- No new Go dependencies. Everything here shells out to binaries already on the host. - -## Correctness risks carried without tests - -Places where a mistake produces a **wrong answer or a silent cost rather than a crash**. Each has a manual check written into its task. - -1. **The protected set (Task 4).** Wrong in the permissive direction and a server can stop its own agent — it goes offline and the only way back is SSH or physical access, which is what this feature exists to avoid needing. -2. **Hash order-independence (Task 3).** `docker ps` output ordering is not stable. An ordering-sensitive hash resends the full list every 60 seconds forever, visible only as traffic. -3. **Log caps (Task 5).** A line count alone does not bound size. 500 lines of 4KB JSON is 2MB through the bus. -4. **`DockerOK` vs empty list (Tasks 2 and 9).** "Docker not installed" and "Docker running nothing" must not render alike. - ---- - -## File Structure - -**Create — agent:** -- `agent/internal/workloads/workloads.go` — `Collect()`, orchestrating both collectors -- `agent/internal/workloads/docker.go` — Docker collection and parsing -- `agent/internal/workloads/systemd.go` — systemd collection and parsing -- `agent/internal/workloads/control.go` — start/stop/restart, and the protected set -- `agent/internal/workloads/logs.go` — bounded log reads - -**Create — server:** -- `server/internal/models/workloads.go` — `ServerWorkloads`, `Workload` -- `server/internal/services/workloads.go` — storage, hash compare, fleet search, dispatch wrappers -- `server/internal/services/workloadresults.go` — `WorkloadResults` registry over the bus -- `server/internal/api/workloads.go` — REST handlers - -**Create — web:** -- `web/app/(app)/workloads/page.tsx` — fleet-wide view and search -- `web/components/workloads/WorkloadList.tsx` — grouped stacks, containers, units -- `web/components/workloads/WorkloadRow.tsx` — one row with its actions -- `web/components/workloads/LogDialog.tsx` — bounded log snapshot - -**Modify:** -- `proto/vantage/v1/vantage.proto` — `ReportWorkloads` RPC, three `ServerCommand` variants, one `AgentMessage` variant -- `agent/internal/sync/sync.go` — the 60s report loop and three command handlers -- `agent/internal/grpc/client.go` — `ReportWorkloads` client method -- `server/internal/grpc/server.go` — `ReportWorkloads` handler; route `WorkloadLogsResult` in `CommandStream` -- `server/internal/services/coreindexes.go` — workload indexes -- `server/internal/api/handlers.go` — register routes -- `web/lib/api.ts` — types and client methods -- `web/components/Sidebar.tsx` — Workloads entry -- `web/app/(app)/servers/[id]/page.tsx` — Workloads tab -- The control plane's instance-deletion collection list — add `server_workloads` -- `CLAUDE.md`, `docsite/docs/vantage/` — document the subsystem - ---- - -### Task 1: Models and indexes - -**Files:** -- Create: `server/internal/models/workloads.go` -- Modify: `server/internal/services/coreindexes.go`, `server/cmd/main.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: `models.ServerWorkloads`, `models.Workload`, `services.EnsureWorkloadIndexes() error`. - -- [ ] **Step 1: Write the models** - -Create `server/internal/models/workloads.go`: - -```go -package models - -import ( - "time" - - "go.mongodb.org/mongo-driver/v2/bson" -) - -// Workload kinds. -const ( - WorkloadContainer = "container" - WorkloadUnit = "unit" -) - -// Control actions. -const ( - WorkloadStart = "start" - WorkloadStop = "stop" - WorkloadRestart = "restart" -) - -// Workload is one container or one systemd unit. -type Workload struct { - Kind string `bson:"kind" json:"kind"` // container | unit - ID string `bson:"id" json:"id"` // container id, or unit name - Name string `bson:"name" json:"name"` - - // State is deliberately NOT collapsed into a shared vocabulary across the - // two kinds. Containers report running/exited/paused/restarting/created; - // units report active/inactive/failed/activating. A failed unit and an - // exited container mean different things, and flattening them loses the - // distinction the operator needs. - State string `bson:"state" json:"state"` - Health string `bson:"health,omitempty" json:"health,omitempty"` - - Image string `bson:"image,omitempty" json:"image,omitempty"` - Stack string `bson:"stack,omitempty" json:"stack,omitempty"` // compose project label - Ports []string `bson:"ports,omitempty" json:"ports,omitempty"` - - Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"` - StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"` - - // Protected is computed agent-side and reported so the UI can render the - // action disabled with a reason rather than offering a button whose refusal - // is already known. The field is the courtesy; the agent's own check is the - // boundary. - Protected bool `bson:"protected" json:"protected"` -} - -// ServerWorkloads holds one server's whole workload list in ONE document. -type ServerWorkloads struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"-"` - InstanceID string `bson:"instance_id" json:"-"` - ServerID string `bson:"server_id" json:"server_id"` - Hash string `bson:"hash" json:"hash"` - Workloads []Workload `bson:"workloads" json:"workloads"` - CollectedAt time.Time `bson:"collected_at" json:"collected_at"` - - // A host with no Docker and a host with Docker running nothing both produce - // an empty list. One should read "not in use here", the other "nothing - // running", and only the second deserves any alarm. - // - // The error strings separate a third case the booleans cannot: installed - // with the daemon down. "Not installed" and "installed but not responding" - // are different problems with different fixes. - DockerOK bool `bson:"docker_ok" json:"docker_ok"` - DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"` - SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"` - SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"` -} -``` - -- [ ] **Step 2: Add the index builder** - -Append to `server/internal/services/coreindexes.go` (or create `server/internal/services/workloadindexes.go` if that file's conventions do not fit): - -```go -// EnsureWorkloadIndexes declares the indexes for the workload registry. -// -// Warns rather than being fatal, matching EnsureSecretIndexes: a missing index -// degrades these queries to a collection scan, which is no reason to refuse to -// serve the fleet. -func EnsureWorkloadIndexes() error { - ctx := context.Background() - - idx := []mongo.IndexModel{ - { - Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "server_id", Value: 1}}, - Options: options.Index().SetUnique(true), - }, - // Multikey, for the fleet-wide "which servers run image X" query, which - // is the reason the snapshot is stored rather than fetched and discarded. - {Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "workloads.image", Value: 1}}}, - } - if _, err := db.Col("server_workloads").Indexes().CreateMany(ctx, idx); err != nil { - log.Printf("warning: server_workloads indexes: %v", err) - } - return nil -} -``` - -- [ ] **Step 3: Wire it into schema setup** - -In `server/cmd/main.go`, find `runSchemaSetup` (near line 75) and the block calling the other `Ensure*Indexes` functions. Add alongside the non-fatal ones: - -```go - if err := services.EnsureWorkloadIndexes(); err != nil { - log.Printf("warning: workload indexes: %v", err) - } -``` - -- [ ] **Step 4: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/models/workloads.go server/internal/services/ server/cmd/main.go -git commit -m "feat: models and indexes for the workload registry" -``` - ---- - -### Task 2: Docker collector - -**Files:** -- Create: `agent/internal/workloads/docker.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: `workloads.Workload` (the agent-side struct), `workloads.collectDocker() ([]Workload, bool, string)` returning workloads, ok, and an error string. - -- [ ] **Step 1: Write the collector** - -Create `agent/internal/workloads/docker.go`: - -```go -package workloads - -import ( - "context" - "encoding/json" - "os/exec" - "strings" - "time" -) - -// Workload is one container or one systemd unit, agent-side. It mirrors -// models.Workload on the server. -type Workload struct { - Kind string - ID string - Name string - State string - Health string - Image string - Stack string - Ports []string - Restarts int - StartedAt time.Time - Protected bool -} - -const dockerTimeout = 30 * time.Second - -// dockerInspect is the subset of `docker inspect` output we read. -// -// We use inspect rather than `docker ps --format '{{json .}}'` because ps -// reports health and uptime inside a human Status string — "Up 2 hours -// (healthy)" — and anything built on that is parsing English that is -// localised, reworded between releases, and silently different for a paused or -// restarting container. inspect gives typed fields instead. -type dockerInspect struct { - ID string `json:"Id"` - Name string `json:"Name"` - State struct { - Status string `json:"Status"` - StartedAt string `json:"StartedAt"` - Restarting bool `json:"Restarting"` - Health *struct { - Status string `json:"Status"` - } `json:"Health"` - } `json:"State"` - Config struct { - Image string `json:"Image"` - Labels map[string]string `json:"Labels"` - } `json:"Config"` - RestartCount int `json:"RestartCount"` - NetworkSettings struct { - Ports map[string][]struct { - HostIP string `json:"HostIp"` - HostPort string `json:"HostPort"` - } `json:"Ports"` - } `json:"NetworkSettings"` -} - -// collectDocker enumerates containers. It returns ok=false with an empty error -// string when Docker is simply not installed — the common case on this fleet, -// and not a fault. -func collectDocker(ctx context.Context) ([]Workload, bool, string) { - if _, err := exec.LookPath("docker"); err != nil { - return nil, false, "" // not installed; not an error - } - - ctx, cancel := context.WithTimeout(ctx, dockerTimeout) - defer cancel() - - idsOut, err := exec.CommandContext(ctx, "docker", "ps", "-aq").Output() - if err != nil { - // Installed but not answering: a different problem with a different - // fix, so it carries a message where "not installed" does not. - return nil, false, "docker ps failed: " + errText(err) - } - - ids := strings.Fields(string(idsOut)) - if len(ids) == 0 { - return []Workload{}, true, "" // Docker present, nothing running - } - - args := append([]string{"inspect", "--format", "{{json .}}"}, ids...) - out, err := exec.CommandContext(ctx, "docker", args...).Output() - if err != nil { - return nil, false, "docker inspect failed: " + errText(err) - } - - var wls []Workload - for _, line := range strings.Split(string(out), "\n") { - line = strings.TrimSpace(line) - if line == "" { - continue - } - var di dockerInspect - if err := json.Unmarshal([]byte(line), &di); err != nil { - continue - } - wls = append(wls, dockerToWorkload(di)) - } - return wls, true, "" -} - -func dockerToWorkload(di dockerInspect) Workload { - w := Workload{ - Kind: "container", - ID: di.ID, - Name: strings.TrimPrefix(di.Name, "/"), - State: di.State.Status, - Image: di.Config.Image, - Restarts: di.RestartCount, - } - if di.State.Health != nil { - w.Health = strings.ToLower(di.State.Health.Status) - } - // The compose project label is what Docker itself treats as authoritative. - // No YAML is read from disk: a compose file there may not be what is running. - if v := di.Config.Labels["com.docker.compose.project"]; v != "" { - w.Stack = v - } - if t, err := time.Parse(time.RFC3339Nano, di.State.StartedAt); err == nil { - w.StartedAt = t - } - for container, bindings := range di.NetworkSettings.Ports { - for _, b := range bindings { - w.Ports = append(w.Ports, b.HostIP+":"+b.HostPort+"->"+container) - } - } - return w -} - -func errText(err error) string { - if ee, ok := err.(*exec.ExitError); ok && len(ee.Stderr) > 0 { - return strings.TrimSpace(string(ee.Stderr)) - } - return err.Error() -} -``` - -- [ ] **Step 2: Verify against a real Docker host** - -Confirm the field names by eye before trusting the struct tags: - -```bash -docker run -d --name plantest --restart=always nginx:alpine -docker inspect --format '{{json .}}' plantest | python3 -m json.tool | head -60 -docker rm -f plantest -``` - -Check that `Id`, `Name`, `State.Status`, `State.StartedAt`, `RestartCount`, `Config.Image`, `Config.Labels` and `NetworkSettings.Ports` all appear with those exact spellings and shapes. If you have a compose project handy, confirm `com.docker.compose.project` is present on its containers. - -- [ ] **Step 3: Build and commit** - -Run: `cd agent && go build ./... && go vet ./...` - -```bash -git add agent/internal/workloads/docker.go -git commit -m "feat: agent enumerates docker containers" -``` - ---- - -### Task 3: systemd collector and hashing - -**Files:** -- Create: `agent/internal/workloads/systemd.go`, `agent/internal/workloads/workloads.go` - -**Interfaces:** -- Consumes: `workloads.Workload`, `workloads.collectDocker` (Task 2). -- Produces: `workloads.Collect(ctx) Result`, `workloads.Hash([]Workload) string`, `workloads.Result{Workloads, DockerOK, DockerError, SystemdOK, SystemdError}`. - -- [ ] **Step 1: Write the systemd collector** - -Create `agent/internal/workloads/systemd.go`: - -```go -package workloads - -import ( - "context" - "os/exec" - "strings" - "time" -) - -const systemdTimeout = 30 * time.Second - -// excludedPrefixes drops the platform's own units. A typical host carries 300+ -// units and systemd accounts for most of them; listing all of them buries the -// ten anyone cares about. -var excludedPrefixes = []string{"systemd-", "user@", "user-", "session-", "init.scope"} - -// collectSystemd enumerates services in two passes, because "running or -// failed" and "enabled but stopped" are different questions — and an enabled -// unit that is not running is exactly the one worth seeing. -func collectSystemd(ctx context.Context) ([]Workload, bool, string) { - if _, err := exec.LookPath("systemctl"); err != nil { - return nil, false, "" - } - - ctx, cancel := context.WithTimeout(ctx, systemdTimeout) - defer cancel() - - // Column output rather than --output=json: the JSON flag needs systemd - // 246+, and this fleet includes older stable distributions. The columns - // have been stable considerably longer than the JSON has existed. - unitsOut, err := exec.CommandContext(ctx, "systemctl", - "list-units", "--type=service", "--state=running,failed", - "--no-legend", "--plain", "--no-pager").Output() - if err != nil { - return nil, false, "systemctl list-units failed: " + errText(err) - } - - seen := map[string]bool{} - var wls []Workload - - for _, line := range strings.Split(string(unitsOut), "\n") { - f := strings.Fields(line) - // UNIT LOAD ACTIVE SUB DESCRIPTION… - if len(f) < 4 { - continue - } - name := f[0] - if excluded(name) || seen[name] { - continue - } - seen[name] = true - wls = append(wls, Workload{ - Kind: "unit", - ID: name, - Name: strings.TrimSuffix(name, ".service"), - State: f[2], // ACTIVE: active | failed | activating | inactive - }) - } - - filesOut, err := exec.CommandContext(ctx, "systemctl", - "list-unit-files", "--type=service", "--state=enabled", - "--no-legend", "--plain", "--no-pager").Output() - if err == nil { - for _, line := range strings.Split(string(filesOut), "\n") { - f := strings.Fields(line) - // UNIT FILE STATE [PRESET] - if len(f) < 2 { - continue - } - name := f[0] - if excluded(name) || seen[name] { - continue - } - seen[name] = true - wls = append(wls, Workload{ - Kind: "unit", - ID: name, - Name: strings.TrimSuffix(name, ".service"), - State: "inactive", // enabled but not currently running - }) - } - } - - return wls, true, "" -} - -func excluded(name string) bool { - for _, p := range excludedPrefixes { - if strings.HasPrefix(name, p) { - return true - } - } - return false -} -``` - -- [ ] **Step 2: Write the orchestrator and hash** - -Create `agent/internal/workloads/workloads.go`: - -```go -package workloads - -import ( - "context" - "crypto/sha256" - "encoding/hex" - "runtime" - "sort" - "strconv" -) - -// Result is one collection pass. -type Result struct { - Workloads []Workload - DockerOK bool - DockerError string - SystemdOK bool - SystemdError string -} - -// Collect enumerates every workload on this host. Linux only. -func Collect(ctx context.Context) Result { - if runtime.GOOS != "linux" { - return Result{} - } - - var r Result - containers, dockerOK, dockerErr := collectDocker(ctx) - units, systemdOK, systemdErr := collectSystemd(ctx) - - r.DockerOK, r.DockerError = dockerOK, dockerErr - r.SystemdOK, r.SystemdError = systemdOK, systemdErr - r.Workloads = append(append([]Workload{}, containers...), units...) - - markProtected(r.Workloads) - return r -} - -// Hash fingerprints a workload set so an unchanged set never has to be sent. -// -// It sorts first: `docker ps` output ordering is not stable, and an -// ordering-sensitive hash would resend the full list every 60 seconds forever -// — a cost visible only as traffic. -// -// StartedAt is deliberately excluded: it does not change while a container -// runs, and including it would add nothing. Restarts IS included, because a -// container cycling is exactly the change worth reporting. -func Hash(wls []Workload) string { - lines := make([]string, 0, len(wls)) - for _, w := range wls { - lines = append(lines, strings.Join([]string{ - w.Kind, w.ID, w.Name, w.State, w.Health, w.Image, w.Stack, - strconv.Itoa(w.Restarts), - }, "\x00")) - } - sort.Strings(lines) - h := sha256.New() - for _, l := range lines { - h.Write([]byte(l)) - h.Write([]byte("\n")) - } - return hex.EncodeToString(h.Sum(nil)) -} -``` - -Add `"strings"` to the import block. `markProtected` arrives in Task 4 — add a temporary empty `func markProtected(_ []Workload) {}` to this file now so it builds, and delete it when Task 4 lands. - -- [ ] **Step 3: Verify the systemd parsing and the hash** - -On a Linux host, confirm the column positions the parser assumes: - -```bash -systemctl list-units --type=service --state=running,failed --no-legend --plain --no-pager | head -5 -systemctl list-unit-files --type=service --state=enabled --no-legend --plain --no-pager | head -5 -``` - -The first must have the unit name in column 1 and `active`/`failed` in column 3. The second must have the unit name in column 1. Confirm `systemd-journald.service` appears in the raw output and would be dropped by `excluded`, while `sshd.service` or `nginx.service` survives. - -For the hash, confirm by inspection that two slices holding the same workloads in different order produce the same string, and that changing one `Restarts` value changes it. - -- [ ] **Step 4: Build and commit** - -Run: `cd agent && go build ./... && go vet ./...` - -```bash -git add agent/internal/workloads/ -git commit -m "feat: agent enumerates systemd services" -``` - ---- - -### Task 4: Control actions and the protected set - -**Files:** -- Create: `agent/internal/workloads/control.go` -- Modify: `agent/internal/workloads/workloads.go` — remove the temporary `markProtected` stub - -**Interfaces:** -- Consumes: `workloads.Workload`. -- Produces: `workloads.Control(ctx, kind, id, action string) error`, `workloads.ErrProtected`, `workloads.markProtected([]Workload)`. - -**This is the task with the unrecoverable failure mode.** A server that stops its own agent goes offline, and the only way back is SSH or physical access — which is what this feature exists to avoid needing. - -- [ ] **Step 1: Write the control layer** - -Create `agent/internal/workloads/control.go`: - -```go -package workloads - -import ( - "context" - "errors" - "fmt" - "os" - "os/exec" - "regexp" - "strings" - "time" -) - -// ErrProtected is returned for a workload the agent will not act on. -var ErrProtected = errors.New("workload is protected") - -// AgentUnit is the systemd unit this agent runs as. -const AgentUnit = "vantage-agent.service" - -// controlTimeout bounds a stop that may never finish on its own. `docker stop` -// waits on a container that may ignore SIGTERM, and `systemctl stop` on a unit -// with a long TimeoutStopSec blocks for exactly as long as that says. A -// timeout must return a real error rather than an ack implying success. -const controlTimeout = 90 * time.Second - -// ownContainerID is read once: the container this agent runs in, if any. -var ownContainerID = detectOwnContainer() - -var cgroupContainerRe = regexp.MustCompile(`[0-9a-f]{64}`) - -// detectOwnContainer returns this process's container ID, or "" on a host -// install. The agent is normally a systemd service, so "" is the common case; -// this exists so containerising it later cannot silently remove the guard. -func detectOwnContainer() string { - b, err := os.ReadFile("/proc/self/cgroup") - if err != nil { - return "" - } - if m := cgroupContainerRe.FindString(string(b)); m != "" { - return m - } - return "" -} - -// isProtected reports whether the agent refuses to act on this workload. -// -// The refusal lives here, in the agent, and not in the control plane. As with -// the console relay hardcoding 127.0.0.1 agent-side: the control plane may name -// a target, but the agent decides what it will do to itself. A server-side -// denylist alone would be bypassed by the next dispatch path someone adds. -func isProtected(kind, id, name string) bool { - if kind == "unit" { - return id == AgentUnit || name == strings.TrimSuffix(AgentUnit, ".service") - } - if ownContainerID == "" { - return false - } - // Container IDs are commonly abbreviated to 12 characters; compare on the - // shorter of the two so a short id still matches a full one. - return strings.HasPrefix(ownContainerID, id) || strings.HasPrefix(id, ownContainerID) -} - -// markProtected stamps the flag onto a collected list so the UI can render the -// action disabled with a reason. -func markProtected(wls []Workload) { - for i := range wls { - wls[i].Protected = isProtected(wls[i].Kind, wls[i].ID, wls[i].Name) - } -} - -// Control starts, stops or restarts a workload. -func Control(ctx context.Context, kind, id, action string) error { - switch action { - case "start", "stop", "restart": - default: - return fmt.Errorf("unknown action %q", action) - } - - // Checked before anything else happens, and checked here rather than only - // on the server. See isProtected. - if isProtected(kind, id, strings.TrimSuffix(id, ".service")) { - return fmt.Errorf("%w: %s", ErrProtected, id) - } - - ctx, cancel := context.WithTimeout(ctx, controlTimeout) - defer cancel() - - var cmd *exec.Cmd - switch kind { - case "container": - cmd = exec.CommandContext(ctx, "docker", action, id) - case "unit": - cmd = exec.CommandContext(ctx, "systemctl", action, id) - default: - return fmt.Errorf("unknown workload kind %q", kind) - } - - if out, err := cmd.CombinedOutput(); err != nil { - if ctx.Err() == context.DeadlineExceeded { - return fmt.Errorf("%s %s timed out after %s", action, id, controlTimeout) - } - return fmt.Errorf("%s %s: %s", action, id, strings.TrimSpace(string(out))) - } - return nil -} -``` - -Delete the temporary `markProtected` stub from `workloads.go`. - -- [ ] **Step 2: Verify the guard** - -**Do not skip this.** From a scratch `main.go` on a dev machine, confirm: - -| Call | Expected | -| ---- | -------- | -| `Control(ctx, "unit", "vantage-agent.service", "restart")` | error wrapping `ErrProtected`, and **no restart happens** | -| `Control(ctx, "unit", "vantage-agent", "restart")` | error wrapping `ErrProtected` — the name form must be caught too | -| `Control(ctx, "unit", "nginx.service", "restart")` | proceeds | -| `Control(ctx, "container", "", "restart")` | proceeds | -| `Control(ctx, "unit", "nginx.service", "reload")` | error, unknown action | - -Then run `markProtected` over a collected list and confirm `vantage-agent.service` comes back `Protected: true` and everything else `false`. - -- [ ] **Step 3: Build and commit** - -Run: `cd agent && go build ./... && go vet ./...` - -```bash -git add agent/internal/workloads/ -git commit -m "feat: agent control actions with self-protection" -``` - ---- - -### Task 5: Bounded log reads - -**Files:** -- Create: `agent/internal/workloads/logs.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: `workloads.Logs(ctx, kind, id string, tail int) (text string, truncated bool, err error)`, `workloads.MaxLogLines`, `workloads.MaxLogBytes`. - -- [ ] **Step 1: Write the log reader** - -Create `agent/internal/workloads/logs.go`: - -```go -package workloads - -import ( - "context" - "fmt" - "os/exec" - "strconv" - "strings" - "time" -) - -const ( - // MaxLogLines and MaxLogBytes are BOTH enforced, whichever binds first. - // - // A line count alone does not bound size: 500 lines of a container printing - // 4KB JSON blobs is 2MB travelling over the bus. This is the same reasoning - // that gave workflow logs a per-line cap as well as a per-run one. - MaxLogLines = 500 - MaxLogBytes = 256 * 1024 - - logTimeout = 60 * time.Second -) - -// Logs returns a bounded snapshot of a workload's recent output. -// -// There is no follow mode. The browser console already offers a real terminal -// on the same server where `docker logs -f` works properly, with its own -// scrollback and cancellation. A snapshot answers "why did this restart", -// which is the question that sends people to the console in the first place. -func Logs(ctx context.Context, kind, id string, tail int) (string, bool, error) { - if tail <= 0 || tail > MaxLogLines { - tail = MaxLogLines - } - - ctx, cancel := context.WithTimeout(ctx, logTimeout) - defer cancel() - - var cmd *exec.Cmd - switch kind { - case "container": - cmd = exec.CommandContext(ctx, "docker", "logs", - "--tail", strconv.Itoa(tail), "--timestamps", id) - case "unit": - cmd = exec.CommandContext(ctx, "journalctl", "-u", id, - "-n", strconv.Itoa(tail), "--no-pager", "--output=short-iso") - default: - return "", false, fmt.Errorf("unknown workload kind %q", kind) - } - - // docker logs writes container stderr to our stderr, so both streams must - // be captured or half the output silently disappears. - out, err := cmd.CombinedOutput() - if err != nil && len(out) == 0 { - return "", false, fmt.Errorf("read logs for %s: %s", id, errText(err)) - } - - return cap(string(out)) -} - -// cap enforces both limits, trimming from the FRONT: the most recent lines are -// the ones worth keeping. -func cap(s string) (string, bool, error) { - truncated := false - - lines := strings.Split(s, "\n") - if len(lines) > MaxLogLines { - lines = lines[len(lines)-MaxLogLines:] - truncated = true - } - s = strings.Join(lines, "\n") - - if len(s) > MaxLogBytes { - s = s[len(s)-MaxLogBytes:] - // Drop the leading partial line left by a byte-wise cut. - if i := strings.IndexByte(s, '\n'); i >= 0 { - s = s[i+1:] - } - truncated = true - } - - return s, truncated, nil -} -``` - -`cap` shadows the builtin. Rename it to `capLog` if that bothers the linter; the behaviour is what matters. - -- [ ] **Step 2: Verify both caps** - -Both directions, because a line-count-only implementation passes the first and fails the second silently: - -```bash -# Line cap: 600 lines in, expect 500 out and truncated=true -docker run -d --name captest alpine sh -c 'for i in $(seq 1 600); do echo line-$i; done; sleep 3600' - -# Byte cap: few lines, each large. Expect truncated=true well under 500 lines. -docker run -d --name bigtest alpine sh -c 'for i in $(seq 1 100); do head -c 4000 /dev/zero | tr "\0" "x"; echo; done; sleep 3600' -``` - -Call `Logs(ctx, "container", "captest", 0)` and `Logs(ctx, "container", "bigtest", 0)` from a scratch `main.go`. Confirm the first returns 500 lines with `truncated` true, and the second returns ≤256KB with `truncated` true. Then `docker rm -f captest bigtest`. - -- [ ] **Step 3: Build and commit** - -Run: `cd agent && go build ./... && go vet ./...` - -```bash -git add agent/internal/workloads/logs.go -git commit -m "feat: agent reads bounded workload logs" -``` - ---- - -### Task 6: Proto - -**Files:** -- Modify: `proto/vantage/v1/vantage.proto` -- Regenerate: `agent/internal/grpc/pb/`, `server/internal/grpc/pb/` - -- [ ] **Step 1: Add the RPC and messages** - -Add to the `Vantage` service block: - -```protobuf - rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse); -``` - -Add these messages at the end of the file: - -```protobuf -// ReportWorkloads carries what a server is running. -// -// Offer-then-send, the same handshake as ReportPackages: the agent calls once -// with workloads empty, and resends with the body only if need_full is set. -message ReportWorkloadsRequest { - string server_id = 1; - string agent_token = 2; - string hash = 3; - bool docker_ok = 4; - string docker_error = 5; - bool systemd_ok = 6; - string systemd_error = 7; - repeated Workload workloads = 8; // empty on the offer call -} - -message ReportWorkloadsResponse { - bool need_full = 1; -} - -message Workload { - string kind = 1; // "container" | "unit" - string id = 2; - string name = 3; - string state = 4; - string health = 5; - string image = 6; - string stack = 7; - repeated string ports = 8; - int32 restarts = 9; - string started_at = 10; // RFC3339, empty when not running - bool protected = 11; -} - -// RefreshWorkloadsCmd carries no payload back. It makes the agent report -// immediately through ReportWorkloads, so there is exactly one writer for the -// server_workloads collection rather than two arriving by different routes. -message RefreshWorkloadsCmd {} - -message ControlWorkloadCmd { - string kind = 1; - string id = 2; - string action = 3; // "start" | "stop" | "restart" -} - -message WorkloadLogsCmd { - string kind = 1; - string id = 2; - int32 tail = 3; -} - -message WorkloadLogsResult { - string command_id = 1; - string text = 2; - bool truncated = 3; - string error = 4; -} -``` - -- [ ] **Step 2: Add the command and result variants** - -In `message ServerCommand`, add three variants to the `oneof command` block using the next free field numbers — **check the file, do not reuse a number**: - -```protobuf - RefreshWorkloadsCmd refresh_workloads = ; - ControlWorkloadCmd control_workload = ; - WorkloadLogsCmd workload_logs = ; -``` - -In `message AgentMessage`, add one variant to its `oneof` using the next free number there: - -```protobuf - WorkloadLogsResult workload_logs_result = ; -``` - -- [ ] **Step 3: Regenerate** - -Run the project's existing protoc generation command. Find it with: - -```bash -grep -rn "protoc" --include=Makefile --include="*.sh" --include="*.yml" . | head -``` - -Output must land in both `agent/internal/grpc/pb/` and `server/internal/grpc/pb/`. - -- [ ] **Step 4: Build both modules and commit** - -Run: `cd agent && go build ./... && cd ../server && go build ./...` - -```bash -git add proto/ agent/internal/grpc/pb/ server/internal/grpc/pb/ -git commit -m "feat: workload registry proto messages" -``` - ---- - -### Task 7: Server storage and the result registry - -**Files:** -- Create: `server/internal/services/workloads.go`, `server/internal/services/workloadresults.go` -- Modify: `server/internal/grpc/server.go` - -**Interfaces:** -- Consumes: `models.ServerWorkloads` (Task 1), the proto types (Task 6). -- Produces: `services.HasWorkloadHash`, `services.StoreWorkloads`, `services.GetWorkloads`, `services.SearchWorkloads`, `services.WorkloadResults.Await/Deliver`, `services.DispatchRefreshWorkloads`, `services.DispatchControlWorkload`, `services.DispatchWorkloadLogs`. - -- [ ] **Step 1: Write the storage service** - -Create `server/internal/services/workloads.go`: - -```go -package services - -import ( - "context" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -func HasWorkloadHash(instanceID, serverID, hash string) (bool, error) { - err := db.Col("server_workloads").FindOne(context.Background(), bson.M{ - "instance_id": instanceID, - "server_id": serverID, - "hash": hash, - }, options.FindOne().SetProjection(bson.M{"_id": 1})).Err() - if err == mongo.ErrNoDocuments { - return false, nil - } - return err == nil, err -} - -// StoreWorkloads replaces a server's workload list. -func StoreWorkloads(instanceID, serverID, hash string, wls []models.Workload, - dockerOK bool, dockerErr string, systemdOK bool, systemdErr string) error { - - _, err := db.Col("server_workloads").UpdateOne(context.Background(), - bson.M{"instance_id": instanceID, "server_id": serverID}, - bson.M{"$set": bson.M{ - "hash": hash, - "workloads": wls, - "collected_at": time.Now(), - "docker_ok": dockerOK, - "docker_error": dockerErr, - "systemd_ok": systemdOK, - "systemd_error": systemdErr, - }}, - options.UpdateOne().SetUpsert(true), - ) - return err -} - -func GetWorkloads(instanceID, serverID string) (*models.ServerWorkloads, error) { - var sw models.ServerWorkloads - err := db.Col("server_workloads").FindOne(context.Background(), bson.M{ - "instance_id": instanceID, - "server_id": serverID, - }).Decode(&sw) - if err == mongo.ErrNoDocuments { - return nil, nil - } - if err != nil { - return nil, err - } - return &sw, nil -} - -type WorkloadHit struct { - ServerID string `json:"server_id"` - Workload models.Workload `json:"workload"` -} - -// SearchWorkloads answers "which servers run image X" — the reason the snapshot -// is stored rather than fetched on demand and discarded. -func SearchWorkloads(instanceID, image, stack, state string) ([]WorkloadHit, error) { - ctx := context.Background() - - filter := bson.M{"instance_id": instanceID} - if image != "" { - filter["workloads.image"] = image - } - - cur, err := db.Col("server_workloads").Find(ctx, filter) - if err != nil { - return nil, err - } - defer cur.Close(ctx) - - var docs []models.ServerWorkloads - if err := cur.All(ctx, &docs); err != nil { - return nil, err - } - - hits := []WorkloadHit{} - for _, d := range docs { - for _, w := range d.Workloads { - if image != "" && w.Image != image { - continue - } - if stack != "" && w.Stack != stack { - continue - } - if state != "" && w.State != state { - continue - } - hits = append(hits, WorkloadHit{ServerID: d.ServerID, Workload: w}) - } - } - return hits, nil -} -``` - -- [ ] **Step 2: Write the result registry** - -Create `server/internal/services/workloadresults.go`, mirroring `stepresults.go` exactly: - -```go -package services - -import ( - "context" - "encoding/json" - "log" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/bus" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/grpc/pb" -) - -// Workload log results travel back over the bus for the same reason commands -// travel out over it: the pod serving the HTTP request and the pod holding the -// agent's stream are two different processes, and a map in one cannot be read -// by the other. -// -// Await MUST be called before the command is dispatched, or a fast agent -// answers into a channel nobody is listening on yet. See stepresults.go. - -type workloadResultRegistry struct{} - -var WorkloadResults = &workloadResultRegistry{} - -func (r *workloadResultRegistry) Await(commandID string) (<-chan *pb.WorkloadLogsResult, func()) { - out := make(chan *pb.WorkloadLogsResult, 1) - - ctx, cancel := context.WithCancel(context.Background()) - raw, unsub, err := bus.Subscribe(ctx, bus.ResultChannel+commandID) - if err != nil { - log.Printf("workload results: subscribe for %s: %v", commandID, err) - cancel() - close(out) - return out, func() {} - } - - go func() { - defer close(out) - select { - case <-ctx.Done(): - return - case b, ok := <-raw: - if !ok { - return - } - var res pb.WorkloadLogsResult - if err := json.Unmarshal(b, &res); err != nil { - log.Printf("workload results: undecodable result for %s: %v", commandID, err) - return - } - out <- &res - } - }() - - return out, func() { - cancel() - unsub() - } -} - -// Deliver publishes a result received from an agent. Called on the pod holding -// that agent's stream, which is not usually the pod waiting for it. -func (r *workloadResultRegistry) Deliver(res *pb.WorkloadLogsResult) { - if res == nil || res.CommandId == "" { - return - } - ctx, cancel := context.WithTimeout(context.Background(), dispatchAckTimeout) - defer cancel() - if _, err := bus.Publish(ctx, bus.ResultChannel+res.CommandId, res); err != nil { - log.Printf("workload results: publish for %s: %v", res.CommandId, err) - } -} -``` - -- [ ] **Step 3: Write the dispatch wrappers** - -Append to `server/internal/services/workloads.go`, following how `DispatchRunStep` in `dispatch.go` builds a command and how `dispatchAndWait` orders subscription before dispatch: - -```go -// DispatchRefreshWorkloads asks an agent to report immediately. It returns as -// soon as the agent acks; the caller refetches the stored document. -func DispatchRefreshWorkloads(serverID string) error { … } - -// DispatchControlWorkload runs a control action and waits for the result. -func DispatchControlWorkload(serverID, kind, id, action string) error { … } - -// DispatchWorkloadLogs fetches a bounded log snapshot. -// -// Await is called BEFORE dispatch. Reversing those two lines introduces a race -// that only shows under load, on a fast agent. -func DispatchWorkloadLogs(serverID, kind, id string, tail int) (text string, truncated bool, err error) { … } -``` - -Fill each in against the actual `commandDispatcher` API in `dispatch.go` — generate a command ID the same way `DispatchRunStep` does, and reuse its ack-error handling so an offline agent surfaces as `ErrAgentNotConnected`. - -- [ ] **Step 4: Add the gRPC handler and route the result** - -In `server/internal/grpc/server.go`, add a `ReportWorkloads` handler following `ReportUpdates`: validate the agent token, answer `need_full: !known` on the offer call (empty `workloads`), otherwise convert to `[]models.Workload` and call `StoreWorkloads`. - -Then in `CommandStream` (near line 165), where `CommandResult` and `StepResult` are already routed, add the new variant: - -```go - if r := msg.GetWorkloadLogsResult(); r != nil { - services.WorkloadResults.Deliver(r) - continue - } -``` - -Match the surrounding switch or if-chain style exactly. - -- [ ] **Step 5: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/services/workloads.go server/internal/services/workloadresults.go server/internal/grpc/server.go -git commit -m "feat: store workload reports and route log results" -``` - ---- - -### Task 8: Agent wiring - -**Files:** -- Modify: `agent/internal/sync/sync.go`, `agent/internal/grpc/client.go` - -**Interfaces:** -- Consumes: `workloads.Collect`, `workloads.Hash`, `workloads.Control`, `workloads.Logs` (Tasks 2–5); the proto types (Task 6). -- Produces: a 60s report loop and three command handlers. - -- [ ] **Step 1: Add the client method** - -In `agent/internal/grpc/client.go`, following `ReportUpdates`: - -```go -// ReportWorkloads sends a workload report and returns whether the server wants -// the full list. -func (c *Client) ReportWorkloads(req *pb.ReportWorkloadsRequest) (bool, error) { - ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) - defer cancel() - resp, err := c.client.ReportWorkloads(ctx, req) - if err != nil { - return false, err - } - return resp.GetNeedFull(), nil -} -``` - -- [ ] **Step 2: Add the report loop** - -In `agent/internal/sync/sync.go`, add a `runWorkloads(ctx, cfg)` following the shape of the existing `runInventory` (line 413), on a 60-second ticker, calling a `reportWorkloads(cfg)` that: - -1. returns immediately on non-Linux -2. calls `workloads.Collect(ctx)` -3. computes `workloads.Hash(result.Workloads)` -4. calls `ReportWorkloads` with `Workloads` empty -5. on `need_full`, calls again with the list populated - -Convert `workloads.Workload` to `*pb.Workload`, formatting `StartedAt` as RFC3339 and sending `""` for the zero time. - -Start it from the same place the other goroutines are started (`Run`, near line 31): - -```go - go runWorkloads(ctx, cfg) -``` - -- [ ] **Step 3: Add the three command handlers** - -In the command stream's dispatch switch, alongside `handleApplyUpdates` and `handleOpenProxy`: - -```go -// handleRefreshWorkloads makes the agent report immediately. It sends nothing -// back beyond the ack: the refresh is a nudge, not a channel, so there is one -// writer for the collection rather than two. -func handleRefreshWorkloads(cfg *config.Config) { - go reportWorkloads(cfg) -} - -func handleControlWorkload(stream pb.Vantage_CommandStreamClient, cfg *config.Config, commandID string, cmd *pb.ControlWorkloadCmd) { - err := workloads.Control(context.Background(), cmd.GetKind(), cmd.GetId(), cmd.GetAction()) - // Report through the existing CommandResult path, matching how - // handleApplyUpdates replies. On success, report immediately so the UI's - // refetch shows the new state rather than the old one. - if err == nil { - go reportWorkloads(cfg) - } - // … send CommandResult{CommandId: commandID, Ok: err == nil, Error: …} -} - -func handleWorkloadLogs(stream pb.Vantage_CommandStreamClient, cfg *config.Config, commandID string, cmd *pb.WorkloadLogsCmd) { - text, truncated, err := workloads.Logs(context.Background(), cmd.GetKind(), cmd.GetId(), int(cmd.GetTail())) - res := &pb.WorkloadLogsResult{CommandId: commandID, Text: text, Truncated: truncated} - if err != nil { - res.Error = err.Error() - } - // … send AgentMessage{WorkloadLogsResult: res} -} -``` - -Fill in the send calls against the exact `AgentMessage` construction the neighbouring handlers use — the stream send is not the same shape in every file. - -- [ ] **Step 4: Verify end to end against a dev instance** - -With a dev server and one Linux agent: - -1. Confirm a `server_workloads` document appears within 60s. -2. Confirm a second cycle with nothing changed sends no body — add a temporary log line at the `need_full` branch if the traffic is not otherwise visible, then remove it. -3. Start a container and confirm the next report includes it. -4. Dispatch a control action and confirm the state changes and a fresh report follows. -5. Dispatch a control action against `vantage-agent.service` and **confirm it is refused and the agent stays up**. - -- [ ] **Step 5: Build and commit** - -Run: `cd agent && go build ./... && go vet ./...` - -```bash -git add agent/internal/sync/sync.go agent/internal/grpc/client.go -git commit -m "feat: agent reports workloads and handles workload commands" -``` - ---- - -### Task 9: REST API - -**Files:** -- Create: `server/internal/api/workloads.go` -- Modify: `server/internal/api/handlers.go` - -- [ ] **Step 1: Write the handlers** - -Create `server/internal/api/workloads.go`, following the conventions in `handlers.go` — `auth.InstanceID(c)` for scoping, `actorFromCtx(c)` for the audit actor: - -- `getServerWorkloads` — `GET /api/servers/:id/workloads`. Returns the stored document. A server with no document yet returns an empty list with `docker_ok: false, systemd_ok: false` rather than 404 — the agent may simply not have reported yet. -- `refreshServerWorkloads` — `POST /api/servers/:id/workloads/refresh`. Calls `DispatchRefreshWorkloads`. Answers **503** when the dispatcher reports the agent is not connected; the client then shows the stored snapshot as stale rather than pretending. -- `controlWorkload` — `POST /api/servers/:id/workloads/:wid/action`, body `{"action":"start|stop|restart"}`. Owner or admin. **Answers 409 when the agent refuses a protected workload**, with the reason in the message — not 500, since nothing failed. Audit event `workload.` naming the target. -- `getWorkloadLogs` — `GET /api/servers/:id/workloads/:wid/logs?tail=`. Owner or admin. Clamps `tail` to 500 server-side rather than erroring. Audit event `workload.logs_read`. -- `listWorkloads` — `GET /api/workloads?image=&stack=&state=`. Fleet-wide, calls `SearchWorkloads`. - -`:wid` arrives URL-encoded — decode it before use. Unit names carry dots and `@`. - -- [ ] **Step 2: Register the routes** - -In `server/internal/api/handlers.go`, inside the `apiGroup` block: - -```go - apiGroup.GET("/workloads", listWorkloads) - apiGroup.GET("/servers/:id/workloads", getServerWorkloads) - apiGroup.POST("/servers/:id/workloads/refresh", refreshServerWorkloads) - apiGroup.POST("/servers/:id/workloads/:wid/action", auth.RequireRole("owner", "admin"), controlWorkload) - apiGroup.GET("/servers/:id/workloads/:wid/logs", auth.RequireRole("owner", "admin"), getWorkloadLogs) -``` - -**Check `auth.RequireRole`'s actual variadic signature** before using it with two arguments. Note also that gin resolves static segments ahead of wildcards, so `/servers/:id/workloads/refresh` and `/servers/:id/workloads/:wid/action` coexist — the same arrangement `/servers/new` and `/servers/:id` already use. - -- [ ] **Step 3: Verify by hand** - -Against a dev instance: a member session gets 200 on the workload list and **403** on both the action and the logs endpoints; an action against the agent's own unit returns **409**; a refresh against a stopped agent returns **503**; `?tail=99999` returns at most 500 lines. - -- [ ] **Step 4: Build and commit** - -Run: `cd server && go build ./... && go vet ./...` - -```bash -git add server/internal/api/ -git commit -m "feat: workload registry REST API" -``` - ---- - -### Task 10: Web UI - -**Files:** -- Create: `web/app/(app)/workloads/page.tsx`, `web/components/workloads/{WorkloadList,WorkloadRow,LogDialog}.tsx` -- Modify: `web/lib/api.ts`, `web/components/Sidebar.tsx`, `web/app/(app)/servers/[id]/page.tsx` - -- [ ] **Step 1: Add API client types and methods** - -In `web/lib/api.ts`, add `Workload`, `ServerWorkloads` and `WorkloadHit` types mirroring the Go JSON tags exactly, plus one client method per Task 9 route. Follow the file's existing conventions. - -- [ ] **Step 2: Build the server-detail tab** - -Add a **Workloads** tab to `web/app/(app)/servers/[id]/page.tsx`, following the existing tab pattern, rendering `WorkloadList`. - -`WorkloadList` orders: **compose stacks first, grouped under the stack name**, then loose containers, then units. Not cosmetic — a stack is one thing to an operator even when it is six containers, and a flat list turns one decision into six rows. - -Opening the tab dispatches a refresh, then refetches. Show the `collected_at` age while it is in flight. - -Three rules from the model: -- **`docker_ok: false` with no `docker_error` reads "Docker not in use on this server"** — never an empty list, and never alarming. With a `docker_error`, show that instead: it is a real problem with a real fix. -- **`protected` rows render their action buttons disabled with the reason**, rather than offering a button whose refusal is already known. -- State never reads by colour alone: every pill carries a distinct shape and a text label, matching the existing monitor pills. Container and unit states use their own vocabularies — do not map them onto a shared one. - -- [ ] **Step 3: Build the log dialog** - -Create `web/components/workloads/LogDialog.tsx`: monospace, on the `--well` token (the floor beneath the ground, used for machine output), scrolled to the bottom on open. When `truncated`, show a banner saying the output was capped — the user must not read a truncated log as a complete one. - -- [ ] **Step 4: Build the fleet view** - -Create `web/app/(app)/workloads/page.tsx`: filters for image, stack and state; rows link to their server. Add a **Workloads** entry to `web/components/Sidebar.tsx`. - -- [ ] **Step 5: Verify** - -Run: `cd web && npm run lint && npm run build` -Expected: exit 0, no type errors. - -Then against a dev instance confirm: stacks group, a protected row's buttons are disabled, a server without Docker says "not in use" rather than showing nothing, a truncated log shows its banner, and a restart updates the row without a manual reload. - -- [ ] **Step 6: Commit** - -```bash -git add web/ -git commit -m "feat: workload registry UI" -``` - ---- - -### Task 11: Instance deletion and documentation - -- [ ] **Step 1: Add the collection to instance deletion** - -Find the instance-deletion routine — the one place that knows which collections carry `instance_id`: - -```bash -cd server && grep -rn "instance_id" internal/services/ | grep -i "delete\|purge\|reap" -``` - -Add `server_workloads` to its collection list. Missing this orphans a tenant's data indefinitely. - -- [ ] **Step 2: Verify the whole tree builds** - -From the repo root: - -```bash -go build ./... && go vet ./... -``` - -Expected: exit 0. - -- [ ] **Step 3: Document it** - -Add a **Workload registry** subsystem section to `CLAUDE.md` covering: why refresh does not return data (one writer); why the protected set lives agent-side; why there is no live log following; and the `DockerOK`/`DockerError` distinction. Add a user-facing page under `docsite/docs/vantage/`. - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "docs: document the workload registry" -``` - ---- - -## Spec coverage - -- Workload as the domain word, containers and units in one collection → Task 1. -- `DockerOK`/`DockerError` distinction → Tasks 1, 2, 10. -- Docker via `ps -aq` + `inspect`, no English parsing → Task 2. -- Compose stacks from the label, no YAML → Task 2. -- systemd two-pass with the exclusion filter, column output not JSON → Task 3. -- Hash order-independence → Task 3. -- Protected set agent-side, `vantage-agent.service` and own container → Task 4. -- Control timeouts returning real errors → Task 4. -- Log caps in both directions, no follow mode → Task 5. -- Refresh returns no data; one writer → Tasks 6, 8. -- `Await` before dispatch → Task 7. -- 503 offline, 409 protected, owner|admin on control and logs → Task 9. -- Stacks grouped first; protected rows disabled; truncation stated → Task 10. -- Instance-deletion collection list → Task 11. - -## Known gaps the executing engineer must close - -- **Three dispatch wrappers are specified by signature and doc comment, not body** (Task 7, Step 3). They must be written against the actual `commandDispatcher` API in `dispatch.go`, reusing how `DispatchRunStep` generates a command ID and handles ack errors. `DispatchWorkloadLogs` must call `Await` **before** dispatch. -- **The agent's stream-send calls are left to match their neighbours** (Task 8, Step 3). `AgentMessage` construction is not identical across handlers, so copy the adjacent one rather than the sketch. -- **`auth.RequireRole`'s variadic signature must be confirmed** before Task 9's two-argument use. -- **No automated tests, by instruction.** The manual checks in Tasks 2, 3, 4, 5, 8, 9 and 10 are the substitute. Task 4's is the one that matters most: get the protected set wrong in the permissive direction and a server can take itself offline in a way the UI cannot undo. diff --git a/docs/superpowers/plans/2026-08-12-api-tokens-openapi.md b/docs/superpowers/plans/2026-08-12-api-tokens-openapi.md deleted file mode 100644 index abb69e0..0000000 --- a/docs/superpowers/plans/2026-08-12-api-tokens-openapi.md +++ /dev/null @@ -1,2065 +0,0 @@ -# API Tokens and OpenAPI Reference Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add scoped, optionally expiring API tokens to the Vantage control plane REST API, and publish a generated OpenAPI 3.1 document rendered by a self-hosted Scalar reference page. - -**Architecture:** Tokens fall back into the existing `auth.Middleware()` rather than getting their own route group, so every handler, `RequireRole`, `RequireActiveLicense`, `RequireFeature` and `actorFromCtx` keep working untouched. A second middleware enforces coarse resource scopes derived from the matched gin route pattern, failing closed on any unmapped route and refusing to boot if an `/api` route is missing from the map. The OpenAPI document is generated from swaggo v2 annotations, committed, embedded, and verified in CI by regenerate-and-diff. - -**Tech Stack:** Go 1.24 (`server` module), gin, MongoDB (mongo-driver v2), Redis (go-redis v9), Next.js 16 + TanStack Query (`web`), swaggo/swag v2, Scalar standalone JS (vendored). - -**Spec:** `docs/superpowers/specs/2026-08-12-api-tokens-openapi-design.md` - -## Global Constraints - -- Module path prefix is `gitea.hostxtra.co.uk/mrhid6/vantage/`. -- Every new document carries `instance_id`, and every query is scoped by it. -- Any new tenant-scoped collection MUST be added to `services.ScopedCollections` in `server/internal/services/migrate_instance.go`, or `AssertNoScopedCollectionMissed` fails at boot and instance purge leaks rows. -- Token plaintext is stored nowhere. Only `sha256` hex, via the existing `services.HashToken`. -- Roles are `owner` > `admin` > `member`, constants `models.RoleOwner`, `models.RoleAdmin`, `models.RoleMember`. -- `web/` components must carry no hex colour values — Tailwind token classes only (`bg-surface-2`, `border-border`, `text-text-primary`, `bg-well`, `text-danger`, `text-warning`, `text-success`, `accent`). -- No test files are added by this plan. Every task is verified with `go build ./...`, `go vet ./...`, and a concrete curl or UI check. -- The repository has no running dev stack assumed; verification commands assume the server is started locally per the existing compose file with MongoDB and Redis reachable. -- Commit style follows the existing log: `feat:`, `fix:`, `docs:` with a capitalised imperative subject. -- Work happens on branch `feat/api-tokens`, which already exists and holds the spec commit. - ---- - -### Task 1: Token model, indexes and scoped-collection registration - -**Files:** -- Create: `server/internal/models/api_token.go` -- Modify: `server/internal/services/migrate_instance.go` (the `ScopedCollections` slice, after `"vuln_alert_rules"`) -- Create: `server/internal/services/tokenindexes.go` -- Modify: `server/cmd/main.go` (schema setup section, beside the other `Ensure*Indexes` calls) - -**Interfaces:** -- Consumes: nothing. -- Produces: `models.APIToken` struct; `models.TokenScopes` type alias `[]string`; `services.EnsureAPITokenIndexes() error`. - -- [ ] **Step 1: Create the model** - -Create `server/internal/models/api_token.go`: - -```go -package models - -import ( - "time" - - "go.mongodb.org/mongo-driver/v2/bson" -) - -// APIToken is a personal access token for the REST API. -// -// The plaintext is shown once at creation and never stored: only TokenHash, -// which is sha256 hex of the value, exactly as servers.agent_token_hash and the -// ESO read token already are. bcrypt is deliberately not used — the value is -// full-entropy random rather than a chosen password, and a per-token salt would -// force a collection scan where an indexed lookup is wanted. -// -// Role and Scopes are immutable after creation. There is no update endpoint: -// editing what a credential already deployed in CI can do, with no record of -// what it could do before, is worse than requiring a rotation. -type APIToken struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"-"` - TokenID string `bson:"token_id" json:"token_id"` - InstanceID string `bson:"instance_id" json:"instance_id"` - UserID string `bson:"user_id" json:"user_id"` - - Name string `bson:"name" json:"name"` - // Hint is the first 8 characters of the plaintext, stored in clear so the - // list can identify a token without revealing it. - Hint string `bson:"hint" json:"hint"` - // TokenHash is never serialised to JSON. - TokenHash string `bson:"token_hash" json:"-"` - - Role string `bson:"role" json:"role"` - Scopes []string `bson:"scopes" json:"scopes"` - - // ExpiresAt nil means the token never expires. Whether that is allowed is - // a per-instance policy, settings.api_token_max_days. - ExpiresAt *time.Time `bson:"expires_at,omitempty" json:"expires_at,omitempty"` - - CreatedAt time.Time `bson:"created_at" json:"created_at"` - LastUsedAt *time.Time `bson:"last_used_at,omitempty" json:"last_used_at,omitempty"` - CreatedByIP string `bson:"created_by_ip,omitempty" json:"created_by_ip,omitempty"` - - // Email of the owning user, joined at read time for the list. Never stored. - UserEmail string `bson:"-" json:"user_email,omitempty"` -} - -// Expired reports whether the token's expiry has passed. A nil ExpiresAt never -// expires. -func (t *APIToken) Expired(now time.Time) bool { - return t.ExpiresAt != nil && now.After(*t.ExpiresAt) -} -``` - -- [ ] **Step 2: Register the collection as tenant-scoped** - -In `server/internal/services/migrate_instance.go`, add `"api_tokens"` to the `ScopedCollections` slice, after `"vuln_alert_rules"`: - -```go - "vuln_alert_rules", - "api_tokens", -``` - -- [ ] **Step 3: Declare the indexes** - -Create `server/internal/services/tokenindexes.go`: - -```go -package services - -import ( - "context" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -// EnsureAPITokenIndexes declares the indexes the token path depends on. -// -// The unique index on token_hash is a security property, not an optimisation: -// it is what makes authentication a single indexed lookup rather than a scan, -// and what makes two tokens hashing to one value impossible to store. -// -// Fatal on failure, like EnsureAuthIndexes and unlike the secrets and workflow -// builders: without the unique index the auth path would still answer, which is -// exactly the wrong kind of degradation. -func EnsureAPITokenIndexes() error { - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - - if _, err := db.Col("api_tokens").Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "token_hash", Value: 1}}, - Options: options.Index().SetUnique(true), - }); err != nil { - return err - } - - if _, err := db.Col("api_tokens").Indexes().CreateOne(ctx, mongo.IndexModel{ - Keys: bson.D{{Key: "instance_id", Value: 1}, {Key: "user_id", Value: 1}}, - }); err != nil { - return err - } - return nil -} -``` - -- [ ] **Step 4: Call it at boot** - -In `server/cmd/main.go`, find the block calling `services.EnsureAuthIndexes()` and the other index builders during schema setup. Add immediately after the `EnsureAuthIndexes` call, matching its fatal-on-error handling: - -```go - if err := services.EnsureAPITokenIndexes(); err != nil { - log.Fatalf("api token indexes: %v", err) - } -``` - -If the surrounding code collects errors rather than calling `log.Fatalf` directly, follow the surrounding pattern instead of introducing a second one. - -- [ ] **Step 5: Verify it builds and the boot check still passes** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output from either command. - -Then start the server against your local MongoDB and Redis and confirm the log shows no `ScopedCollections` complaint: - -```bash -cd /go-projects/vantage/server && VANTAGE_MIGRATE_ONLY=1 go run ./cmd 2>&1 | tail -20 -``` - -Expected: schema setup runs and the process exits 0. Specifically, no line containing `holds ... document(s) with org_id but is not in ScopedCollections`. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/models/api_token.go server/internal/services/tokenindexes.go server/internal/services/migrate_instance.go server/cmd/main.go -git commit -m "feat: Add the api_tokens collection and its indexes - -The unique index on token_hash is what makes authentication an indexed -lookup rather than a scan, so this builder is fatal on failure like -EnsureAuthIndexes rather than warning like the secrets one. - -Registered in ScopedCollections so instance purge reaches it." -``` - ---- - -### Task 2: Scope vocabulary and validation - -**Files:** -- Create: `server/internal/services/scopes.go` - -**Interfaces:** -- Consumes: nothing. -- Produces: `services.AllScopes() []string`; `services.ValidScopes(scopes []string) error`; `services.ScopeSatisfied(held []string, required string) bool`; `services.ErrInvalidScope`. - -- [ ] **Step 1: Write the scope vocabulary** - -Create `server/internal/services/scopes.go`: - -```go -package services - -import ( - "errors" - "fmt" - "sort" - "strings" -) - -// ErrInvalidScope is returned when a token is requested with a scope outside -// the vocabulary below. -var ErrInvalidScope = errors.New("invalid scope") - -// ScopeResources is the whole vocabulary. Eight resources, each with :read and -// :write, and write implies read on the same resource. -// -// It is deliberately coarse. A scope per endpoint is a table nobody maintains, -// and a route added without an entry either fails closed and breaks, or -// defaults open and is pointless. -var ScopeResources = []string{ - "servers", - "keys", - "secrets", - "workflows", - "monitors", - "vulns", - "workloads", - "settings", -} - -const ( - ScopeRead = "read" - ScopeWrite = "write" -) - -// AllScopes returns every valid scope string, sorted, for the API to advertise -// to the token-creation UI. -func AllScopes() []string { - out := make([]string, 0, len(ScopeResources)*2) - for _, r := range ScopeResources { - out = append(out, r+":"+ScopeRead, r+":"+ScopeWrite) - } - sort.Strings(out) - return out -} - -func validScope(s string) bool { - resource, action, ok := strings.Cut(s, ":") - if !ok || (action != ScopeRead && action != ScopeWrite) { - return false - } - for _, r := range ScopeResources { - if r == resource { - return true - } - } - return false -} - -// ValidScopes rejects an unknown scope and an empty list. A token with no -// scopes can reach nothing, so creating one is a mistake worth naming rather -// than a credential worth issuing. -func ValidScopes(scopes []string) error { - if len(scopes) == 0 { - return fmt.Errorf("%w: at least one scope is required", ErrInvalidScope) - } - for _, s := range scopes { - if !validScope(s) { - return fmt.Errorf("%w: %q", ErrInvalidScope, s) - } - } - return nil -} - -// ScopeSatisfied reports whether the held scopes cover the required one. -// Holding "servers:write" satisfies a requirement of "servers:read"; the -// converse is false. -func ScopeSatisfied(held []string, required string) bool { - resource, action, ok := strings.Cut(required, ":") - if !ok { - return false - } - for _, h := range held { - if h == required { - return true - } - if action == ScopeRead && h == resource+":"+ScopeWrite { - return true - } - } - return false -} -``` - -- [ ] **Step 2: Verify it builds** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. - -- [ ] **Step 3: Commit** - -```bash -git add server/internal/services/scopes.go -git commit -m "feat: Define the API token scope vocabulary - -Eight resources with read and write, write implying read. Coarse on -purpose: a scope per endpoint is a table nobody maintains, and a route -added without an entry either breaks or is unguarded." -``` - ---- - -### Task 3: Token lifetime policy in settings - -**Files:** -- Modify: `shared/models/settings.go` -- Modify: `server/internal/services/settings.go` (`SaveSettings`) -- Modify: `server/internal/api/handlers.go` (`saveSettings` handler, lines 564-584) - -**Interfaces:** -- Consumes: nothing. -- Produces: `Settings.APITokenMaxDays *int` field; `models.APITokenMaxDays(s *Settings) int` reader returning 0 for "no cap"; `services.SaveSettings(instanceID string, alerts models.AlertSettings, retentionDays *int, localLoginEnabled *bool, apiTokenMaxDays *int) error` — note the added final parameter. - -- [ ] **Step 1: Add the field to the shared settings model** - -In `shared/models/settings.go`, add to the `Settings` struct after `VulnFindingRetentionDays`: - -```go - // APITokenMaxDays caps how long a newly created API token may live. - // - // A pointer for the same reason the retention fields are: absent must mean - // the default, and the default here is no cap at all — never-expire tokens - // are allowed until an instance decides otherwise, so an upgrade changes - // nothing. Nil or 0 is no cap. A positive value refuses both a longer - // expiry and a token with no expiry. - // - // It is a policy on issuance, not on use: raising or lowering it never - // invalidates a token that already exists. - APITokenMaxDays *int `bson:"api_token_max_days,omitempty" json:"api_token_max_days,omitempty"` -``` - -And add the reader beside `LocalLoginEnabled`: - -```go -// APITokenMaxDays reads the token lifetime cap with its absent-means-uncapped -// default. 0 means no cap. Every caller must go through this rather than -// dereferencing the field. -func APITokenMaxDays(s *Settings) int { - if s == nil || s.APITokenMaxDays == nil || *s.APITokenMaxDays < 0 { - return 0 - } - return *s.APITokenMaxDays -} -``` - -- [ ] **Step 2: Persist it in SaveSettings** - -In `server/internal/services/settings.go`, change the signature and the `set` document: - -```go -func SaveSettings(instanceID string, alerts models.AlertSettings, retentionDays *int, localLoginEnabled *bool, apiTokenMaxDays *int) error { -``` - -and after the `local_login_enabled` block, before the `UpdateOne`: - -```go - if apiTokenMaxDays != nil { - set["api_token_max_days"] = *apiTokenMaxDays - } -``` - -- [ ] **Step 3: Accept it in the handler** - -In `server/internal/api/handlers.go`, extend the `saveSettings` body struct and the call: - -```go - var body struct { - Alerts models.AlertSettings `json:"alerts"` - WorkflowLogRetentionDays *int `json:"workflow_log_retention_days"` - LocalLoginEnabled *bool `json:"local_login_enabled"` - APITokenMaxDays *int `json:"api_token_max_days"` - } -``` - -```go - if body.APITokenMaxDays != nil && *body.APITokenMaxDays < 0 { - c.JSON(http.StatusBadRequest, gin.H{"error": "api_token_max_days cannot be negative"}) - return - } - if err := services.SaveSettings(auth.InstanceID(c), body.Alerts, body.WorkflowLogRetentionDays, body.LocalLoginEnabled, body.APITokenMaxDays); err != nil { -``` - -Then add an audit line immediately after the existing `settings.updated` event, so the policy change is distinguishable from an alert change: - -```go - if body.APITokenMaxDays != nil { - services.LogEvent(auth.InstanceID(c), "settings.token_policy_updated", actorFromCtx(c), "", "", - fmt.Sprintf("API token maximum lifetime set to %d day(s); 0 means no cap", *body.APITokenMaxDays)) - } -``` - -- [ ] **Step 4: Fix every other caller of SaveSettings** - -```bash -cd /go-projects/vantage && grep -rn "SaveSettings(" --include=*.go . -``` - -Pass `nil` as the new final argument at every call site other than the handler above. Do not change their behaviour. - -- [ ] **Step 5: Verify** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. - -With the server running and a browser session cookie in `cookies.txt`: - -```bash -curl -s -b cookies.txt -X PUT localhost:8080/api/settings \ - -H 'Content-Type: application/json' \ - -d '{"alerts":{"offline_threshold_minutes":5,"offline_channel_ids":[]},"api_token_max_days":90}' -curl -s -b cookies.txt localhost:8080/api/settings | grep api_token_max_days -``` - -Expected: `{"saved":true}` then a line containing `"api_token_max_days":90`. - -- [ ] **Step 6: Commit** - -```bash -git add shared/models/settings.go server/internal/services/settings.go server/internal/api/handlers.go -git commit -m "feat: Add a per-instance API token lifetime cap - -A pointer with absent meaning no cap, so an upgrade allows never-expire -tokens exactly as before and an instance opts into the policy. It governs -issuance only: changing it never invalidates a token that already exists." -``` - ---- - -### Task 4: Token service — mint, resolve, list, revoke - -**Files:** -- Create: `server/internal/services/tokens.go` -- Modify: `server/internal/services/users.go` (`DeleteUser`, around line 157-180) - -**Interfaces:** -- Consumes: `models.APIToken`, `services.ValidScopes`, `services.HashToken` (already in `servers.go`), `models.APITokenMaxDays`, `services.GetUserInInstance` (already in `users.go`), `services.GetSettings`. -- Produces: - - `services.CreateAPIToken(instanceID, userID, name, role string, scopes []string, expiresInDays *int, ip string) (*models.APIToken, string, error)` - - `services.ResolveAPIToken(plaintext string) (*models.APIToken, error)` - - `services.ListAPITokens(instanceID string, userID string, all bool) ([]models.APIToken, error)` - - `services.RevokeAPIToken(instanceID, tokenID string, requester *models.User) (*models.APIToken, error)` - - `services.DeleteTokensForUser(instanceID, userID string) error` - - Errors: `ErrTokenExpired`, `ErrTokenNotFound`, `ErrTokenNameTaken`, `ErrTokenRoleTooHigh`, `ErrTokenExpiryPolicy`. - -- [ ] **Step 1: Write the service** - -Create `server/internal/services/tokens.go`: - -```go -package services - -import ( - "context" - "errors" - "fmt" - "strings" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/db" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "github.com/google/uuid" - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" -) - -var ( - ErrTokenNotFound = errors.New("token not found") - ErrTokenExpired = errors.New("token expired") - ErrTokenNameTaken = errors.New("a token with that name already exists") - ErrTokenRoleTooHigh = errors.New("cannot create a token above your own role") - ErrTokenExpiryPolicy = errors.New("expiry exceeds this instance's maximum token lifetime") - - // ErrTokenInvalid marks a caller mistake, as distinct from a backend - // failure. It is what lets the handler answer 400 for a bad request and - // 500 for a database that is down, rather than blaming the caller for both. - ErrTokenInvalid = errors.New("invalid token request") -) - -// TokenPrefix is on every plaintext so a leaked value is recognisable in a log -// or a paste, and so a wrong credential fails at the prefix check rather than -// as an anonymous 401. -const TokenPrefix = "vt_" - -const tokenNameMax = 64 - -// roleRank orders the three roles so a token can be capped at its owner's. -func roleRank(role string) int { - switch role { - case models.RoleOwner: - return 3 - case models.RoleAdmin: - return 2 - case models.RoleMember: - return 1 - } - return 0 -} - -// LowerRole returns whichever of the two roles grants less. It is what makes a -// token's authority follow its owner: demote the person and the token demotes -// with them, because this is recomputed on every request rather than frozen at -// creation. -func LowerRole(a, b string) string { - if roleRank(a) <= roleRank(b) { - return a - } - return b -} - -// CreateAPIToken mints a token and returns the document plus the plaintext. -// The plaintext is the only copy: it is returned once and never stored. -func CreateAPIToken(instanceID, userID, name, role string, scopes []string, expiresInDays *int, ip string) (*models.APIToken, string, error) { - name = strings.TrimSpace(name) - if name == "" || len(name) > tokenNameMax { - return nil, "", fmt.Errorf("token name must be 1 to %d characters", tokenNameMax) - } - if !models.ValidRole(role) { - return nil, "", fmt.Errorf("invalid role %q", role) - } - if err := ValidScopes(scopes); err != nil { - return nil, "", err - } - - owner, err := GetUserInInstance(instanceID, userID) - if err != nil { - return nil, "", fmt.Errorf("user not found") - } - if roleRank(role) > roleRank(owner.Role) { - return nil, "", ErrTokenRoleTooHigh - } - - settings, err := GetSettings(instanceID) - if err != nil { - return nil, "", err - } - maxDays := models.APITokenMaxDays(settings) - - var expiresAt *time.Time - switch { - case expiresInDays != nil: - if *expiresInDays <= 0 { - return nil, "", fmt.Errorf("expires_in_days must be positive") - } - if maxDays > 0 && *expiresInDays > maxDays { - return nil, "", fmt.Errorf("%w: maximum is %d day(s)", ErrTokenExpiryPolicy, maxDays) - } - t := time.Now().UTC().AddDate(0, 0, *expiresInDays) - expiresAt = &t - case maxDays > 0: - // A policy is set, so a token with no expiry is refused rather than - // silently capped: the caller asked for something the instance does not - // allow, and quietly giving them something else is worse than a 422. - return nil, "", fmt.Errorf("%w: an expiry of at most %d day(s) is required", ErrTokenExpiryPolicy, maxDays) - } - - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - existing := db.Col("api_tokens").FindOne(ctx, bson.M{"instance_id": instanceID, "user_id": userID, "name": name}) - if existing.Err() == nil { - return nil, "", ErrTokenNameTaken - } else if !errors.Is(existing.Err(), mongo.ErrNoDocuments) { - return nil, "", existing.Err() - } - - secret, err := generateToken(32) - if err != nil { - return nil, "", err - } - plaintext := TokenPrefix + secret - - tok := &models.APIToken{ - TokenID: uuid.NewString(), - InstanceID: instanceID, - UserID: userID, - Name: name, - Hint: plaintext[:8], - TokenHash: HashToken(plaintext), - Role: role, - Scopes: scopes, - ExpiresAt: expiresAt, - CreatedAt: time.Now().UTC(), - CreatedByIP: ip, - } - - if _, err := db.Col("api_tokens").InsertOne(ctx, tok); err != nil { - return nil, "", err - } - return tok, plaintext, nil -} - -// ResolveAPIToken looks a plaintext up by hash. -// -// It returns ErrTokenExpired distinctly from ErrTokenNotFound so the auth layer -// can say which happened: a forgotten CI job hitting an expired token is worth -// seeing in the audit log, and an anonymous 401 hides it. -func ResolveAPIToken(plaintext string) (*models.APIToken, error) { - if !strings.HasPrefix(plaintext, TokenPrefix) { - return nil, ErrTokenNotFound - } - - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - var tok models.APIToken - err := db.Col("api_tokens").FindOne(ctx, bson.M{"token_hash": HashToken(plaintext)}).Decode(&tok) - if errors.Is(err, mongo.ErrNoDocuments) { - return nil, ErrTokenNotFound - } - if err != nil { - return nil, err - } - if tok.Expired(time.Now().UTC()) { - return &tok, ErrTokenExpired - } - return &tok, nil -} - -// TouchAPIToken records use, but only when the stored value is more than a -// minute stale. Without the check this is a Mongo write on every API call. -func TouchAPIToken(tok *models.APIToken) { - now := time.Now().UTC() - if tok.LastUsedAt != nil && now.Sub(*tok.LastUsedAt) < time.Minute { - return - } - ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) - defer cancel() - _, _ = db.Col("api_tokens").UpdateOne(ctx, - bson.M{"token_id": tok.TokenID, "instance_id": tok.InstanceID}, - bson.M{"$set": bson.M{"last_used_at": now}}, - ) - tok.LastUsedAt = &now -} - -// ListAPITokens returns a user's own tokens, or every token in the instance -// when all is true. The caller decides whether all is permitted. -func ListAPITokens(instanceID string, userID string, all bool) ([]models.APIToken, error) { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - filter := bson.M{"instance_id": instanceID} - if !all { - filter["user_id"] = userID - } - cursor, err := db.Col("api_tokens").Find(ctx, filter) - if err != nil { - return nil, err - } - defer cursor.Close(ctx) - - var tokens []models.APIToken - if err := cursor.All(ctx, &tokens); err != nil { - return nil, err - } - if tokens == nil { - tokens = []models.APIToken{} - } - - // Join the owning email so an admin's list names people rather than UUIDs. - users, err := ListUsers(instanceID) - if err == nil { - byID := make(map[string]string, len(users)) - for _, u := range users { - byID[u.UserID] = u.Email - } - for i := range tokens { - tokens[i].UserEmail = byID[tokens[i].UserID] - } - } - return tokens, nil -} - -// RevokeAPIToken deletes a token. A member may revoke only their own; owner and -// admin may revoke any token in the instance. -func RevokeAPIToken(instanceID, tokenID string, requester *models.User) (*models.APIToken, error) { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - var tok models.APIToken - err := db.Col("api_tokens").FindOne(ctx, bson.M{"instance_id": instanceID, "token_id": tokenID}).Decode(&tok) - if errors.Is(err, mongo.ErrNoDocuments) { - return nil, ErrTokenNotFound - } - if err != nil { - return nil, err - } - - elevated := requester.Role == models.RoleOwner || requester.Role == models.RoleAdmin - if tok.UserID != requester.UserID && !elevated { - // Not 403: confirming the token exists tells a member about somebody - // else's credential. Same argument as admin's customer endpoints. - return nil, ErrTokenNotFound - } - - if _, err := db.Col("api_tokens").DeleteOne(ctx, bson.M{"instance_id": instanceID, "token_id": tokenID}); err != nil { - return nil, err - } - return &tok, nil -} - -// DeleteTokensForUser removes every token belonging to a user. Offboarding is -// one action, not two: a token that outlives its owner is an access path with -// nobody attached to it. -func DeleteTokensForUser(instanceID, userID string) error { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - _, err := db.Col("api_tokens").DeleteMany(ctx, bson.M{"instance_id": instanceID, "user_id": userID}) - return err -} -``` - -If `github.com/google/uuid` is not already a dependency of the `server` module, check how other services generate IDs first: - -```bash -cd /go-projects/vantage && grep -rn "uuid\." server/internal/services/servers.go | head -3 -``` - -Use whatever that file uses rather than adding a dependency. - -- [ ] **Step 2: Cascade the delete from user removal** - -In `server/internal/services/users.go`, inside `DeleteUser`, immediately after the successful `db.Col("users").DeleteOne(...)` and before the function returns, add: - -```go - // Offboarding is one action. A token outliving its owner is an access path - // with nobody attached to it. - if err := DeleteTokensForUser(instanceID, userID); err != nil { - log.Printf("delete tokens for user %s: %v", userID, err) - } -``` - -Add `"log"` to the file's imports if it is not already there. - -- [ ] **Step 3: Verify** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. - -- [ ] **Step 4: Commit** - -```bash -git add server/internal/services/tokens.go server/internal/services/users.go -git commit -m "feat: Add the API token service - -Mint, resolve, list and revoke, with the effective role capped at the -owner's and recomputed per request rather than frozen at creation. - -Deleting a user deletes their tokens in the same call, so offboarding is -one action. Revoking somebody else's token answers not-found rather than -forbidden, since a 403 confirms the credential exists." -``` - ---- - -### Task 5: Bearer fallback in the session middleware - -**Files:** -- Modify: `server/internal/auth/session.go` (the `Session` struct, lines 19-25) -- Modify: `server/internal/auth/middleware.go` (`Middleware`, lines 51-79; add accessors) - -**Interfaces:** -- Consumes: `services.ResolveAPIToken`, `services.TouchAPIToken`, `services.LowerRole`, `services.GetUserInInstance`, `services.LogEvent`, `services.ErrTokenExpired`. -- Produces: `Session.TokenID string`, `Session.TokenName string`, `Session.Scopes []string`; `auth.TokenID(c *gin.Context) string`; `auth.Scopes(c *gin.Context) []string`; `auth.IsToken(c *gin.Context) bool`. - -- [ ] **Step 1: Extend the session struct** - -In `server/internal/auth/session.go`: - -```go -type Session struct { - UserID string `json:"user_id"` - InstanceID string `json:"instance_id"` - Role string `json:"role"` - Email string `json:"email"` - Name string `json:"name"` - - // The three fields below are set only when the request authenticated with - // an API token. They are never persisted to Redis — a token authenticates - // per request and mints no session, so a revoked token stops working - // immediately rather than at the end of a session TTL. - TokenID string `json:"-"` - TokenName string `json:"-"` - Scopes []string `json:"-"` -} -``` - -- [ ] **Step 2: Add the bearer fallback** - -Replace `Middleware()` in `server/internal/auth/middleware.go` with: - -```go -// Middleware authenticates a request by session cookie or by API token. -// -// Both paths end by putting a *Session in the context, which is why no handler, -// role guard, licence gate or audit call needed changing: the token path is a -// second way to arrive at the same value, not a second way through the API. -func Middleware() gin.HandlerFunc { - return func(c *gin.Context) { - sess, ok := sessionFromCookie(c) - if !ok { - // A cookie that was presented and rejected has already been - // answered. Falling through would put a second JSON body on the - // wire for the ordinary expired-session case. - if c.IsAborted() { - return - } - sess, ok = sessionFromToken(c) - } - if !ok { - // sessionFromCookie and sessionFromToken have already written the - // response describing which credential failed and why. - return - } - - if sess.InstanceID == "" { - c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "session has no organization"}) - return - } - - c.Set(ctxSessionKey, sess) - - // The host guard applies to both credential kinds. A token carries an - // instance, and the tenant boundary must not have a token-shaped hole. - if hostInstance, ok := InstanceFromHost(c); ok && hostInstance.InstanceID != sess.InstanceID { - c.AbortWithStatusJSON(http.StatusForbidden, gin.H{"error": "instance host mismatch"}) - return - } - - c.Next() - } -} - -// sessionFromCookie returns false without writing a response when there is no -// cookie at all, so the token path gets its turn. It writes and aborts only -// when a cookie was presented and was not usable. -func sessionFromCookie(c *gin.Context) (*Session, bool) { - cookie, err := c.Request.Cookie(sessionCookieName) - if err != nil { - return nil, false - } - sess, err := GetSession(c.Request.Context(), cookie.Value) - if err != nil { - // A stale cookie plus a valid bearer token is a real combination — - // a browser tab left open beside a curl. Fall through rather than - // refusing a credential that would have worked. - if bearerToken(c) != "" { - return nil, false - } - c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "session expired"}) - return nil, false - } - return sess, true -} - -func bearerToken(c *gin.Context) string { - const prefix = "Bearer " - h := c.GetHeader("Authorization") - if len(h) <= len(prefix) || !strings.EqualFold(h[:len(prefix)], prefix) { - return "" - } - return h[len(prefix):] -} - -func sessionFromToken(c *gin.Context) (*Session, bool) { - raw := bearerToken(c) - if raw == "" { - c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "not authenticated"}) - return nil, false - } - - tok, err := services.ResolveAPIToken(raw) - if errors.Is(err, services.ErrTokenExpired) { - // Recorded rather than only refused: an expired token still being - // presented is how a forgotten CI job becomes visible. - services.LogEvent(tok.InstanceID, "token.expired_use", tok.Name, "", "", - fmt.Sprintf("expired token '%s' was used", tok.Name)) - c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "token expired", "code": "token_expired"}) - return nil, false - } - if err != nil { - c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "invalid token"}) - return nil, false - } - - user, err := services.GetUserInInstance(tok.InstanceID, tok.UserID) - if err != nil { - // The owner is gone. DeleteUser removes tokens, so this is the - // belt-and-braces path for a row deleted some other way. - c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "invalid token"}) - return nil, false - } - - services.TouchAPIToken(tok) - - return &Session{ - UserID: tok.UserID, - InstanceID: tok.InstanceID, - // Recomputed per request, so demoting the person demotes the token. - Role: services.LowerRole(user.Role, tok.Role), - Email: user.Email, - Name: user.Email, - TokenID: tok.TokenID, - TokenName: tok.Name, - Scopes: tok.Scopes, - }, true -} -``` - -Update the file's import block to: - -```go -import ( - "errors" - "fmt" - "net/http" - "strings" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "github.com/gin-gonic/gin" -) -``` - -- [ ] **Step 3: Add the accessors** - -Append to `server/internal/auth/middleware.go`: - -```go -// TokenID is empty for a cookie session and the token's ID for a token -// request. It is what lets audit detail record which credential acted. -func TokenID(c *gin.Context) string { - if s := GetSessionFromContext(c); s != nil { - return s.TokenID - } - return "" -} - -func TokenName(c *gin.Context) string { - if s := GetSessionFromContext(c); s != nil { - return s.TokenName - } - return "" -} - -func Scopes(c *gin.Context) []string { - if s := GetSessionFromContext(c); s != nil { - return s.Scopes - } - return nil -} - -// IsToken reports whether this request authenticated with an API token rather -// than a browser session. -func IsToken(c *gin.Context) bool { return TokenID(c) != "" } -``` - -- [ ] **Step 4: Verify the build and that both credentials work** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. In particular, no import cycle — `auth` already imports `services` (`local.go`, `oidc.go`, `instancehost.go`) and `services` imports `auth` nowhere. - -Confirm the cookie path is unchanged with the server running: - -```bash -curl -s -o /dev/null -w '%{http_code}\n' -b cookies.txt localhost:8080/api/servers -curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/api/servers -curl -s -w '\n%{http_code}\n' -H 'Authorization: Bearer vt_deadbeef' localhost:8080/api/servers -``` - -Expected: `200`, then `401`, then `{"error":"invalid token"}` with `401`. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/auth/session.go server/internal/auth/middleware.go -git commit -m "feat: Authenticate the API with a bearer token as well as a cookie - -One middleware, two ways to arrive at the same *Session, so every handler, -role guard, licence gate and audit call is untouched. The host guard -applies to both: a token carries an instance, and the tenant boundary must -not have a token-shaped hole in it. - -The effective role is min(user, token) recomputed per request, so demoting -somebody demotes their tokens with them. A stale cookie beside a valid -bearer falls through rather than refusing a credential that would work." -``` - ---- - -### Task 6: Scope enforcement with a boot-time completeness check - -**Files:** -- Create: `server/internal/api/scopes.go` -- Modify: `server/internal/api/handlers.go` (`RegisterRoutes`, line 43-48; end of function) -- Modify: `server/cmd/main.go` (after routes are registered, before serving) - -**Interfaces:** -- Consumes: `auth.IsToken`, `auth.Scopes`, `services.ScopeSatisfied`. -- Produces: `api.RequireScopes() gin.HandlerFunc`; `api.AssertScopeMapComplete(r *gin.Engine) error`. - -- [ ] **Step 1: Write the scope map and middleware** - -Create `server/internal/api/scopes.go`: - -```go -package api - -import ( - "fmt" - "net/http" - "sort" - "strings" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "github.com/gin-gonic/gin" -) - -// routeScopes maps a registered gin route — " " — to -// the scope an API token must hold to reach it. -// -// It is keyed on the route pattern rather than declared per route with a -// decorator, because a route registered without a decorator would be -// unguarded. AssertScopeMapComplete refuses to boot if any /api route is -// missing here, so the failure lands at deploy rather than as a surprise 403 -// in production. -// -// GET is read, everything else is write. The exceptions are written out rather -// than derived, because two of them are not obvious: reading a private key is -// still reading a key, and reading a container's logs is a write-level action -// because container output is arbitrary and cannot be masked. -var routeScopes = map[string]string{ - "GET /api/license": "settings:read", - "POST /api/license": "settings:write", - - "GET /api/servers": "servers:read", - "GET /api/servers/tags": "servers:read", - "POST /api/servers": "servers:write", - "GET /api/servers/new": "servers:write", - "POST /api/servers/new": "servers:write", - "GET /api/servers/:id": "servers:read", - "DELETE /api/servers/:id": "servers:write", - "POST /api/servers/:id/generate-key": "keys:write", - "POST /api/servers/:id/update-agent": "servers:write", - "POST /api/servers/:id/apply-updates": "servers:write", - "PUT /api/servers/:id/tags": "servers:write", - - "GET /api/agent/latest-version": "servers:read", - "GET /api/audit": "settings:read", - - "GET /api/settings": "settings:read", - "PUT /api/settings": "settings:write", - "POST /api/settings/secrets-token": "settings:write", - - "GET /api/secrets": "secrets:read", - "POST /api/secrets": "secrets:write", - "GET /api/secrets/:group": "secrets:read", - "PUT /api/secrets/:group": "secrets:write", - "POST /api/secrets/:group/reveal": "secrets:read", - "DELETE /api/secrets/:group": "secrets:write", - "DELETE /api/secrets/:group/:key": "secrets:write", - - "GET /api/keys": "keys:read", - "POST /api/keys": "keys:write", - "GET /api/keys/:id": "keys:read", - "GET /api/keys/:id/private-key": "keys:read", - "DELETE /api/keys/:id": "keys:write", - "POST /api/keys/:id/assign": "keys:write", - "DELETE /api/keys/:id/assign/:serverId": "keys:write", - - "POST /api/console/connect": "servers:write", - "GET /api/console/tunnel": "servers:write", - - "GET /api/vulnerabilities": "vulns:read", - "GET /api/vulnerabilities/summary": "vulns:read", - "POST /api/vulnerabilities/rescan": "vulns:write", - "POST /api/vulnerabilities/:id/accept": "vulns:write", - "DELETE /api/vulnerabilities/:id/accept": "vulns:write", - "GET /api/servers/:id/vulnerabilities": "vulns:read", - "GET /api/servers/:id/packages": "vulns:read", - "GET /api/packages/search": "vulns:read", - "GET /api/vuln-rules": "vulns:read", - "POST /api/vuln-rules": "vulns:write", - "PUT /api/vuln-rules/:id": "vulns:write", - "DELETE /api/vuln-rules/:id": "vulns:write", - - "GET /api/workloads": "workloads:read", - "GET /api/servers/:id/workloads": "workloads:read", - "POST /api/servers/:id/workloads/refresh": "workloads:read", - "POST /api/servers/:id/workloads/:wid/action": "workloads:write", - "GET /api/servers/:id/workloads/:wid/logs": "workloads:write", - - "GET /api/tokens": "settings:read", - "POST /api/tokens": "settings:write", - "DELETE /api/tokens/:id": "settings:write", -} - -// RequireScopes enforces routeScopes for token-authenticated requests and does -// nothing at all for cookie sessions, whose authority is their role. -func RequireScopes() gin.HandlerFunc { - return func(c *gin.Context) { - if !auth.IsToken(c) { - c.Next() - return - } - - key := c.Request.Method + " " + c.FullPath() - required, ok := routeScopes[key] - if !ok { - // Fail closed. An unmapped route reached by a token is a route - // nobody decided the authority for. - c.AbortWithStatusJSON(http.StatusForbidden, gin.H{ - "error": "this endpoint is not available to API tokens", - "code": "scope_unmapped", - }) - return - } - - if !services.ScopeSatisfied(auth.Scopes(c), required) { - c.AbortWithStatusJSON(http.StatusForbidden, gin.H{ - "error": fmt.Sprintf("token is missing the %q scope", required), - "code": "scope_missing", - "required_scope": required, - }) - return - } - c.Next() - } -} - -// AssertScopeMapComplete fails boot when a registered /api route has no scope. -// -// Without it, adding a route silently makes it unreachable by every token, and -// the report arrives as a customer asking why their script gets 403. -func AssertScopeMapComplete(r *gin.Engine) error { - var missing []string - for _, route := range r.Routes() { - if !strings.HasPrefix(route.Path, "/api/") { - continue - } - // The ESO endpoint keeps its own bearer scheme and is deliberately - // outside the token vocabulary. - if route.Path == "/api/secrets/:group/values" { - continue - } - if _, ok := routeScopes[route.Method+" "+route.Path]; !ok { - missing = append(missing, route.Method+" "+route.Path) - } - } - if len(missing) > 0 { - sort.Strings(missing) - return fmt.Errorf("routes missing from the API token scope map: %s", strings.Join(missing, ", ")) - } - return nil -} -``` - -- [ ] **Step 2: Mount the middleware** - -In `server/internal/api/handlers.go`, extend the `/api` group's middleware stack: - -```go - apiGroup := r.Group("/api") - apiGroup.Use(auth.Middleware()) - // Scope enforcement sits between authentication and the licence gate, and - // no-ops for cookie sessions. It is mounted here rather than per route so - // a route added later is covered by where it lives, not by memory. - apiGroup.Use(RequireScopes()) - apiGroup.Use(RequireActiveLicense()) -``` - -- [ ] **Step 3: Call the completeness check at boot** - -In `server/cmd/main.go`, after `api.RegisterRoutes(r)` and before the HTTP server starts: - -```go - if err := api.AssertScopeMapComplete(r); err != nil { - log.Fatalf("api scope map: %v", err) - } -``` - -- [ ] **Step 4: Verify** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. - -Start the server. If it exits with `api scope map: routes missing from the API token scope map: …`, add each listed route to `routeScopes` with the scope its resource implies, then start again. Boot must reach the normal listening log line. - -The workflow, monitor and channel routes are registered by `registerWorkflowRoutes`, `registerMonitorRoutes` and `registerChannelRoutes`, so their exact patterns come from this check rather than from guessing. Map every workflow, step and run route to `workflows:read` for GET and `workflows:write` otherwise; every monitor and incident route to `monitors:read`/`monitors:write`; every channel route to `monitors:read`/`monitors:write`, since channels exist to serve alerts. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/api/scopes.go server/internal/api/handlers.go server/cmd/main.go -git commit -m "feat: Enforce API token scopes from the route map - -Keyed on the registered gin route pattern rather than a per-route -decorator, because a route registered without a decorator would be -unguarded. An unmapped route reached by a token is a 403, and a boot-time -check refuses to start when any /api route is missing, so the failure -lands at deploy rather than as a customer's surprise 403." -``` - ---- - -### Task 7: Token endpoints and audit - -**Files:** -- Create: `server/internal/api/tokens.go` -- Modify: `server/internal/api/handlers.go` (register the three routes inside `apiGroup`) -- Modify: `server/internal/services/audit.go` only if it holds a category or label table that new event names must join — check first with `grep -n "token\|category" server/internal/services/audit.go` -- Modify: `web/lib/auditEvents.ts` (label the new events) - -**Interfaces:** -- Consumes: `services.CreateAPIToken`, `services.ListAPITokens`, `services.RevokeAPIToken`, `services.AllScopes`, `services.ErrTokenNameTaken`, `services.ErrTokenRoleTooHigh`, `services.ErrTokenExpiryPolicy`, `services.ErrInvalidScope`, `services.ErrTokenNotFound`, `services.GetUserInInstance`. -- Produces: `GET /api/tokens`, `POST /api/tokens`, `DELETE /api/tokens/:id`, `GET /api/tokens/scopes`. - -- [ ] **Step 1: Write the handlers** - -Create `server/internal/api/tokens.go`: - -```go -package api - -import ( - "errors" - "fmt" - "net/http" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/services" - "github.com/gin-gonic/gin" -) - -func elevated(c *gin.Context) bool { - r := auth.Role(c) - return r == models.RoleOwner || r == models.RoleAdmin -} - -// listTokens returns the caller's own tokens. Owner and admin may ask for every -// token in the instance with ?all=true. -func listTokens(c *gin.Context) { - all := c.Query("all") == "true" && elevated(c) - tokens, err := services.ListAPITokens(auth.InstanceID(c), auth.UserID(c), all) - if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - c.JSON(http.StatusOK, gin.H{"tokens": tokens, "all": all}) -} - -// listTokenScopes advertises the vocabulary so the UI never hardcodes it. -func listTokenScopes(c *gin.Context) { - c.JSON(http.StatusOK, gin.H{"scopes": services.AllScopes()}) -} - -func createToken(c *gin.Context) { - var body struct { - Name string `json:"name" binding:"required"` - Role string `json:"role" binding:"required"` - Scopes []string `json:"scopes" binding:"required"` - ExpiresInDays *int `json:"expires_in_days"` - } - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - - tok, plaintext, err := services.CreateAPIToken( - auth.InstanceID(c), auth.UserID(c), - body.Name, body.Role, body.Scopes, body.ExpiresInDays, c.ClientIP(), - ) - switch { - case errors.Is(err, services.ErrTokenNameTaken): - c.JSON(http.StatusConflict, gin.H{"error": err.Error(), "code": "name_taken"}) - return - case errors.Is(err, services.ErrTokenRoleTooHigh): - c.JSON(http.StatusForbidden, gin.H{"error": err.Error(), "code": "role_too_high"}) - return - case errors.Is(err, services.ErrTokenExpiryPolicy): - c.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error(), "code": "expiry_policy"}) - return - case errors.Is(err, services.ErrInvalidScope): - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error(), "code": "invalid_scope"}) - return - case errors.Is(err, services.ErrTokenInvalid): - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - case err != nil: - // Anything left is a backend failure. Reporting it as 400 tells the - // caller their request was malformed while the database is down. - c.JSON(http.StatusInternalServerError, gin.H{"error": "could not create token"}) - return - } - - expiry := "no expiry" - if tok.ExpiresAt != nil { - expiry = "expires " + tok.ExpiresAt.Format("2006-01-02") - } - services.LogEvent(auth.InstanceID(c), "token.created", actorFromCtx(c), "", "", - fmt.Sprintf("API token '%s' created with role %s, scopes %v, %s", tok.Name, tok.Role, tok.Scopes, expiry)) - - // The plaintext is returned exactly once and is not stored anywhere. - c.JSON(http.StatusCreated, gin.H{"token": plaintext, "record": tok}) -} - -func revokeToken(c *gin.Context) { - requester, err := services.GetUserInInstance(auth.InstanceID(c), auth.UserID(c)) - if err != nil { - c.JSON(http.StatusUnauthorized, gin.H{"error": "user not found"}) - return - } - - tok, err := services.RevokeAPIToken(auth.InstanceID(c), c.Param("id"), requester) - if errors.Is(err, services.ErrTokenNotFound) { - c.JSON(http.StatusNotFound, gin.H{"error": "token not found"}) - return - } - if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - - services.LogEvent(auth.InstanceID(c), "token.revoked", actorFromCtx(c), "", "", - fmt.Sprintf("API token '%s' revoked", tok.Name)) - c.JSON(http.StatusOK, gin.H{"revoked": true}) -} -``` - -- [ ] **Step 2: Register the routes** - -In `server/internal/api/handlers.go`, inside the `apiGroup` block, after the `/audit` line: - -```go - apiGroup.GET("/tokens", listTokens) - apiGroup.GET("/tokens/scopes", listTokenScopes) - apiGroup.POST("/tokens", createToken) - apiGroup.DELETE("/tokens/:id", revokeToken) -``` - -Then add the matching entry for `/api/tokens/scopes` to `routeScopes` in `server/internal/api/scopes.go`: - -```go - "GET /api/tokens/scopes": "settings:read", -``` - -- [ ] **Step 3: Record the acting credential on every audit event** - -The spec asks for `via: "token:"` on every event written during a token request. It is implemented on the **actor** field rather than appended to each event's detail, because detail is composed at ~60 call sites and the actor is composed in one function. Same information, one edit instead of sixty. In `server/internal/api/handlers.go`, change `actorFromCtx` so a token-authenticated request is distinguishable from a person clicking: - -```go -func actorFromCtx(c *gin.Context) string { - sess := auth.GetSessionFromContext(c) - if sess == nil || sess.Email == "" { - return "admin" - } - // The actor stays the human, because a token acts on their behalf and the - // log has to name somebody. The credential is appended so a person clicking - // and their CI job are told apart. - if sess.TokenID != "" { - return fmt.Sprintf("%s (via token:%s)", sess.Email, sess.TokenName) - } - return sess.Email -} -``` - -- [ ] **Step 4: Label the new events in the web audit page** - -In `web/lib/auditEvents.ts`, follow the existing shape of the file and add entries for `token.created`, `token.revoked`, `token.expired_use` and `settings.token_policy_updated`. Read the file first and match whatever structure it uses — a map of event key to label and category. Suggested labels: "API token created", "API token revoked", "Expired API token used", "API token policy updated". Category: whatever the file uses for `settings.updated`. - -- [ ] **Step 5: Verify the whole loop with curl** - -Restart the server, then with a browser session cookie in `cookies.txt`: - -```bash -# create -curl -s -b cookies.txt -X POST localhost:8080/api/tokens \ - -H 'Content-Type: application/json' \ - -d '{"name":"ci","role":"member","scopes":["servers:read"],"expires_in_days":30}' -``` - -Expected: 201 with `{"token":"vt_…","record":{…}}`. Copy the plaintext into `$VT`. - -```bash -# it authenticates -curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $VT" localhost:8080/api/servers -# scope enforcement -curl -s -w '\n%{http_code}\n' -H "Authorization: Bearer $VT" localhost:8080/api/keys -# role cap: member cannot reach an owner|admin route even with the scope -curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $VT" localhost:8080/api/settings -# list, then revoke -curl -s -b cookies.txt localhost:8080/api/tokens | head -c 400 -curl -s -b cookies.txt -X DELETE localhost:8080/api/tokens/ -curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $VT" localhost:8080/api/servers -``` - -Expected in order: `200`; `403` with `"required_scope":"keys:read"`; `403` from the role guard; the JSON list; `{"revoked":true}`; `401`. - -Check the audit page shows `token.created` and `token.revoked` with the actor as your email. - -- [ ] **Step 6: Commit** - -```bash -git add server/internal/api/tokens.go server/internal/api/handlers.go server/internal/api/scopes.go web/lib/auditEvents.ts -git commit -m "feat: Add the API token endpoints - -Create, list and revoke, with no update: editing what a credential already -deployed in CI can do, with no record of what it could do before, is worse -than requiring a rotation. Revoking a token that is not yours answers -not-found, since a 403 confirms it exists. - -The audit actor stays the human and names the credential alongside, so a -person clicking and their CI job are told apart." -``` - ---- - -### Task 8: Per-token rate limit - -**Files:** -- Create: `server/internal/api/ratelimit.go` -- Modify: `server/internal/api/handlers.go` (mount after `RequireScopes()`) -- Modify: `server/internal/auth/session.go` — add an exported accessor for the Redis client if none exists; check with `grep -n "func.*redis.Client\|var rdb" server/internal/auth/session.go` - -**Interfaces:** -- Consumes: `auth.IsToken`, `auth.TokenID`, the session Redis client. -- Produces: `api.RateLimitTokens() gin.HandlerFunc`; `auth.Redis() *redis.Client`. - -- [ ] **Step 1: Expose the Redis client** - -In `server/internal/auth/session.go`, add: - -```go -// Redis exposes the session client for callers that need a counter rather than -// a session. There is one Redis in this deployment and adding a second client -// would double the connection pool for no reason. -func Redis() *redis.Client { return rdb } -``` - -- [ ] **Step 2: Write the limiter** - -Create `server/internal/api/ratelimit.go`: - -```go -package api - -import ( - "net/http" - "strconv" - "time" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/auth" - "github.com/gin-gonic/gin" -) - -// tokenRateLimit is per token per minute. It is not the general API -// rate-limiting project: it is only enough that a runaway script cannot take an -// instance down, and cookie sessions are deliberately untouched. -const tokenRateLimit = 600 - -// RateLimitTokens counts requests per token in a one-minute fixed window. -// -// A fixed window rather than a sliding one because the cost of a burst at a -// boundary is a script running twice as fast for one second, and a sliding -// window is a sorted set per token for that. -func RateLimitTokens() gin.HandlerFunc { - return func(c *gin.Context) { - if !auth.IsToken(c) { - c.Next() - return - } - rdb := auth.Redis() - if rdb == nil { - c.Next() - return - } - - window := time.Now().UTC().Unix() / 60 - key := "vantage:tokenrate:" + auth.TokenID(c) + ":" + strconv.FormatInt(window, 10) - - count, err := rdb.Incr(c.Request.Context(), key).Result() - if err != nil { - // Redis is already required for sessions, so it being down is a - // larger problem than this. Do not turn it into a second outage. - c.Next() - return - } - if count == 1 { - rdb.Expire(c.Request.Context(), key, 2*time.Minute) - } - if count > tokenRateLimit { - c.Header("Retry-After", "60") - c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{ - "error": "rate limit exceeded for this API token", - "code": "rate_limited", - }) - return - } - c.Next() - } -} -``` - -- [ ] **Step 3: Mount it** - -In `server/internal/api/handlers.go`: - -```go - apiGroup.Use(RequireScopes()) - apiGroup.Use(RateLimitTokens()) - apiGroup.Use(RequireActiveLicense()) -``` - -- [ ] **Step 4: Verify** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. Then, with `$VT` a valid token: - -```bash -for i in $(seq 1 610); do - curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $VT" localhost:8080/api/servers -done | sort | uniq -c -``` - -Expected: roughly 600 lines of `200` and the remainder `429`. - -- [ ] **Step 5: Commit** - -```bash -git add server/internal/api/ratelimit.go server/internal/api/handlers.go server/internal/auth/session.go -git commit -m "feat: Rate limit API token requests - -600 per minute per token, in the Redis that sessions already require. -Cookie sessions are untouched. A Redis failure falls through rather than -refusing traffic — it is already a larger problem and should not become a -second outage." -``` - ---- - -### Task 9: API tokens card in web settings - -**Files:** -- Create: `web/components/settings/ApiTokensCard.tsx` -- Modify: `web/lib/api.ts` (types and client methods) -- Modify: `web/app/(app)/settings/page.tsx` (render the card in the Access group; add the policy field to the settings form) - -**Interfaces:** -- Consumes: `GET/POST /api/tokens`, `DELETE /api/tokens/:id`, `GET /api/tokens/scopes`, `api_token_max_days` on settings. -- Produces: `ApiToken` type; `api.listApiTokens`, `api.listTokenScopes`, `api.createApiToken`, `api.revokeApiToken`; the `` component. - -- [ ] **Step 1: Add the client methods** - -In `web/lib/api.ts`, beside the existing settings methods, add the type and the four calls, matching the file's existing `request` style: - -```ts -export type ApiToken = { - token_id: string; - name: string; - hint: string; - role: Role; - scopes: string[]; - expires_at?: string | null; - created_at: string; - last_used_at?: string | null; - user_id: string; - user_email?: string; -}; -``` - -```ts - listApiTokens(all = false): Promise<{ tokens: ApiToken[]; all: boolean }> { - return request<{ tokens: ApiToken[]; all: boolean }>(`/tokens${all ? "?all=true" : ""}`); - }, - - listTokenScopes(): Promise<{ scopes: string[] }> { - return request<{ scopes: string[] }>("/tokens/scopes"); - }, - - createApiToken(body: { name: string; role: Role; scopes: string[]; expires_in_days?: number | null }): Promise<{ token: string; record: ApiToken }> { - return request<{ token: string; record: ApiToken }>("/tokens", { - method: "POST", - body: JSON.stringify(body), - }); - }, - - revokeApiToken(tokenId: string): Promise<{ revoked: boolean }> { - return request<{ revoked: boolean }>(`/tokens/${tokenId}`, { method: "DELETE" }); - }, -``` - -Also extend the `Settings` type with `api_token_max_days?: number | null` and the `saveSettings` argument type with the same field. - -- [ ] **Step 2: Build the card** - -Create `web/components/settings/ApiTokensCard.tsx`, modelled directly on `MembersCard.tsx` — same imports, same `SectionCard` wrapper, same `Table`/`Badge`/`Modal`/`ConfirmDialog`/`useToast` usage, same `Field` and `inputClass`. - -Requirements the component must meet: - -- `useQuery({ queryKey: ["api-tokens", showAll], queryFn: () => api.listApiTokens(showAll) })`. -- Owner and admin see an "All tokens" toggle; members do not. Read the current user from `useAuth()` exactly as `MembersCard` does. -- Table columns: Name (with `hint` beneath in `font-mono text-xs text-text-secondary`), Owner (only when `showAll`), Role badge reusing `MembersCard`'s `roleVariant`, Scopes as `Badge` chips, Last used, Expires, and a Revoke button. -- Expiry cell states: `—` and the text "never" when `expires_at` is null; the date in `text-warning` when it is within 7 days; `text-danger` with the word "expired" when it has passed. -- When `settings.api_token_max_days > 0` and a listed token has no expiry or one beyond the cap, show a `text-warning` note on that row reading "outside the current policy — rotate when convenient". The policy is not applied retroactively, so this is a prompt, not a failure. -- Create opens a `Modal` with: name input, role select limited to roles at or below the current user's, a checkbox grid of scopes from `api.listTokenScopes()` grouped by resource with read and write side by side, and an expiry select of 30 / 60 / 90 / 365 days plus "Never". "Never" is disabled with an explanatory note when the policy caps lifetime, and any option above the cap is likewise disabled. -- On success the modal switches to a result state showing the plaintext in `bg-well rounded p-3 font-mono text-sm break-all`, a Copy button using `navigator.clipboard.writeText`, and the line "This is the only time this token will be shown. Store it now." Closing the result state invalidates the query. -- Errors surface through `friendlyMessage(error)` as `MembersCard` does. The 409, 403 and 422 codes must produce readable messages rather than raw JSON. -- No hex colours anywhere. Token classes only. - -- [ ] **Step 3: Render it and add the policy field** - -In `web/app/(app)/settings/page.tsx`: - -```tsx -import { ApiTokensCard } from "@/components/settings/ApiTokensCard"; -``` - -Render `` inside the Access group, immediately after `` and before ``. - -Add a number input for `api_token_max_days` to the existing settings form, following the `numberInputClass` and `Field` pattern already in that file. Label: "Maximum API token lifetime (days)". Helper text: "0 means no cap, and tokens may be created with no expiry. Changing this affects new tokens only." Visible to owner and admin only, matching how the file already gates owner/admin fields. - -- [ ] **Step 4: Verify** - -```bash -cd /go-projects/vantage/web && npm run lint && npx tsc --noEmit && npm run build -``` - -Expected: all three succeed with no errors. - -Then in the browser at `/settings`: - -1. Access group shows the API tokens card. -2. Create a token named `laptop`, role `member`, scopes `servers:read` and `workflows:write`, expiry 30 days. The plaintext appears once; Copy works. -3. Reload. The token is listed with its hint, role and scope chips, and no plaintext. -4. Set "Maximum API token lifetime" to 7, save, reopen the create modal. "Never", 30, 60, 90 and 365 are all disabled or refused with the policy message. -5. Revoke the token. It disappears and the audit page shows `token.revoked`. - -- [ ] **Step 5: Commit** - -```bash -git add web/components/settings/ApiTokensCard.tsx web/lib/api.ts "web/app/(app)/settings/page.tsx" -git commit -m "feat: Manage API tokens from settings - -A card in the Access group beside Members and single sign-on rather than a -new nav entry — /settings/instance was folded back in for exactly this -reason. The plaintext is shown once in a well block and never again. - -Tokens outside a newly tightened lifetime policy are flagged rather than -broken, because the policy governs issuance, not existing credentials." -``` - ---- - -### Task 10: OpenAPI 3.1 generation and the Scalar reference page - -**Files:** -- Create: `server/internal/api/docs/doc.go` (embed directives) -- Create: `server/internal/api/docs/scalar.standalone.js` (vendored) -- Create: `server/internal/api/docs/openapi.json` (generated, committed) -- Create: `server/internal/api/openapi.go` (the two routes and the HTML page) -- Modify: every handler file under `server/internal/api/` (annotations) -- Modify: `server/cmd/main.go` (top-level swag annotations) -- Modify: `server/internal/api/handlers.go` (register the doc routes, add them to `routeScopes`) -- Modify: `.gitea/workflows/server-deploy.yml` (regenerate-and-diff check) - -**Interfaces:** -- Consumes: everything above. -- Produces: `GET /api/openapi.json`, `GET /api/docs`. - -- [ ] **Step 1: Install swag v2 and confirm the version** - -```bash -cd /go-projects/vantage && go install github.com/swaggo/swag/v2/cmd/swag@latest && swag --version -``` - -Expected: a version line beginning with `v2.`. If the binary reports v1, stop and pin explicitly with `@v2.0.0-rc4` or the newest v2 tag from https://github.com/swaggo/swag/releases — v1 emits Swagger 2.0, which Scalar renders poorly, and the whole reason for choosing generation over a hand-written document was that the output is trustworthy. - -- [ ] **Step 2: Add the top-level API annotations** - -In `server/cmd/main.go`, above `func main()`: - -```go -// @title Vantage API -// @version 1.0 -// @description The Vantage control plane REST API. Authenticate with a browser session cookie, or with an API token created under Settings → API tokens. -// @BasePath /api -// -// @securityDefinitions.apikey cookieAuth -// @in cookie -// @name km_session -// -// @securityDefinitions.apikey bearerAuth -// @in header -// @name Authorization -// @description An API token, sent as "Bearer vt_…". Scoped and optionally expiring. -// -// @securityDefinitions.apikey esoAuth -// @in header -// @name Authorization -// @description The External Secrets read token, rotated under Settings. It reaches /api/secrets/{group}/values and nothing else. It is a different credential from an API token, and the two must never be substituted for one another. -``` - -- [ ] **Step 3: Annotate the handlers** - -Work file by file through `server/internal/api/`: `handlers.go`, `tokens.go`, `secrets.go`, `workflows.go`, `monitors.go`, `channels.go`, `console.go`, `vulnerabilities.go`, `workloads.go`, `licence.go`, `auth_providers.go`, `instance.go`. - -Each exported route handler gets a block in this shape — `listTokens` as the worked example: - -```go -// listTokens godoc -// -// @Summary List API tokens -// @Description Returns the caller's own tokens. Owner and admin may pass all=true to see every token in the instance. -// @Tags tokens -// @Produce json -// @Param all query bool false "Include every token in the instance (owner and admin only)" -// @Success 200 {object} ListTokensResponse -// @Failure 401 {object} ErrorResponse -// @Failure 403 {object} ErrorResponse -// @Security cookieAuth -// @Security bearerAuth -// @Router /tokens [get] -func listTokens(c *gin.Context) { -``` - -Anonymous inline response structs must become named types for swag to describe them. Create `server/internal/api/types.go` and move them there as they are converted, starting with: - -```go -package api - -import "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/models" - -// ErrorResponse is the shape every failing endpoint answers with. Some also -// carry a machine-readable code; it is omitted when absent rather than empty. -type ErrorResponse struct { - Error string `json:"error"` - Code string `json:"code,omitempty"` -} - -type ListTokensResponse struct { - Tokens []models.APIToken `json:"tokens"` - All bool `json:"all"` -} - -type CreateTokenRequest struct { - Name string `json:"name"` - Role string `json:"role"` - Scopes []string `json:"scopes"` - ExpiresInDays *int `json:"expires_in_days,omitempty"` -} - -type CreateTokenResponse struct { - // Token is the plaintext, returned exactly once and stored nowhere. - Token string `json:"token"` - Record models.APIToken `json:"record"` -} -``` - -Replace the corresponding `gin.H{...}` literals in the handlers with these types as you go, so the annotation and the response cannot disagree. This is the bulk of the work in this plan and it touches handler code the feature otherwise has no business in — that is the accepted cost of generated documentation over a hand-written file. - -- [ ] **Step 4: Generate the document** - -```bash -cd /go-projects/vantage/server && swag init \ - --generalInfo cmd/main.go \ - --dir ./,../shared \ - --output internal/api/docs \ - --outputTypes json \ - --v3.1 -``` - -Expected: `internal/api/docs/openapi.json` written, no `cannot find type definition` errors. Each such error names a type still declared inline; convert it in `types.go` and rerun. - -Delete any `docs.go` or `swagger.yaml` swag also emits — only `openapi.json` is wanted, and a stray generated Go file in that package will fight the hand-written `doc.go` in the next step. - -- [ ] **Step 5: Vendor Scalar** - -```bash -cd /go-projects/vantage/server/internal/api/docs && \ - curl -fsSL -o scalar.standalone.js https://cdn.jsdelivr.net/npm/@scalar/api-reference@latest/dist/browser/standalone.js && \ - ls -lh scalar.standalone.js -``` - -Expected: a file of roughly 1MB. Record the version by pinning the URL you actually used in a comment in the next step — no CDN reference remains at runtime, because air-gapped self-hosted installs are supported and a reference page that fails closed offline is a support ticket. - -- [ ] **Step 6: Embed and serve** - -Create `server/internal/api/docs/doc.go`: - -```go -// Package docs holds the generated OpenAPI document and the vendored Scalar -// bundle that renders it. -// -// openapi.json is generated by `swag init` and committed rather than built into -// the image: server/Dockerfile produces a scratch runtime from a Go build -// stage, and adding codegen there means putting the toolchain in the image. -// server-deploy.yml regenerates and diffs it, so an annotation edited without -// regenerating fails the build. -// -// scalar.standalone.js is vendored from -// https://cdn.jsdelivr.net/npm/@scalar/api-reference@latest/dist/browser/standalone.js -// and refreshed by hand. Fetched at build time it would break an air-gapped -// install; fetched at page load it would break an air-gapped install more -// visibly. -package docs - -import _ "embed" - -//go:embed openapi.json -var OpenAPI []byte - -//go:embed scalar.standalone.js -var ScalarJS []byte -``` - -Create `server/internal/api/openapi.go`: - -```go -package api - -import ( - "net/http" - - "gitea.hostxtra.co.uk/mrhid6/vantage/server/internal/api/docs" - "github.com/gin-gonic/gin" -) - -// scalarPage renders the reference against this instance's own spec, so "Try -// it" acts on the reader's API with the reader's session. -const scalarPage = ` - - - Vantage API - - - - -
- - - -` - -func getOpenAPI(c *gin.Context) { - c.Data(http.StatusOK, "application/json; charset=utf-8", docs.OpenAPI) -} - -func getScalarJS(c *gin.Context) { - c.Data(http.StatusOK, "application/javascript; charset=utf-8", docs.ScalarJS) -} - -func getAPIDocs(c *gin.Context) { - c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(scalarPage)) -} -``` - -Register in `server/internal/api/handlers.go` inside `apiGroup`: - -```go - apiGroup.GET("/openapi.json", getOpenAPI) - apiGroup.GET("/docs", getAPIDocs) - apiGroup.GET("/docs/scalar.js", getScalarJS) -``` - -and add to `routeScopes` in `scopes.go`: - -```go - "GET /api/openapi.json": "settings:read", - "GET /api/docs": "settings:read", - "GET /api/docs/scalar.js": "settings:read", -``` - -- [ ] **Step 7: Add the CI drift check** - -In `.gitea/workflows/server-deploy.yml`, add a step to the job that builds the `server` image, before the docker build, following the file's existing step style: - -```yaml - - name: Verify the OpenAPI document is current - run: | - go install github.com/swaggo/swag/v2/cmd/swag@v2.0.0-rc4 - cd server - swag init --generalInfo cmd/main.go --dir ./,../shared \ - --output internal/api/docs --outputTypes json --v3.1 - git diff --exit-code internal/api/docs/openapi.json -``` - -Pin the same version you installed in Step 1. Without this step the annotations drift while appearing authoritative, which is worse than a hand-written file nobody claimed was generated. - -- [ ] **Step 8: Verify** - -```bash -cd /go-projects/vantage && go build ./... && go vet ./server/... -``` - -Expected: no output. - -With the server running and a session cookie: - -```bash -curl -s -b cookies.txt localhost:8080/api/openapi.json | head -c 200 -curl -s -o /dev/null -w '%{http_code} %{size_download}\n' -b cookies.txt localhost:8080/api/docs/scalar.js -``` - -Expected: JSON beginning `{"openapi":"3.1.0"`, then `200` with a size near 1000000. - -Open `http://localhost:8080/api/docs` in a browser: the Scalar reference renders, lists the tag groups, and shows both `cookieAuth` and `bearerAuth`. Confirm the network tab makes no request to any external host. - -Then confirm the drift check does its job: - -```bash -cd /go-projects/vantage/server && sed -i 's/@Summary List API tokens/@Summary List all API tokens/' internal/api/tokens.go -swag init --generalInfo cmd/main.go --dir ./,../shared --output internal/api/docs --outputTypes json --v3.1 -git diff --stat internal/api/docs/openapi.json -``` - -Expected: a non-empty diff. Revert the `sed` edit, regenerate, and confirm `git diff --exit-code internal/api/docs/openapi.json` is clean before committing. - -- [ ] **Step 9: Commit** - -```bash -git add server/internal/api/docs server/internal/api/openapi.go server/internal/api/types.go server/internal/api server/cmd/main.go .gitea/workflows/server-deploy.yml -git commit -m "feat: Publish an OpenAPI 3.1 document and a Scalar reference - -Generated from swaggo v2 annotations, committed rather than built into the -image: the runtime stage is scratch and adding codegen puts the toolchain -in the build. CI regenerates and diffs, so an annotation edited without -regenerating fails the build — without that the annotations would drift -while still looking authoritative. - -Scalar is vendored rather than loaded from a CDN, because air-gapped -self-hosted installs are supported and a reference page that fails closed -offline is a support ticket." -``` - ---- - -### Task 11: Documentation - -**Files:** -- Create: `docsite/docs/reference/api-tokens.md` -- Modify: `docsite/sidebars.ts` -- Modify: `CLAUDE.md` - -**Interfaces:** -- Consumes: everything above. -- Produces: user documentation. - -- [ ] **Step 1: Write the reference page** - -Create `docsite/docs/reference/api-tokens.md`, matching the front-matter and heading style of the other files in `docsite/docs/reference/`. Cover, in this order: - -1. What a token is and where to create one (Settings → API tokens). -2. That the value is shown once and stored nowhere, so a lost token is rotated, not recovered. -3. The scope table — all eight resources, read and write, and that write implies read. -4. That a token never exceeds its owner's role, and that demoting or removing the person demotes or removes the token. -5. Expiry, and the instance-wide maximum lifetime setting, including that changing the setting affects new tokens only. -6. A curl example: - - ```bash - curl -H "Authorization: Bearer vt_…" https://acme.vantage.example.com/api/servers - ``` - -7. The rate limit — 600 requests per minute per token, answered as 429 with `Retry-After`. -8. Rotation guidance: create the replacement, deploy it, then revoke the old one. -9. A pointer to `/api/docs` on the reader's own instance for the full reference. -10. An explicit warning that the External Secrets read token is a different credential and an API token must not be used in its place. - -Add the page to `docsite/sidebars.ts` in the Reference section, which is authored by hand. - -- [ ] **Step 2: Update CLAUDE.md** - -Three edits, each placed with the material it belongs to: - -- Under **REST API**, in the session-authed table, add: - - ``` - tokens GET /tokens · GET /tokens/scopes · POST /tokens · DELETE /tokens/:id - GET /openapi.json · GET /docs - ``` - -- Under **MongoDB Collections**, add `api_tokens` to the collection list, and a note beneath the existing bullets: - - > - `api_tokens` stores only `sha256` of the token, like `servers.agent_token_hash`. A token's effective role is `min(user.role, token.role)` **recomputed per request**, so demoting somebody demotes their tokens; deleting the user deletes them. Scopes are enforced from a map keyed on the registered gin route pattern, and `AssertScopeMapComplete` **fails boot** when an `/api` route is missing from it — a route added without an entry would otherwise be silently unreachable by every token. - -- Add a short subsection under **Subsystems** describing API tokens and the OpenAPI document, including that `openapi.json` is generated by swag v2, committed, and verified in CI by regenerate-and-diff, and that the Scalar bundle is vendored because air-gapped installs are supported. - -- [ ] **Step 3: Verify the docs build** - -```bash -cd /go-projects/vantage/docsite && npm run build -``` - -Expected: build succeeds, and the output mentions no broken links. A broken link is fatal in Docusaurus's default config, so a wrong sidebar path fails here rather than in production. - -- [ ] **Step 4: Commit** - -```bash -git add docsite/docs/reference/api-tokens.md docsite/sidebars.ts CLAUDE.md -git commit -m "docs: Document API tokens and the OpenAPI reference" -``` - ---- - -### Task 12: Refresh the knowledge graph - -**Files:** -- Modify: `graphify-out/` (regenerated) - -- [ ] **Step 1: Update the graph** - -```bash -cd /go-projects/vantage && graphify update . -``` - -Expected: completes without error; `graphify-out/graph.json` is newer than before. - -- [ ] **Step 2: Commit** - -```bash -git add graphify-out -git commit -m "chore: Refresh the knowledge graph after API tokens" -``` diff --git a/docs/superpowers/plans/2026-08-12-instance-rename.md b/docs/superpowers/plans/2026-08-12-instance-rename.md deleted file mode 100644 index 37f6363..0000000 --- a/docs/superpowers/plans/2026-08-12-instance-rename.md +++ /dev/null @@ -1,938 +0,0 @@ -# Instance Rename in Vantage HQ — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let an HQ customer (owner or admin) rename a cloud instance, which re-derives its slug and moves it to a new DNS host, with staff able to do the same without the cooldown. - -**Architecture:** Slug derivation stays in `shared/provision`, beside the create path that already owns it. Admin reaches the control plane only through `cloudprov`, writing `instances` — a collection it already writes. Admin's own row (`admin_instances`) is updated second and carries the 24h cooldown timestamp, because the cooldown is admin's policy and the control plane has no opinion about it. The portal shows the new host and asks the customer to click through; it does not redirect. - -**Note:** This repo has no automated test suite and the user has ruled out adding test files. Every task verifies by build, vet and (Task 8) manual exercise. - -**Tech Stack:** Go 1.x (gin, mongo-driver v2), Next.js 16 App Router + TanStack Query + Tailwind 3 (`adminsite`). - -**Spec:** `docs/superpowers/specs/2026-08-12-instance-rename-design.md` - -## Global Constraints - -- A licence binds an instance **UUID**, not a slug. A rename must not issue a licence, call Paddle, or touch `licenses`, `subscriptions` or `entitlements`. -- Admin's control-plane write boundary is unchanged: `cloudprov` writes `instances` and `users` only. Do not add a write to any other control-plane collection. -- Customer rename is **cloud only**. Self-hosted is refused with the existing `selfHostedRefusal` constant and HTTP **400**, matching `members.go`. -- Cooldown for customers is **24 hours**, tracked by `admin_instances.renamed_at`. Staff bypass it and must **not** write `renamed_at`. -- No `-2` suffix loop on rename. A taken slug is a refusal (`ErrSlugTaken` → HTTP 409). -- No component in `adminsite` may carry a hex colour; use the existing token classes (`text-ink-2`, `text-ink-3`, `border-rule`, `text-accent`, `text-expired`, `bg-panel-2`). -- The host domain used for display is `vantage.hostxtra.co.uk`, already hardcoded in `adminsite/components/InstanceRecord.tsx` and the customer instance page. -- Commit messages follow the repo's existing style: `feat: Sentence case summary` / `fix: …` / `docs: …`. - ---- - -### Task 1: Slug derivation and the control-plane rename - -**Files:** -- Modify: `shared/provision/instance.go` -**Interfaces:** -- Consumes: `BaseSlug(name string) (string, error)`, `ErrNameRejected` — both already in `shared/provision`. -- Produces: - - `provision.ErrSlugTaken` (`error`) - - `provision.RenameSlug(name, currentSlug string) (string, error)` - - `provision.RenameInstance(ctx context.Context, db *mongo.Database, instanceID, name string) (*models.Instance, error)` - - `provision.RestoreInstanceIdentity(ctx context.Context, db *mongo.Database, instanceID, name, slug string) error` - -Behaviour `RenameSlug` must have, verified by reading rather than by test (this -repo has no Go test suite and the user has ruled out adding one): - -| Input name | Current slug | Result | -|---|---|---| -| `Acme Ltd` | `acme` | `acme-ltd` | -| `ACME!` | `acme` | `acme` — still derives to the current slug, so not a move | -| `Acme` | `acme-2` | `acme` — a creation-time collision suffix derives from no name, so moving off it is a real move | -| `ab` | any | `ErrNameRejected` | -| `Admin` | any | `ErrNameRejected` (reserved) | -| `!!!` | any | `ErrNameRejected` | -| 50 `a`s | any | truncated to `MaxSlugLength`, exactly as `BaseSlug` truncates on create | - -- [ ] **Step 1: Write the implementation** - -Append to `shared/provision/instance.go`: - -```go -// ErrSlugTaken means the slug a new name derives to already belongs to another -// instance. -// -// Rename refuses rather than appending a counter the way creation does. Creation -// appends because the customer is waiting on an instance and any free slug will -// do; a rename is a request for one specific host, and silently landing them on -// "acme-2" answers a question they did not ask. -var ErrSlugTaken = errors.New("slug taken") - -// RenameSlug derives the slug a rename to name would move an instance to, given -// the slug it holds now. -// -// It returns the current slug unchanged when the name still derives to it, so a -// cosmetic edit — capitalisation, punctuation, a trailing "Ltd." — is not a move -// and cannot collide with the instance's own slug. -func RenameSlug(name, currentSlug string) (string, error) { - base, err := BaseSlug(name) - if err != nil { - return "", fmt.Errorf("%w: %s", ErrNameRejected, err.Error()) - } - if base == currentSlug { - return currentSlug, nil - } - return base, nil -} - -// RenameInstance changes an instance's name and re-derives its slug from it. -// -// The count-then-update is racy on its own, and is safe for the same reason -// CreateInstanceWithID's loop is: instances.slug carries a unique index, so a -// lost race surfaces as a duplicate-key error. Unlike creation there is nothing -// to retry with — the caller asked for one specific name — so it becomes -// ErrSlugTaken. Do not remove the duplicate-key branch, and do not remove the -// index. -func RenameInstance(ctx context.Context, db *mongo.Database, instanceID, name string) (*models.Instance, error) { - var inst models.Instance - if err := db.Collection("instances").FindOne(ctx, - bson.M{"instance_id": instanceID}).Decode(&inst); err != nil { - return nil, err - } - - slug, err := RenameSlug(name, inst.Slug) - if err != nil { - return nil, err - } - - if slug != inst.Slug { - n, err := db.Collection("instances").CountDocuments(ctx, bson.M{ - "slug": slug, - "instance_id": bson.M{"$ne": instanceID}, - }) - if err != nil { - return nil, err - } - if n > 0 { - return nil, fmt.Errorf("%w: %s", ErrSlugTaken, slug) - } - } - - if _, err := db.Collection("instances").UpdateOne(ctx, - bson.M{"instance_id": instanceID}, - bson.M{"$set": bson.M{"name": name, "slug": slug}}); err != nil { - if mongo.IsDuplicateKeyError(err) { - return nil, fmt.Errorf("%w: %s", ErrSlugTaken, slug) - } - return nil, err - } - - inst.Name = name - inst.Slug = slug - return &inst, nil -} - -// RestoreInstanceIdentity writes an exact name and slug back, unwinding a rename -// whose caller-side bookkeeping then failed. -// -// It derives nothing. The values being restored may include a creation-time -// collision suffix that no name derives to, so re-running RenameInstance with the -// old name would not reproduce them. -func RestoreInstanceIdentity(ctx context.Context, db *mongo.Database, instanceID, name, slug string) error { - _, err := db.Collection("instances").UpdateOne(ctx, - bson.M{"instance_id": instanceID}, - bson.M{"$set": bson.M{"name": name, "slug": slug}}) - return err -} -``` - -- [ ] **Step 2: Build and vet** - -Run: `cd /go-projects/vantage && go build ./shared/... && go vet ./shared/provision/` -Expected: clean. - -- [ ] **Step 3: Commit** - -```bash -git add shared/provision/instance.go -git commit -m "feat: Add instance rename to shared provisioning" -``` - ---- - -### Task 2: Admin's row and the cloudprov wrappers - -**Files:** -- Modify: `admin/internal/models/models.go` (the `Instance` struct, ~line 129; constants block near `RenewWindow`, ~line 105) -- Modify: `admin/internal/cloudprov/cloudprov.go` - -**Interfaces:** -- Consumes: `provision.RenameInstance`, `provision.RestoreInstanceIdentity` (Task 1). -- Produces: - - `models.RenameCooldown` (`time.Duration`) - - `models.Instance.RenamedAt *time.Time` (bson `renamed_at`, json `renamed_at`) - - `cloudprov.RenameInstance(ctx context.Context, instanceID, name string) (*sharedmodels.Instance, error)` - - `cloudprov.RestoreInstanceIdentity(ctx context.Context, instanceID, name, slug string) error` - -- [ ] **Step 1: Add the cooldown constant** - -In `admin/internal/models/models.go`, directly beneath the `RenewWindow` block: - -```go -// RenameCooldown is how long a customer must wait between renames of one -// instance. -// -// A rename moves the instance's DNS host and invalidates every saved link to it, -// so this exists to make that a considered act rather than a slider. Staff are -// not subject to it: a support conversation about a name is already a human -// deciding. -const RenameCooldown = 24 * time.Hour -``` - -- [ ] **Step 2: Add the field to `Instance`** - -In the same file, inside the `Instance` struct, after `RelinkCount`: - -```go - // RenamedAt is when this instance last changed name, and backs the customer - // rename cooldown. It is a pointer because absent means "never renamed"; a - // zero time.Time would read as year 1 — an inert cooldown, but only by - // accident. Staff renames deliberately leave it alone. - RenamedAt *time.Time `bson:"renamed_at,omitempty" json:"renamed_at,omitempty"` -``` - -- [ ] **Step 3: Add the cloudprov wrappers** - -Append to `admin/internal/cloudprov/cloudprov.go`: - -```go -// RenameInstance changes a cloud instance's name and moves it to the slug that -// name derives to. -// -// It writes `instances` and nothing else, so admin's control-plane write -// boundary is unchanged. It issues no licence: a licence binds the instance -// UUID, which a rename never touches. -func RenameInstance(ctx context.Context, instanceID, name string) (*sharedmodels.Instance, error) { - return provision.RenameInstance(ctx, db.ControlDB(), instanceID, name) -} - -// RestoreInstanceIdentity puts an instance's previous name and slug back, for a -// caller unwinding a rename whose admin-side write failed. Leaving the two -// databases disagreeing would have HQ print a host that is not the host. -func RestoreInstanceIdentity(ctx context.Context, instanceID, name, slug string) error { - return provision.RestoreInstanceIdentity(ctx, db.ControlDB(), instanceID, name, slug) -} -``` - -- [ ] **Step 4: Build** - -Run: `cd /go-projects/vantage && go build ./admin/... ./shared/...` -Expected: clean build, no output. - -- [ ] **Step 5: Commit** - -```bash -git add admin/internal/models/models.go admin/internal/cloudprov/cloudprov.go -git commit -m "feat: Add rename cooldown field and cloudprov rename" -``` - ---- - -### Task 3: Customer rename endpoint - -**Files:** -- Modify: `admin/internal/api/customer.go` (add handler; `loginURLFor` at ~line 442 is already in this file) -- Modify: `admin/internal/api/routes.go` (~line 77, beside the other `/instances/:id/*` customer routes) - -**Interfaces:** -- Consumes: `ownedInstance(c, id) (*models.Instance, bool)`, `selfHostedRefusal` (`members.go`), `loginURLFor(slug) string`, `cloudprov.RenameInstance`, `cloudprov.RestoreInstanceIdentity`, `models.RenameCooldown`, `provision.ErrSlugTaken`, `provision.ErrNameRejected`. -- Produces: `PUT /api/instances/:id/name` returning `{instance_id, name, slug, login_url}`. - -- [ ] **Step 1: Write the handler** - -Append to `admin/internal/api/customer.go`: - -```go -// renameInstance changes a cloud instance's name and moves it to the slug that -// name derives to. -// -// The control plane is written FIRST, because instances.slug carries the unique -// index and that index is what actually settles a race between two accounts -// reaching for the same name. Admin's own row follows; if that write fails the -// control plane is put back, because HQ printing a host that is not the host is -// worse than a failed rename. -// -// No licence is issued and Paddle is not called: a licence binds the instance -// UUID, and a rename does not change it. -func renameInstance(c *gin.Context) { - inst, ok := ownedInstance(c, c.Param("id")) - if !ok { - return - } - if inst.Deployment != license.DeploymentCloud { - c.JSON(http.StatusBadRequest, gin.H{"error": selfHostedRefusal}) - return - } - if inst.Placeholder { - c.JSON(http.StatusConflict, gin.H{"error": "this instance is not provisioned yet"}) - return - } - - var body struct { - Name string `json:"name"` - } - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": "name is required"}) - return - } - name := strings.TrimSpace(body.Name) - if name == "" { - c.JSON(http.StatusBadRequest, gin.H{"error": "name is required"}) - return - } - - if inst.RenamedAt != nil { - if until := inst.RenamedAt.Add(models.RenameCooldown); time.Now().UTC().Before(until) { - c.JSON(http.StatusTooManyRequests, gin.H{ - "error": fmt.Sprintf("this instance was renamed recently; it can be renamed again after %s UTC", until.Format("2 Jan 2006 15:04")), - "retry_after": until, - }) - return - } - } - - ctx := c.Request.Context() - renamed, err := cloudprov.RenameInstance(ctx, inst.InstanceID, name) - switch { - case errors.Is(err, provision.ErrSlugTaken): - c.JSON(http.StatusConflict, gin.H{"error": "that name is already in use — try another"}) - return - case errors.Is(err, provision.ErrNameRejected): - c.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()}) - return - case err != nil: - log.Printf("renameInstance: control plane rename of %s: %v", inst.InstanceID, err) - c.JSON(http.StatusInternalServerError, gin.H{"error": "could not rename the instance"}) - return - } - - if _, err := db.Admin("admin_instances").UpdateOne(ctx, - bson.M{"instance_id": inst.InstanceID}, - bson.M{"$set": bson.M{ - "name": renamed.Name, - "slug": renamed.Slug, - "renamed_at": time.Now().UTC(), - }}); err != nil { - if rbErr := cloudprov.RestoreInstanceIdentity(ctx, inst.InstanceID, inst.Name, inst.Slug); rbErr != nil { - log.Printf("renameInstance: rollback of %s failed: %v", inst.InstanceID, rbErr) - } - log.Printf("renameInstance: record rename of %s: %v", inst.InstanceID, err) - c.JSON(http.StatusInternalServerError, gin.H{"error": "could not rename the instance"}) - return - } - - s := auth.Current(c) - audit.Write(ctx, models.AuditEntry{ - Actor: s.Email, Action: "instance.renamed", AccountID: s.AccountID, - Target: inst.InstanceID, Detail: inst.Slug + " -> " + renamed.Slug, IP: c.ClientIP()}) - - c.JSON(http.StatusOK, gin.H{ - "instance_id": inst.InstanceID, - "name": renamed.Name, - "slug": renamed.Slug, - // The same builder the licence emails use, rather than a second opinion - // about how a tenant host is spelled. Empty when APP_LOGIN_URL is unset. - "login_url": loginURLFor(renamed.Slug), - }) -} -``` - -- [ ] **Step 2: Check the imports** - -`customer.go` must import `errors`, `fmt`, `log`, `net/http`, `strings`, `time`, `audit`, `auth`, `cloudprov`, `db`, `models`, `license`, `provision`, `gin`, `bson`. Most are already there — add only what the compiler asks for. `provision` is `gitea.hostxtra.co.uk/mrhid6/vantage/shared/provision`; `license` is `gitea.hostxtra.co.uk/mrhid6/vantage/shared/license`. - -- [ ] **Step 3: Mount the route** - -In `admin/internal/api/routes.go`, in the `cust` group beside the other instance routes (after `cust.POST("/instances/:id/claim-free", …)`): - -```go - // Renaming moves the instance's DNS host, so it is owner-or-admin like - // every other instance mutation. Cloud only; the handler refuses the rest. - cust.PUT("/instances/:id/name", - auth.RequireAccountRole(models.AccountRoleOwner, models.AccountRoleAdmin), - renameInstance) -``` - -- [ ] **Step 4: Build** - -Run: `cd /go-projects/vantage && go build ./admin/... && go vet ./admin/internal/api/` -Expected: clean. - -- [ ] **Step 5: Commit** - -```bash -git add admin/internal/api/customer.go admin/internal/api/routes.go -git commit -m "feat: Add customer instance rename endpoint" -``` - ---- - -### Task 4: Staff rename endpoint - -**Files:** -- Modify: `admin/internal/api/staff.go` -- Modify: `admin/internal/api/routes.go` (the `staff` group, beside `staff.POST("/instances/:id/relink", …)`) - -**Interfaces:** -- Consumes: everything Task 3 consumes, plus `db.Admin`. -- Produces: `PUT /api/staff/instances/:id/name` returning `{instance_id, name, slug}`. - -- [ ] **Step 1: Write the handler** - -Append to `admin/internal/api/staff.go`: - -```go -// staffRenameInstance renames any instance, with no cooldown. -// -// It does NOT write renamed_at: a staff rename must not start the customer's -// 24h clock, or fixing a name for someone locks them out of fixing it further. -// -// On self-hosted it changes admin's label only. There is no control-plane row to -// write — the install is the customer's — and no slug, because self-hosted has -// no tenant subdomain. -func staffRenameInstance(c *gin.Context) { - var body struct { - Name string `json:"name"` - } - if err := c.ShouldBindJSON(&body); err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": "name is required"}) - return - } - name := strings.TrimSpace(body.Name) - if name == "" { - c.JSON(http.StatusBadRequest, gin.H{"error": "name is required"}) - return - } - - ctx := c.Request.Context() - var inst models.Instance - if err := db.Admin("admin_instances").FindOne(ctx, - bson.M{"instance_id": c.Param("id")}).Decode(&inst); err != nil { - c.JSON(http.StatusNotFound, gin.H{"error": "not found"}) - return - } - - set := bson.M{"name": name} - slug := inst.Slug - - if inst.Deployment == license.DeploymentCloud && !inst.Placeholder { - renamed, err := cloudprov.RenameInstance(ctx, inst.InstanceID, name) - switch { - case errors.Is(err, provision.ErrSlugTaken): - c.JSON(http.StatusConflict, gin.H{"error": "that name is already in use"}) - return - case errors.Is(err, provision.ErrNameRejected): - c.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()}) - return - case err != nil: - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - slug = renamed.Slug - set["slug"] = renamed.Slug - } - - if _, err := db.Admin("admin_instances").UpdateOne(ctx, - bson.M{"instance_id": inst.InstanceID}, bson.M{"$set": set}); err != nil { - if inst.Deployment == license.DeploymentCloud && !inst.Placeholder { - if rbErr := cloudprov.RestoreInstanceIdentity(ctx, inst.InstanceID, inst.Name, inst.Slug); rbErr != nil { - log.Printf("staffRenameInstance: rollback of %s failed: %v", inst.InstanceID, rbErr) - } - } - c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) - return - } - - audit.Write(ctx, models.AuditEntry{ - Actor: auth.Current(c).Email, Action: "instance.renamed", AccountID: inst.AccountID, - Target: inst.InstanceID, Detail: inst.Slug + " -> " + slug, IP: c.ClientIP()}) - - c.JSON(http.StatusOK, gin.H{"instance_id": inst.InstanceID, "name": name, "slug": slug}) -} -``` - -`staff.go` will need `errors`, `log`, `cloudprov` and `provision` added to its imports; `fmt`, `net/http`, `strings`, `time`, `audit`, `auth`, `db`, `models`, `license`, `bson` are already there. - -- [ ] **Step 2: Mount the route** - -In `routes.go`, in the `staff` group after `staff.POST("/instances/:id/relink", staffRelink)`: - -```go - staff.PUT("/instances/:id/name", staffRenameInstance) -``` - -- [ ] **Step 3: Build** - -Run: `cd /go-projects/vantage && go build ./admin/... && go vet ./admin/internal/api/` -Expected: clean. - -- [ ] **Step 4: Commit** - -```bash -git add admin/internal/api/staff.go admin/internal/api/routes.go -git commit -m "feat: Add staff instance rename endpoint" -``` - ---- - -### Task 5: `adminsite` API client and slug preview - -**Files:** -- Create: `adminsite/lib/slug.ts` -- Modify: `adminsite/lib/api.ts` (the `Instance` interface ~line 123; the `api` object's instance calls ~line 305; `api.staff` ~line 360) - -**Interfaces:** -- Consumes: `PUT /api/instances/:id/name`, `PUT /api/staff/instances/:id/name` (Tasks 3 and 4). -- Produces: - - `INSTANCE_DOMAIN`, `slugify(name: string): string`, `slugError(name: string): string | undefined` from `@/lib/slug` - - `RenameResult` interface, `api.renameInstance(id, name): Promise`, `api.staff.renameInstance(id, name): Promise` - - `Instance.renamed_at?: string` - -- [ ] **Step 1: Create the slug mirror** - -Create `adminsite/lib/slug.ts`: - -```ts -/* - * A TypeScript mirror of shared/provision's slug rules, used ONLY to preview the - * host a rename would move an instance to while the customer types. - * - * It is a second implementation of Slugify, BaseSlug and ReservedSlugs, and it - * must change in the same commit as the Go one — the same hazard as - * web/lib/targets.ts. The preview is a courtesy; the server's 409 is the - * boundary, and the two are allowed to disagree without anything breaking. - */ - -/** Mirrors provision.MinSlugLength / MaxSlugLength. */ -export const MIN_SLUG_LENGTH = 3; -export const MAX_SLUG_LENGTH = 40; - -/** Mirrors provision.ReservedSlugs. */ -const RESERVED = new Set([ - "www", "api", "app", "admin", "auth", - "install", "static", "_next", "default", -]); - -/* - * The tenant subdomain namespace. Also hardcoded in InstanceRecord.tsx and the - * customer instance page; those predate this file and are left alone rather than - * refactored under a rename change. - */ -export const INSTANCE_DOMAIN = "vantage.hostxtra.co.uk"; - -/** Mirrors provision.Slugify. */ -export function slugify(name: string): string { - return name - .toLowerCase() - .replace(/[^a-z0-9]+/g, "-") - .replace(/^-+|-+$/g, ""); -} - -/** Mirrors provision.BaseSlug's truncation. */ -export function baseSlug(name: string): string { - return slugify(name).slice(0, MAX_SLUG_LENGTH); -} - -/** The reason a name cannot become a slug, or undefined when it can. */ -export function slugError(name: string): string | undefined { - const base = slugify(name); - if (base.length < MIN_SLUG_LENGTH) { - return `Needs at least ${MIN_SLUG_LENGTH} letters or digits.`; - } - if (RESERVED.has(base.slice(0, MAX_SLUG_LENGTH))) { - return "That name is reserved."; - } - return undefined; -} - -/** The host an instance on this slug is reached at. */ -export function hostFor(slug: string): string { - return `${slug}.${INSTANCE_DOMAIN}`; -} -``` - -- [ ] **Step 2: Extend the API client** - -In `adminsite/lib/api.ts`, add `renamed_at` to `Instance` (after `relink_count`): - -```ts - renamed_at?: string; -``` - -Add the response type beside the other interfaces: - -```ts -export interface RenameResult { - instance_id: string; - name: string; - slug: string; - /** Empty when APP_LOGIN_URL is unset on the server. */ - login_url?: string; -} -``` - -Add the call to the `api` object, after `renewInstance`: - -```ts - renameInstance: (id: string, name: string) => - put(`/api/instances/${id}/name`, { name }), -``` - -And to `api.staff`, after `relink`: - -```ts - renameInstance: (id: string, name: string) => - put(`/api/staff/instances/${id}/name`, { name }), -``` - -- [ ] **Step 3: Type-check** - -Run: `cd /go-projects/vantage/adminsite && npx tsc --noEmit` -Expected: no errors. - -- [ ] **Step 4: Commit** - -```bash -git add adminsite/lib/slug.ts adminsite/lib/api.ts -git commit -m "feat: Add rename calls and slug preview to the HQ client" -``` - ---- - -### Task 6: The rename panel and the customer instance page - -**Files:** -- Create: `adminsite/components/RenamePanel.tsx` -- Modify: `adminsite/app/(customer)/instances/[id]/page.tsx` - -**Interfaces:** -- Consumes: `api.renameInstance` / `api.staff.renameInstance`, `RenameResult` (Task 5); `slugError`, `baseSlug`, `hostFor` (Task 5); `Panel`, `Note` (`@/components/Panel`), `Button` (`@/components/Button`), `Field` (`@/components/Field`), `ApiError` (`@/lib/api`). -- Produces: `RenamePanel({ currentName, currentSlug, onRename })` — a default-collapsed control; `onRename` is `(name: string) => Promise`. - -- [ ] **Step 1: Create the component** - -Create `adminsite/components/RenamePanel.tsx`: - -```tsx -"use client"; - -import { useState } from "react"; -import { Button } from "./Button"; -import { Field } from "./Field"; -import { Note } from "./Panel"; -import { ApiError, type RenameResult } from "@/lib/api"; -import { baseSlug, hostFor, slugError } from "@/lib/slug"; - -/* - * The rename control, and only the control — the same shape as RelinkPanel: an - * input that expands in place rather than a modal, because this app has no modal - * and one action with one field does not need one. - * - * The host preview is drawn from lib/slug.ts, a mirror of the Go rules. It can - * disagree with the server; the 409 that comes back is the answer that counts. - */ -export function RenamePanel({ - currentName, - currentSlug, - onRename, -}: { - currentName: string; - currentSlug: string; - onRename: (name: string) => Promise; -}) { - const [open, setOpen] = useState(false); - const [value, setValue] = useState(currentName); - const [error, setError] = useState(); - const [busy, setBusy] = useState(false); - const [done, setDone] = useState(); - - const name = value.trim(); - const derived = baseSlug(name); - const invalid = slugError(name); - // A cosmetic edit that lands on the same slug is still a rename worth doing — - // the name is what the customer reads. Only an empty or unchanged name is - // nothing to submit. - const unchanged = name === currentName.trim(); - - async function submit() { - setError(undefined); - setBusy(true); - try { - const res = await onRename(name); - setDone(res); - setOpen(false); - } catch (err) { - setError(err instanceof ApiError ? err.message : "Rename failed. Try again."); - } finally { - setBusy(false); - } - } - - if (done) { - const host = done.login_url || `https://${hostFor(done.slug)}`; - return ( - - - - This instance is now {done.name}, at{" "} - {hostFor(done.slug)}. The old address has stopped working, and your - sign-in does not follow it — you will need to sign in again there. - - - Open {hostFor(done.slug)} → - - - - ); - } - - return ( -
- {open && ( - setValue(e.target.value)} - error={error ?? (name ? invalid : undefined)} - hint={ - name && !invalid ? ( - <> - Moves to {hostFor(derived)} - {derived === currentSlug && " — the address does not change"} - - ) : ( - "Letters and digits; everything else becomes a hyphen." - ) - } - /> - )} -
- - {open && ( - - Anyone signed in will need to sign in again at the new address, and links to the old one stop working. - - )} -
-
- ); -} -``` - -`Note` is `({ tone = "accent" | "warn" | "expired", children })` and renders a `

`, which is why the success state wraps its two lines in a `` rather than block elements. - -- [ ] **Step 2: Mount it on the customer instance page** - -In `adminsite/app/(customer)/instances/[id]/page.tsx`: - -Add the imports: - -```tsx -import { RenamePanel } from "@/components/RenamePanel"; -``` - -and - -```tsx -import { useSession } from "@/lib/session"; -``` - -Inside `InstancePage`, with the other hooks (hooks must precede the early returns already in this component): - -```tsx - // useSession is the app's one way to ask who the caller is — it shares the - // ["me"] query, so this adds no request. - const { session } = useSession(); -``` - -and after the `cloud` const: - -```tsx - const mayRename = session?.account_role === "owner" || session?.account_role === "admin"; -``` - -Then add the panel to `PageFrame`'s children, directly after the `MembersPanel` line: - -```tsx - {/* - * Address rather than "Rename": the panel is about where this - * instance lives, and the rename is how you change it. Cloud - * only — a self-hosted install has no tenant subdomain for us to - * move. - */} - {cloud && mayRename && ( - -

- The instance name is where its address comes from. Renaming moves it to a new address and releases the old - one, so saved links and bookmarks to it stop working. -

- { - const res = await api.renameInstance(instance.instance_id, name); - qc.invalidateQueries({ queryKey: ["account"] }); - return res; - }} - /> - - )} -``` - -- [ ] **Step 3: Build** - -Run: `cd /go-projects/vantage/adminsite && npm run build` -Expected: build succeeds. - -- [ ] **Step 4: Commit** - -```bash -git add adminsite/components/RenamePanel.tsx "adminsite/app/(customer)/instances/[id]/page.tsx" -git commit -m "feat: Let a customer rename a cloud instance from HQ" -``` - ---- - -### Task 7: Staff instance page rename - -**Files:** -- Modify: `adminsite/app/(staff)/staff/instances/[id]/page.tsx` - -**Interfaces:** -- Consumes: `RenamePanel` (Task 6), `api.staff.renameInstance` (Task 5). -- Produces: nothing later tasks depend on. - -- [ ] **Step 1: Add the panel** - -In `adminsite/app/(staff)/staff/instances/[id]/page.tsx`, add the imports: - -```tsx -import { RenamePanel } from "@/components/RenamePanel"; -``` - -and, inside `StaffInstancePage`, add `const qc = useQueryClient();` at the top of the component if it is not already there (`useQueryClient` is already imported for `EntitlementSection`). - -Add this panel after the "Licence history" panel: - -```tsx - {/* - * Staff rename has no cooldown and does not start the customer's: - * fixing a name on someone's behalf must not spend their next 24 - * hours. - */} - - { - const res = await api.staff.renameInstance(data.instance.instance_id, name); - qc.invalidateQueries({ queryKey: ["staff-instance", id] }); - return res; - }} - /> - -``` - -- [ ] **Step 2: Build** - -Run: `cd /go-projects/vantage/adminsite && npm run build` -Expected: build succeeds. - -- [ ] **Step 3: Commit** - -```bash -git add "adminsite/app/(staff)/staff/instances/[id]/page.tsx" -git commit -m "feat: Let staff rename an instance" -``` - ---- - -### Task 8: Documentation and end-to-end verification - -**Files:** -- Modify: `CLAUDE.md` (the Admin REST API route list, and the `admin_instances` note under MongoDB Collections) - -**Interfaces:** -- Consumes: everything above. -- Produces: nothing. - -- [ ] **Step 1: Update the Admin REST API route list** - -In `CLAUDE.md`, in the customer-session block, after the `POST /instances/:id/claim-free` line: - -``` -PUT /instances/:id/name # rename a cloud instance; moves its slug (owner|admin, 24h cooldown) -``` - -and in the staff-session block, after `POST /instances/:id/issue · /instances/:id/relink`: - -``` -PUT /instances/:id/name # rename any instance, no cooldown -``` - -- [ ] **Step 2: Add the design note** - -In `CLAUDE.md`, under "Grants project, they do not federate" (admin's control-plane write boundary is described nearby), add a short paragraph: - -```markdown -**A rename moves the host, and the licence does not care.** `PUT -/api/instances/:id/name` re-derives the slug from the new name through -`provision.RenameSlug` — the same rules that named the instance at creation — -and writes the control plane first, because `instances.slug`'s unique index is -what settles a race between two accounts reaching for one name. A taken slug is -a refusal, not an `acme-2`: creation appends a counter because any free slug -will do, and a rename is a request for one specific host. A licence binds the -instance UUID, so nothing is reissued and Paddle is not called. The old host -keeps resolving for up to 60s (`instancehost.go`'s cache, which admin cannot -reach into), and `km_session` is host-only, so the customer signs in again on -the new address — the portal says so rather than redirecting them into a login -screen with no explanation. The 24h cooldown lives on `admin_instances.renamed_at` -because it is admin's policy; staff bypass it and must not write the field. -``` - -- [ ] **Step 3: Full build** - -Run: -```bash -cd /go-projects/vantage && go build ./... && go vet ./admin/... ./shared/... && (cd adminsite && npm run build) -``` -Expected: all clean. - -- [ ] **Step 4: Manual verification against a running stack** - -Work through each and record the result: - -1. Rename a cloud instance from `/instances/` as an owner. Panel reports the new host. -2. In Mongo: `db.instances.findOne({instance_id})` and `db.admin_instances.findOne({instance_id})` agree on `name` and `slug`; `admin_instances.renamed_at` is set. -3. The new host serves a login page. The old host stops resolving to the instance within ~60 seconds. -4. A second rename inside 24 hours answers `429` with the unlock time. -5. Renaming onto a slug another instance holds answers `409` and changes neither database. -6. `PUT /api/instances/:id/name` on a self-hosted instance answers `400` with the `selfHostedRefusal` message. -7. `GET /api/staff/audit` shows `instance.renamed` with `old-slug -> new-slug`. -8. Staff rename of the same instance succeeds immediately and leaves `renamed_at` unchanged. - -- [ ] **Step 5: Commit** - -```bash -git add CLAUDE.md -git commit -m "docs: Document instance rename in HQ" -``` - -- [ ] **Step 6: Refresh the knowledge graph** - -```bash -graphify update . -``` diff --git a/docs/superpowers/specs/2026-08-03-multi-auth-providers-design.md b/docs/superpowers/specs/2026-08-03-multi-auth-providers-design.md deleted file mode 100644 index 5b1df95..0000000 --- a/docs/superpowers/specs/2026-08-03-multi-auth-providers-design.md +++ /dev/null @@ -1,307 +0,0 @@ -# Multiple auth providers - -Date: 2026-08-03 - -## Problem - -An instance can configure exactly one OIDC provider. `instance_oidc` holds one -document per instance, `/auth/oidc/start` takes no argument, and `/login` -renders an unconditional "Sign in with your instance's SSO" button whether or -not anything is configured behind it. Customers who federate with more than one -identity source cannot, and customers who federate with none are shown a button -that leads to an error. - -## Goals - -- N auth providers per instance, each independently enabled and named. -- Login page renders one button per enabled provider, and none when there are - none. -- Local email/password login can be turned off per instance. -- Presets for the common identity providers, so a customer supplies a tenant ID - rather than an issuer URL. -- Existing configured SSO keeps working across the upgrade with no customer - action. - -## Non-goals - -- SAML. Different protocol, metadata parsing and certificate handling; not in - this work. -- Per-provider role or group mapping. Provisioned users remain `member`, as - today. -- Provider-specific account linking. An email address is an email address; the - existing instance-scoped lookup stands. - -## Data model - -New collection `auth_providers`, one document per provider: - -```go -type AuthProvider struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"_id,omitempty"` - InstanceID string `bson:"instance_id" json:"instance_id"` - ProviderID string `bson:"provider_id" json:"provider_id"` - Name string `bson:"name" json:"name"` - Kind string `bson:"kind" json:"kind"` // "oidc" | "oauth2" - Preset string `bson:"preset" json:"preset"` // "" for custom - Issuer string `bson:"issuer" json:"issuer"` - ClientID string `bson:"client_id" json:"client_id"` - ClientSecretEnc string `bson:"client_secret_enc,omitempty" json:"-"` - Scopes []string `bson:"scopes" json:"scopes"` - Enabled bool `bson:"enabled" json:"enabled"` - CallbackNotice bool `bson:"callback_notice" json:"callback_notice"` - Order int `bson:"order" json:"order"` - CreatedAt time.Time `bson:"created_at" json:"created_at"` - UpdatedAt time.Time `bson:"updated_at" json:"updated_at"` -} -``` - -`ProviderID` is a short random identifier, not the Mongo `_id`: it appears in -the callback URL a customer pastes into their IdP, and an `_id` there would -publish a database key. - -Unique index on `(instance_id, provider_id)`. Index build is fatal on failure, -matching `EnsureAuthIndexes` — a duplicate `provider_id` within an instance -would make the callback ambiguous. - -`ClientSecretEnc` is AES-256-GCM under `KEY_ENCRYPTION_KEY`, as -`instance_oidc.client_secret_enc` is today, and is never serialised. - -### Presets - -A Go table in `server/internal/auth/presets.go`, not database rows — adding one -is a commit, not a migration. - -| Preset | Kind | Issuer | Input asked of the customer | Default scopes | -| ------- | -------- | ------------------------------------------------- | --------------------------- | ----------------------------- | -| `entra` | `oidc` | `https://login.microsoftonline.com/{tenant}/v2.0` | Directory (tenant) ID | `openid profile email` | -| `google`| `oidc` | `https://accounts.google.com` | none | `openid profile email` | -| `okta` | `oidc` | `https://{domain}/oauth2/default` | Okta org domain | `openid profile email` | -| `github`| `oauth2` | n/a | none | `read:user user:email` | -| `` (custom) | `oidc` | supplied verbatim | Issuer URL | `openid profile email` | - -The issuer template is expanded server-side on save; the stored `Issuer` is -always the resolved URL, so nothing downstream has to know a preset existed. - -### Settings - -`settings.local_login_enabled bool`, defaulting true. Absent on existing -documents, and Go's zero value for `bool` is false, so the field is read through -a `*bool` and a nil pointer means enabled. A plain `bool` would silently -disable password login on every instance in the fleet at upgrade. - -## Migration - -`0005_auth_providers` — the next free number; `0004_instance_rename` is the -highest recorded today. For each document in `instance_oidc`, insert one -`auth_providers` document: - -- `Name: "Single sign-on"` -- `Preset: ""`, `Kind: "oidc"` -- `Issuer`, `ClientID`, `Enabled` copied -- `ClientSecretEnc` copied **verbatim**, not decrypted and re-encrypted — a - migration that needs `KEY_ENCRYPTION_KEY` fails on an instance that has none - and strands the SSO configuration. -- `Scopes: ["openid", "profile", "email"]`, matching what `oidc.go` hardcodes - today. -- `ProviderID` freshly generated. -- `CallbackNotice: true` — this provider's redirect URI has changed and an - administrator has not yet acknowledged it. Set only by the migration; cleared - by the settings UI. New providers are created `false`. - -`instance_oidc` is left in place and no longer read. Idempotent by skipping any -instance that already has an `auth_providers` document, so a re-run after a -partial failure completes rather than duplicating. - -## Auth flow - -Routes: - -``` -GET /auth/oidc/:providerId/start -GET /auth/oidc/:providerId/callback -``` - -The old unparameterised `/auth/oidc/start` and `/auth/oidc/callback` are -**removed**, not retained. See Upgrade impact below — this breaks configured SSO -until the customer updates their IdP, and that is accepted deliberately rather -than carried as a compatibility path. - -The state token in Redis stores `{instance_id, provider_id}` rather than the -bare instance ID. The callback resolves its provider from the consumed state -and cross-checks it against `:providerId` in the path, refusing a mismatch — -the path alone is attacker-controlled, and the state is the half that was -issued by the start handler. - -`providerForInstance` becomes `providerFor(ctx, c, instanceID, providerID)`. -The `go-oidc` provider cache keys on `provider_id`, not instance. Saving, -disabling or deleting a provider evicts that key. - -`redirectURL(c, providerID)` returns the one per-provider shape, and returns the -same URL in the start and callback halves of a flow — an IdP rejects the token -exchange if they differ. - -### OIDC providers - -Unchanged from the current implementation: `AuthCodeURL` with the stored -scopes, exchange, `id_token` verified against the provider's key set with -`ClientID` as audience, `email` and `name` claims extracted. - -### GitHub (`kind: "oauth2"`) - -GitHub is OAuth2 and issues no `id_token`, so it takes a separate branch: -exchange the code, then `GET https://api.github.com/user/emails` with the access -token and take the address that is both `primary` and `verified`. An -unverified-only response is refused — an unverified address is not proof of -control, and accepting one would let anyone holding a GitHub account claim any -address in the instance. `name` comes from `GET https://api.github.com/user`. - -Both branches converge on one function: - -```go -func completeSSOLogin(c *gin.Context, instanceID, email, name string) error -``` - -which holds today's lookup-or-provision, session creation, `TouchLastLogin` and -cookie set, verbatim. Email is lower-cased before lookup, and the lookup stays -`GetUserInInstanceByEmail` — instance-scoped, as it is now. - -### Licence gate - -`services.GetLicenseState(instanceID).Feature("oidc")` continues to gate both -the start and the callback, for every provider kind, and is checked on the -callback against the instance named by the consumed state rather than the host. -Unchanged behaviour, applied to more providers. - -## REST API - -Unauthenticated: - -``` -GET /auth/providers - -> {"local_enabled": true, - "providers": [{"id": "...", "name": "...", "preset": "entra"}]} -``` - -Instance is resolved from the host, as `/auth/bootstrap-status` already does. -The response carries **no issuer, no client ID and no secret** — it is served to -anyone who can reach the login page. - -Session-authed, `owner|admin`, under `/api`: - -``` -GET,POST /auth/providers -PUT,DELETE /auth/providers/:id -POST /auth/providers/:id/test -``` - -`test` fetches the provider's discovery document (or, for GitHub, calls the API -with the stored credentials) and reports reachability. It does not sign anyone -in. - -`GET,PUT /api/org/oidc` is removed along with the old auth routes. Its only -caller is `OIDCCard.tsx`, which this work replaces, and a compatibility shim -over a one-of-many model would have to invent which provider it means. - -Every mutation writes an audit event, as every mutating path does. - -### Lockout guards - -Both refused with 409 and a distinct error code: - -- `local_login_required` — disabling local login while zero providers are - enabled. -- `last_provider` — disabling or deleting the last enabled provider while local - login is off. - -These are enforced in the service layer, not the handler, so the two endpoints -that can reach the condition cannot disagree. - -## Frontend - -### Settings - -`web/components/settings/OIDCCard.tsx` becomes `AuthProvidersCard`, in the -Access group of `/settings` where the OIDC card already lives. It renders the -provider list with per-row enable toggle, edit, delete and drag ordering, an -Add flow that asks for the preset first and then only the fields that preset -needs, and the local-login toggle beneath the list. A guard violation surfaces -the 409's message rather than a generic failure. - -Every provider row shows its **callback URL** with click-to-copy — that is the -value the customer pastes into their IdP, it now differs per provider, and after -the upgrade every migrated provider needs it re-pasted. A migrated provider -additionally carries a warning until an administrator dismisses it, naming the -change and the URL. Dismissal is per provider, stored on the document. - -### Login page - -`web/app/login/page.tsx` calls `/auth/providers` on mount alongside the existing -`bootstrapStatus` call, and renders on the result: - -| `local_enabled` | providers | Rendered | -| --------------- | --------- | --------------------------------------------------- | -| true | none | Password form only. No divider, no buttons. | -| true | some | Password form, divider, one button per provider. | -| false | some | Buttons only. No form, no divider. | -| false | none | Password form (see below). | - -The last row cannot be reached through the API — the guards above prevent it — -but a hand-edited database could produce it, and a login page that renders -nothing at all is unrecoverable without database access. It therefore falls back -to the password form. - -The current unconditional SSO button and its "SSO must be enabled for this -instance by an administrator" note are both removed; the button now only exists -when it works. - -Buttons are labelled with the provider's `Name` and carry the preset's icon -where there is one, a neutral key glyph otherwise. Presets never override the -name — a customer who calls their Entra provider "Staff" gets "Staff". - -Errors keep the existing `/login?error=` redirect convention. - -## Testing - -- Migration: an `instance_oidc` document produces one enabled provider with the - ciphertext byte-identical; a re-run inserts nothing further. -- `local_login_enabled` absent decodes as enabled. -- Guards: both 409 paths, and the enable/disable sequences that approach them - without crossing. -- Per-provider callback: two providers in one instance, each resolving to its - own configuration; a `provider_id` from another instance answers 404. -- A callback whose `:providerId` disagrees with the consumed state is refused, - and the state is consumed rather than left replayable. -- The removed routes (`/auth/oidc/start`, `/auth/oidc/callback`, - `/api/org/oidc`) answer 404. -- GitHub: primary+verified selected; verified-only-absent refused. -- `/auth/providers` response contains no issuer, client ID or secret. - -## Upgrade impact - -**This release breaks configured SSO until each customer updates their identity -provider.** The old `/auth/oidc/callback` is gone, migrated providers are -reachable only at `/auth/oidc//callback`, and an IdP still pointing -at the old URL fails the flow. - -It is a deliberate trade: one callback shape rather than two, no -`legacy_callback` branch through `redirectURL`, and no permanently retained -route whose only purpose is a single past upgrade. - -Mitigations, in order of who sees them first: - -- The settings card shows the new callback URL per provider with click-to-copy, - and a migrated provider carries a dismissable warning naming the change. -- The failure is visible rather than silent: an IdP rejects the redirect URI - before Vantage is reached, so the customer sees their own provider's error. -- Local password login is unaffected, so no instance is locked out — an - administrator can always sign in to fix the URL. This is why - `local_login_enabled` defaults to true and why nothing in this migration - turns it off. -- Release notes and `docsite/docs/vantage/settings.md` state the required - action. - -## Deployment notes - -No new environment variables. No agent change. `KEY_ENCRYPTION_KEY` is already -required wherever OIDC was configured, and the migration does not add a -dependency on it. diff --git a/docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md b/docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md deleted file mode 100644 index 3cc9609..0000000 --- a/docs/superpowers/specs/2026-08-04-server-tags-and-scheduled-workflows-design.md +++ /dev/null @@ -1,235 +0,0 @@ -# Server tags and scheduled workflows - -Date: 2026-08-04 - -Two features, designed together because the second is worth much less without -the first. Tags make a target set describable; schedules make it recur. A -nightly job that patches "everything tagged `env:staging`" needs both halves, -and neither half is large on its own. - ---- - -## Part A — Server tags - -### Model - -`models.Server` gains one field: - -```go -Tags map[string]string `bson:"tags,omitempty" json:"tags,omitempty"` -``` - -Keys and values are lowercase `[a-z0-9_-]`. Keys are capped at 32 characters, -values at 64, and a server holds at most 20 tags. Validation lives in the -service layer rather than the handler, so the tag endpoint, the server-create -path and anything added later cannot disagree about what a valid tag is. - -There is **no `tags` collection.** A tag is a property of a server, not an -entity with a lifecycle: a registry would need reference counting to know when -a tag stopped existing, and garbage collection to act on it, which is work -bought for nothing. The list of known keys and values that the UI offers for -autocomplete is a distinct aggregation over `servers`, cached for 60 seconds — -the same treatment org lookups already get. - -No reserved keys ship in this change. If inventory-derived tags (`os`, `arch`) -are added later they take a `sys:` key prefix, so a user tag written today can -never collide with a system tag invented tomorrow. - -Index: `{instance_id: 1, "tags.$**": 1}` — a wildcard index over the tag -subdocument, because the queried key is chosen by the user at request time and -cannot be named in advance. - -### API - -``` -PUT /api/servers/:id/tags # replace the whole map -GET /api/servers/tags # known keys and values, for pickers -GET /api/servers?tag=env:prod # repeatable; AND across keys -``` - -`PUT` replaces the entire map rather than patching one tag. A tag set is small -enough that sending all of it is free, and last-write-wins over a whole map is -easier to reason about than merge semantics between two people editing the same -server. The audit event records the map before and after. - -`?tag=` is repeatable and ANDs: `?tag=env:prod&tag=role:web` matches servers -carrying both. A malformed value (no colon, unknown characters) is a 400 rather -than a silent empty result — a filter that matches nothing and a filter that is -nonsense look identical in a list, and only one of them is the user's fault. - -### Targeting - -`models.Workflow` gains `TargetTags map[string]string` beside the existing -`TargetServerIDs`. One function in `services` resolves them: - -```go -ResolveTargets(ctx, instanceID string, ids []string, tags map[string]string) ([]Server, error) -``` - -- Result is the **distinct union** of the explicit IDs and the tag matches. -- Tag matching ANDs across keys. -- Offline servers are included. The dispatcher already answers 503 per server, - and a patch run that silently omits an unreachable machine is worse than one - that visibly fails on it. -- Empty IDs **and** empty tags returns `ErrNoTargets` (400). A workflow that - matches nothing must say so rather than report success over zero servers. - -The resolved set is snapshotted into `WorkflowRun.ServerRuns` exactly as today. -History records what actually ran, not what the selector would match when the -run is later read back — the same reason `steps_snapshot` exists. - -### Frontend - -- **Server detail**: tag chips in the header with an inline editor. Keys - autocomplete from `GET /api/servers/tags`, values autocomplete per key. -- **`/servers`**: a filter bar that reads and writes the same `?tag=` query - params the API takes, so a filtered fleet view is a URL someone can send. -- **Workflow designer**: a target section holding both inputs, with a live - "runs on 14 servers" readout that lists them on hover. The union model costs - us the at-a-glance answer to "what will this touch"; this readout buys it - back, and it is the reason the union is acceptable. - ---- - -## Part B — Scheduled workflows - -### Model - -```go -type Schedule struct { - Enabled bool `bson:"enabled" json:"enabled"` - Cron string `bson:"cron" json:"cron"` // 5-field - TZ string `bson:"tz" json:"tz"` // IANA name -} - -type Skip struct { - Reason string `bson:"reason" json:"reason"` // "missed" | "already_running" - Due time.Time `bson:"due" json:"due"` - At time.Time `bson:"at" json:"at"` -} -``` - -On `Workflow`: - -```go -Schedule *Schedule `bson:"schedule,omitempty"` -NextRunAt *time.Time `bson:"next_run_at,omitempty"` // UTC, indexed -LastRunAt *time.Time `bson:"last_run_at,omitempty"` -LastSkipped *Skip `bson:"last_skipped,omitempty"` -``` - -`next_run_at` is **persisted, not held in memory.** A leader handover between -computing the next occurrence and firing it would otherwise either lose the -occurrence or fire it twice. Coordination state has to live where every replica -can see it — the same argument that put `workflow_log_seq` in MongoDB. - -Cron parsing uses `robfig/cron/v3`'s **parser only** — `Parse` and -`Next(time)`. Its scheduler and goroutines are not used; the loop below is ours -and has to be, because it runs under the leader lock. - -**Alpine ships no tzdata.** `server/Dockerfile` builds a slim image, so -`time.LoadLocation("Europe/London")` returns an error and every schedule -falls back to UTC — an hour wrong for half the year, in the direction nobody -notices until a maintenance window lands in business hours. `main` therefore -imports `_ "time/tzdata"`, embedding the database in the binary. Zone names are -also validated at save time, so an unknown zone is a 400 rather than a surprise -at 2am. - -### Scheduler - -A new `server/internal/workflowsched` package, started inside the **existing** -`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, `StartReaper` -and the sweepers. One role, one lock. It takes the same cancellable context and -returns the instant leadership is lost. - -The loop ticks every 30 seconds: - -1. `find({schedule.enabled: true, next_run_at: {$lte: now}})`. -2. **Claim atomically.** `findOneAndUpdate` matching the document *and* its - current `next_run_at`, setting the recomputed next occurrence. A process - that reaches the same document after another has claimed it matches nothing - and does nothing. The claim is what makes this correct; the leader lock only - makes it cheap. -3. **Grace check.** If `now - due > 1h`, record - `last_skipped{reason: "missed"}`, write an audit event, and do not run. A - job missed by ten minutes during a deploy should still run; one missed by - two days should not fire at lunchtime. -4. **Overlap check.** If a run for this workflow is still active, record - `last_skipped{reason: "already_running"}`, audit, and do not run. A patch - workflow must never run twice at once, and a silent skip is how a week goes - by before anyone notices nothing ran. -5. Otherwise start the run through the **same** `RunWorkflow` path a person - uses, with `TriggeredBy: "schedule"`. - -Step 5 is the design. A scheduled run is an ordinary run with a different -trigger: no second dispatch path, no second snapshot format, and the run detail -page needs no changes to display one. - -### API - -``` -PUT /api/workflows/:id/schedule # {enabled, cron, tz} -GET /api/workflows/:id/schedule/preview?cron=…&tz=… # next 3 occurrences -``` - -`PUT` validates the expression and the zone, then computes and stores -`next_run_at`. The preview endpoint exists so the browser and the scheduler -agree on what a cron string means — a client-side cron parser that disagrees -with the server by one field is a bug found in production, at night. - -### Frontend - -- **Workflow page**: a schedule card with preset buttons (hourly, nightly at - HH:MM, weekly on DAY at HH:MM) that write cron underneath, a raw cron field - for anything else, a timezone select, and the next three occurrences rendered - from the preview endpoint in mono. -- **Workflows list**: a schedule chip and the next run as relative time. -- **Skips are surfaced**, not just stored: a warning line reading - "Skipped Sun 02:00 — previous run still active". Recording a reason nobody - reads is the same as not recording one. - ---- - -## Out of scope - -**Notification on scheduled-run failure.** It needs the monitor channel -machinery pointed at workflow outcomes and its own answer to what counts as -failure — a non-zero exit on a step with `on_failure: continue` is not -obviously an alert. Visibility in this change is the run list and the recorded -skip reason. Excluded deliberately, not overlooked. - -**Tag-scoped permissions.** Roles stay instance-wide. Tags describe servers; -they do not yet gate who may act on them. - -**Inventory-derived tags.** Reserved via the `sys:` prefix, not implemented. - ---- - -## Migration and compatibility - -No migration is required. `Tags`, `TargetTags` and `Schedule` are all -`omitempty` and absent means what it meant before: no tags, no selector, no -schedule. Existing workflows keep their explicit server lists and behave -identically. - -The wildcard tag index and the `next_run_at` index are declared by a new -`EnsureServerIndexes`, following the convention `EnsureSecretIndexes` and -`EnsureWorkflowIndexes` already set: it warns rather than aborting boot, -because a missing index degrades -tag filtering to a collection scan on a small collection rather than breaking -the fleet list. - -## Testing - -- `ResolveTargets`: union deduplicates; AND across tag keys; empty/empty - returns `ErrNoTargets`; offline servers are included. -- Tag validation: charset, length caps, tag count cap, malformed `?tag=` is a - 400. -- Schedule validation: bad cron and unknown zone both 400; `next_run_at` is - computed in the stored zone, verified across a DST boundary. -- Scheduler claim: two concurrent claims of the same due workflow start exactly - one run. -- Grace window: due 10 minutes ago runs; due 2 hours ago records `missed`. -- Overlap: an active run yields `already_running` and no second run. -- Preview endpoint and the scheduler agree on the next occurrence for a table - of expressions, including a DST-crossing one. diff --git a/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md b/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md deleted file mode 100644 index 147d9fb..0000000 --- a/docs/superpowers/specs/2026-08-06-package-inventory-and-cve-findings-design.md +++ /dev/null @@ -1,538 +0,0 @@ -# Package inventory and CVE findings - -Date: 2026-08-06 - -Agents report the packages installed on each server. The control plane matches -them against distro security feeds and raises findings that link straight to -the patching path that already exists. A finding nobody can fix today can be -accepted with a reason and an expiry date rather than sitting red forever. - -This is one of four sub-projects sketched together and deliberately separated: - -| # | Sub-project | Depends on | -| - | ----------- | ---------- | -| A | **Package inventory + CVE findings** — this spec | nothing | -| B | Container/service registry | nothing | -| C | Container image scanning | A and B | -| D | Compliance profiles (baseline assertions) | shares A's findings UI only | - -A and B are independent of one another. C is the joiner and must not be -designed before both exist. D shares a page with A and nothing else — a -different collector, a different evaluation model and a different remediation -story — so folding it in here would double the size for no shared machinery. - -Scope of this spec is **A, Linux only.** Windows needs a separate source -(MSRC CVRF), a separate collector (`Get-HotFix` plus registry) and a KB -supersedence matcher that shares no code with the Linux path. That matches the -existing position that Windows agents are second-class by design, and the six -package managers `updates.go` already detects cover the whole Linux surface. - ---- - -## The trap this design is built around - -Distributions **backport** security fixes without changing the upstream -version. Ubuntu ships `openssl 3.0.2-0ubuntu1.15` patched against -CVE-2023-0286; NVD says version 3.0.2 is vulnerable. Matching installed -versions against NVD or CPE ranges therefore reports a fleet full of criticals -that are all already fixed. - -That is not merely noisy. It is fatal to the feature: once the first report is -mostly wrong, nobody reads the second one, and a genuine finding is lost in the -noise it created. Everything below follows from refusing to make that mistake. - -The correct source is the **distribution's own security feed**, keyed on the -distribution's own version string — Debian and Ubuntu OVAL/USN, Red Hat OVAL -v2, Alpine secdb. `trivy-db` is those feeds pre-merged into one BoltDB -artifact, rebuilt every six hours and published as an OCI artifact. - ---- - -## Where the vulnerability data comes from - -`trivy-db`, pulled server-side from `ghcr.io/aquasecurity/trivy-db:2`. - -The alternative considered was querying OSV.dev per scan, which needs no -storage and no puller. It was rejected on two counts: it requires outbound -internet on every scan, which breaks air-gapped installs; and it sends the -package list of a customer's entire fleet to a third party. The audience most -likely to buy vulnerability scanning is the audience least willing to do that. - -The blob is roughly 50MB, read-only, reproducible, and identified by a version -number. **It is not stored in Mongo and not written to `/data`** — -`server.persistence` defaults to off and nothing writes to `/data` any more. -It does not need durable storage: whichever pod needs it pulls it to its own -ephemeral temp directory. Nothing shared, nothing to back up, nothing to -migrate. - -`VANTAGE_TRIVY_DB_REF` overrides the default reference so a customer can mirror -the artifact into their own registry. It also covers the anonymous ghcr rate -limit, which the six-hourly pull cadence already makes unlikely to bite. - ---- - -## Only the leader matches - -This is the crux, and it falls out of the replica model already in the -codebase. - -Two things trigger matching, and they happen on different pods: - -1. a fleet-wide rescan when `trivy-db` updates — naturally the leader's job -2. a server's package list changing — handled by whichever pod holds *that - agent's* command stream - -If (2) matched inline, **every replica would need the 50MB database resident**, -and a database refresh would have N pods racing to rescan the same fleet and N -digests reaching the customer. That is the exact failure `RunAsLeader` exists -to prevent, and it is the same argument that put `monitorsched` behind the -lock. - -So `ReportPackages` does not match. It upserts the package list and sets -`scan_pending: true`. That is all it does. - -`server/internal/vulnsched` then runs inside the **existing** -`bus.RunAsLeader("housekeeping", …)` alongside `monitorsched`, -`workflowsched` and the sweepers — one role, one lock. Every 60 seconds it: - -1. pulls `trivy-db` if the local copy is older than six hours -2. if the pulled version differs from `vulndb_meta.db_version`, marks **every** - server `scan_pending` -3. matches all `scan_pending` servers, clears the flag, diffs against existing - findings -4. emits **one** digest per tick covering everything newly opened - -Step 4 is why batching is structural rather than bolted on. A `trivy-db` -refresh can open several hundred findings across a fleet at once; one message -per finding would rate-limit the webhook or get the channel muted, and either -way the customer stops receiving the alerts they are paying for. The tick is -already the natural batch boundary, so **the failure cannot occur by -construction** rather than by a debounce someone has to maintain. - -`scan_pending` lives on the document rather than in memory, for the same reason -`next_run_at` and `workflow_log_seq` do: a leader handover between marking and -scanning would otherwise lose it. A handover costs the new leader one re-pull -of the database. - -The cost of this indirection is up to 60 seconds between an agent reporting a -changed package set and its findings updating. For vulnerability data that is -nothing, and it buys a single matching path instead of two. - ---- - -## Components - -``` -agent/internal/packages/ collect installed packages + /etc/os-release -proto/ ReportPackages RPC -server/internal/vulndb/ puller, BoltDB access, matcher -server/internal/vulnsched/ leader-owned tick: pull, scan, digest -server/internal/services/ findings, acceptance, alert rules -web/app/(app)/vulnerabilities/ fleet board; plus two server-detail tabs -``` - -`vulnsched` takes the dependencies it needs — `LogEvent` and the notification -dispatch — as a `vulnsched.Deps` injected from `main.go`, following -`workflowsched`. The manual rescan endpoint does not call into `vulnsched` at -all: it sets `scan_pending` on every server and lets the next tick find them, -so there is no path by which `services` imports the scheduler and no cycle to -avoid later. - ---- - -## The wire path - -A new `ReportPackages` RPC on the agent's existing hourly loop — the same -`runUpdateCheck` cadence, reusing `updates.go`'s `detectPM()`. - -```protobuf -rpc ReportPackages(ReportPackagesRequest) returns (ReportPackagesResponse); - -message ReportPackagesRequest { - string server_id = 1; - string agent_token = 2; - string hash = 3; // sha256 of the sorted list - OSRelease os = 4; - repeated InstalledPackage packages = 5; // omitted when only offering a hash -} - -message ReportPackagesResponse { - bool need_full = 1; // hash differs; resend with packages populated -} -``` - -The agent calls once with `packages` empty. `need_full` true means the hash -differs from what the server holds, and the agent immediately calls again with -the list populated. - -The agent sends a SHA-256 of its sorted package list first. If it matches what -the server already holds, the server answers `unchanged` and the ~150KB body is -never sent. A machine's package set changes rarely, so almost every hour costs -one small message, and the rare changed hour costs one extra round trip. - -Folding the list into the existing 15-minute `InventoryReport` static snapshot -was rejected: it would re-send ~150KB per server every 15 minutes regardless of -change, roughly 40MB/hour of gRPC traffic on a 100-server fleet to transmit -data that is almost always identical. - ---- - -## Data model - -Four new collections. Every one carries `instance_id` except `vulndb_meta`, -which is explained below. - -### `server_packages` — one document per server, not per package - -```go -type ServerPackages struct { - ID primitive.ObjectID `bson:"_id"` - InstanceID primitive.ObjectID `bson:"instance_id"` - ServerID string `bson:"server_id"` - OS OSRelease `bson:"os"` // family, version_id, arch - Hash string `bson:"hash"` // sha256 of the sorted list - Packages []InstalledPackage `bson:"packages"` - CollectedAt time.Time `bson:"collected_at"` - ScanPending bool `bson:"scan_pending"` - ScannedAt time.Time `bson:"scanned_at"` - Status string `bson:"status"` // ok | unsupported - DBVersion int `bson:"db_version"` // last matched against -} - -type InstalledPackage struct { - Name string `bson:"name"` - Version string `bson:"version"` // distro version string, verbatim - Epoch int `bson:"epoch,omitempty"` - Arch string `bson:"arch"` - SourceName string `bson:"source_name,omitempty"` -} -``` - -One document rather than two thousand is what makes a report a **single atomic -upsert with no delta logic** — the hash already established that something -changed, so there is nothing to reconcile field by field. A typical Linux host -lands near 150KB, comfortably inside the 16MB document limit. - -Indexes: `{instance_id, server_id}` unique, and a multikey -`{instance_id, "packages.name"}` for fleet-wide package search. - -`SourceName` is not decoration. **Debian and Ubuntu advisories are keyed on the -source package**: a CVE against `openssl` covers the binaries `libssl3`, -`openssl` and `libssl-dev`, so matching on binary name alone misses two of the -three. - -`OS.VersionID` selects the feed. Ubuntu 22.04 and 24.04 publish different fixed -versions for the same CVE, so a scan without it is guesswork. - -### `vuln_findings` — one document per (server, CVE, package) - -```go -type VulnFinding struct { - ID primitive.ObjectID `bson:"_id"` - InstanceID primitive.ObjectID `bson:"instance_id"` - ServerID string `bson:"server_id"` - - CVEID string `bson:"cve_id"` - PackageName string `bson:"package_name"` - Installed string `bson:"installed_version"` - FixedIn string `bson:"fixed_in,omitempty"` - Severity string `bson:"severity"` - CVSSScore float64 `bson:"cvss_score,omitempty"` - Title string `bson:"title,omitempty"` - References []string `bson:"references,omitempty"` - - State string `bson:"state"` // open | fixed | accepted - FirstSeen time.Time `bson:"first_seen"` - LastSeen time.Time `bson:"last_seen"` - FixedAt *time.Time `bson:"fixed_at,omitempty"` - Accepted *Acceptance `bson:"accepted,omitempty"` -} - -type Acceptance struct { - By primitive.ObjectID `bson:"by"` - Reason string `bson:"reason"` - Until time.Time `bson:"until"` - At time.Time `bson:"at"` -} -``` - -Unique on `{instance_id, server_id, cve_id, package_name}`. That key is what -makes a rescan an idempotent upsert rather than a duplicate factory, and it is -what lets `first_seen` survive across scans. Query index -`{instance_id, state, severity}`. - -**An empty `FixedIn` is a real and common state** and must never be conflated -with "not vulnerable". A CVE with no vendor fix published yet is exactly the -finding people most need to see, and also the one that most needs acceptance, -because there is nothing to patch. - -Findings are **not deleted when a package is patched**. State moves to `fixed` -with `fixed_at` set, so "what did we remediate last quarter" remains -answerable — which is the question an auditor asks. - -### `vulndb_meta` — singleton, deliberately unscoped - -`db_version`, `pulled_at`, `last_full_scan_at`, `last_error`. It carries no -`instance_id` because the vulnerability database is a property of the -deployment, not of a tenant. Same reasoning as `migrations`. - -### `vuln_alert_rules` - -`instance_id`, `name`, `enabled`, `min_severity`, `tags map[string]string`, -`channel_ids []`, timestamps. - -The tag filter resolves through **`services.ResolveTargets`**, not a second -matcher. That function is already the single answer to which servers a -selector touches, and an alert rule that disagreed with a workflow about what -`env:prod` means would be worse than having no filter at all. - ---- - -## The matching engine - -``` -server/internal/vulndb/ - pull.go OCI fetch → temp dir, version compare against vulndb_meta - db.go BoltDB open, advisory lookup by (ecosystem, source, version) - match.go per-family matching, severity resolution - version.go dispatch to deb/rpm/apk comparator by OS family -``` - -Dependencies: `github.com/aquasecurity/trivy-db` for the BoltDB schema, plus -`go-deb-version`, `go-rpm-version` and `go-apk-version` — each a small -standalone module doing one job. The roughly 200 lines of per-distro advisory -lookup are ours. - -Importing `trivy` itself was rejected: it would pull a very large transitive -dependency tree into the server binary for one feature, and its Go API carries -no stability guarantee across minor versions. Shelling out to the `trivy` -binary against a generated SBOM was rejected for shipping a second binary in -the image and turning a library call into subprocess lifecycle, timeouts and -output-format drift. - -### Why the comparators are bought rather than written - -Version ordering is where this feature lives or dies, and its failure mode is -silent. `dpkg` ordering has epochs, and `~` sorts *before* the empty string, so -`3.0.2-0ubuntu1.15~rc1` precedes `3.0.2-0ubuntu1.15`. `rpmvercmp` has its own -segment rules and treats `~` and `^` differently again. A `strings.Compare` or -a semver parse orders `1.9` above `1.10` and reports a vulnerable fleet as -clean — a false negative, which nobody notices until it matters. - -### Scanning one server - -1. Load `server_packages`; resolve OS family and version to a `trivy-db` - ecosystem. -2. **Unsupported ecosystem → record `status: unsupported`, clear the flag, - write no findings.** -3. For each package: resolve source name, look up advisories, compare versions. -4. Upsert vulnerable results as `open`, preserving `first_seen`. -5. Any currently-`open` finding absent from this result set → `fixed`, stamp - `fixed_at`. -6. Any `accepted` finding past its `until` → back to `open`. -7. Clear `scan_pending`, stamp `scanned_at` and `db_version`. - -Steps 5 and 6 must run in that order, so a finding that is both absent and -expired settles as `fixed` rather than reopening on a package that no longer -carries it. - -Step 2 matters as much as any of the matching. Arch has no feed in `trivy-db`, -so an Arch host must report **unsupported**, never "0 findings". Reporting -clean when the truth is unknown is the same class of lie as a silently stale -database, and it is the reason `vulndb_meta.pulled_at` appears on screen rather -than only in a log. - -### Severity - -Resolved **vendor → NVD → unknown**, in that order, never invented. - -This will surface as "why is this critical CVE marked low", and the answer is -that Debian and Red Hat routinely downgrade an NVD score because the vulnerable -code path is not reachable in their build. Their rating is the accurate one for -that package, and showing NVD's above it would manufacture work that does not -need doing. - ---- - -## Findings lifecycle - -`open | fixed | accepted`. - -An accepted finding is suppressed from counts and alerts until its `until` -date, then reopens automatically. A reason is required. - -Acceptance with a mandatory expiry, rather than permanent dismissal, is what -keeps the feature usable in both directions. Without any acceptance mechanism, -a kernel CVE awaiting a reboot window sits red indefinitely and trains people -to ignore the page. With permanent dismissal, accepted findings accumulate -silently and nobody revisits them — the dismissal list becomes where risk goes -to be forgotten, which is precisely what an auditor asks to see. - -Retention: `settings.vuln_finding_retention_days`, a `*int` on the same pattern -as `workflow_log_retention_days` — nil means 90 days, 0 means forever. Only -`fixed` findings are swept, by a `StartVulnSweeper` inside the same -`RunAsLeader("housekeeping", …)` as the existing sweepers. `open` and -`accepted` findings are never swept at any setting. - ---- - -## Alerting - -Per-org rules over the existing `notification_channels`: severity threshold, -optional tag filter, target channels. - -A rescan emits one message summarising what newly opened — "12 new critical -across 4 servers" — never one message per finding. See the leader section for -why the tick boundary makes this structural. - -Modelling findings as a monitor type was rejected. It would reuse monitors' -state machine and channel wiring for free, but monitors are up/down for one -endpoint with retries and hourly rollups, none of which means anything for a -CVE; most fields would be disabled in the UI and the uptime graphs would be -polluted with a signal that is not uptime. - -This adds one `notify` payload type and a `vuln_digest.html.tmpl` / -`vuln_digest.txt.tmpl` pair in `shared/mail`. Note that `shared/mail` templates -are parsed in `init()`, so a mistyped field is a boot-time panic — CLAUDE.md -describes a `render_test.go` guarding against exactly this, but **that file does -not exist**; the repository has no Go tests at all, and by instruction this -feature adds none. The template pair must therefore be verified by starting the -binary and sending one digest through a real channel. - ---- - -## Entitlement - -The feature name is `vuln_scanning`, and it crosses the two services the way -every other feature does: - -- **admin** carries it as a per-instance entitlement toggle, so it can later be - priced as a catalogue `feature` component without a second migration; -- **the licence** snapshots it into `License.Features []string` at issue time; -- **the server** asks `lic.HasFeature("vuln_scanning")` and never switches on - tier, so changing what a tier includes needs no server release. - -Off on Free. - -**The gate is checked at `ReportPackages`, not at display.** Gating only the UI -would still pay every write cost, and storage is the expensive half. - -The agent learns of it through the existing 30-second `SyncKeys` poll: -`SyncResponse` gains a `collect_packages` bool, and the hourly loop skips -collection entirely when it is false. So an ungated instance produces no -collection, no gRPC body, no document and no storage. `ReportPackages` still -re-checks the entitlement server-side and refuses — the agent flag is an -optimisation, the server check is the boundary. - -Turning the feature off does not delete existing findings; they stop being -served and stop updating. Deletion is the instance-deletion path's job. - ---- - -## REST API - -``` -GET /api/vulnerabilities # filter: severity, state, server, tags -GET /api/vulnerabilities/summary # severity counts + database freshness -POST /api/vulnerabilities/rescan # marks all scan_pending (owner|admin) -POST /api/vulnerabilities/:id/accept # reason + until (owner|admin) -DELETE /api/vulnerabilities/:id/accept # (owner|admin) -GET /api/servers/:id/vulnerabilities -GET /api/servers/:id/packages -GET /api/packages/search?name= # fleet-wide -GET,POST /api/vuln-rules · PUT,DELETE /api/vuln-rules/:id -``` - -Every mutating path writes an audit event, as all of them do. Acceptance is the -one decision people will be asked to justify, so `by`, `reason`, `until` and -`at` land in the audit record and not only on the document. - ---- - -## UI - -`/vulnerabilities` is a fleet board **grouped by CVE** — one row per CVE with -an affected-server count, expandable to the individual servers. The same CVE -across 40 servers is one decision, and a flat list of findings makes it look -like forty. - -Server detail gains **Vulnerabilities** and **Packages** tabs. Alert rules go -on `/settings/notifications`, beside the channels they consume. - -Remediation introduces no new mechanism: a finding carrying `fixed_in` renders -an **Apply updates** action calling the existing -`POST /api/servers/:id/apply-updates`, which is already `ApplyUpdatesCmd`. See -it, patch it, one place — and no second patching path to keep consistent with -the first. - -Database freshness is shown wherever findings are, not tucked into settings. A -fleet scanning against a three-week-old database must say so rather than -quietly report all-clear. - ---- - -## Verification - -**No automated tests.** The repository has none today, and by explicit -instruction this feature adds none — no `*_test.go`, no frontend test files. -That is a deliberate decision by the repository owner, recorded here so the -absence reads as a choice rather than an omission. - -It does change the risk profile, and the places it changes it are worth naming, -because each fails by producing a **wrong answer rather than a crash**: - -- **Version comparison.** The backport case — installed `1:3.0.2-0ubuntu1.15` - against advisory fixed-in `1:3.0.2-0ubuntu1.15` resolving to *not - vulnerable* — plus tilde ordering (`1.0~rc1` < `1.0`), epoch dominance - (`1:1.0` > `2.0`) and `1.9` < `1.10`. Wrong here means a vulnerable fleet - reported clean. -- **Source-package fan-out.** One advisory against `openssl` must flag - `libssl3`, `openssl` and `libssl-dev`. Matching on binary name alone silently - finds one of three. -- **`first_seen` preservation.** An upsert that overwrites it makes every - finding look discovered today, and nothing surfaces that until someone reads - a report. -- **Fixed-before-reopen ordering.** A finding both absent from a scan and past - its acceptance expiry must settle `fixed`, not reopen. - -The implementation plan carries a manual verification table for each, to be -walked before the relevant task is committed. They are the substitute for the -tests, not a formality. - ---- - -## Failure modes - -| Failure | Behaviour | -| ------- | --------- | -| Database pull fails | Keep the last good copy and serve stale. Record `last_error`, surface `pulled_at` age. **Never clear findings** — a network blip must not read as "all fixed" | -| Unsupported distribution | `status: unsupported`, not zero findings | -| Agent stops reporting | Findings persist and `collected_at` age is shown. No auto-expiry: a silent agent is not a patched server | -| `trivy-db` schema version bumps | The puller refuses an unknown schema rather than mis-parsing it | -| ghcr anonymous rate limit | Backoff; `VANTAGE_TRIVY_DB_REF` mirrors to a private registry | -| Leadership lost mid-scan | The context is cancelled and the scan returns; `scan_pending` is still set, so the next leader picks it up | -| Instance deleted | **`server_packages` and `vuln_findings` must be added to the control plane's instance-deletion collection list.** Easy to miss, and missing it orphans a tenant's package data indefinitely | - ---- - -## Environment variables - -| Name | Required | Notes | -| ---- | -------- | ----- | -| `VANTAGE_TRIVY_DB_REF` | no | default `ghcr.io/aquasecurity/trivy-db:2`. Point at a mirror for air-gapped installs or to avoid the anonymous ghcr rate limit | -| `VANTAGE_VULNDB_DISABLED` | no | disables the puller and the scheduler entirely. Findings already written are still served and still marked stale | - ---- - -## Deliberately out of scope - -- **Windows.** Separate source, collector and matcher; its own spec. -- **Container image scanning.** Sub-project C; needs the container registry. -- **Compliance baseline assertions.** Sub-project D; shares this findings UI - and nothing else. -- **Language-level dependency scanning** (npm, pip, Go modules). `trivy-db` - covers these ecosystems, but finding the manifests on a host is a different - collection problem from asking the package manager what is installed. -- **Automatic patching on a finding.** Remediation is one click, not zero. An - unattended upgrade triggered by a CVE feed is a fleet-wide change driven by a - third party's data, which is not a decision to take away from an operator. diff --git a/docs/superpowers/specs/2026-08-06-workload-registry-design.md b/docs/superpowers/specs/2026-08-06-workload-registry-design.md deleted file mode 100644 index cbae7d6..0000000 --- a/docs/superpowers/specs/2026-08-06-workload-registry-design.md +++ /dev/null @@ -1,441 +0,0 @@ -# Workload registry - -Date: 2026-08-06 - -Agents enumerate what each server actually runs — Docker containers, the -compose stacks grouping them, and systemd services — and report it to the -control plane. Containers and units can be started, stopped and restarted from -the UI, and a bounded snapshot of their logs can be read without opening a -console. - -This is **sub-project B** of the four sketched in -`2026-08-06-package-inventory-and-cve-findings-design.md`: - -| # | Sub-project | Depends on | -| - | ----------- | ---------- | -| A | Package inventory + CVE findings — its own spec | nothing | -| B | **Workload registry** — this spec | nothing | -| C | Container image scanning | A and B | -| D | Compliance profiles | shares A's findings UI only | - -A and B are independent. C is the joiner and must not be designed before both -exist: it needs B's image list and A's findings model. - -**Workload** is the domain word throughout: one container or one systemd unit. -It gives the collection, the commands and the page a single honest name rather -than saying "container or service" in every identifier. - -Scope is **Linux only**, matching sub-project A and the existing position that -Windows agents are second-class by design. Docker runs on Windows; systemd does -not, and half a feature per platform is worse than a clear line. - ---- - -## What this is for - -The control plane can manage a fleet's keys, run workflows across it and watch -its endpoints, but it has no idea what any of those servers actually *runs*. -"Restart nginx on that box" means opening a console. "Which of these 80 servers -is still on the old image" is unanswerable. - ---- - -## Reporting and refresh are one path - -The agent reports on its own 60-second ticker through a `ReportWorkloads` RPC, -using the same hash short-circuit as the package report: it offers a SHA-256 of -the sorted workload list, and sends the body only when the server does not -already hold that hash. An unchanged list costs one small message, which on a -60-second cadence is the common case by a wide margin. - -The on-demand refresh **does not return data**. `RefreshWorkloadsCmd` carries -no payload back; it makes the agent report immediately through the normal RPC, -and the UI refetches the stored document. - -That is deliberate. A refresh that returned workloads inline would be a second -writer for the same collection, arriving by a different route, with its own -serialisation and its own opportunity to disagree with the periodic one. One -writer, one shape; the refresh is a nudge, not a channel. - -Opening a server's Workloads tab dispatches a refresh, so what is on screen is -live rather than up to a minute stale. That matters because the page has a -Restart button on it: a stale list is not merely a wrong impression, it is a -wrong action aimed at a container that already died. - -## What does answer back - -Two operations genuinely return something: - -| Command | Answers with | -| ------- | ------------ | -| `ControlWorkloadCmd{kind, id, action}` | the existing `CommandResult` — ok or error | -| `WorkloadLogsCmd{kind, id, tail}` | a new `WorkloadLogsResult{command_id, text, truncated}` | - -Both ride the proven path: `commandDispatcher.send()` for request and ack, and -a `WorkloadResults` registry mirroring `StepResults.Await`/`Deliver` over the -bus. **`Await` must subscribe before the command is dispatched** — the pod -driving the request is usually not the pod holding the agent's stream, and a -fast agent otherwise answers into a channel nobody has joined. This is not a -new hazard; it is the one `stepresults.go` already documents. - -```protobuf -rpc ReportWorkloads(ReportWorkloadsRequest) returns (ReportWorkloadsResponse); - -message ReportWorkloadsRequest { - string server_id = 1; - string agent_token = 2; - string hash = 3; - bool docker_ok = 4; - string docker_error = 5; - bool systemd_ok = 6; - string systemd_error = 7; - repeated Workload workloads = 8; // empty on the offer call -} - -message ReportWorkloadsResponse { - bool need_full = 1; -} - -// ServerCommand gains three variants. -message RefreshWorkloadsCmd {} - -message ControlWorkloadCmd { - string kind = 1; // "container" | "unit" - string id = 2; - string action = 3; // "start" | "stop" | "restart" -} - -message WorkloadLogsCmd { - string kind = 1; - string id = 2; - int32 tail = 3; -} - -// AgentMessage gains one variant. -message WorkloadLogsResult { - string command_id = 1; - string text = 2; - bool truncated = 3; - string error = 4; -} -``` - -The offer-then-send handshake is the package report's, unchanged: the agent -calls once with `workloads` empty, and resends with the body only if the -response sets `need_full`. - -An agent whose stream no pod holds gets a 503 from the dispatcher, as -everything else does. Commands are not queued: a command whose owner died must -fail loudly rather than be delivered to nobody while the operator is told it -worked. - ---- - -## Not gated by licence - -Unlike CVE scanning, this reads as core fleet management rather than a premium -add-on, so v1 ships to every instance with no entitlement check. - -If that changes it is a one-line `HasFeature` check at `ReportWorkloads`, -gating collection rather than display — the same placement and the same -reasoning as sub-project A, where gating the UI alone would still pay every -write cost. - ---- - -## Data model - -One new collection, `server_workloads`, one document per server, mirroring -`server_packages`. - -```go -type ServerWorkloads struct { - ID bson.ObjectID `bson:"_id,omitempty" json:"-"` - InstanceID string `bson:"instance_id" json:"-"` - ServerID string `bson:"server_id" json:"server_id"` - Hash string `bson:"hash" json:"hash"` - Workloads []Workload `bson:"workloads" json:"workloads"` - CollectedAt time.Time `bson:"collected_at" json:"collected_at"` - - DockerOK bool `bson:"docker_ok" json:"docker_ok"` - DockerError string `bson:"docker_error,omitempty" json:"docker_error,omitempty"` - SystemdOK bool `bson:"systemd_ok" json:"systemd_ok"` - SystemdError string `bson:"systemd_error,omitempty" json:"systemd_error,omitempty"` -} - -type Workload struct { - Kind string `bson:"kind" json:"kind"` // "container" | "unit" - ID string `bson:"id" json:"id"` // container id, or unit name - Name string `bson:"name" json:"name"` - State string `bson:"state" json:"state"` - Health string `bson:"health,omitempty" json:"health,omitempty"` - Image string `bson:"image,omitempty" json:"image,omitempty"` - Stack string `bson:"stack,omitempty" json:"stack,omitempty"` - Ports []string `bson:"ports,omitempty" json:"ports,omitempty"` - Restarts int `bson:"restarts,omitempty" json:"restarts,omitempty"` - StartedAt time.Time `bson:"started_at,omitempty" json:"started_at,omitempty"` - Protected bool `bson:"protected" json:"protected"` -} -``` - -`State` is normalised across the two kinds: containers report `running`, -`exited`, `paused`, `restarting`, `created`; units report `active`, `inactive`, -`failed`, `activating`. They are deliberately **not** collapsed into a shared -vocabulary — a failed unit and an exited container mean different things, and -flattening them would lose the distinction the operator needs. - -Indexes: `{instance_id, server_id}` unique, plus multikey -`{instance_id, "workloads.image"}` for the fleet-wide "which servers run image -X" query. - -### Why the OK/Error pairs exist - -A host with no Docker installed and a host where Docker is installed and -running nothing both produce an empty list. One should read "not in use here", -the other "nothing running", and only the second deserves any alarm. - -The error strings separate a third case the booleans alone cannot: Docker -installed with the daemon down. "Not installed" and "installed but not -responding" are different problems with different fixes, and collapsing them -into one false boolean throws away the only thing that tells them apart. - -### Why `Protected` is reported rather than derived - -The agent already knows which unit and container it is. Sending that up lets -the UI render the action disabled with a reason instead of offering a button -whose refusal is already known. - -The field is the courtesy; the agent's own check is the boundary. See the -control section. - -### No history - -A workload list is state, not a record. Nobody asks what containers ran last -Tuesday, and keeping it would grow a collection per server per minute in -exchange for a question nobody has. - ---- - -## Collectors - -### Docker: two commands, no English parsing - -``` -docker ps -aq -docker inspect --format '{{json .}}' -``` - -Not `docker ps --format '{{json .}}'` alone. That reports health and uptime -inside a human `Status` string — `"Up 2 hours (healthy)"` — and anything built -on it is parsing English that is localised, reworded between releases, and -silently different for a paused or restarting container. `inspect` returns -`State.Health.Status`, `State.StartedAt` and `RestartCount` as typed fields. -Two execs instead of one, and no parser to be wrong. - -`RestartCount` justifies the second call by itself: a container cycling is the -single thing this page most needs to show, and it is invisible in a list that -only ever says "Up". - -Compose stacks come from the `com.docker.compose.project` label. **No YAML is -read from disk** — the label is what Docker itself treats as authoritative, and -a compose file on disk may not be what is actually running. - -Docker absent, or a socket that cannot be reached, sets `DockerOK: false`. It -is not an error and produces no log line: most servers in a fleet built around -SSH key management will not have Docker, and treating the normal case as a -fault makes the feature look broken on the majority of the estate. - -### systemd: filtered on purpose - -``` -systemctl list-units --type=service --state=running,failed --no-legend --plain --no-pager -systemctl list-unit-files --type=service --state=enabled --no-legend --plain --no-pager -``` - -Two calls because "running or failed" and "enabled but stopped" are different -questions, and an enabled unit that is not running is exactly the one worth -seeing. - -Excluded by prefix: `systemd-`, `user@`, `session-`, `init.scope`. A typical -host carries 300+ units, the platform's own accounting for most of them. -Listing all of them buries the ten anyone cares about — the same failure mode -as an unfiltered vulnerability report, and the same fix. - -Column output rather than `--output=json`: the JSON flag requires systemd 246+, -and this fleet includes older stable distributions. The column format has been -stable considerably longer than the JSON one has existed. - ---- - -## Control actions - -``` -container: docker {start|stop|restart} -unit: systemctl {start|stop|restart} -``` - -Owner or admin only. Every action writes an audit event naming the actor, the -server and the target. - -### The protected set - -Computed agent-side: `vantage-agent.service`, plus the container ID read from -`/proc/self/cgroup` should the agent ever be run inside a container. - -The agent refuses those before doing anything. As with the console relay -hardcoding `127.0.0.1` agent-side, **the control plane may name a target, but -the agent decides what it will do to itself**. A server-side denylist alone -would be bypassed by the next dispatch path someone adds, and the failure is -unrecoverable from the UI: a server that stops its own agent goes offline, and -the way back is SSH or physical access — precisely what this feature exists to -avoid needing. - -### Timeouts - -`docker stop` waits on a container that may ignore SIGTERM. `systemctl stop` -on a unit with a long `TimeoutStopSec` blocks for exactly as long as that says. -Both run under a 90-second context, and a timeout returns a real error rather -than an ack implying success. - ---- - -## Logs - -``` -container: docker logs --tail 500 --timestamps -unit: journalctl -u -n 500 --no-pager --output=short-iso -``` - -Capped at **500 lines and 256KB, whichever binds first**, with `truncated` set -so the UI can say so. Two caps because 500 lines of a container emitting 4KB -JSON blobs is 2MB, and a line count alone does not stop it — the same reasoning -that gave workflow logs both a per-line and a per-run cap. - -Live following is deliberately absent. The browser console already offers a -real terminal on the same server, where `docker logs -f` works properly with -its own scrollback and cancellation. Building a second streaming path — a -relay listener, proxy bus keys, a WebSocket upgrade and a cancellation story -for a follow nobody closed — to duplicate that would be a large amount of -machinery aimed at a capability already shipped. A bounded snapshot answers -"why did this restart", which is the question that sends people to the console -in the first place. - -### Log reads are owner or admin only, and audited - -Unlike workflow logs, these cannot be masked. A workflow's logs can be masked -because the run injected the secrets and therefore knows their values. A -container's stdout is arbitrary and may contain credentials nobody declared — -a connection string in a startup banner, a token in a stack trace. - -So log reads sit behind the same role check as control actions and are audited. -A member who can see the fleet cannot read its logs. This is a deliberate -access decision, not an oversight, and it is why log reading is not simply -folded in with the read-only snapshot endpoints. - ---- - -## REST API - -``` -GET /api/servers/:id/workloads # stored snapshot -POST /api/servers/:id/workloads/refresh # dispatch, then refetch -POST /api/servers/:id/workloads/:wid/action # {"action":"start|stop|restart"} (owner|admin) -GET /api/servers/:id/workloads/:wid/logs?tail= # (owner|admin) -GET /api/workloads?image=&stack=&state= # fleet-wide -``` - -`:wid` is a container ID or a unit name, URL-encoded. Unit names carry dots and -`@`, which are legal in a path segment but not worth relying on unencoded. - -`tail` is clamped to the 500-line cap server-side; a client asking for more -gets 500, not an error. - ---- - -## UI - -Server detail gains a **Workloads** tab, ordered compose stacks first — grouped -under the stack name — then loose containers, then units. - -That ordering is not cosmetic. A stack is one thing to an operator even when it -is six containers, and a flat list turns one decision into six rows. It is the -same argument that groups the vulnerabilities board by CVE rather than by -finding. - -A `/workloads` fleet view answers "which servers run image X", which is the -reason the snapshot is stored at all rather than fetched on demand and -discarded. - -Three rules that follow directly from the model: - -- **Protected rows render their actions disabled, with the reason**, rather - than offering a button whose refusal is already known. -- **`DockerOK: false` reads "Docker not in use on this server"**, never an - empty list, and `DockerError` when present is shown as a distinct problem. -- State never reads by colour alone: every pill carries a distinct shape and a - text label, matching the existing monitor and severity pills. - ---- - -## Verification - -**No automated tests.** The repository has none today, and by explicit -instruction this feature adds none — no `*_test.go`, no frontend test files. -A deliberate decision by the repository owner, recorded so the absence reads as -a choice rather than an omission. - -The behaviours that would otherwise have been tested are the ones that fail -quietly, and the implementation plan carries a manual check for each: - -- **Parser output against real command output.** `docker inspect` must yield - `RestartCount`, health and the compose label as `Stack`; the `systemctl` - exclusion filter must drop `systemd-*` and `user@*` while keeping - `nginx.service`. Both are verified against a live host rather than a fixture. -- **Protected-set computation.** `vantage-agent.service` marked, - `nginx.service` not. Getting this wrong in the permissive direction lets a - server stop its own agent, which is unrecoverable from the UI. -- **Hash order-independence.** An ordering-sensitive hash resends the full list - every 60 seconds, which is invisible except as traffic. -- **Log capping in both directions.** 600 lines in → 500 out with `truncated`; - a 300KB blob of fewer than 500 lines → capped, `truncated`. The second is the - case a line-count-only implementation silently fails, and it fails by sending - megabytes rather than by erroring. - ---- - -## Failure modes - -| Failure | Behaviour | -| ------- | --------- | -| Docker not installed | `DockerOK: false`, no error, UI reads "not in use" | -| Docker installed, daemon down | `DockerOK: false` **plus** `DockerError` — different message, different fix | -| Agent offline | 503 from the existing dispatcher. No queueing: a command whose owner died must fail loudly | -| Action on a protected workload | Agent refuses; API answers 409 naming the reason | -| `stop` exceeds its timeout | Real error surfaced, never a hopeful ack. Snapshot refreshed afterwards | -| Container removed between snapshot and action | Docker's "No such container" surfaced and a refresh dispatched — this is what on-demand refresh is for | -| Log exceeds either cap | Truncated, flagged, and stated in the UI | -| Instance deleted | **`server_workloads` must be added to the control plane's instance-deletion collection list**, alongside sub-project A's two collections | - ---- - -## Deliberately out of scope - -- **Live log following.** The console already does it. See the logs section. -- **Creating, deleting or updating containers and units.** This is a control - and visibility surface, not a deployment tool — workflows already exist for - changing what a server runs, with snapshots, audit and rollback. -- **`docker exec` into a container.** The console reaches the host; exec from - the control plane is a second remote-execution path with its own audit and - authorisation story, and it belongs in its own spec if anywhere. -- **Kubernetes and containerd.** The Docker collector shells to the `docker` - CLI, so a node whose runtime is containerd or CRI-O reports nothing from it — - `DockerOK: false`, correctly, since Docker genuinely is not in use. Covering - those runtimes means a `crictl`/`nerdctl` collector, and talking to a - Kubernetes API server is a different subsystem again. Neither is v1. -- **Podman as a supported runtime.** Its `docker`-compatible CLI means an - aliased install will largely work, and that is a happy accident rather than a - claim: nothing here is tested against Podman and its `RestartCount` and - compose-label behaviour are not verified. -- **Windows.** No systemd, and a different container story. -- **Image vulnerability scanning.** Sub-project C, which needs this spec's - image list and sub-project A's findings model. diff --git a/docs/superpowers/specs/2026-08-12-api-tokens-openapi-design.md b/docs/superpowers/specs/2026-08-12-api-tokens-openapi-design.md deleted file mode 100644 index 849c09e..0000000 --- a/docs/superpowers/specs/2026-08-12-api-tokens-openapi-design.md +++ /dev/null @@ -1,297 +0,0 @@ -# API tokens and OpenAPI reference - -Date: 2026-08-12 -Status: approved, ready for implementation planning - -## Problem - -The only programmatic credential the control plane issues is the ESO secrets-read -bearer token, which reaches exactly one endpoint. Everything else requires a -browser session cookie. There is therefore no supported way to drive Vantage from -CI, a script, or infrastructure-as-code, and no machine-readable description of -the REST API for anyone who wants to try. - -This spec covers two deliverables that ship together: scoped API tokens, and an -OpenAPI 3.1 document rendered as a live reference page. A Terraform provider is -the intended follow-on and is explicitly out of scope here — it depends on both -of these being settled, and it is a separate Go module with its own release -cycle. - -## Goals - -- A person can mint a scoped, optionally expiring token and use it against the - existing REST API with no new endpoints to learn. -- A leaked token is bounded by role, by scope, and by expiry policy. -- Offboarding a person removes their tokens as a side effect of removing them. -- The API has a machine-readable description that cannot silently drift from the - handlers it describes. -- The reference page works on an air-gapped self-hosted install. - -## Non-goals - -- Token editing. Role and scopes are immutable; rotation replaces amendment. -- OAuth device flow or any browser-based authorisation grant. -- Per-server or per-tag restrictions on a token. -- Instance-owned service tokens that outlive their creator. -- The Terraform provider. -- General API rate limiting beyond the per-token limit described below. - -## Part 1 — API tokens - -### Token format and storage - -A token is `vt_` followed by 32 random bytes, hex encoded. It is displayed once, -at creation, and never again. - -Only the SHA-256 hash is stored, in a unique index. This follows the precedent -already set by `servers.agent_token_hash` and the ESO read token. bcrypt is -deliberately not used: the value is full-entropy random rather than a -user-chosen password, so a fast hash is sufficient, and a per-token salt would -force a collection scan where an indexed lookup is wanted. - -The first eight characters are stored in clear as `hint`, so the list can -identify a token without revealing it. - -### Authentication path - -`auth.Middleware()` gains a fallback. When there is no `km_session` cookie it -looks for `Authorization: Bearer vt_…`. Both paths end by placing a `*Session` in -the gin context, so every handler, `auth.RequireRole`, `RequireActiveLicense`, -`RequireFeature` and `actorFromCtx` continue to work unmodified. - -``` -Session{ - UserID: token.UserID - InstanceID: token.InstanceID - Role: min(user.Role, token.Role) // owner > admin > member - Email: user.Email - TokenID: token.TokenID // "" for cookie sessions - Scopes: token.Scopes // nil for cookie sessions -} -``` - -The effective role is recomputed on every request rather than frozen at -creation. Demoting the user demotes the token with them. No caching is required -because the user document is already read to confirm the user still exists. - -The existing host guard applies identically. A token carries an `instance_id`, -and a request arriving at a different instance's host is rejected exactly as a -mismatched cookie session is. The tenant boundary must not have a token-shaped -hole in it. - -`last_used_at` is written best-effort and only when the stored value is more -than 60 seconds old, so it does not become a Mongo write per request. - -Rejections: - -| Condition | Status | Body | -| -------------------- | ------ | -------------------------------------- | -| No credential at all | 401 | `not authenticated` | -| Unknown token | 401 | `invalid token` | -| Expired token | 401 | `code: token_expired` | -| Owning user deleted | 401 | `invalid token` | -| Missing scope | 403 | names the required scope | -| Wrong instance host | 403 | `instance host mismatch` | - -### Data model - -New collection `api_tokens`, added to `services.ScopedCollections` so instance -purge reaches it. - -``` -instance_id string -token_id string -user_id string -name string 1-64 chars, unique per user -hint string first 8 chars of the plaintext -token_hash string sha256 -role string owner|admin|member -scopes []string -expires_at *time.Time nil means never -created_at time.Time -last_used_at *time.Time -created_by_ip string -``` - -Indexes: unique on `token_hash`; compound on `(instance_id, user_id)`. - -Deleting a user deletes their tokens as part of the same service call as -`DeleteInstanceUser`, so offboarding is one action rather than two. - -### Expiry policy - -Expiry is optional by default: a token may be created with no expiry at all. -Instance settings gain `api_token_max_days *int`, editable by owner and admin: - -- `nil` — no cap; never-expire is allowed. This is the default, so an upgrade - changes nothing. -- `n > 0` — a new token must expire within `n` days, and a never-expire token is - refused. - -Changing the setting does not retroactively invalidate existing tokens; it is a -policy on issuance. Tokens already outside the new cap are flagged in the UI so -that someone can rotate them deliberately, rather than discovering the change -when a pipeline breaks. - -### Scopes - -Eight resources, each with `:read` and `:write`. Write implies read on the same -resource. - -``` -servers keys secrets workflows -monitors vulns workloads settings -``` - -Scope enforcement is a single middleware, `RequireScopes()`, mounted once in the -`/api` stack. It derives the required resource from the matched gin route -pattern using a map, rather than from a per-route decorator: a route registered -without a decorator would otherwise be unguarded, and this repo already prefers -guards that come from where a route is mounted rather than from someone -remembering. - -- Cookie sessions skip the check entirely. -- A token-authenticated request whose route pattern is absent from the map is - denied with 403. Fail closed. -- A startup check fails boot if any registered `/api` route pattern is missing - from the map, so the failure surfaces at deploy rather than at the first call. - -Deliberate placements: - -- `keys:read` covers `GET /keys/:id/private-key`. Reading a private key is - reading a key. -- `secrets:read` does not cover `GET /api/secrets/:group/values`. That endpoint - keeps its separate ESO bearer path and is unaffected by this work. -- `workloads:write` covers both container control actions and log reads, which - are already restricted to owner and admin. -- The token endpoints themselves map to the `settings` resource: `GET - /api/tokens` requires `settings:read`, and `POST` and `DELETE` require - `settings:write`. A token can therefore mint or revoke tokens only when - explicitly granted that scope, and never above its own role. - -### Endpoints - -``` -GET /api/tokens list; a member sees their own, owner|admin see all -POST /api/tokens create; returns the plaintext once -DELETE /api/tokens/:id revoke; own always, owner|admin any -``` - -There is no `PUT`. Editing a token's role or scopes changes what a credential -already deployed in a CI system can do, with no record of what it could do -before. Rotation replaces amendment. - -`POST` body: `name`, `role`, `scopes[]`, `expires_in_days` (omitted means never, -and is refused when `api_token_max_days` is set). - -Refusals: 400 for an unknown scope, 409 for a duplicate name for that user, 403 -for a role above the creator's own, 422 for an expiry beyond policy. - -### Web UI - -A new "API tokens" card in the Access group of `/settings`, alongside Members -and single sign-on. Not a new nav entry — `/settings/instance` was folded back -into `/settings` for precisely this reason, and the card lives in -`web/components/settings/` with the others, reusing the shared `Field` and -`inputClass`. - -The card lists name, hint, role, scope chips, last used, and expiry with a -distinct state for expired and for over-policy. Revoke is per row and confirms. - -Create opens a modal. The plaintext is shown once in a `--well` block with -copy-to-clipboard and an explicit line saying it will not be shown again. - -Members see only their own rows. Owner and admin get an "All tokens" toggle. - -`api_token_max_days` is a field on the same card, visible to owner and admin -only. - -### Audit - -New events: - -- `token.created` -- `token.revoked` -- `token.expired_use` — a rejected expired token, which is how a forgotten CI - job becomes visible -- `settings.token_policy_updated` - -The actor is the human's email throughout, so `actorFromCtx` needs no change. -Every existing audit event written during a token-authenticated request gains -`via: "token:"` in its detail, so the log distinguishes a person clicking -from their credential acting. - -### Rate limiting - -Token-authenticated requests are limited per token in Redis at 600 per minute, -answering 429 with `Retry-After`. Cookie sessions are untouched. This is narrow -on purpose: it is not the general API rate-limiting project, only enough that a -runaway script cannot take an instance down. - -## Part 2 — OpenAPI and the reference page - -### Generation - -`swaggo/swag` v2, pinned, emitting OpenAPI 3.1. v1 emits Swagger 2.0, which -Scalar renders poorly. - -Handlers in `server/internal/api/*.go` gain annotation comments. Request and -response bodies that are currently anonymous inline structs become named -structs. This is real churn across roughly fifteen files and is the honest cost -of choosing generation over a hand-written document. - -The generated `server/internal/api/docs/openapi.json` is committed and embedded -with `go:embed`, not generated during the image build: `server/Dockerfile` -produces a `scratch` runtime from a Go build stage, and adding codegen there -means putting the toolchain in the build image. - -`server-deploy.yml` gains a check that regenerates the spec and runs -`git diff --exit-code`. An annotation edited without regenerating fails the -build. Without this check the annotations are worth less than a hand-written -document, because they would drift while appearing authoritative. - -### Serving - -``` -GET /api/openapi.json the spec, session or token authenticated -GET /api/docs HTML page loading a vendored Scalar bundle -``` - -The Scalar standalone bundle is vendored under `server/internal/api/docs/`, with -its version recorded in a comment beside it and refreshed by hand. No CDN: -air-gapped self-hosted installs are supported, and a reference page that fails -closed on an offline site is a support ticket. - -Because the page is served by the instance itself, "Try it" acts against the -reader's own API with their own session. - -### Documented auth schemes - -Three, kept distinct: - -- `cookieAuth` — the `km_session` cookie. -- `bearerAuth` — a `vt_…` API token. -- The ESO secrets endpoint is marked as its own separate scheme, so nobody wires - a personal access token into External Secrets Operator. - -## Documentation - -- `docsite/docs/reference/api-tokens.md`: creating a token, the scope table, - curl examples, rotation, and the maximum-lifetime policy. -- `CLAUDE.md`: the three token routes under REST API, the `api_tokens` - collection, and a note that `openapi.json` is generated and CI-verified. - -## Risks - -- The anonymous-struct-to-named-struct conversion is the bulk of the work and - touches handler code this feature otherwise has no business in. -- The vendored Scalar bundle is a manual refresh that nobody will remember. The - version comment is the only mitigation. -- A scope map keyed on gin route patterns breaks if a route path is renamed. The - boot-time completeness check is what turns that into a startup failure rather - than a silent 403 in production. - -## Follow-on work - -A Terraform provider, as its own spec and plan, consuming the tokens and the -OpenAPI document produced here. diff --git a/docs/superpowers/specs/2026-08-12-instance-rename-design.md b/docs/superpowers/specs/2026-08-12-instance-rename-design.md deleted file mode 100644 index fc1ac47..0000000 --- a/docs/superpowers/specs/2026-08-12-instance-rename-design.md +++ /dev/null @@ -1,236 +0,0 @@ -# Instance rename in Vantage HQ - -**Date:** 2026-08-12 -**Status:** approved, not yet implemented - -## Problem - -A cloud instance is named once, at creation, and never again. The name is -chosen in the first thirty seconds of a customer's relationship with the -product — before they have decided whether this is "Acme" or "Acme -Production" — and it is the name that becomes their DNS host, appears in every -sign-in link and heads every page of their control plane. Today the only way to -change it is to create a second instance and move, or to open a support ticket -that has no tooling behind it. - -## What a rename is - -One customer-initiated action on a **cloud** instance: a new name, from which a -new slug is derived, which moves the instance to a new DNS host. - -Name and slug move together. The slug is re-derived through -`provision.BaseSlug`, so the rules that named the instance at creation are the -rules that rename it — the same reserved-label list, the same 3–40 character -bound, the same `Slugify` collapse of non-alphanumeric runs. There is no -separate slug field for the customer to edit, because two fields invite the -state where the name says one thing and the host says another, and that -divergence is exactly what a rename exists to fix. - -A licence binds an instance **UUID**, not a slug. A rename therefore issues no -licence, calls Paddle not at all, and consumes no relink. This is the property -that makes the whole feature cheap, and it should be stated in any future change -that tempts someone to touch the licence from this path. - -### What breaks, deliberately - -- **The old host stops working.** The old slug is released the moment the rename - commits; another account may take it. Bookmarks, saved sign-in links and any - agent install one-liner that named the web host are stale. Agents themselves - are unaffected — they dial `GRPC_HOST`, which is not per-tenant. -- **The old host keeps working for up to 60 seconds.** `server/internal/auth/instancehost.go` - caches slug-to-instance lookups for 60s, and admin has no path to invalidate - another process's memory. The released slug can be claimed by another account - inside that window, so for up to a minute a replica still maps that host to the - previous tenant. No data is exposed — the host/session guard rejects a session - belonging to a different instance — but the new owner's users can briefly reach - the old tenant's instance on their own host, and see its login page rather than - theirs. Adding a cross-service invalidation channel for a 60-second window is - not worth the coupling. -- **The customer must sign in again.** `km_session` is set with no `Domain` - attribute, so it is host-only and does not follow the instance to its new - subdomain. The UI says so rather than letting the customer discover it. - -## Scope - -| | Customer (owner or admin) | Staff | -|---|---|---| -| Cloud instance | rename, 24h cooldown | rename, no cooldown | -| Self-hosted instance | refused, 400 | name only; there is no slug | -| Cloud placeholder | refused, 409 | refused, 409 | - -Self-hosted is refused on the customer side for the same reason the member -endpoints refuse it: there is no control-plane row to write. The install is the -customer's, on their own database, and admin cannot reach it. Staff may still -correct the label on admin's own row, because that label is what staff search -by. - -## Data flow - -Two writes, in this order: - -1. **Control plane `instances`** — `{name, slug}`. -2. **Admin `admin_instances`** — `{name, slug, renamed_at}`. - -The control plane goes first because `instances.slug` carries the unique index, -and that index is what actually decides a race between two accounts reaching for -the same name. Deciding it anywhere else would be guessing. - -If the second write fails, the first is rolled back best-effort — restoring the -previous name and slug — and the request answers 500. Leaving them divergent -would have HQ print a host that is not the host, which is worse than a failed -rename. - -## Backend - -### `shared/provision/instance.go` - -```go -// ErrSlugTaken means the derived slug belongs to another instance. -var ErrSlugTaken = errors.New("slug taken") - -// RenameInstance changes an instance's name and re-derives its slug. -func RenameInstance(ctx context.Context, db *mongo.Database, instanceID, name string) (*models.Instance, error) -``` - -It lives beside `CreateInstanceWithID` so slug derivation keeps one home, and it -behaves as that function's rules imply: - -- `BaseSlug(name)` failures wrap `ErrNameRejected` — too short, too long, - reserved. -- The derived slug is compared against the instance's current one. If they are - equal, only the name is written; a cosmetic capitalisation change is not a - move, and must not fail on its own slug. -- **No `-2` suffix loop.** Creation appends a counter because the customer is - waiting on an instance and any free slug will do. A rename is a request for a - specific host, and silently landing the customer on `acme-2` is a worse answer - than refusing. -- A duplicate-key error on the update surfaces as `ErrSlugTaken`, exactly as the - create path treats it as "that slug is taken". The pre-check is a courtesy; - the index is the boundary. - -### `admin/internal/cloudprov` - -```go -func RenameInstance(ctx context.Context, instanceID, name string) (*sharedmodels.Instance, error) -``` - -A thin wrapper over `provision.RenameInstance` on `db.ControlDB()`. It writes -`instances` and nothing else, so admin's documented control-plane write boundary -— `instances` and `users`, from `cloudprov` and `inject` only — is unchanged. - -### `admin/internal/models` - -`Instance` gains: - -```go -// RenamedAt is when this instance last changed name, and backs the 24h -// customer cooldown. The cooldown is admin's policy, so it lives on admin's -// row rather than in the control plane, which has no opinion about how often -// a customer may move. -RenamedAt *time.Time `bson:"renamed_at,omitempty" json:"renamed_at,omitempty"` -``` - -A pointer because absent means "never renamed", and a zero `time.Time` would -read as 1 January year 1 — far enough in the past that the cooldown is inert, -but only by accident. - -### `PUT /api/instances/:id/name` (customer) - -Mounted in the `cust` group behind `auth.RequireAccountRole(owner, admin)`, and -resolving the instance through `ownedInstance` like every other instance route, -so another account's instance answers 404 rather than 403. - -Body: `{"name": "..."}`, trimmed before use. - -Refusals, in the order checked: - -| Condition | Status | Body | -|---|---|---| -| `deployment != cloud` | 400 | `selfHostedRefusal`, the same constant and status the member endpoints already answer with | -| `placeholder` | 409 | instance is not provisioned yet | -| within 24h of `renamed_at` | 429 | includes the UTC time it unlocks | -| `provision.ErrNameRejected` | 422 | the wrapped reason, verbatim | -| `provision.ErrSlugTaken` | 409 | that name is already in use | - -Success returns `{"instance_id", "name", "slug", "login_url"}` and writes an -audit entry `instance.renamed` with detail ` -> `, so the -history of a host is answerable from the audit log alone. - -`login_url` comes from the existing `loginURLFor(slug)`, which fills `{slug}` -into `APP_LOGIN_URL` — the same builder the licence emails already use, rather -than a second opinion about how a tenant host is spelled. It is empty when -`APP_LOGIN_URL` is unset, and the portal then falls back to the host string it -already composes from the slug in `InstanceRecord` and the instance page. - -### `PUT /api/staff/instances/:id/name` - -The same core, without the cooldown, actor recorded as the staff user. On a -self-hosted instance it updates `admin_instances.name` only and does not call -`cloudprov`. - -## Frontend (`adminsite`) - -### `lib/slug.ts` - -A TypeScript mirror of `provision.Slugify` and the length/reserved checks, used -only to preview the resulting host while the customer types. It carries the same -warning as `web/lib/targets.ts`: it is a second implementation and must change in -the same commit as the Go one. The preview can disagree with the server — the -409 is the answer that counts. - -### `components/RenamePanel.tsx` - -An inline panel, not a modal — `adminsite` has no modal component, and the -codebase's idiom for a destructive-ish action with one input is `RelinkPanel`: -a control that expands in place inside a `Panel`. - -Prefilled with the current name. Below the input, a live line reading -`acme-ltd.vantage.hostxtra.co.uk` as the customer types, and a note that they -will need to sign in again on the new host. Submit is disabled while the derived -slug is unchanged or invalid. - -It lives in an "Address" panel on `app/(customer)/instances/[id]/page.tsx`, -rendered only when the instance is cloud and `account_role` is `owner` or -`admin`. The staff instance page mounts the same component against the staff -route. - -`InstanceRecord` on the Overview page is not touched: it stays a summary, and -the rename is a decision that deserves the detail page. - -### After a successful rename - -Invalidate `["account"]`, collapse the panel, and let the page redraw with the new -name and host. The Console rail card shows the new host, with a note: - -> This instance now lives at `acme-ltd.vantage.hostxtra.co.uk`. You will need to -> sign in again there. - -**No automatic redirect.** Sending the browser to the new host lands the customer -on a login screen with no explanation, having just lost the HQ page they were -standing on. The link is right there; they click it when they are ready. - -## Verification - -The repository has no Go test suite, so verification is build plus manual -exercise, matching existing practice: - -- `go build ./...` in `shared` and `admin`; `npm run build` in `adminsite`. -- Rename a cloud instance; confirm `instances` and `admin_instances` agree on - name and slug. -- The new host serves a login page; the old host stops resolving to the instance - within ~60 seconds. -- A second rename within 24 hours answers 429. -- A rename onto an occupied slug answers 409 and changes nothing. -- A rename attempt on a self-hosted instance from the customer portal answers - 400, the same status and constant the member endpoints already answer with. -- The audit log carries `instance.renamed` with both slugs. - -## Out of scope - -- Slug aliases or redirects from the old host. The control plane resolves one - slug per instance, and an alias table is a second identity to keep correct for - the sake of stale bookmarks. -- Renaming from inside the control plane's own `/settings`. HQ owns instance - identity, the same way it owns licences and `hq`-sourced users; a second - writer would need the same collision handling and the same cooldown. -- Any change to the licence, subscription or Paddle line items.