feat(server): store inventory and handle ReportInventory RPC
Server Deploy / deploy (push) Successful in 1m4s
Server Deploy / deploy (push) Successful in 1m4s
This commit is contained in:
@@ -1,267 +0,0 @@
|
||||
# Ad-hoc Steps, Step Import/Export, and Default Steps — Design
|
||||
|
||||
Date: 2026-07-21
|
||||
|
||||
## Summary
|
||||
|
||||
Four related additions to the workflow step system:
|
||||
|
||||
1. **Ad-hoc steps** — steps defined inline in a single workflow, not written to the
|
||||
shared step library.
|
||||
2. **Import/Export** — single-step portable JSON (`vantage.step/v1`). Import can
|
||||
target the shared library or a workflow as an inline ad-hoc step.
|
||||
3. **Default steps** — JSON files in a bind-mounted directory, seeded into the
|
||||
library on boot and re-syncable on demand. Org-ready for a future SaaS plan.
|
||||
4. **Auto-derived outputs** — `declared_outputs` is scanned from the script
|
||||
(writes to `$WORKFLOW_ENV`) instead of being entered by hand.
|
||||
|
||||
Existing model: shared steps live in the `workflow_steps` collection; a
|
||||
`Workflow.Steps[]` is a list of `WorkflowStepRef` that reference a library step by
|
||||
`step_id` and may carry `Overrides` + `Inputs`. `resolveSteps` freezes each ref
|
||||
into a `ResolvedStep` snapshot at run time.
|
||||
|
||||
## 1. Data model
|
||||
|
||||
`server/internal/models/workflow.go`.
|
||||
|
||||
### WorkflowStep
|
||||
|
||||
Add a provenance field:
|
||||
|
||||
```go
|
||||
Source string `bson:"source" json:"source"` // "user" | "default"
|
||||
Slug string `bson:"slug" json:"slug"` // kebab of name; stable key for default seeding
|
||||
```
|
||||
|
||||
`Slug` is set for `source="default"` steps (used as the upsert key by the seeder).
|
||||
For `source="user"` steps it may be empty. Existing steps default to
|
||||
`source="user"` (absent field decodes to "").
|
||||
|
||||
### WorkflowStepRef
|
||||
|
||||
Add an inline definition. A ref is **either** a library ref (`StepID` set) **or**
|
||||
ad-hoc (`Inline` set). Never both.
|
||||
|
||||
```go
|
||||
type WorkflowStepRef struct {
|
||||
StepID string `bson:"step_id,omitempty" json:"step_id"`
|
||||
Inline *WorkflowStep `bson:"inline,omitempty" json:"inline,omitempty"`
|
||||
Order int `bson:"order" json:"order"`
|
||||
OnFailure string `bson:"on_failure" json:"on_failure"`
|
||||
MaxRetries int `bson:"max_retries" json:"max_retries"`
|
||||
Overrides *StepOverride `bson:"overrides,omitempty" json:"overrides,omitempty"` // library-ref only
|
||||
Inputs map[string]string `bson:"inputs,omitempty" json:"inputs,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
`Inline` reuses `WorkflowStep` (name, description, interpreter, script,
|
||||
declared_inputs, declared_outputs, secret_refs). Its `ID`, `StepID`, `Slug`,
|
||||
`Source`, and timestamps stay empty and are never persisted to `workflow_steps`.
|
||||
|
||||
Validation on workflow create/update: for each ref, exactly one of `StepID` /
|
||||
`Inline` must be set. `Overrides` is ignored when `Inline` is set.
|
||||
|
||||
## 2. Resolve at run time
|
||||
|
||||
`server/internal/services/workflow_runner.go`, `resolveSteps`.
|
||||
|
||||
For each ref:
|
||||
|
||||
- If `ref.Inline != nil`: build `ResolvedStep` from `ref.Inline` directly
|
||||
(name/interpreter/script/secret_refs), apply `ref.Inputs` against
|
||||
`Inline.DeclaredInputs` defaults. Skip `getStep`, skip `Overrides`.
|
||||
- Else: current library path unchanged (load step, apply overrides).
|
||||
|
||||
`ResolvedStep` output shape and the run snapshot are unchanged, so the runner and
|
||||
the run-history UI need no changes.
|
||||
|
||||
`DeleteStep` cascade is unaffected — ad-hoc refs carry no `step_id`, so they never
|
||||
match the cascade query.
|
||||
|
||||
## 3. Import / Export
|
||||
|
||||
Portable single-step JSON, `kind: "vantage.step/v1"`:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "...",
|
||||
"description": "...",
|
||||
"interpreter": "bash",
|
||||
"script": "...",
|
||||
"declared_inputs": [ { "name": "...", "default": "...", "description": "..." } ],
|
||||
"declared_outputs": ["..."],
|
||||
"secret_refs": ["NAME"]
|
||||
}
|
||||
```
|
||||
|
||||
Export strips `_id`, `step_id`, `slug`, `source`, and timestamps.
|
||||
Secret refs are exported as names only. On import, dangling secret refs are kept
|
||||
verbatim (not auto-created).
|
||||
|
||||
### Service functions (`services/workflows.go`)
|
||||
|
||||
- `ExportStep(stepID string) ([]byte, error)` — load library step, marshal to the
|
||||
v1 shape.
|
||||
- `ParseStepDoc(b []byte) (models.WorkflowStep, error)` — validate `kind`, decode
|
||||
into a `WorkflowStep` (no id/source). Shared by both import targets.
|
||||
- `ImportStepToLibrary(b []byte) (*models.WorkflowStep, error)` — `ParseStepDoc`
|
||||
then `CreateStep` (fresh `step_id`, `source="user"`).
|
||||
|
||||
Import-to-inline needs no new service fn: the web editor calls `ParseStepDoc`'s
|
||||
API equivalent (see routes) and drops the returned step object into a new
|
||||
`WorkflowStepRef.Inline` in the workflow it's editing, then saves the workflow
|
||||
normally.
|
||||
|
||||
### Routes (`api/workflows.go`)
|
||||
|
||||
- `GET /api/steps/:id/export` — returns JSON as a downloadable attachment
|
||||
(`Content-Disposition`).
|
||||
- `POST /api/steps/import` — body is the v1 JSON; imports to library; returns the
|
||||
created step. (Used by the "import to library" flow.)
|
||||
- `POST /api/steps/parse` — body is the v1 JSON; validates and returns the
|
||||
normalized step object **without** persisting. Used by "import to inline" so the
|
||||
editor can insert it as an ad-hoc ref. (Keeps parsing/validation server-side.)
|
||||
|
||||
Audit: `workflow.step_imported` logged on library import.
|
||||
|
||||
## 4. Default steps (seed + admin re-sync)
|
||||
|
||||
Mirrors the existing `WorkflowLogDir` pattern — a bind-mounted directory, no Go
|
||||
`embed`.
|
||||
|
||||
### Directory
|
||||
|
||||
```go
|
||||
// DefaultStepsDir returns the directory holding default step JSON files, creating it.
|
||||
func DefaultStepsDir() string {
|
||||
dir := os.Getenv("VANTAGE_DEFAULT_STEPS_DIR")
|
||||
if dir == "" {
|
||||
dir = filepath.Join("data", "default-steps")
|
||||
}
|
||||
_ = os.MkdirAll(dir, 0700)
|
||||
return dir
|
||||
}
|
||||
```
|
||||
|
||||
Compose already bind-mounts `./data:/data`. Set
|
||||
`VANTAGE_DEFAULT_STEPS_DIR=/data/default-steps` for explicitness (optional).
|
||||
Operator drops `*.json` (`vantage.step/v1`) files into that folder.
|
||||
|
||||
### Seeder
|
||||
|
||||
`SeedDefaultSteps() (created, updated int, err error)`:
|
||||
|
||||
1. Glob `DefaultStepsDir()/*.json`.
|
||||
2. For each file: `ParseStepDoc`, compute `slug = kebab(name)`.
|
||||
3. Upsert into `workflow_steps` keyed on `{ slug, source: "default" }`:
|
||||
- absent → insert with fresh `step_id`, `source="default"`, `slug`. (`created++`)
|
||||
- present → `$set` name/description/interpreter/script/declared_*/secret_refs +
|
||||
`updated_at`. (`updated++`)
|
||||
|
||||
**Override rule (confirmed):** re-sync is authoritative for `source="default"`
|
||||
steps and overwrites their content, reverting any user edits to those steps.
|
||||
`source="user"` steps are never touched by the seeder, even on a slug collision
|
||||
(the seeder query is scoped to `source: "default"`).
|
||||
|
||||
Add a partial unique index on `slug` where `source == "default"` (or enforce
|
||||
uniqueness in the seeder loop) to keep default slugs unambiguous.
|
||||
|
||||
### Boot
|
||||
|
||||
Call `SeedDefaultSteps()` from server startup after `EnsureWorkflowIndexes()`
|
||||
(alongside index setup in `server/cmd/main.go`). Log the created/updated counts;
|
||||
a seed error is logged but non-fatal (server still boots).
|
||||
|
||||
### Route
|
||||
|
||||
- `POST /api/steps/seed-defaults` (admin) — runs `SeedDefaultSteps()`, returns
|
||||
`{ "created": n, "updated": m }`. Audit `workflow.defaults_synced`.
|
||||
|
||||
### Org readiness
|
||||
|
||||
Signature stays global today. When Orgs land, `SeedDefaultSteps(orgID)` seeds
|
||||
per-org and the upsert key becomes `{ org_id, slug, source }`. No schema churn
|
||||
blocks that later change.
|
||||
|
||||
## 5. Auto-derived outputs
|
||||
|
||||
Today `WorkflowStep.DeclaredOutputs` is entered by hand and consumed only by the
|
||||
UI (no runtime reads it — outputs are captured at run time by `parseEnvFile` on
|
||||
the agent). Replace manual entry with a server-side scan of the script.
|
||||
|
||||
At run time the agent exposes an env file path in `$WORKFLOW_ENV` (bash) /
|
||||
`$env:WORKFLOW_ENV` (powershell); a step emits an output by appending a
|
||||
`KEY=value` line to it, e.g. `echo "test=123" >> $WORKFLOW_ENV`.
|
||||
|
||||
### Scanner
|
||||
|
||||
`services.DeriveOutputs(script string) []string`:
|
||||
|
||||
- Scan line by line. For each line that references `WORKFLOW_ENV`, extract every
|
||||
`KEY=` assignment target on that line, where `KEY` matches
|
||||
`[A-Za-z_][A-Za-z0-9_]*`.
|
||||
- Covers the common forms across both interpreters (line mentions `WORKFLOW_ENV`
|
||||
and contains `KEY=...`):
|
||||
- `echo "test=123" >> $WORKFLOW_ENV`
|
||||
- `echo "test=123" >> "$WORKFLOW_ENV"`
|
||||
- `printf 'k=v\n' >> $WORKFLOW_ENV`
|
||||
- `"k=v" >> $env:WORKFLOW_ENV` / `Add-Content $env:WORKFLOW_ENV "k=v"`
|
||||
- Deduplicate, preserve first-seen order. Best-effort heuristic — false positives
|
||||
are acceptable (they only widen the documented output list); it never affects
|
||||
what the agent actually captures.
|
||||
|
||||
### Wiring
|
||||
|
||||
- `CreateStep` and `UpdateStep` set `DeclaredOutputs = DeriveOutputs(s.Script)`,
|
||||
ignoring any client-sent value.
|
||||
- Inline ad-hoc steps: `DeriveOutputs` is applied when the workflow is saved (for
|
||||
each `ref.Inline`), so inline outputs are derived too.
|
||||
- `ParseStepDoc` (import) also derives outputs, so `declared_outputs` in an
|
||||
imported/exported file is informational and always recomputed on import.
|
||||
- `SeedDefaultSteps` derives outputs the same way when upserting.
|
||||
|
||||
`DeclaredOutputs` stays in the model and JSON (still shown in the UI and used to
|
||||
wire step-to-step input references), it is just no longer user-authored.
|
||||
|
||||
### Web
|
||||
|
||||
The step editor's "declared outputs" input becomes a read-only, auto-populated
|
||||
display (derived from the script, refreshed on save / on script edit). No manual
|
||||
add/remove.
|
||||
|
||||
## 6. Web
|
||||
|
||||
`web/app/workflows/[id]/page.tsx` and the steps list page.
|
||||
|
||||
- **Steps list:** per-row **Export** (downloads JSON) and top-level **Import**
|
||||
(file picker → `POST /api/steps/import` → library). **Sync defaults** admin
|
||||
button → `POST /api/steps/seed-defaults`, toast the counts. `source="default"`
|
||||
rows get a "default" badge.
|
||||
- **Workflow editor — Add step:** existing "add from library" plus **Add ad-hoc
|
||||
step** (inline mini-form: name, interpreter, script, optional inputs) stored as
|
||||
a `WorkflowStepRef.Inline`. Also **Import ad-hoc from file** → `POST
|
||||
/api/steps/parse` → inserts the returned step as a new inline ref.
|
||||
- Ad-hoc rows in the editor are visually distinguished from library refs (badge)
|
||||
and are editable in place; library refs keep the existing override UI.
|
||||
|
||||
## Testing
|
||||
|
||||
- `resolveSteps`: inline ref resolves without touching the library; inputs apply
|
||||
from `Inline.DeclaredInputs` defaults and ref overrides; library path unchanged.
|
||||
- Workflow validation: rejects a ref with both `StepID` and `Inline`, and one with
|
||||
neither.
|
||||
- Export → import round-trips to an equivalent library step with a new `step_id`.
|
||||
- `ParseStepDoc` rejects a wrong/missing `kind`.
|
||||
- `SeedDefaultSteps`: insert-then-update idempotency; user steps untouched;
|
||||
user-edited default step reverted on re-sync; counts correct.
|
||||
- `DeleteStep` cascade ignores ad-hoc refs.
|
||||
- `DeriveOutputs`: extracts keys from each interpreter form above, dedupes,
|
||||
preserves order, ignores lines not referencing `WORKFLOW_ENV`; create/update/
|
||||
import/seed all populate `declared_outputs` from it and ignore client input.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Whole-workflow export/import.
|
||||
- Auto-creating secret refs on import.
|
||||
- Multi-tenant Org model (design is forward-compatible only).
|
||||
Reference in New Issue
Block a user