docs: add Server Workflows design spec
This commit is contained in:
@@ -0,0 +1,238 @@
|
||||
# Server Workflows — Design
|
||||
|
||||
**Date:** 2026-07-20
|
||||
**Status:** Approved (design) — ready for implementation planning
|
||||
**Scope:** Server Workflows only. Fleet Inventory and SaaS/local-auth are separate sub-projects with their own specs.
|
||||
|
||||
Approved UI mockup: three-pane builder (Step Library · Canvas · Inspector), env vars shown riding the wire between nodes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
|
||||
Let operators compose **reusable shell steps** (Bash or PowerShell) into **workflows** and run them across many managed servers in parallel. Steps pass data to later steps through a `$WORKFLOW_ENV` file (GitHub-Actions style). Every run is recorded with full per-step logs. Steps can reference org secrets, injected as environment variables at runtime.
|
||||
|
||||
Builds directly on the existing `CommandStream` gRPC infrastructure (`dispatch.go`, `ServerCommand` oneof, agent command loop).
|
||||
|
||||
---
|
||||
|
||||
## 2. Locked decisions
|
||||
|
||||
| Topic | Decision |
|
||||
|-------|----------|
|
||||
| Data passing | Implicit. Every step's `$WORKFLOW_ENV` outputs merge into the run's env and are exposed to **all** later steps as `$KEY`. No explicit port wiring. |
|
||||
| Failure model | Per-step policy: `stop` (default), `continue`, `retry` (with max attempt count). |
|
||||
| Targets | Fan-out. Same step sequence runs on N target servers **in parallel**. Steps within one server run **sequentially**. |
|
||||
| History/logs | Every run persisted: status, timing, per-server per-step stdout/stderr/exit code, captured output env. |
|
||||
| Secrets | Steps declare needed secret keys; resolved from existing `secrets` store and injected as env vars at exec time. Never persisted into run logs. |
|
||||
| Testing | **Skipped** for this iteration per request. No test files written. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model (MongoDB)
|
||||
|
||||
### `workflow_steps` — reusable step library
|
||||
```json
|
||||
{
|
||||
"_id": "ObjectId",
|
||||
"step_id": "uuid",
|
||||
"name": "Restart service",
|
||||
"description": "Restart-Service by name, wait ready",
|
||||
"interpreter": "bash | powershell",
|
||||
"script": "Restart-Service vantage-api\n...",
|
||||
"declared_outputs": ["STARTED_AT"], // documentation/UI hints; not enforced
|
||||
"secret_refs": ["DEPLOY_TOKEN"], // secret keys this step needs injected
|
||||
"org_id": "uuid", // for future multi-tenant; single-org for now
|
||||
"created_at": "ISODate",
|
||||
"updated_at": "ISODate"
|
||||
}
|
||||
```
|
||||
|
||||
### `workflows` — ordered composition
|
||||
```json
|
||||
{
|
||||
"_id": "ObjectId",
|
||||
"workflow_id": "uuid",
|
||||
"name": "Deploy & Restart API",
|
||||
"target_server_ids": ["uuid", "uuid"],
|
||||
"steps": [
|
||||
{
|
||||
"step_id": "uuid", // reference to library step
|
||||
"order": 0,
|
||||
"on_failure": "stop | continue | retry",
|
||||
"max_retries": 0, // used when on_failure = retry
|
||||
"overrides": { // optional local fork of the library step
|
||||
"script": null,
|
||||
"secret_refs": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"created_at": "ISODate",
|
||||
"updated_at": "ISODate"
|
||||
}
|
||||
```
|
||||
Editing a library step from the Inspector writes an `overrides` block on that workflow step (a local fork) rather than mutating the shared step.
|
||||
|
||||
### `workflow_runs` — execution records
|
||||
```json
|
||||
{
|
||||
"_id": "ObjectId",
|
||||
"run_id": "uuid",
|
||||
"workflow_id": "uuid",
|
||||
"workflow_snapshot": { }, // frozen copy of workflow + resolved steps at trigger time
|
||||
"status": "running | success | failed | cancelled",
|
||||
"triggered_by": "user-id",
|
||||
"started_at": "ISODate",
|
||||
"finished_at": "ISODate | null",
|
||||
"server_runs": [
|
||||
{
|
||||
"server_id": "uuid",
|
||||
"status": "queued | running | success | failed | skipped",
|
||||
"started_at": "ISODate | null",
|
||||
"finished_at": "ISODate | null",
|
||||
"run_env": { "VERSION": "a1b9f0" }, // accumulated non-secret output env
|
||||
"steps": [
|
||||
{
|
||||
"order": 0,
|
||||
"name": "Git pull & build",
|
||||
"status": "success | failed | running | queued | skipped",
|
||||
"attempts": 1,
|
||||
"exit_code": 0,
|
||||
"stdout": "…",
|
||||
"stderr": "…",
|
||||
"output_env": { "VERSION": "a1b9f0" },
|
||||
"started_at": "ISODate",
|
||||
"finished_at": "ISODate"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
Secret values are never written to `stdout`/`stderr`/`run_env` by us; masking of known secret values in captured output is applied before persistence.
|
||||
|
||||
---
|
||||
|
||||
## 4. gRPC protocol changes (`proto/vantage/v1/vantage.proto`)
|
||||
|
||||
### New command in the `ServerCommand` oneof
|
||||
```protobuf
|
||||
message RunStepCmd {
|
||||
string interpreter = 1; // "bash" | "powershell"
|
||||
string script = 2;
|
||||
map<string, string> env = 3; // inputs = accumulated run env + injected secrets
|
||||
int32 timeout_seconds = 4;
|
||||
}
|
||||
```
|
||||
Add `RunStepCmd run_step = 6;` to the `ServerCommand` oneof.
|
||||
|
||||
### Richer result — new `AgentMessage` payload
|
||||
Current `CommandResult{command_id, success, message}` is too thin. Add a dedicated step result:
|
||||
```protobuf
|
||||
message StepResult {
|
||||
string command_id = 1;
|
||||
int32 exit_code = 2;
|
||||
string stdout = 3;
|
||||
string stderr = 4;
|
||||
map<string, string> output_env = 5; // parsed $WORKFLOW_ENV KEY=value lines
|
||||
}
|
||||
```
|
||||
Add `StepResult step_result = 5;` to the `AgentMessage` oneof (alongside existing `ready` / `result`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Agent execution (`agent/internal/...`)
|
||||
|
||||
New handler for `RunStepCmd` in the agent command loop:
|
||||
|
||||
1. Create a temp dir; create empty `WORKFLOW_ENV` file inside it.
|
||||
2. Write `script` to a temp script file.
|
||||
3. Build the process environment: inherited env + `cmd.env` (run env + secrets) + `WORKFLOW_ENV=<path to env file>`.
|
||||
4. Execute:
|
||||
- `bash` → `bash <script>`
|
||||
- `powershell` → `pwsh -NoProfile -File <script>` (fallback `powershell.exe` on Windows if `pwsh` absent).
|
||||
5. Capture stdout, stderr, exit code. Enforce `timeout_seconds` (kill on exceed → non-zero exit, stderr note).
|
||||
6. Parse the `WORKFLOW_ENV` file: each `KEY=value` line becomes an `output_env` entry (last write wins; supports multi-line via simple `KEY<<EOF` heredoc form, optional for v1 — start with single-line `KEY=value`).
|
||||
7. Reply with `StepResult`. Delete temp dir.
|
||||
|
||||
Agent runs as root (existing), so no privilege change. Script content is trusted operator input.
|
||||
|
||||
---
|
||||
|
||||
## 6. Server orchestration (`server/internal/services/workflows.go`)
|
||||
|
||||
Runner responsibilities:
|
||||
|
||||
1. On trigger: snapshot the workflow (resolve each library step + overrides), create a `workflow_runs` doc with one `server_run` per target, all `queued`.
|
||||
2. Spawn one goroutine **per target server** (parallel fan-out). Each goroutine:
|
||||
- Verifies the agent is connected (`Dispatcher.IsConnected`); if not → `server_run.status = skipped`, reason recorded.
|
||||
- Maintains a `run_env map[string]string`, seeded empty.
|
||||
- For each step in order:
|
||||
- Resolve `secret_refs` from the secrets service → merge into the command env (kept separate from persisted `run_env`).
|
||||
- Dispatch `RunStepCmd{env: run_env + secrets}` via a **correlated** send — needs a way to await the matching `StepResult` by `command_id` (see §7).
|
||||
- On result: persist step record (stdout/stderr/exit, masked); merge `output_env` into `run_env`.
|
||||
- Apply `on_failure` on non-zero exit: `stop` (fail server_run, break), `continue` (mark failed, proceed), `retry` (re-dispatch up to `max_retries`).
|
||||
3. Aggregate: run `status = success` if all server_runs succeeded, else `failed`. Set `finished_at`.
|
||||
|
||||
### Concurrency / queue
|
||||
- One workflow run per workflow at a time (reject or queue concurrent triggers — v1: reject with clear error).
|
||||
- Per-server step dispatch is serial; servers are parallel.
|
||||
|
||||
---
|
||||
|
||||
## 7. Correlated command results
|
||||
|
||||
The existing dispatcher is fire-and-forget; workflows need request/response by `command_id`. Add a small **pending-result registry** alongside `Dispatcher`:
|
||||
|
||||
- `AwaitResult(commandID) <-chan *pb.StepResult` — registers a channel before dispatch.
|
||||
- The `CommandStream` receive loop, on a `StepResult`, looks up the pending channel by `command_id` and delivers it (falls back to existing `CommandResult` handling for other command types).
|
||||
- Timeout guard on the server side (step `timeout_seconds` + grace) so a dead agent can't hang a run.
|
||||
|
||||
This is additive; existing `CommandResult` flow for key/update commands is unchanged.
|
||||
|
||||
---
|
||||
|
||||
## 8. REST API (`server/internal/api/workflows.go`)
|
||||
|
||||
| Method + path | Purpose |
|
||||
|---------------|---------|
|
||||
| `GET /api/steps` / `POST` / `PUT /:id` / `DELETE /:id` | Reusable step library CRUD |
|
||||
| `GET /api/workflows` / `POST` / `PUT /:id` / `DELETE /:id` | Workflow CRUD (name, targets, ordered steps) |
|
||||
| `POST /api/workflows/:id/run` | Trigger a run; returns `run_id` |
|
||||
| `GET /api/workflows/:id/runs` | Run history (summary list) |
|
||||
| `GET /api/runs/:run_id` | Full run detail incl. per-server per-step logs |
|
||||
| `POST /api/runs/:run_id/cancel` | Best-effort cancel |
|
||||
|
||||
Secrets are referenced by key only through these APIs; values never returned.
|
||||
|
||||
---
|
||||
|
||||
## 9. Frontend (`web/app/workflows/`)
|
||||
|
||||
- `/workflows` — list workflows, last run status/time, Run button.
|
||||
- `/workflows/[id]` — the three-pane builder from the approved mockup:
|
||||
- **Library** (left): reusable steps, `bash`/`pwsh` badges, search, add.
|
||||
- **Canvas** (center): ordered nodes, env chips on wires, live status pills.
|
||||
- **Inspector** (right): name, command editor, declared inputs/outputs, `secret_refs` picker, `on_failure` + retry count.
|
||||
- `/workflows/[id]/runs/[runId]` — run detail: per-server columns, expandable per-step stdout/stderr, exit codes, timing. Live-updating while `running` (poll, consistent with existing 30s-poll ethos — or reuse whatever the console screen uses).
|
||||
|
||||
Reuse existing web components/styling patterns (there is already `servers`, `secrets`, `audit`, console UI to match).
|
||||
|
||||
---
|
||||
|
||||
## 10. Security notes
|
||||
|
||||
- Scripts are trusted operator input executed as root — same trust level as the existing console feature. No new sandbox in v1.
|
||||
- Secret values injected as env only; masked from all persisted logs (`stdout`/`stderr`/`run_env`) by literal replacement before write.
|
||||
- Run triggering and step/workflow CRUD gated behind existing auth (`server/internal/auth`).
|
||||
- Audit: emit audit-log entries (existing `audit` service) on workflow create/edit/delete and run trigger.
|
||||
|
||||
---
|
||||
|
||||
## 11. Out of scope (this iteration)
|
||||
|
||||
- Tests (explicitly skipped).
|
||||
- Branching/conditional steps, matrix per-server conditionals (fan-out only).
|
||||
- Scheduled/cron triggers (manual run only for v1).
|
||||
- Multi-org isolation enforcement (schema carries `org_id` for later; single-org behavior now).
|
||||
- Artifact upload/collection beyond env vars.
|
||||
Reference in New Issue
Block a user