This commit is contained in:
@@ -1,241 +0,0 @@
|
||||
# Vantage Web Console (Guacamole Replacement) — Design
|
||||
|
||||
**Date:** 2026-07-17
|
||||
**Status:** Approved design, pre-implementation
|
||||
|
||||
## Goal
|
||||
|
||||
Add a browser-based remote-access console to Vantage — SSH, RDP, and VNC into
|
||||
managed servers — as a self-hosted Guacamole replacement. Users select an SSH
|
||||
key to connect over SSH. RDP targets are reachable from a new Windows agent that
|
||||
registers the host and reports status. Windows agent ships as an MSI installer
|
||||
produced by CI.
|
||||
|
||||
## Non-Goals (YAGNI)
|
||||
|
||||
- Session recording / replay (may be added later).
|
||||
- Native Go RDP implementation (guacd handles protocol translation).
|
||||
- Per-user Linux/Windows account management from the agent.
|
||||
- Tunneling console traffic through the agent (direct network path assumed).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser (guacamole-common-js, vendored — no CDN)
|
||||
│ Guacamole protocol over WebSocket
|
||||
▼
|
||||
Go server: /api/console/tunnel (github.com/wwt/guac)
|
||||
│ Guacamole protocol over TCP :4822
|
||||
▼
|
||||
guacd container (Apache Guacamole daemon)
|
||||
│ SSH :22 / RDP :3389 / VNC :5900 — direct to target IP
|
||||
▼
|
||||
Target host (LAN / VPN line-of-sight from server)
|
||||
```
|
||||
|
||||
- **Browser:** loads vendored `guacamole-common-js`, renders RDP/VNC display and
|
||||
SSH terminal. No external CDN (matches existing infra rules).
|
||||
- **Go server:** exposes a WebSocket tunnel endpoint using `github.com/wwt/guac`
|
||||
(Go Guacamole tunnel library). No Java `guacamole-client` required.
|
||||
- **guacd:** new container in `deploy/docker-compose.yml`, bound to the internal
|
||||
docker network only, reachable by the server on `:4822`.
|
||||
- **Network path:** guacd connects **directly** to the target IP. Requires the
|
||||
central server to have network line-of-sight to hosts (homelab LAN / VPN). The
|
||||
agent's outbound-only guarantee is unchanged — the console path is
|
||||
server→target, not agent-mediated.
|
||||
|
||||
---
|
||||
|
||||
## Data Model Changes
|
||||
|
||||
### `keys` — extend to hold private material
|
||||
|
||||
```json
|
||||
{
|
||||
"key_id": "uuid",
|
||||
"label": "dom-macbook",
|
||||
"public_key": "ssh-ed25519 AAAA...",
|
||||
"private_key_enc": "<AES-256-GCM ciphertext | null>",
|
||||
"has_private": true,
|
||||
"passphrase_enc": "<AES-256-GCM ciphertext | null>",
|
||||
"fingerprint": "SHA256:...",
|
||||
"source": "uploaded|generated",
|
||||
"created_at": "ISODate"
|
||||
}
|
||||
```
|
||||
|
||||
- A key may be created from an uploaded **private+public** pair, upload of a
|
||||
public key only, or agent generation.
|
||||
- Agent key generation now also uploads `private_key_enc` (reuses the existing
|
||||
AES-256 key used for at-rest encryption). Private key no longer stays local
|
||||
only — it is stored encrypted so the console can reuse it.
|
||||
- Optional `passphrase_enc` for passphrase-protected private keys.
|
||||
- Console lists only keys where `has_private = true`.
|
||||
|
||||
### `servers` — extend with console metadata
|
||||
|
||||
```json
|
||||
{
|
||||
"...": "...existing fields...",
|
||||
"os_type": "linux|windows",
|
||||
"console_protocols": ["ssh"],
|
||||
"ssh_port": 22,
|
||||
"rdp_port": 3389
|
||||
}
|
||||
```
|
||||
|
||||
- `os_type` set at registration from the agent.
|
||||
- `console_protocols` lists enabled protocols per server (`ssh`, `rdp`, `vnc`).
|
||||
- Port fields default to standard ports, overridable in the UI.
|
||||
|
||||
### `console_sessions` — new collection (audit)
|
||||
|
||||
```json
|
||||
{
|
||||
"session_id": "uuid",
|
||||
"server_id": "uuid",
|
||||
"protocol": "ssh|rdp|vnc",
|
||||
"key_id": "uuid | null",
|
||||
"user": "who opened it",
|
||||
"started_at": "ISODate",
|
||||
"ended_at": "ISODate | null",
|
||||
"client_ip": "string"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Broker + Connection Flow
|
||||
|
||||
New service: `server/internal/services/console.go`.
|
||||
|
||||
1. Browser `POST /api/console/connect`
|
||||
`{ server_id, protocol, key_id?, rdp_username?, rdp_password? }`.
|
||||
2. Broker validates request, loads the server (host IP, port for protocol),
|
||||
loads the key and **decrypts `private_key_enc` in memory only**.
|
||||
3. Builds the guacd connection parameter map:
|
||||
- **SSH:** `hostname`, `port`, `username`, `private-key` (decrypted),
|
||||
`passphrase` (if any).
|
||||
- **RDP:** `hostname`, `port`, `username`, `password`, `security=any`,
|
||||
`ignore-cert=true`.
|
||||
- **VNC:** `hostname`, `port`, `password`.
|
||||
4. Creates a `console_sessions` document, returns a short-lived signed session
|
||||
token.
|
||||
5. Browser opens WebSocket `/api/console/tunnel?token=…`. The `wwt/guac` handler
|
||||
validates the token, dials guacd `:4822`, and pipes bytes in both directions.
|
||||
6. On socket close, the broker sets `ended_at` on the session doc.
|
||||
|
||||
### Security
|
||||
|
||||
- Decrypted private keys and RDP passwords are **never persisted, never logged,
|
||||
never sent to the browser** — passed only to guacd.
|
||||
- Session token: short TTL (~60s to open the WebSocket), single-use,
|
||||
HMAC-signed, bound to the authenticated user.
|
||||
- guacd is bound to the internal docker network only; not exposed publicly.
|
||||
- At-rest encryption (`private_key_enc`, `passphrase_enc`) reuses the existing
|
||||
AES-256 key already used for agent-generated private keys.
|
||||
|
||||
---
|
||||
|
||||
## Windows Agent
|
||||
|
||||
Same Go codebase as the Linux agent, with a reduced role: **register +
|
||||
heartbeat + status only**. No `authorized_keys` management (meaningless on
|
||||
Windows).
|
||||
|
||||
- Build target: `GOOS=windows GOARCH=amd64` → `vantage-agent-windows-amd64.exe`.
|
||||
- Agent detects OS at registration and sends `os_type=windows`.
|
||||
- The key-sync loop is disabled on Windows via a runtime OS check (or build tag)
|
||||
— no `authorized_keys` writes are ever attempted.
|
||||
- Config file: `C:\ProgramData\vantage\config.yaml`, locked down via ACL to the
|
||||
equivalent of `0600`.
|
||||
- Runs as a Windows service via **nssm**.
|
||||
|
||||
---
|
||||
|
||||
## Windows Installer (MSI)
|
||||
|
||||
Agent ships as a WiX v4 MSI produced in CI.
|
||||
|
||||
- **WiX v4** chosen because it is a dotnet tool that builds MSIs
|
||||
**cross-platform** — runs on the Linux Gitea act_runner. (Inno Setup is
|
||||
Windows-only and does not fit the runner.)
|
||||
- MSI bundles `vantage-agent.exe`, installs it to `C:\Program Files\Vantage\`,
|
||||
and registers the nssm service (ships nssm or uses a CustomAction).
|
||||
- Accepts install parameters as MSI properties for silent/headless install:
|
||||
```
|
||||
msiexec /i vantage-agent.msi /qn SERVERID=<id> TOKEN=<token> SERVERURL=vantage..:9090
|
||||
```
|
||||
- GUI install (double-click) prompts for server-id / token / server-url via a
|
||||
dialog.
|
||||
|
||||
### Two install paths
|
||||
|
||||
1. **Installer direct** — user downloads `vantage-agent.msi`, double-clicks,
|
||||
fills the dialog. No script required.
|
||||
2. **PowerShell one-liner** — served dynamically (like the existing bash
|
||||
`/install`). Script downloads the `.msi`, verifies SHA-256, then runs
|
||||
`msiexec /qn` with injected `SERVERID` / `TOKEN` / `SERVERURL`. Used by the
|
||||
copy-paste "Add Server" flow.
|
||||
|
||||
The PowerShell script (`/install.ps1`) steps:
|
||||
1. Detect arch.
|
||||
2. Download `vantage-agent.msi` from the latest Gitea `agent/v*` release.
|
||||
3. Verify SHA-256 against `checksums.txt`.
|
||||
4. Run `msiexec /i vantage-agent.msi /qn SERVERID=.. TOKEN=.. SERVERURL=..`.
|
||||
|
||||
---
|
||||
|
||||
## Frontend Routes
|
||||
|
||||
| Route | Change |
|
||||
| ------------------------- | ------------------------------------------------------------- |
|
||||
| `/servers` | Show `os_type` badge, enabled console protocols |
|
||||
| `/servers/[id]` | Add **Connect** button(s) per enabled protocol |
|
||||
| `/servers/[id]/console` | New — full-screen console (guacamole-common-js), key picker |
|
||||
| `/servers/new` | Offer Windows (MSI) vs Linux (bash) install instructions |
|
||||
|
||||
Console page: select protocol + SSH key (SSH) or enter RDP credentials, call
|
||||
`/api/console/connect`, open the tunnel WebSocket, mount the Guacamole client.
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Changes
|
||||
|
||||
### `agent-release.yml`
|
||||
|
||||
- Add `windows/amd64` build: `vantage-agent-windows-amd64.exe`.
|
||||
- Add WiX v4 MSI build job → `vantage-agent.msi`.
|
||||
- Add both to `checksums.txt` and release assets.
|
||||
|
||||
Release assets become:
|
||||
- `vantage-agent-linux-amd64`
|
||||
- `vantage-agent-linux-arm64`
|
||||
- `vantage-agent-windows-amd64.exe`
|
||||
- `vantage-agent.msi`
|
||||
- `checksums.txt`
|
||||
|
||||
### `server-deploy.yml`
|
||||
|
||||
- Add guacd service to `deploy/docker-compose.yml` (deployed alongside server).
|
||||
|
||||
---
|
||||
|
||||
## New Dependencies
|
||||
|
||||
- **Go:** `github.com/wwt/guac` (Guacamole tunnel/WebSocket in Go).
|
||||
- **Container:** `guacamole/guacd` official image.
|
||||
- **Frontend:** vendored `guacamole-common-js` (no CDN).
|
||||
- **CI:** WiX v4 dotnet tool; nssm binary bundled for the MSI.
|
||||
|
||||
---
|
||||
|
||||
## Open Implementation Notes
|
||||
|
||||
- Confirm `wwt/guac` API surface for connection-parameter passing and token auth
|
||||
binding during implementation.
|
||||
- nssm packaging inside MSI: bundle the nssm binary as a payload + CustomAction,
|
||||
or run `sc.exe`-based service install if nssm proves awkward in WiX.
|
||||
- ACL hardening of `C:\ProgramData\vantage\config.yaml` in the MSI CustomAction.
|
||||
@@ -1,238 +0,0 @@
|
||||
# Server Workflows — Design
|
||||
|
||||
**Date:** 2026-07-20
|
||||
**Status:** Approved (design) — ready for implementation planning
|
||||
**Scope:** Server Workflows only. Fleet Inventory and SaaS/local-auth are separate sub-projects with their own specs.
|
||||
|
||||
Approved UI mockup: three-pane builder (Step Library · Canvas · Inspector), env vars shown riding the wire between nodes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
|
||||
Let operators compose **reusable shell steps** (Bash or PowerShell) into **workflows** and run them across many managed servers in parallel. Steps pass data to later steps through a `$WORKFLOW_ENV` file (GitHub-Actions style). Every run is recorded with full per-step logs. Steps can reference org secrets, injected as environment variables at runtime.
|
||||
|
||||
Builds directly on the existing `CommandStream` gRPC infrastructure (`dispatch.go`, `ServerCommand` oneof, agent command loop).
|
||||
|
||||
---
|
||||
|
||||
## 2. Locked decisions
|
||||
|
||||
| Topic | Decision |
|
||||
|-------|----------|
|
||||
| Data passing | Implicit. Every step's `$WORKFLOW_ENV` outputs merge into the run's env and are exposed to **all** later steps as `$KEY`. No explicit port wiring. |
|
||||
| Failure model | Per-step policy: `stop` (default), `continue`, `retry` (with max attempt count). |
|
||||
| Targets | Fan-out. Same step sequence runs on N target servers **in parallel**. Steps within one server run **sequentially**. |
|
||||
| History/logs | Every run persisted: status, timing, per-server per-step stdout/stderr/exit code, captured output env. |
|
||||
| Secrets | Steps declare needed secret keys; resolved from existing `secrets` store and injected as env vars at exec time. Never persisted into run logs. |
|
||||
| Testing | **Skipped** for this iteration per request. No test files written. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model (MongoDB)
|
||||
|
||||
### `workflow_steps` — reusable step library
|
||||
```json
|
||||
{
|
||||
"_id": "ObjectId",
|
||||
"step_id": "uuid",
|
||||
"name": "Restart service",
|
||||
"description": "Restart-Service by name, wait ready",
|
||||
"interpreter": "bash | powershell",
|
||||
"script": "Restart-Service vantage-api\n...",
|
||||
"declared_outputs": ["STARTED_AT"], // documentation/UI hints; not enforced
|
||||
"secret_refs": ["DEPLOY_TOKEN"], // secret keys this step needs injected
|
||||
"org_id": "uuid", // for future multi-tenant; single-org for now
|
||||
"created_at": "ISODate",
|
||||
"updated_at": "ISODate"
|
||||
}
|
||||
```
|
||||
|
||||
### `workflows` — ordered composition
|
||||
```json
|
||||
{
|
||||
"_id": "ObjectId",
|
||||
"workflow_id": "uuid",
|
||||
"name": "Deploy & Restart API",
|
||||
"target_server_ids": ["uuid", "uuid"],
|
||||
"steps": [
|
||||
{
|
||||
"step_id": "uuid", // reference to library step
|
||||
"order": 0,
|
||||
"on_failure": "stop | continue | retry",
|
||||
"max_retries": 0, // used when on_failure = retry
|
||||
"overrides": { // optional local fork of the library step
|
||||
"script": null,
|
||||
"secret_refs": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"created_at": "ISODate",
|
||||
"updated_at": "ISODate"
|
||||
}
|
||||
```
|
||||
Editing a library step from the Inspector writes an `overrides` block on that workflow step (a local fork) rather than mutating the shared step.
|
||||
|
||||
### `workflow_runs` — execution records
|
||||
```json
|
||||
{
|
||||
"_id": "ObjectId",
|
||||
"run_id": "uuid",
|
||||
"workflow_id": "uuid",
|
||||
"workflow_snapshot": { }, // frozen copy of workflow + resolved steps at trigger time
|
||||
"status": "running | success | failed | cancelled",
|
||||
"triggered_by": "user-id",
|
||||
"started_at": "ISODate",
|
||||
"finished_at": "ISODate | null",
|
||||
"server_runs": [
|
||||
{
|
||||
"server_id": "uuid",
|
||||
"status": "queued | running | success | failed | skipped",
|
||||
"started_at": "ISODate | null",
|
||||
"finished_at": "ISODate | null",
|
||||
"run_env": { "VERSION": "a1b9f0" }, // accumulated non-secret output env
|
||||
"steps": [
|
||||
{
|
||||
"order": 0,
|
||||
"name": "Git pull & build",
|
||||
"status": "success | failed | running | queued | skipped",
|
||||
"attempts": 1,
|
||||
"exit_code": 0,
|
||||
"stdout": "…",
|
||||
"stderr": "…",
|
||||
"output_env": { "VERSION": "a1b9f0" },
|
||||
"started_at": "ISODate",
|
||||
"finished_at": "ISODate"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
Secret values are never written to `stdout`/`stderr`/`run_env` by us; masking of known secret values in captured output is applied before persistence.
|
||||
|
||||
---
|
||||
|
||||
## 4. gRPC protocol changes (`proto/vantage/v1/vantage.proto`)
|
||||
|
||||
### New command in the `ServerCommand` oneof
|
||||
```protobuf
|
||||
message RunStepCmd {
|
||||
string interpreter = 1; // "bash" | "powershell"
|
||||
string script = 2;
|
||||
map<string, string> env = 3; // inputs = accumulated run env + injected secrets
|
||||
int32 timeout_seconds = 4;
|
||||
}
|
||||
```
|
||||
Add `RunStepCmd run_step = 6;` to the `ServerCommand` oneof.
|
||||
|
||||
### Richer result — new `AgentMessage` payload
|
||||
Current `CommandResult{command_id, success, message}` is too thin. Add a dedicated step result:
|
||||
```protobuf
|
||||
message StepResult {
|
||||
string command_id = 1;
|
||||
int32 exit_code = 2;
|
||||
string stdout = 3;
|
||||
string stderr = 4;
|
||||
map<string, string> output_env = 5; // parsed $WORKFLOW_ENV KEY=value lines
|
||||
}
|
||||
```
|
||||
Add `StepResult step_result = 5;` to the `AgentMessage` oneof (alongside existing `ready` / `result`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Agent execution (`agent/internal/...`)
|
||||
|
||||
New handler for `RunStepCmd` in the agent command loop:
|
||||
|
||||
1. Create a temp dir; create empty `WORKFLOW_ENV` file inside it.
|
||||
2. Write `script` to a temp script file.
|
||||
3. Build the process environment: inherited env + `cmd.env` (run env + secrets) + `WORKFLOW_ENV=<path to env file>`.
|
||||
4. Execute:
|
||||
- `bash` → `bash <script>`
|
||||
- `powershell` → `pwsh -NoProfile -File <script>` (fallback `powershell.exe` on Windows if `pwsh` absent).
|
||||
5. Capture stdout, stderr, exit code. Enforce `timeout_seconds` (kill on exceed → non-zero exit, stderr note).
|
||||
6. Parse the `WORKFLOW_ENV` file: each `KEY=value` line becomes an `output_env` entry (last write wins; supports multi-line via simple `KEY<<EOF` heredoc form, optional for v1 — start with single-line `KEY=value`).
|
||||
7. Reply with `StepResult`. Delete temp dir.
|
||||
|
||||
Agent runs as root (existing), so no privilege change. Script content is trusted operator input.
|
||||
|
||||
---
|
||||
|
||||
## 6. Server orchestration (`server/internal/services/workflows.go`)
|
||||
|
||||
Runner responsibilities:
|
||||
|
||||
1. On trigger: snapshot the workflow (resolve each library step + overrides), create a `workflow_runs` doc with one `server_run` per target, all `queued`.
|
||||
2. Spawn one goroutine **per target server** (parallel fan-out). Each goroutine:
|
||||
- Verifies the agent is connected (`Dispatcher.IsConnected`); if not → `server_run.status = skipped`, reason recorded.
|
||||
- Maintains a `run_env map[string]string`, seeded empty.
|
||||
- For each step in order:
|
||||
- Resolve `secret_refs` from the secrets service → merge into the command env (kept separate from persisted `run_env`).
|
||||
- Dispatch `RunStepCmd{env: run_env + secrets}` via a **correlated** send — needs a way to await the matching `StepResult` by `command_id` (see §7).
|
||||
- On result: persist step record (stdout/stderr/exit, masked); merge `output_env` into `run_env`.
|
||||
- Apply `on_failure` on non-zero exit: `stop` (fail server_run, break), `continue` (mark failed, proceed), `retry` (re-dispatch up to `max_retries`).
|
||||
3. Aggregate: run `status = success` if all server_runs succeeded, else `failed`. Set `finished_at`.
|
||||
|
||||
### Concurrency / queue
|
||||
- One workflow run per workflow at a time (reject or queue concurrent triggers — v1: reject with clear error).
|
||||
- Per-server step dispatch is serial; servers are parallel.
|
||||
|
||||
---
|
||||
|
||||
## 7. Correlated command results
|
||||
|
||||
The existing dispatcher is fire-and-forget; workflows need request/response by `command_id`. Add a small **pending-result registry** alongside `Dispatcher`:
|
||||
|
||||
- `AwaitResult(commandID) <-chan *pb.StepResult` — registers a channel before dispatch.
|
||||
- The `CommandStream` receive loop, on a `StepResult`, looks up the pending channel by `command_id` and delivers it (falls back to existing `CommandResult` handling for other command types).
|
||||
- Timeout guard on the server side (step `timeout_seconds` + grace) so a dead agent can't hang a run.
|
||||
|
||||
This is additive; existing `CommandResult` flow for key/update commands is unchanged.
|
||||
|
||||
---
|
||||
|
||||
## 8. REST API (`server/internal/api/workflows.go`)
|
||||
|
||||
| Method + path | Purpose |
|
||||
|---------------|---------|
|
||||
| `GET /api/steps` / `POST` / `PUT /:id` / `DELETE /:id` | Reusable step library CRUD |
|
||||
| `GET /api/workflows` / `POST` / `PUT /:id` / `DELETE /:id` | Workflow CRUD (name, targets, ordered steps) |
|
||||
| `POST /api/workflows/:id/run` | Trigger a run; returns `run_id` |
|
||||
| `GET /api/workflows/:id/runs` | Run history (summary list) |
|
||||
| `GET /api/runs/:run_id` | Full run detail incl. per-server per-step logs |
|
||||
| `POST /api/runs/:run_id/cancel` | Best-effort cancel |
|
||||
|
||||
Secrets are referenced by key only through these APIs; values never returned.
|
||||
|
||||
---
|
||||
|
||||
## 9. Frontend (`web/app/workflows/`)
|
||||
|
||||
- `/workflows` — list workflows, last run status/time, Run button.
|
||||
- `/workflows/[id]` — the three-pane builder from the approved mockup:
|
||||
- **Library** (left): reusable steps, `bash`/`pwsh` badges, search, add.
|
||||
- **Canvas** (center): ordered nodes, env chips on wires, live status pills.
|
||||
- **Inspector** (right): name, command editor, declared inputs/outputs, `secret_refs` picker, `on_failure` + retry count.
|
||||
- `/workflows/[id]/runs/[runId]` — run detail: per-server columns, expandable per-step stdout/stderr, exit codes, timing. Live-updating while `running` (poll, consistent with existing 30s-poll ethos — or reuse whatever the console screen uses).
|
||||
|
||||
Reuse existing web components/styling patterns (there is already `servers`, `secrets`, `audit`, console UI to match).
|
||||
|
||||
---
|
||||
|
||||
## 10. Security notes
|
||||
|
||||
- Scripts are trusted operator input executed as root — same trust level as the existing console feature. No new sandbox in v1.
|
||||
- Secret values injected as env only; masked from all persisted logs (`stdout`/`stderr`/`run_env`) by literal replacement before write.
|
||||
- Run triggering and step/workflow CRUD gated behind existing auth (`server/internal/auth`).
|
||||
- Audit: emit audit-log entries (existing `audit` service) on workflow create/edit/delete and run trigger.
|
||||
|
||||
---
|
||||
|
||||
## 11. Out of scope (this iteration)
|
||||
|
||||
- Tests (explicitly skipped).
|
||||
- Branching/conditional steps, matrix per-server conditionals (fan-out only).
|
||||
- Scheduled/cron triggers (manual run only for v1).
|
||||
- Multi-org isolation enforcement (schema carries `org_id` for later; single-org behavior now).
|
||||
- Artifact upload/collection beyond env vars.
|
||||
@@ -1,150 +0,0 @@
|
||||
# Workflow Builder v2 — Design
|
||||
|
||||
**Date:** 2026-07-20
|
||||
**Status:** Approved (design) — ready for implementation planning
|
||||
**Scope:** Overhaul the workflow builder UI to match the approved mockup, add drag-and-drop (library→canvas + reorder), base-step editing/deletion with cascade, step input parameters, an Edit-Workflow modal, runs navigation, and fix the save crash. Enhancement to the merged Server Workflows feature. Independent of the in-flight log-streaming work.
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
|
||||
The shipped builder diverges from the approved mockup and is missing interactions. This iteration:
|
||||
|
||||
1. **Restyle** the builder (`/workflows/[id]`) to the approved mockup: dotted-grid canvas, 340px node cards with index badge + shell badge + status, wire connectors with dashed-amber "passes" env chips, a kicker/field inspector, and a library of grabbable step cards with descriptions.
|
||||
2. **Drag-and-drop**: drag a library step onto the canvas to add it; drag nodes to reorder. Remove the up/down/remove buttons.
|
||||
3. **Base-step editing**: an "Edit base step" modal edits the shared library step (name/interpreter/script/outputs/inputs/secret refs) and **saves**; deleting a shared step **cascades** — it is pulled from every workflow that references it.
|
||||
4. **Input parameters**: a base step can declare inputs (`name` + `default` + `description`); when placed, each placement sets values; the runner injects them into the step's environment.
|
||||
5. **Env visibility**: show the output variables passed between steps as chips on the wires (already partially present — align to the mockup).
|
||||
6. **Edit-Workflow modal**: edit name, target servers, delete the workflow, and other workflow settings.
|
||||
7. **Runs navigation**: a runs list page per workflow, linked from the builder and the workflows list.
|
||||
8. **Bug fix**: `updateWorkflow` returns `{updated:true}`, which the builder stores as the workflow and then crashes on `[...wf.steps]` ("d.steps is not iterable"). Fix the endpoint to return the updated workflow and harden the client.
|
||||
|
||||
---
|
||||
|
||||
## 2. Locked decisions
|
||||
|
||||
| Topic | Decision |
|
||||
|-------|----------|
|
||||
| Visual target | The approved mockup (artifact `61ab5256`). Adopt its layout + **amber (`#f5a524`) as the builder signal/focus color**, plus bash-green (`#3fb950`) / pwsh-blue (`#5b9bff`) badge colors. Keep the app's existing `surface`/`border`/`text-*` tokens for panels so it integrates with the dark theme. |
|
||||
| Drag-and-drop | Native HTML5 DnD. Library cards are `draggable`; the canvas has drop targets (between nodes + end zone) to insert; nodes are `draggable` to reorder. Clicking a library card still appends (keyboard/fallback). |
|
||||
| Step removal | No per-node buttons. Remove a placed step from the **inspector** ("Remove from workflow"). |
|
||||
| Input params | `WorkflowStep.declared_inputs: [{name, default, description}]`. `WorkflowStepRef.inputs: map[name]value`. Runner resolves `value = ref.inputs[name] ?? default` and injects as env vars. |
|
||||
| Edit scope | Inspector script edit = **per-placement override** (existing `overrides`, "forks a local copy"). A separate **Edit base step** modal updates the shared library step for all workflows. |
|
||||
| Cascade delete | Deleting a library step pulls its `step_id` from every `workflow.steps` and re-sequences remaining `order`s. |
|
||||
| Env chips | Names passed between steps = union of prior steps' `declared_outputs`. Shown on the wire between nodes. |
|
||||
| Edit workflow | Modal launched from the topbar: name, target-servers multiselect, delete workflow. |
|
||||
| Runs nav | New page `/workflows/[id]/runs` (list); links from the builder topbar and the workflows list page. |
|
||||
| Save fix | `PUT /api/workflows/:id` returns the full updated `Workflow`. Client also guards against non-workflow responses. |
|
||||
| Proto | **No proto change** — input params travel through the existing `RunStepCmd.Env`. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model (`server/internal/models/workflow.go`)
|
||||
|
||||
Add an input-parameter type and fields:
|
||||
|
||||
```go
|
||||
type InputParam struct {
|
||||
Name string `bson:"name" json:"name"`
|
||||
Default string `bson:"default" json:"default"`
|
||||
Description string `bson:"description" json:"description"`
|
||||
}
|
||||
```
|
||||
|
||||
- `WorkflowStep` gains: `DeclaredInputs []InputParam `bson:"declared_inputs" json:"declared_inputs"``.
|
||||
- `WorkflowStepRef` gains: `Inputs map[string]string `bson:"inputs,omitempty" json:"inputs,omitempty"`` (per-placement values).
|
||||
- `ResolvedStep` gains: `Inputs map[string]string `bson:"inputs" json:"inputs"`` (frozen resolved input env for the run).
|
||||
|
||||
`declared_inputs` defaults to `[]` on create (like `declared_outputs`).
|
||||
|
||||
---
|
||||
|
||||
## 4. Services
|
||||
|
||||
### 4.1 Step CRUD (`server/internal/services/workflows.go`)
|
||||
|
||||
- `CreateStep`: default `DeclaredInputs` to `[]InputParam{}` when nil; persist it.
|
||||
- `UpdateStep`: add `declared_inputs` to the `$set`.
|
||||
- `DeleteStep` → **cascade**. New behavior: within the delete, also update every workflow that references the step:
|
||||
1. `DeleteOne` on `workflow_steps` by `step_id` (as today).
|
||||
2. Load all workflows containing the step (`workflows` where `steps.step_id == stepID`); for each, remove the matching `WorkflowStepRef`(s), re-sequence remaining `order` values to `0..n-1`, and `UpdateWorkflow`.
|
||||
Keep it a single service call `DeleteStep(stepID)` so the handler is unchanged. Audit both the step deletion and each affected workflow via `LogEvent`.
|
||||
|
||||
### 4.2 Workflow update returns the workflow (`server/internal/services/workflows.go` + handler)
|
||||
|
||||
- `UpdateWorkflow(id, w)` stays, but the **handler** `updateWorkflow` re-fetches and returns the full workflow: after `services.UpdateWorkflow`, call `services.GetWorkflow(id)` and return it (200 with the `Workflow` JSON) instead of `{"updated": true}`. This is the crash fix's server half.
|
||||
|
||||
### 4.3 Runner input injection (`server/internal/services/workflow_runner.go`)
|
||||
|
||||
- `resolveSteps`: when freezing each `ResolvedStep`, compute `Inputs`: for each `InputParam` on the library step, `value = ref.Inputs[name]` if present else `param.Default`; store the resulting `map[string]string` on `ResolvedStep.Inputs`.
|
||||
- `runServer`: when building `cmdEnv`, merge `step.Inputs` **first** (base layer), then `runEnv` (upstream outputs), then `secretVals` (highest precedence). Input values are not secrets and are not masked (they are user-provided config, not secret material) — unless a value coincidentally equals a secret literal, existing masking still catches it in logs.
|
||||
|
||||
---
|
||||
|
||||
## 5. REST API
|
||||
|
||||
No new endpoints. Changes:
|
||||
- `updateWorkflow` handler returns the `Workflow` (see §4.2).
|
||||
- `deleteStep` handler unchanged (cascade lives in the service).
|
||||
- Existing `GET /api/workflows/:id/runs`, `GET /api/runs/:runId` power the runs pages.
|
||||
- `createStep`/`updateStep` accept `declared_inputs` via the existing `ShouldBindJSON(&models.WorkflowStep)` (no handler change once the model has the field).
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend
|
||||
|
||||
### 6.1 Tokens + primitives
|
||||
|
||||
- `tailwind.config.ts`: add `bash: "#3fb950"`, `pwsh: "#5b9bff"`. Amber signal reuses a new `signal: "#f5a524"` token (add it) for the builder's focus ring / env chips / run button; `signal-ink: "#241800"` for text on amber chips.
|
||||
- Add a `Modal` primitive to `components/ui` (overlay + centered panel, `onClose`, ESC + backdrop click, `title`, children, exported from `index.ts`). Used by the Edit-base-step and Edit-workflow modals.
|
||||
|
||||
### 6.2 API client (`web/lib/api.ts`)
|
||||
|
||||
- `WorkflowStep`: add `declared_inputs: InputParam[]`.
|
||||
- New `InputParam { name: string; default: string; description: string }`.
|
||||
- `WorkflowStepRef`: add `inputs?: Record<string, string>`.
|
||||
- `updateWorkflow` return type stays `Workflow` (now actually returns one). Add a client guard: if a mutation is expected to return a workflow but the body lacks `steps`, treat it as an error / refetch.
|
||||
- No other method changes; `deleteStep`, `updateStep`, `createStep`, `listRuns`, `getRun` already exist.
|
||||
|
||||
### 6.3 Builder restyle + DnD (`web/app/workflows/[id]/page.tsx`)
|
||||
|
||||
Rebuild the three-pane builder to the mockup:
|
||||
|
||||
- **Topbar:** brand dot + `Workflows /` crumb + workflow name + `· draft · edited …`; right side a **Targets** chip (read-only summary, e.g. "3 servers"), a **Runs** link (→ `/workflows/[id]/runs`), an **Edit** button (opens Edit-workflow modal), **Save**, **Run workflow** (amber). Remove the inline name input and the in-canvas Target-servers card (both move into the Edit-workflow modal).
|
||||
- **Library (left):** section header with `+` (opens Edit-base-step modal in "new" mode), a search box that filters by name, grouped `Shared · Bash` / `Shared · PowerShell`, each a `.step-card` with shell badge, name, description, and a grip glyph; `draggable`. Clicking still appends. Each card has an edit affordance (pencil / context) that opens the Edit-base-step modal for that step.
|
||||
- **Canvas (center):** dotted-grid background. Render placed steps as 340px node cards: index badge, title, shell badge, (build-time) no status pill — status is a run concern; in the builder show the interpreter badge only. Node body shows a syntax-lite script preview. Between nodes render a wire + a dashed-amber `passes` chip row listing the union of prior `declared_outputs` (names only in the builder). Nodes are `draggable` to reorder; drop targets sit on the wires and at the end ("+ Drop a step here"). Selecting a node opens it in the inspector.
|
||||
- **Inspector (right):** kicker "Step N · Inspector", title with shell badge + name. Fields: **Step name** (edits the placement label? — placements don't have a name; show the library name read-only, edit happens in the base-step modal), **Command** (script `<textarea>` — this is the per-placement override; empty = inherit base), a hint "Write `KEY=value` to `$WORKFLOW_ENV`…", **Inputs** (one row per `declared_input` with an input to set the placement value, showing the default as placeholder), **Inputs · from upstream** (read-only list of upstream `declared_outputs` available to this step), **Outputs · to $WORKFLOW_ENV** (read-only list of this step's `declared_outputs`), **Secret refs** (existing group/KEY checklist), **On failure** (stop/continue/retry + max retries), and **Remove from workflow**.
|
||||
|
||||
DnD detail: use `dataTransfer` with a payload discriminating "library step" (carries `step_id`) vs "reorder" (carries the placement index). On drop at position `k`, insert/move and re-sequence `order`. Keep everything in React state; **Save** persists via `updateWorkflow` (which now returns the workflow → `setWf(returned)` no longer crashes).
|
||||
|
||||
### 6.4 Edit base step modal
|
||||
|
||||
A `Modal` with fields: name, interpreter (bash/powershell), script (`<textarea>` mono), declared **outputs** (chip/list editor — add/remove names), declared **inputs** (rows of name/default/description, add/remove), secret refs (optional). Actions: **Save** (`api.createStep` in new mode / `api.updateStep` in edit mode) then invalidate `["steps"]`; **Delete** (edit mode only) → confirm, `api.deleteStep` (cascades server-side), invalidate `["steps"]` and `["workflow", id]` (a deleted step vanishes from the canvas after refetch). Editing here changes the shared step for all workflows (surface the "shared across all workflows" hint).
|
||||
|
||||
### 6.5 Edit workflow modal
|
||||
|
||||
A `Modal` launched from the topbar **Edit** button: workflow **name** input, **target servers** multiselect (the chips currently in the canvas), and a **Delete workflow** action (confirm → `api.deleteWorkflow` → route to `/workflows`). Save applies name/targets to local `wf` state (persisted on the builder's Save) or immediately via `updateWorkflow` — immediate is simpler and avoids losing the change; use immediate save for the modal, then `setWf(returned)`.
|
||||
|
||||
### 6.6 Runs list page (`web/app/workflows/[id]/runs/page.tsx`)
|
||||
|
||||
New page: header "Runs · <workflow name>", a table of `api.listRuns(id)` rows — run id (short), status badge, started_at, triggered_by, server count — each linking to `/workflows/[id]/runs/[runId]`. A "Back to builder" link. Also add a **Runs** action/link on the workflows list page (`web/app/workflows/page.tsx`) per row and the **Runs** link in the builder topbar.
|
||||
|
||||
---
|
||||
|
||||
## 7. Security
|
||||
|
||||
- Input parameter values are user config, injected as env; not masked (not secret). Secret masking (existing) still applies to logs and to any value equal to a secret literal.
|
||||
- Cascade delete is an authenticated mutation; audited via `LogEvent` for the step and each affected workflow.
|
||||
- Modals perform the same session-authed API calls; no new trust boundary.
|
||||
|
||||
---
|
||||
|
||||
## 8. Out of scope
|
||||
|
||||
- Live run status pills inside the builder canvas (status belongs to the run detail page).
|
||||
- Typed inputs (all inputs are strings), required/validation rules, secret-typed inputs.
|
||||
- Multi-select drag of several steps, copy/paste of steps, undo/redo.
|
||||
- Reworking the run-detail page (covered by the separate log-streaming iteration).
|
||||
- Reordering via keyboard.
|
||||
- Tests (skipped, consistent with prior iterations).
|
||||
```
|
||||
@@ -1,193 +0,0 @@
|
||||
# Workflow Log Streaming — Design
|
||||
|
||||
**Date:** 2026-07-20
|
||||
**Status:** Approved (design) — ready for implementation planning
|
||||
**Scope:** Stream step stdout/stderr live from agent to server-side log files, tail them live in the UI, and auto-expire them on a retention period. Enhancement to the already-merged Server Workflows feature. No auth/orgs, no inventory.
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
|
||||
Today a workflow step buffers all stdout/stderr in agent RAM, ships it in one terminal `StepResult`, and the server persists the whole body into the `workflow_runs` Mongo document. Long/chatty steps risk: agent memory blow-up, the gRPC 4MB message ceiling, and the Mongo 16MB document cap.
|
||||
|
||||
Change to **live streaming**:
|
||||
|
||||
1. Agent streams output chunks over the existing `CommandStream` as the process runs.
|
||||
2. Server appends chunks (secret-masked) to a **per-server-run log file** on disk — not Mongo.
|
||||
3. UI tails the file live via **SSE** while a server-run is running; slices per-step by byte offset after completion.
|
||||
4. A **retention sweeper** deletes old run-log directories on a configurable period (default 30 days, set in Settings).
|
||||
|
||||
`workflow_runs` documents shrink: they no longer carry `stdout`/`stderr` bodies, only status/exit/attempts/output_env/timestamps plus a per-step `log_offset`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Locked decisions
|
||||
|
||||
| Topic | Decision |
|
||||
|-------|----------|
|
||||
| Transport | Reuse bidirectional `CommandStream`. New `AgentMessage.StepOutput` chunk message. |
|
||||
| Chunk shape | `{command_id, seq, data, eof}`. Interleaved stdout+stderr in execution order. |
|
||||
| Terminal result | `StepResult` still sent at step end, now carries only `exit_code` + `output_env` (no stdout/stderr). |
|
||||
| Log granularity | **One file per server-run**: `<logdir>/<run_id>/<server_id>.log`, with a marker line before each step. |
|
||||
| Streams | **Interleaved** — single synchronized writer on the agent, terminal-order output. |
|
||||
| Masking | **Server-side** (agent can't tell secret env from normal env). Per-stream carry buffer of `maxSecretLen-1` bytes so a secret split across a chunk boundary still masks; flushed on EOF. |
|
||||
| Live tail | **SSE** at per-server-run granularity: `GET /api/runs/:runId/servers/:serverId/logs/stream`. Post-run whole-file fetch + per-step offset slice. |
|
||||
| Retention | `settings.workflow_log_retention_days`, default **30**, editable in `/settings`. Hourly sweeper deletes `<logdir>/<run_id>/` dirs older than retention by run `finished_at`. |
|
||||
| Log dir | Env `VANTAGE_WORKFLOW_LOG_DIR`, default `<data>/workflow-logs`. Created `0700`. |
|
||||
| Mongo | No log bodies in `workflow_runs`. Disk is the source of truth for output. |
|
||||
|
||||
---
|
||||
|
||||
## 3. gRPC protocol (`proto/vantage/v1/vantage.proto` + both `pb.go` files)
|
||||
|
||||
Add to the `AgentMessage` oneof: `StepOutputChunk step_output = 6;`
|
||||
|
||||
```protobuf
|
||||
message StepOutputChunk {
|
||||
string command_id = 1;
|
||||
uint64 seq = 2; // monotonic per command_id, 0-based
|
||||
bytes data = 3; // raw interleaved stdout+stderr bytes
|
||||
bool eof = 4; // true on the final (empty) chunk
|
||||
}
|
||||
```
|
||||
|
||||
`StepResult` is unchanged in shape but `stdout`/`stderr` are now left empty by the agent (kept in the message for backward-compat / error notes only — server ignores them for log content). The server still reads `exit_code` and `output_env` from `StepResult`.
|
||||
|
||||
Hand-written JSON-codec struct added to **both** `server/internal/grpc/pb/vantage.pb.go` and `agent/internal/grpc/pb/vantage.pb.go`, identical. `AgentMessage` gains `StepOutput *StepOutputChunk` in both.
|
||||
|
||||
`data` is `[]byte` in the Go structs (JSON-codec base64-encodes it, which is fine).
|
||||
|
||||
---
|
||||
|
||||
## 4. Agent (`agent/internal/exec/exec.go`)
|
||||
|
||||
`RunStep` signature gains a chunk sink:
|
||||
|
||||
```go
|
||||
func RunStep(cmd *pb.RunStepCmd, emit func(seq uint64, data []byte)) *pb.StepResult
|
||||
```
|
||||
|
||||
- Replace the two `bytes.Buffer`s with a single `streamWriter` set as **both** `c.Stdout` and `c.Stderr`. Its `Write` takes a mutex (so stdout+stderr interleave without interleaving *within* a write), assigns the next `seq`, and calls `emit(seq, copyOfBytes)`. Chunks are whatever the OS pipe delivers (typically ≤64KB); no extra buffering/line-assembly.
|
||||
- `StepResult` returns with `Stdout`/`Stderr` empty; `ExitCode` and `OutputEnv` populated as today (env parsing unchanged).
|
||||
- On timeout/exec error, put the short note in `StepResult.Stderr` (terminal, not streamed) so the runner can still surface a failure reason even if nothing streamed.
|
||||
|
||||
Agent loop (`agent/internal/sync/sync.go`, the `cmd.RunStep != nil` goroutine): pass an `emit` closure that sends `AgentMessage{ServerId, AgentToken, StepOutput: &pb.StepOutputChunk{CommandId, Seq, Data}}` through the existing mutex-guarded `send()`. After `RunStep` returns, send a final `StepOutput{eof:true, seq:last+1}` then the terminal `StepResult` (both via `send()`). Ordering: all chunks, then eof, then StepResult.
|
||||
|
||||
---
|
||||
|
||||
## 5. Server write path
|
||||
|
||||
### 5.1 Log writer registry (`server/internal/services/steplogs.go`)
|
||||
|
||||
Parallel to `StepResults`. Keyed by `command_id`:
|
||||
|
||||
```go
|
||||
type stepLogWriter struct {
|
||||
f *os.File
|
||||
mu sync.Mutex
|
||||
carry []byte // held-back tail for boundary-safe masking
|
||||
secrets []string // secret literals to mask
|
||||
maxSecret int
|
||||
}
|
||||
var StepLogs = &stepLogRegistry{ ... }
|
||||
func (r *stepLogRegistry) Open(commandID, path string, secrets []string) (*stepLogWriter, error)
|
||||
func (r *stepLogRegistry) Append(commandID string, data []byte) // masked write
|
||||
func (r *stepLogRegistry) Close(commandID string) // flush carry, close file
|
||||
```
|
||||
|
||||
- `Append` masking: concatenate `carry+data`, mask all secret literals (`ReplaceAll(v,"***")`), then write everything except the last `maxSecret-1` bytes; keep those as the new `carry`. `Close` masks+writes the remaining carry. If `secrets` empty, write straight through (no carry).
|
||||
- The file handle is opened append-only (`O_APPEND|O_CREATE|O_WRONLY`, `0600`); dir `0700`.
|
||||
|
||||
### 5.2 Stream delivery (`server/internal/grpc/server.go`)
|
||||
|
||||
In the receive loop, after the `m.StepResult` block, add:
|
||||
|
||||
```go
|
||||
if m.StepOutput != nil {
|
||||
if m.StepOutput.Eof {
|
||||
services.StepLogs.Close(m.StepOutput.CommandId)
|
||||
} else {
|
||||
services.StepLogs.Append(m.StepOutput.CommandId, m.StepOutput.Data)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 Runner changes (`server/internal/services/workflow_runner.go`)
|
||||
|
||||
- Resolve the log dir + run/server file path once per server-run; ensure `<logdir>/<run_id>/` exists.
|
||||
- Before dispatching each step: write the step marker line to the file (`\n===== step <order>: <name> =====\n`), record the current file byte offset as the step's `log_offset` (persisted on the `StepRun`), and `StepLogs.Open(commandID, path, secretVals)` **before** `DispatchRunStep` (same ordering rule as `StepResults.Await`).
|
||||
- `dispatchAndWait` no longer expects stdout/stderr in the result. On terminal `StepResult`, `StepLogs.Close(commandID)` is driven by the agent's eof; the runner also calls `Close` defensively on timeout/dispatch-failure (idempotent).
|
||||
- **Drop** `stdout`/`stderr` from `finishStep` persistence. Masking of the streamed body is done in `Append`; `run_env`/`output_env` masking (existing, from the merged fix) stays.
|
||||
- The shared server-run file is written by two writers that never overlap in time (steps are serial, and the runner writes each step marker *before* `StepLogs.Open`): (a) the runner writes markers directly to the path, serially between steps; (b) `StepLogs` writes chunks during a step. **Resolved approach:** `Open(commandID, path, secrets)` opens the path fresh with `O_APPEND|O_CREATE|O_WRONLY` for that step and `Close(commandID)` closes it on eof. One handle live at a time per server-run (serial steps guarantee this), so there is no shared-handle race and no ref-counting. The runner's marker write is a separate short `O_APPEND` open/write/close on the same path.
|
||||
|
||||
### 5.4 Data model (`server/internal/models/workflow.go`)
|
||||
|
||||
`StepRun`:
|
||||
- **Remove** `Stdout`, `Stderr` string fields.
|
||||
- **Add** `LogOffset int64 `bson:"log_offset" json:"log_offset"`` — byte offset in the server-run file where this step's marker begins.
|
||||
|
||||
`ServerRun` gains nothing structural (its file path is derivable: `<logdir>/<run_id>/<server_id>.log`).
|
||||
|
||||
---
|
||||
|
||||
## 6. REST API (`server/internal/api/workflows.go`)
|
||||
|
||||
- `GET /api/runs/:runId/servers/:serverId/logs` — returns the whole server-run log file (`text/plain`). 404 if absent. Used post-run and as SSE fallback.
|
||||
- `GET /api/runs/:runId/servers/:serverId/logs/stream` — **SSE**. Opens the file, streams existing content as `data:` events, then polls for appends (~500ms) emitting new bytes, until the server-run status is terminal (success/failed/skipped/cancelled) AND no more bytes, then sends a final `event: done` and closes. Sets `Content-Type: text/event-stream`, disables gin's buffering. Guards against path traversal (runId/serverId are used as literal path segments — validate they are UUIDs / contain no separators).
|
||||
|
||||
Log content served by these endpoints is already masked (masking happens at write time), so no masking needed on read.
|
||||
|
||||
---
|
||||
|
||||
## 7. Retention
|
||||
|
||||
### 7.1 Setting
|
||||
|
||||
`settings` collection gains `workflow_log_retention_days int` (default 30 when unset). Read/write via the existing settings service + surfaced in `/settings` UI as a number input. `0` or negative disables sweeping (keep forever) — document this.
|
||||
|
||||
### 7.2 Sweeper (`server/internal/services/steplogs.go` or `logsweeper.go`)
|
||||
|
||||
- `StartLogSweeper()` launched at server startup (next to index setup): hourly `time.Ticker`.
|
||||
- Each tick: read retention setting; if ≤0 skip. Compute cutoff = `now - retentionDays`. For each `<logdir>/<run_id>/` dir, look up the run's `finished_at` (query `workflow_runs` by run_id); if finished and older than cutoff, `os.RemoveAll` the dir. Fallback to dir mtime if the run doc is gone.
|
||||
- Also run once at startup.
|
||||
|
||||
---
|
||||
|
||||
## 8. Frontend
|
||||
|
||||
### 8.1 API client (`web/lib/api.ts`)
|
||||
|
||||
- `StepRun`: remove `stdout`/`stderr`; add `log_offset: number`.
|
||||
- Add `getServerRunLog(runId, serverId): Promise<string>` (GET .../logs).
|
||||
- SSE consumed directly via `EventSource` in the component (not through the `request` helper), URL built from the same base.
|
||||
- Settings type gains `workflow_log_retention_days`.
|
||||
|
||||
### 8.2 Run detail (`web/app/workflows/[id]/runs/[runId]/page.tsx`)
|
||||
|
||||
- Per-server card: while the server-run is `running`, open an `EventSource` to the stream endpoint and render a live `<pre>` terminal that appends incoming chunks (auto-scroll). Close the source on `event: done`, unmount, or terminal status.
|
||||
- After completion: fetch the whole file once and render it; step `<details>` still list status/exit/attempts pills. (Per-step slicing by `log_offset` is optional polish — v1 may show the whole server log under the card and keep step pills as the status summary.)
|
||||
- Remove reliance on `st.stdout`/`st.stderr` (fields gone).
|
||||
|
||||
### 8.3 Settings (`web/app/settings/page.tsx`)
|
||||
|
||||
- Add a "Workflow log retention (days)" number input bound to `workflow_log_retention_days`, saved via the existing settings mutation. Note that `0` = keep forever.
|
||||
|
||||
---
|
||||
|
||||
## 9. Security
|
||||
|
||||
- Secret masking moves to the streaming write path but remains server-side and boundary-safe (carry buffer). Same `***` replacement.
|
||||
- Log files `0600`, dirs `0700`, under a dedicated log dir.
|
||||
- SSE/read endpoints validate `runId`/`serverId` as UUID-shaped path segments to prevent traversal; they are session-authed (same `apiGroup`).
|
||||
- Terminal `StepResult.Stderr` (error notes only) is still masked before any persistence (it is no longer persisted as log body; if surfaced, mask against secretVals).
|
||||
|
||||
---
|
||||
|
||||
## 10. Out of scope
|
||||
|
||||
- Per-step (rather than per-server) live SSE channels.
|
||||
- Log compression / rotation within a run, remote log storage (S3), download-as-zip.
|
||||
- Full-text search over logs.
|
||||
- Backfilling/migrating already-existing `workflow_runs` stdout/stderr into files (pre-existing runs keep whatever they had; new field just won't be set — acceptable, feature is new).
|
||||
- Tests (skipped, consistent with the Workflows iteration).
|
||||
```
|
||||
Reference in New Issue
Block a user