This commit is contained in:
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,747 +0,0 @@
|
||||
# Workflow Builder v2 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Rebuild the workflow builder to match the approved mockup with drag-and-drop, base-step editing/deletion (cascade), step input parameters, an edit-workflow modal, runs navigation, and fix the save crash.
|
||||
|
||||
**Architecture:** Backend gains input-parameter fields on step/ref/resolved models, cascade delete, runner env injection, and a workflow-update handler that returns the updated workflow. Frontend adds `bash`/`pwsh`/`signal` tokens and a `Modal` primitive, then rebuilds the builder page (dotted canvas, 340px node cards, wire env chips, kicker/field inspector, HTML5 drag-and-drop), plus Edit-base-step and Edit-workflow modals and a runs list page.
|
||||
|
||||
**Tech Stack:** Go (gin, mongo-driver v2), Next.js 16 app-router + react-query + Tailwind, HTML5 Drag-and-Drop, MongoDB.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **No tests this iteration** — no `*_test.go` or frontend tests. Verify with `go build ./...`, `go vet ./...`, `npm run build`.
|
||||
- Mongo access pattern: `db.Col("collection_name")` + `context.WithTimeout`. Follow `server/internal/services/workflows.go`.
|
||||
- Audit every mutation with `services.LogEvent(action, actor, serverID, targetID, message)`.
|
||||
- Interpreter literals are `"bash"` and `"powershell"`.
|
||||
- Go module path: `github.com/mrhid6/vantage`.
|
||||
- Visual target: approved mockup. Builder palette adds amber signal `#f5a524` (`signal`), `#241800` (`signal-ink`), bash `#3fb950`, pwsh `#5b9bff`; keep existing `surface`/`surface-2`/`border`/`text-primary`/`text-secondary`/`danger`/`accent` tokens for panels. Node cards 340px; dotted-grid canvas; dashed-amber `passes` chips on wires; kicker/field inspector.
|
||||
- Frontend uses `@/lib/api` typed client, `@/components/ui`, react-query. Input styling follows the existing `inputClass` pattern in `web/app/secrets/page.tsx`.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Models + step CRUD + cascade delete
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/internal/models/workflow.go`
|
||||
- Modify: `server/internal/services/workflows.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `models.InputParam{Name,Default,Description}`; `WorkflowStep.DeclaredInputs`, `WorkflowStepRef.Inputs`, `ResolvedStep.Inputs`. `DeleteStep(stepID)` now cascades to workflows.
|
||||
|
||||
- [ ] **Step 1: Add the model fields**
|
||||
|
||||
In `server/internal/models/workflow.go` add:
|
||||
|
||||
```go
|
||||
type InputParam struct {
|
||||
Name string `bson:"name" json:"name"`
|
||||
Default string `bson:"default" json:"default"`
|
||||
Description string `bson:"description" json:"description"`
|
||||
}
|
||||
```
|
||||
|
||||
- In `WorkflowStep`, add after `DeclaredOutputs`: `DeclaredInputs []InputParam `bson:"declared_inputs" json:"declared_inputs"``
|
||||
- In `WorkflowStepRef`, add: `Inputs map[string]string `bson:"inputs,omitempty" json:"inputs,omitempty"``
|
||||
- In `ResolvedStep`, add: `Inputs map[string]string `bson:"inputs" json:"inputs"``
|
||||
|
||||
- [ ] **Step 2: Persist declared_inputs in CreateStep/UpdateStep**
|
||||
|
||||
In `server/internal/services/workflows.go`:
|
||||
- In `CreateStep`, after the `SecretRefs` nil-guard add:
|
||||
|
||||
```go
|
||||
if s.DeclaredInputs == nil {
|
||||
s.DeclaredInputs = []models.InputParam{}
|
||||
}
|
||||
```
|
||||
|
||||
- In `UpdateStep`'s `$set`, add: `"declared_inputs": s.DeclaredInputs,`
|
||||
|
||||
- [ ] **Step 3: Cascade DeleteStep to workflows**
|
||||
|
||||
Replace the body of `DeleteStep` with a version that also strips the step from every workflow:
|
||||
|
||||
```go
|
||||
func DeleteStep(stepID string) error {
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
if _, err := db.Col("workflow_steps").DeleteOne(ctx, bson.M{"step_id": stepID}); err != nil {
|
||||
return err
|
||||
}
|
||||
// Cascade: remove this step from every workflow that references it, re-sequencing orders.
|
||||
cur, err := db.Col("workflows").Find(ctx, bson.M{"steps.step_id": stepID})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
var wfs []models.Workflow
|
||||
if err := cur.All(ctx, &wfs); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, w := range wfs {
|
||||
kept := make([]models.WorkflowStepRef, 0, len(w.Steps))
|
||||
for _, ref := range w.Steps {
|
||||
if ref.StepID == stepID {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, ref)
|
||||
}
|
||||
for i := range kept {
|
||||
kept[i].Order = i
|
||||
}
|
||||
if _, err := db.Col("workflows").UpdateOne(ctx,
|
||||
bson.M{"workflow_id": w.WorkflowID},
|
||||
bson.M{"$set": bson.M{"steps": kept, "updated_at": time.Now()}},
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
Confirm `models` and `time` are imported in this file (they are).
|
||||
|
||||
- [ ] **Step 4: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./...`
|
||||
Expected: success.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add server/internal/models/workflow.go server/internal/services/workflows.go
|
||||
git commit -m "feat(workflows): step input params model + cascade step delete"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Runner input injection + update handler returns workflow
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/internal/services/workflow_runner.go`
|
||||
- Modify: `server/internal/api/workflows.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `models.ResolvedStep.Inputs`, `WorkflowStep.DeclaredInputs`, `WorkflowStepRef.Inputs` (Task 1).
|
||||
- Produces: runner injects input env; `PUT /api/workflows/:id` returns the `Workflow`.
|
||||
|
||||
- [ ] **Step 1: Resolve inputs in `resolveSteps`**
|
||||
|
||||
In `server/internal/services/workflow_runner.go`, in `resolveSteps`, after loading `lib` and before/where the `ResolvedStep` is built, compute the input env and set it. Add inside the loop (after `lib, err := getStep(...)` succeeds):
|
||||
|
||||
```go
|
||||
inputs := map[string]string{}
|
||||
for _, p := range lib.DeclaredInputs {
|
||||
if ref.Inputs != nil {
|
||||
if v, ok := ref.Inputs[p.Name]; ok {
|
||||
inputs[p.Name] = v
|
||||
continue
|
||||
}
|
||||
}
|
||||
inputs[p.Name] = p.Default
|
||||
}
|
||||
```
|
||||
|
||||
and set `Inputs: inputs,` on the `models.ResolvedStep{...}` literal.
|
||||
|
||||
- [ ] **Step 2: Inject inputs into cmdEnv in `runServer`**
|
||||
|
||||
In `runServer`, where `cmdEnv` is built (currently: copy `runEnv`, then overlay `secretVals`), change the layering so inputs are the base layer:
|
||||
|
||||
```go
|
||||
cmdEnv := map[string]string{}
|
||||
for k, v := range step.Inputs {
|
||||
cmdEnv[k] = v
|
||||
}
|
||||
for k, v := range runEnv {
|
||||
cmdEnv[k] = v
|
||||
}
|
||||
for k, v := range secretVals {
|
||||
cmdEnv[k] = v
|
||||
}
|
||||
```
|
||||
|
||||
(Inputs first = lowest precedence; upstream outputs override; secrets win. Keep the existing `secretVals`/`allSecrets` masking logic unchanged.)
|
||||
|
||||
- [ ] **Step 3: `updateWorkflow` handler returns the workflow**
|
||||
|
||||
In `server/internal/api/workflows.go`, change `updateWorkflow` so that after a successful `services.UpdateWorkflow`, it re-fetches and returns the workflow:
|
||||
|
||||
```go
|
||||
func updateWorkflow(c *gin.Context) {
|
||||
var w models.Workflow
|
||||
if err := c.ShouldBindJSON(&w); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if err := services.UpdateWorkflow(c.Param("id"), w); err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
services.LogEvent("workflow.updated", actorFromCtx(c), "", c.Param("id"), "workflow updated")
|
||||
updated, err := services.GetWorkflow(c.Param("id"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, updated)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./...`
|
||||
Expected: success.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add server/internal/services/workflow_runner.go server/internal/api/workflows.go
|
||||
git commit -m "feat(workflows): inject step inputs into env; update returns workflow"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Frontend tokens, Modal primitive, API types
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/tailwind.config.ts`
|
||||
- Create: `web/components/ui/Modal.tsx`
|
||||
- Modify: `web/components/ui/index.ts`
|
||||
- Modify: `web/lib/api.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: tokens `bash`/`pwsh`/`signal`/`signal-ink`; `<Modal open title onClose>children</Modal>`; TS `InputParam`, `WorkflowStep.declared_inputs`, `WorkflowStepRef.inputs`.
|
||||
|
||||
- [ ] **Step 1: Add tokens**
|
||||
|
||||
In `web/tailwind.config.ts`, add to `theme.extend.colors`:
|
||||
|
||||
```ts
|
||||
bash: "#3fb950",
|
||||
pwsh: "#5b9bff",
|
||||
signal: "#f5a524",
|
||||
"signal-ink": "#241800",
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Create the Modal primitive**
|
||||
|
||||
`web/components/ui/Modal.tsx`:
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
|
||||
import { useEffect } from "react";
|
||||
|
||||
export function Modal({
|
||||
open,
|
||||
title,
|
||||
onClose,
|
||||
children,
|
||||
wide,
|
||||
}: {
|
||||
open: boolean;
|
||||
title: string;
|
||||
onClose: () => void;
|
||||
children: React.ReactNode;
|
||||
wide?: boolean;
|
||||
}) {
|
||||
useEffect(() => {
|
||||
if (!open) return;
|
||||
const onKey = (e: KeyboardEvent) => e.key === "Escape" && onClose();
|
||||
window.addEventListener("keydown", onKey);
|
||||
return () => window.removeEventListener("keydown", onKey);
|
||||
}, [open, onClose]);
|
||||
|
||||
if (!open) return null;
|
||||
|
||||
return (
|
||||
<div className="fixed inset-0 z-50 flex items-center justify-center p-4">
|
||||
<div className="absolute inset-0 bg-black/60" onClick={onClose} />
|
||||
<div
|
||||
className={`relative z-10 w-full ${wide ? "max-w-2xl" : "max-w-md"} max-h-[90vh] overflow-auto rounded-xl border border-border bg-surface shadow-2xl`}
|
||||
role="dialog"
|
||||
aria-modal="true"
|
||||
>
|
||||
<div className="flex items-center justify-between border-b border-border px-5 py-3">
|
||||
<h2 className="text-sm font-bold text-text-primary">{title}</h2>
|
||||
<button onClick={onClose} className="text-text-secondary hover:text-text-primary" aria-label="Close">
|
||||
✕
|
||||
</button>
|
||||
</div>
|
||||
<div className="p-5">{children}</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Export it**
|
||||
|
||||
In `web/components/ui/index.ts` add: `export { Modal } from "./Modal";`
|
||||
|
||||
- [ ] **Step 4: API types**
|
||||
|
||||
In `web/lib/api.ts`:
|
||||
- Add interface:
|
||||
|
||||
```ts
|
||||
export interface InputParam {
|
||||
name: string;
|
||||
default: string;
|
||||
description: string;
|
||||
}
|
||||
```
|
||||
|
||||
- In `WorkflowStep`, add `declared_inputs: InputParam[];`
|
||||
- In `WorkflowStepRef`, add `inputs?: Record<string, string>;`
|
||||
|
||||
(`updateWorkflow` already typed to return `Workflow`; the backend now honors it.)
|
||||
|
||||
- [ ] **Step 5: Verify build**
|
||||
|
||||
Run: `cd web && npm run build`
|
||||
Expected: success.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add web/tailwind.config.ts web/components/ui/Modal.tsx web/components/ui/index.ts web/lib/api.ts
|
||||
git commit -m "feat(web): builder tokens, Modal primitive, input-param types"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Edit-base-step modal
|
||||
|
||||
**Files:**
|
||||
- Create: `web/components/workflows/EditStepModal.tsx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `api.createStep`, `api.updateStep`, `api.deleteStep`, `Modal`, types (Task 3).
|
||||
- Produces: `<EditStepModal open step onClose />` where `step` is a `WorkflowStep` (edit) or `null` (new).
|
||||
|
||||
- [ ] **Step 1: Write the modal**
|
||||
|
||||
Client component. Local form state seeded from `step` (or blank for new). Fields:
|
||||
- **Name** (text), **Interpreter** (select bash/powershell), **Script** (`<textarea>` mono).
|
||||
- **Outputs** (`declared_outputs`): a list of text chips with add/remove — an input + "Add" appends a name; each name shows an `✕` to remove.
|
||||
- **Inputs** (`declared_inputs`): rows, each with `name` / `default` / `description` inputs and a remove button; an "Add input" button appends a blank `{name:"",default:"",description:""}`.
|
||||
- Hint: "Reusable steps are shared across all workflows. Editing here changes it everywhere."
|
||||
|
||||
Actions:
|
||||
- **Save**: build the `WorkflowStep` payload (`declared_outputs`, `declared_inputs`, `secret_refs: step?.secret_refs ?? []`, `description: ""` if absent). If editing (`step` truthy) call `api.updateStep(step.step_id, payload)`, else `api.createStep(payload)`. On success `queryClient.invalidateQueries({queryKey:["steps"]})` and `onClose()`.
|
||||
- **Delete** (edit mode only): a confirm (`window.confirm`) then `api.deleteStep(step.step_id)`, invalidate `["steps"]` AND `["workflow"]` (broad — deleted step is pulled from workflows server-side), `onClose()`.
|
||||
|
||||
Use the `inputClass` styling pattern and `Button` variants (`primary` save, `danger` delete, `ghost` cancel). Follow existing token names. Filter empty input rows (blank `name`) out of the payload on save.
|
||||
|
||||
Skeleton:
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { useQueryClient } from "@tanstack/react-query";
|
||||
import { api, WorkflowStep, InputParam } from "@/lib/api";
|
||||
import { Button, Modal } from "@/components/ui";
|
||||
|
||||
const inputClass =
|
||||
"w-full rounded-lg border border-border bg-surface-2 px-3 py-2 text-sm text-text-primary placeholder-text-secondary/50 focus:border-signal focus:outline-none focus:ring-1 focus:ring-signal";
|
||||
|
||||
export function EditStepModal({ open, step, onClose }: { open: boolean; step: WorkflowStep | null; onClose: () => void }) {
|
||||
const qc = useQueryClient();
|
||||
const [name, setName] = useState(step?.name ?? "");
|
||||
const [interpreter, setInterpreter] = useState<"bash" | "powershell">(step?.interpreter ?? "bash");
|
||||
const [script, setScript] = useState(step?.script ?? "");
|
||||
const [outputs, setOutputs] = useState<string[]>(step?.declared_outputs ?? []);
|
||||
const [inputs, setInputs] = useState<InputParam[]>(step?.declared_inputs ?? []);
|
||||
const [newOut, setNewOut] = useState("");
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
// NOTE: because state is seeded from props, render the modal conditionally
|
||||
// (parent mounts it only when opening) OR key it by step_id so it re-seeds.
|
||||
|
||||
const save = async () => {
|
||||
setBusy(true); setError(null);
|
||||
try {
|
||||
const payload = {
|
||||
name: name.trim(), description: step?.description ?? "", interpreter, script,
|
||||
declared_outputs: outputs, declared_inputs: inputs.filter((i) => i.name.trim() !== ""),
|
||||
secret_refs: step?.secret_refs ?? [],
|
||||
};
|
||||
if (step) await api.updateStep(step.step_id, payload);
|
||||
else await api.createStep(payload);
|
||||
qc.invalidateQueries({ queryKey: ["steps"] });
|
||||
onClose();
|
||||
} catch (e) { setError((e as Error).message); } finally { setBusy(false); }
|
||||
};
|
||||
|
||||
const del = async () => {
|
||||
if (!step || !window.confirm("Delete this step? It will be removed from every workflow that uses it.")) return;
|
||||
setBusy(true); setError(null);
|
||||
try {
|
||||
await api.deleteStep(step.step_id);
|
||||
qc.invalidateQueries({ queryKey: ["steps"] });
|
||||
qc.invalidateQueries({ queryKey: ["workflow"] });
|
||||
onClose();
|
||||
} catch (e) { setError((e as Error).message); } finally { setBusy(false); }
|
||||
};
|
||||
|
||||
return (
|
||||
<Modal open={open} onClose={onClose} title={step ? "Edit base step" : "New step"} wide>
|
||||
<div className="space-y-4">
|
||||
{error && <div className="rounded border border-danger/30 bg-danger/10 px-3 py-2 text-sm text-danger">{error}</div>}
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Name</label>
|
||||
<input className={inputClass} value={name} onChange={(e) => setName(e.target.value)} />
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Interpreter</label>
|
||||
<select className={inputClass} value={interpreter} onChange={(e) => setInterpreter(e.target.value as "bash" | "powershell")}>
|
||||
<option value="bash">bash</option>
|
||||
<option value="powershell">powershell</option>
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Script</label>
|
||||
<textarea className={`${inputClass} h-40 font-mono text-xs`} value={script} onChange={(e) => setScript(e.target.value)} />
|
||||
<p className="mt-1 text-xs text-text-secondary">Write <code className="text-signal">KEY=value</code> to <code className="text-signal">$WORKFLOW_ENV</code> to expose it to later steps.</p>
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Outputs</label>
|
||||
<div className="mb-2 flex flex-wrap gap-1">
|
||||
{outputs.map((o) => (
|
||||
<span key={o} className="flex items-center gap-1 rounded bg-signal px-2 py-0.5 font-mono text-[11px] text-signal-ink">
|
||||
{o}<button onClick={() => setOutputs(outputs.filter((x) => x !== o))}>✕</button>
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<input className={inputClass} placeholder="OUTPUT_NAME" value={newOut} onChange={(e) => setNewOut(e.target.value)} />
|
||||
<Button variant="ghost" size="sm" onClick={() => { if (newOut.trim()) { setOutputs([...outputs, newOut.trim()]); setNewOut(""); } }}>Add</Button>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Inputs</label>
|
||||
<div className="space-y-2">
|
||||
{inputs.map((inp, i) => (
|
||||
<div key={i} className="flex gap-2">
|
||||
<input className={inputClass} placeholder="name" value={inp.name} onChange={(e) => setInputs(inputs.map((x, j) => j === i ? { ...x, name: e.target.value } : x))} />
|
||||
<input className={inputClass} placeholder="default" value={inp.default} onChange={(e) => setInputs(inputs.map((x, j) => j === i ? { ...x, default: e.target.value } : x))} />
|
||||
<input className={inputClass} placeholder="description" value={inp.description} onChange={(e) => setInputs(inputs.map((x, j) => j === i ? { ...x, description: e.target.value } : x))} />
|
||||
<Button variant="ghost" size="sm" onClick={() => setInputs(inputs.filter((_, j) => j !== i))}>✕</Button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<Button variant="ghost" size="sm" className="mt-2" onClick={() => setInputs([...inputs, { name: "", default: "", description: "" }])}>Add input</Button>
|
||||
</div>
|
||||
<div className="flex items-center justify-between pt-2">
|
||||
{step ? <Button variant="danger" onClick={del} loading={busy}>Delete step</Button> : <span />}
|
||||
<div className="flex gap-2">
|
||||
<Button variant="ghost" onClick={onClose}>Cancel</Button>
|
||||
<Button variant="primary" onClick={save} loading={busy} disabled={!name.trim()}>Save</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
**Important:** because the form state seeds from `step` props, the parent must either mount `EditStepModal` only while open, or pass a React `key={step?.step_id ?? "new"}` so the state re-initializes each time a different step is edited. Note this in the component with a comment; the builder (Task 6) will mount it with a `key`.
|
||||
|
||||
- [ ] **Step 2: Verify build**
|
||||
|
||||
Run: `cd web && npm run build`
|
||||
Expected: success (the component may be unused until Task 6 — that's fine, but an unused import-free component builds).
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add web/components/workflows/EditStepModal.tsx
|
||||
git commit -m "feat(web): edit-base-step modal with inputs/outputs editor"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Edit-workflow modal
|
||||
|
||||
**Files:**
|
||||
- Create: `web/components/workflows/EditWorkflowModal.tsx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `api.updateWorkflow`, `api.deleteWorkflow`, `api.listServers`, `Modal`, `Workflow` (Task 3).
|
||||
- Produces: `<EditWorkflowModal open workflow onSaved onClose />` — saves name/targets immediately, returns the updated workflow via `onSaved`.
|
||||
|
||||
- [ ] **Step 1: Write the modal**
|
||||
|
||||
Client component. Seeds name + selected target servers from `workflow`. Uses `api.listServers` (react-query) for the multiselect (checkbox chips keyed by `server_id`, labelled `hostname`). Actions:
|
||||
- **Save**: `const updated = await api.updateWorkflow(workflow.workflow_id, { ...workflow, name, target_server_ids })`; call `onSaved(updated)`; `onClose()`. (Backend now returns the workflow.)
|
||||
- **Delete workflow**: confirm, `api.deleteWorkflow(workflow.workflow_id)`, then `router.push("/workflows")`.
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { useRouter } from "next/navigation";
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { api, Workflow } from "@/lib/api";
|
||||
import { Button, Modal } from "@/components/ui";
|
||||
|
||||
const inputClass =
|
||||
"w-full rounded-lg border border-border bg-surface-2 px-3 py-2 text-sm text-text-primary focus:border-signal focus:outline-none focus:ring-1 focus:ring-signal";
|
||||
|
||||
export function EditWorkflowModal({ open, workflow, onSaved, onClose }: { open: boolean; workflow: Workflow; onSaved: (w: Workflow) => void; onClose: () => void }) {
|
||||
const router = useRouter();
|
||||
const [name, setName] = useState(workflow.name);
|
||||
const [targets, setTargets] = useState<string[]>(workflow.target_server_ids);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const { data: servers } = useQuery({ queryKey: ["servers"], queryFn: api.listServers });
|
||||
|
||||
const toggle = (id: string) => setTargets((t) => (t.includes(id) ? t.filter((x) => x !== id) : [...t, id]));
|
||||
|
||||
const save = async () => {
|
||||
setBusy(true); setError(null);
|
||||
try {
|
||||
const updated = await api.updateWorkflow(workflow.workflow_id, { ...workflow, name, target_server_ids: targets });
|
||||
onSaved(updated); onClose();
|
||||
} catch (e) { setError((e as Error).message); } finally { setBusy(false); }
|
||||
};
|
||||
|
||||
const del = async () => {
|
||||
if (!window.confirm("Delete this workflow? This cannot be undone.")) return;
|
||||
setBusy(true); setError(null);
|
||||
try { await api.deleteWorkflow(workflow.workflow_id); router.push("/workflows"); }
|
||||
catch (e) { setError((e as Error).message); setBusy(false); }
|
||||
};
|
||||
|
||||
return (
|
||||
<Modal open={open} onClose={onClose} title="Edit workflow">
|
||||
<div className="space-y-4">
|
||||
{error && <div className="rounded border border-danger/30 bg-danger/10 px-3 py-2 text-sm text-danger">{error}</div>}
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Name</label>
|
||||
<input className={inputClass} value={name} onChange={(e) => setName(e.target.value)} />
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Target servers</label>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{servers?.map((s) => {
|
||||
const on = targets.includes(s.server_id);
|
||||
return (
|
||||
<label key={s.server_id} className={`flex cursor-pointer items-center gap-2 rounded-lg border px-2 py-1 text-sm ${on ? "border-signal bg-signal/10 text-text-primary" : "border-border text-text-secondary"}`}>
|
||||
<input type="checkbox" className="accent-signal" checked={on} onChange={() => toggle(s.server_id)} />
|
||||
{s.hostname}
|
||||
</label>
|
||||
);
|
||||
})}
|
||||
{servers && servers.length === 0 && <p className="text-xs text-text-secondary">No servers registered.</p>}
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex items-center justify-between pt-2">
|
||||
<Button variant="danger" onClick={del} loading={busy}>Delete workflow</Button>
|
||||
<div className="flex gap-2">
|
||||
<Button variant="ghost" onClick={onClose}>Cancel</Button>
|
||||
<Button variant="primary" onClick={save} loading={busy} disabled={!name.trim()}>Save</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify build**
|
||||
|
||||
Run: `cd web && npm run build`
|
||||
Expected: success.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add web/components/workflows/EditWorkflowModal.tsx
|
||||
git commit -m "feat(web): edit-workflow modal (name/targets/delete)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Builder rewrite — mockup styling + drag-and-drop + inspector
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/app/workflows/[id]/page.tsx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `api.getWorkflow/updateWorkflow/listSteps/runWorkflow`, `EditStepModal` (Task 4), `EditWorkflowModal` (Task 5), tokens (Task 3).
|
||||
|
||||
Rebuild the page to the approved mockup. Reference values from the mockup:
|
||||
- Canvas dotted grid: `background: radial-gradient(circle at 1px 1px, <border> 1px, transparent 0) 0 0 / 22px 22px` over `bg-background`.
|
||||
- Node card: width `340px`, `rounded-[10px] border border-border bg-surface`, selected → `border-signal` + `ring-2 ring-signal/40`. Top row: index badge (mono, boxed), title, shell badge, then body with a mono script preview (`bg-surface-2 border border-border rounded p-2 text-xs`).
|
||||
- Shell badges: bash → `text-bash bg-bash/15`, pwsh (powershell) → `text-pwsh bg-pwsh/15`, mono uppercase `text-[10px] px-1.5 py-0.5 rounded`.
|
||||
- Wire: a 2px vertical `bg-border` segment ~26px tall between nodes.
|
||||
- `passes` chip row: dashed amber pill `border border-dashed border-signal/55 bg-surface rounded-full px-3 py-1`, label "passes" (`text-[10px] uppercase text-text-secondary`), each output name a chip `bg-signal text-signal-ink font-mono text-[11px] rounded px-2 py-0.5`.
|
||||
- Inspector fields: kicker (`text-[11px] uppercase tracking-wide text-text-secondary font-bold`), title with shell badge; each field block separated by `border-b border-border` with an uppercase label.
|
||||
|
||||
- [ ] **Step 1: Rewrite the builder page**
|
||||
|
||||
Requirements (keep everything in React state; **Save** persists via `api.updateWorkflow` and sets `wf` to the returned workflow):
|
||||
|
||||
1. **State/data:** load workflow (`["workflow", id]`), library (`["steps"]`), servers (`["servers"]`). Seed local `wf` from the query once. Keep `selected` (index in sorted order). `sortedSteps = [...wf.steps].sort(order)`.
|
||||
2. **Topbar:** brand dot + `Workflows /` crumb + `wf.name` + `· draft`. Right: a **Targets** chip showing `${wf.target_server_ids.length} servers`; a **Runs** link (`<Link href={`/workflows/${id}/runs`}>`); an **Edit** button (opens `EditWorkflowModal`); **Save** (`variant="secondary"`, persists, `setWf(returned)`); **Run workflow** (amber: `className` using `bg-signal text-signal-ink`, or `variant="primary"` acceptable) → `api.runWorkflow` then route to the run detail.
|
||||
3. **Library (left):** header "Step Library" + `+` button opening `EditStepModal` in new mode (`step={null}`). A search input filters `library` by name (case-insensitive). Group by interpreter with labels `Shared · Bash` / `Shared · PowerShell`. Each card: shell badge, name, `description`, a grip glyph (`⠿`), `cursor-grab`, and `draggable`. On `dragstart` set `dataTransfer` to `JSON.stringify({kind:"lib", stepId})`. Clicking the card still appends the step. A small edit affordance (e.g. a pencil `✎` button on hover) opens `EditStepModal` with that library step.
|
||||
4. **Canvas (center):** dotted grid. For each `sortedSteps[i]`: render a drop target above it (a thin zone; on `drop` insert at `i`), then the node card. Node card is `draggable` (`dragstart` sets `{kind:"move", from:i}`). Between consecutive nodes render the wire + `passes` chips = union of `sortedSteps.slice(0,i)` `declared_outputs` (deduped). After the last node render an end drop zone styled `+ Drop a step here` (dashed). Handle `drop`:
|
||||
- parse `dataTransfer`; if `kind==="lib"` insert a new `WorkflowStepRef{step_id, order:<pos>, on_failure:"stop", max_retries:0}` at the drop position; if `kind==="move"` move `from`→`pos`; then re-sequence all `order` to array index. Update `wf`.
|
||||
- `dragover` must `e.preventDefault()` on drop zones to allow dropping.
|
||||
- Selecting a node (click) sets `selected`.
|
||||
5. **Inspector (right):** for the selected placement (`selectedRef` = `sortedSteps[selected]`, `selectedLib` = library by `step_id`):
|
||||
- Kicker `Step ${selected+1} · Inspector`, title = shell badge + `selectedLib.name`.
|
||||
- **Command** `<textarea>` bound to `selectedRef.overrides?.script ?? selectedLib.script`; edits write `overrides.script` on the ref (per-placement override). Hint about `$WORKFLOW_ENV`.
|
||||
- **Inputs** (this step's `declared_inputs`): one row per param — label = `param.name` (+ description as sub-text), an input bound to `selectedRef.inputs?.[name] ?? ""` with `placeholder={param.default}`; edits write `ref.inputs[name]`.
|
||||
- **Inputs · from upstream** (read-only): the union of prior steps' `declared_outputs` as `IN` rows.
|
||||
- **Outputs · to $WORKFLOW_ENV** (read-only): this step's `declared_outputs` as `OUT` rows.
|
||||
- **Secret refs**: keep the existing group/KEY checklist behavior (port it over) writing `overrides.secret_refs`.
|
||||
- **On failure** select + **Max retries** (when retry).
|
||||
- **Remove from workflow** button (danger) → remove the ref, re-sequence orders, clear selection.
|
||||
6. **Modals:** render `<EditWorkflowModal>` (open state) and `<EditStepModal key={editingStep?.step_id ?? "new"} open step={editingStep} onClose>`; the library `+` and per-card edit set `editingStep`.
|
||||
7. **Bug fix:** `save()` sets `wf` to the awaited `updateWorkflow` result (now a full workflow); no direct crash. Additionally guard: if the returned object lacks `steps`, keep the prior `wf` and surface an error.
|
||||
|
||||
Provide a complete, working implementation (this is a full rewrite of the file). Preserve the secret-group lazy-fetch (`api.getSecretGroup`) logic from the current file for the secret-refs checklist. Use `inputClass` styling. Do not leave TODOs.
|
||||
|
||||
- [ ] **Step 2: Verify build**
|
||||
|
||||
Run: `cd web && npm run build`
|
||||
Expected: type-checks and builds. Manually confirm no `st.stdout`-style dead references and no unused imports.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add web/app/workflows/[id]/page.tsx
|
||||
git commit -m "feat(web): rebuild workflow builder — mockup styling, drag-and-drop, inputs inspector"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Runs list page + navigation links
|
||||
|
||||
**Files:**
|
||||
- Create: `web/app/workflows/[id]/runs/page.tsx`
|
||||
- Modify: `web/app/workflows/page.tsx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `api.listRuns`, `api.getWorkflow`.
|
||||
|
||||
- [ ] **Step 1: Runs list page**
|
||||
|
||||
`web/app/workflows/[id]/runs/page.tsx` — client component. Load `api.getWorkflow(id)` (for the name) and `api.listRuns(id)`. Render a header "Runs · <name>" with a "← Back to builder" `Link` to `/workflows/${id}`, and a table (`@/components/ui` Table) of runs: short run id, a status `Badge` (map success→success, failed→danger, running→accent/warning, cancelled→neutral), `started_at` (localized), `triggered_by`, and server count (`server_runs.length`). Each row links to `/workflows/${id}/runs/${run.run_id}`. Empty state "No runs yet."
|
||||
|
||||
Use the existing Badge variants (success/warning/danger/neutral/accent) — verify names in `web/components/ui/Badge.tsx`.
|
||||
|
||||
Skeleton:
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
|
||||
import Link from "next/link";
|
||||
import { useParams } from "next/navigation";
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { api, WorkflowRun } from "@/lib/api";
|
||||
import { Card, Table, Thead, Tbody, Tr, Th, Td, Badge } from "@/components/ui";
|
||||
|
||||
const statusVariant: Record<string, string> = {
|
||||
success: "success", failed: "danger", running: "warning", cancelled: "neutral", queued: "neutral",
|
||||
};
|
||||
|
||||
export default function WorkflowRunsPage() {
|
||||
const { id } = useParams<{ id: string }>();
|
||||
const { data: wf } = useQuery({ queryKey: ["workflow", id], queryFn: () => api.getWorkflow(id) });
|
||||
const { data: runs } = useQuery({ queryKey: ["runs", id], queryFn: () => api.listRuns(id) });
|
||||
|
||||
return (
|
||||
<div className="p-8">
|
||||
<div className="mb-6">
|
||||
<Link href={`/workflows/${id}`} className="text-sm text-text-secondary hover:text-text-primary">← Back to builder</Link>
|
||||
<h1 className="mt-2 text-2xl font-bold text-text-primary">Runs · {wf?.name ?? ""}</h1>
|
||||
</div>
|
||||
<Card padding={false}>
|
||||
{runs && runs.length > 0 ? (
|
||||
<Table>
|
||||
<Thead><Tr><Th>Run</Th><Th>Status</Th><Th>Started</Th><Th>By</Th><Th>Servers</Th></Tr></Thead>
|
||||
<Tbody>
|
||||
{runs.map((r: WorkflowRun) => (
|
||||
<Tr key={r.run_id}>
|
||||
<Td><Link href={`/workflows/${id}/runs/${r.run_id}`} className="font-mono text-text-primary hover:text-signal">{r.run_id.slice(0, 8)}</Link></Td>
|
||||
<Td><Badge variant={(statusVariant[r.status] ?? "neutral") as never}>{r.status}</Badge></Td>
|
||||
<Td className="text-text-secondary">{new Date(r.started_at).toLocaleString()}</Td>
|
||||
<Td className="text-text-secondary">{r.triggered_by}</Td>
|
||||
<Td className="text-text-secondary">{r.server_runs.length}</Td>
|
||||
</Tr>
|
||||
))}
|
||||
</Tbody>
|
||||
</Table>
|
||||
) : (
|
||||
<div className="py-16 text-center text-text-secondary">No runs yet.</div>
|
||||
)}
|
||||
</Card>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
Adapt `Card padding={false}` / Badge variant prop to the real component signatures (check `Card.tsx`/`Badge.tsx`).
|
||||
|
||||
- [ ] **Step 2: Add a Runs link on the workflows list**
|
||||
|
||||
In `web/app/workflows/page.tsx`, in each workflow row add a "Runs" link/button next to "Open" → `Link href={`/workflows/${w.workflow_id}/runs`}`. Match the existing row action styling.
|
||||
|
||||
- [ ] **Step 3: Verify build**
|
||||
|
||||
Run: `cd web && npm run build`
|
||||
Expected: success.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add web/app/workflows/[id]/runs/page.tsx web/app/workflows/page.tsx
|
||||
git commit -m "feat(web): workflow runs list page and navigation links"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: End-to-end verification
|
||||
|
||||
**Files:** none (verification only).
|
||||
|
||||
- [ ] **Step 1: Build everything**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./... && cd ../web && npm run build`
|
||||
Expected: all succeed.
|
||||
|
||||
- [ ] **Step 2: Manual smoke (documented, run if an environment is available)**
|
||||
|
||||
1. Open a workflow builder; confirm it matches the mockup (dotted canvas, node cards, amber passes chips, kicker/field inspector).
|
||||
2. Drag a library step onto the canvas; drag to reorder; confirm order persists after Save (no "steps is not iterable" crash).
|
||||
3. Open Edit base step; add an input param (name/default/description) and an output; Save. Place the step; set the input value in the inspector; Run; confirm the script sees the input env var and downstream steps see the output.
|
||||
4. Delete a shared step from the Edit-base-step modal; confirm it disappears from every workflow that used it.
|
||||
5. Open Edit workflow; rename, change target servers, Save; confirm persisted. Delete a throwaway workflow; confirm redirect to `/workflows`.
|
||||
6. From the workflows list and the builder topbar, navigate to Runs; open a run.
|
||||
|
||||
- [ ] **Step 3: Commit any fixes found**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: workflow builder v2 e2e fixes"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- **Spec coverage:** §3 model → T1; §4.1 cascade/CRUD → T1; §4.2 update-returns-workflow → T2; §4.3 runner inputs → T2; §6.1 tokens/Modal → T3; §6.2 api types → T3; §6.4 edit-step modal → T4; §6.5 edit-workflow modal → T5; §6.3 builder restyle/DnD/inspector → T6; §6.6 runs page + nav → T7. Save crash fixed by T2 (server) + T6 (client guard). Tests omitted per Global Constraints.
|
||||
- **Dependency order:** modals (T4, T5) land before the builder (T6) that imports them; tokens/Modal/api (T3) first.
|
||||
- **Interpreter literals** `"bash"`/`"powershell"` consistent across T1/T4/T6.
|
||||
- **Modal state seeding:** EditStepModal re-seeds via `key` in the builder (documented in T4/T6).
|
||||
- **Open follow-ups (out of scope):** typed/required inputs, drag-to-trash removal, live status in canvas, keyboard reordering.
|
||||
```
|
||||
@@ -1,887 +0,0 @@
|
||||
# Workflow Log Streaming Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Stream workflow step output live from agents to per-server-run log files on the server, tail them live in the UI over SSE, and auto-expire them on a configurable retention period.
|
||||
|
||||
**Architecture:** Agent streams interleaved stdout/stderr chunks over the existing `CommandStream` (`AgentMessage.StepOutput`). Server appends secret-masked chunks to `<logdir>/<run_id>/<server_id>.log` via a per-command log-writer registry, records a per-step byte offset, and stops persisting log bodies in Mongo. UI tails via an SSE endpoint while running and fetches the whole file after. An hourly sweeper deletes run-log dirs older than the retention setting.
|
||||
|
||||
**Tech Stack:** Go (gin, mongo-driver v2), hand-written JSON-codec gRPC structs (no protoc), Next.js 16 app-router + react-query + EventSource, MongoDB, local filesystem for logs.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **No tests this iteration** — do not write `*_test.go` or frontend tests. Verify each task with `go build ./...`, `go vet ./...`, and (frontend) `npm run build`.
|
||||
- gRPC uses a **JSON codec** — proto messages are hand-written Go structs in **two** files that must stay identical: `server/internal/grpc/pb/vantage.pb.go` and `agent/internal/grpc/pb/vantage.pb.go`. No codegen. Also update `proto/vantage/v1/vantage.proto` as documentation.
|
||||
- Mongo access pattern: `db.Col("collection_name")` with `context.WithTimeout`. Follow `server/internal/services/workflows.go`.
|
||||
- Secret values must never be written into log files unmasked — mask by literal `***` replacement at write time, boundary-safe via a carry buffer.
|
||||
- Interpreter values are the literals `"bash"` and `"powershell"`.
|
||||
- Go module path: `github.com/mrhid6/vantage`.
|
||||
- Log dir from env `VANTAGE_WORKFLOW_LOG_DIR`, default `<data>/workflow-logs`; files `0600`, dirs `0700`.
|
||||
- Retention default **30** days, stored `settings.workflow_log_retention_days`; `0`/negative = keep forever.
|
||||
- The agent's stream `Send` is only safe through the existing per-connection mutex-guarded `send()` closure in `connectAndHandleStream` — all `StepOutput`/`StepResult` sends MUST go through it.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Proto/pb — StepOutputChunk
|
||||
|
||||
**Files:**
|
||||
- Modify: `proto/vantage/v1/vantage.proto`
|
||||
- Modify: `server/internal/grpc/pb/vantage.pb.go`
|
||||
- Modify: `agent/internal/grpc/pb/vantage.pb.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `pb.StepOutputChunk{CommandId string, Seq uint64, Data []byte, Eof bool}`; `pb.AgentMessage` gains `StepOutput *StepOutputChunk`.
|
||||
|
||||
- [ ] **Step 1: Document in the proto file**
|
||||
|
||||
In `proto/vantage/v1/vantage.proto`, add to the `AgentMessage` oneof: `StepOutputChunk step_output = 6;` and add the message:
|
||||
|
||||
```protobuf
|
||||
message StepOutputChunk {
|
||||
string command_id = 1;
|
||||
uint64 seq = 2;
|
||||
bytes data = 3;
|
||||
bool eof = 4;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add struct + field to server pb file**
|
||||
|
||||
In `server/internal/grpc/pb/vantage.pb.go`, add to `type AgentMessage struct { ... }`:
|
||||
|
||||
```go
|
||||
StepOutput *StepOutputChunk `json:"step_output,omitempty"`
|
||||
```
|
||||
|
||||
and add the new struct:
|
||||
|
||||
```go
|
||||
type StepOutputChunk struct {
|
||||
CommandId string `json:"command_id"`
|
||||
Seq uint64 `json:"seq"`
|
||||
Data []byte `json:"data,omitempty"`
|
||||
Eof bool `json:"eof,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Mirror identical additions into the agent pb file**
|
||||
|
||||
Apply the identical `AgentMessage.StepOutput` field and `StepOutputChunk` struct to `agent/internal/grpc/pb/vantage.pb.go`.
|
||||
|
||||
- [ ] **Step 4: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && cd ../agent && go build ./...`
|
||||
Expected: both succeed.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add proto/vantage/v1/vantage.proto server/internal/grpc/pb/vantage.pb.go agent/internal/grpc/pb/vantage.pb.go
|
||||
git commit -m "feat(proto): add StepOutputChunk streaming message"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Agent — stream step output
|
||||
|
||||
**Files:**
|
||||
- Modify: `agent/internal/exec/exec.go`
|
||||
- Modify: `agent/internal/sync/sync.go` (the `cmd.RunStep != nil` goroutine)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `pb.RunStepCmd`, `pb.StepResult`, `pb.StepOutputChunk` (Task 1).
|
||||
- Produces: `exec.RunStep(cmd *pb.RunStepCmd, emit func(seq uint64, data []byte)) *pb.StepResult` — streams output via `emit`, returns terminal result with empty stdout/stderr but populated exit_code/output_env.
|
||||
|
||||
- [ ] **Step 1: Rework `exec.RunStep` to stream**
|
||||
|
||||
In `agent/internal/exec/exec.go`, change the signature and replace the two `bytes.Buffer`s with a single mutex-guarded streaming writer. Full new body of the run/capture section (keep the existing temp-dir, env-file, interpreter-selection, timeout, and `parseEnvFile` logic exactly as-is):
|
||||
|
||||
Add this type at package scope:
|
||||
|
||||
```go
|
||||
// streamWriter forwards every write to emit() as an ordered chunk. Used as both
|
||||
// Stdout and Stderr so output interleaves in real execution order. The mutex
|
||||
// ensures a single stdout/stderr write is not interleaved mid-slice with another.
|
||||
type streamWriter struct {
|
||||
mu sync.Mutex
|
||||
seq uint64
|
||||
emit func(seq uint64, data []byte)
|
||||
}
|
||||
|
||||
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.emit != nil {
|
||||
buf := make([]byte, len(p))
|
||||
copy(buf, p)
|
||||
w.emit(w.seq, buf)
|
||||
w.seq++
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
```
|
||||
|
||||
Add `"sync"` to the imports. Change the signature to:
|
||||
|
||||
```go
|
||||
func RunStep(cmd *pb.RunStepCmd, emit func(seq uint64, data []byte)) *pb.StepResult {
|
||||
```
|
||||
|
||||
Replace the block that currently declares `var stdout, stderr bytes.Buffer`, assigns `c.Stdout`/`c.Stderr`, and sets `res.Stdout`/`res.Stderr` from them, with:
|
||||
|
||||
```go
|
||||
sw := &streamWriter{emit: emit}
|
||||
c.Stdout = sw
|
||||
c.Stderr = sw
|
||||
runErr := c.Run()
|
||||
|
||||
// stdout/stderr are streamed via emit, not returned in the result.
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
res.ExitCode = 124
|
||||
res.Stderr = "[vantage] step timed out"
|
||||
} else if ee, ok := runErr.(*exec.ExitError); ok {
|
||||
res.ExitCode = ee.ExitCode()
|
||||
} else if runErr != nil {
|
||||
res.ExitCode = 1
|
||||
res.Stderr = "[vantage] " + runErr.Error()
|
||||
}
|
||||
|
||||
res.OutputEnv = parseEnvFile(envFile)
|
||||
return res
|
||||
```
|
||||
|
||||
Remove the now-unused `"bytes"` and `"bufio"` imports **only if** they are no longer referenced (`parseEnvFile` uses `bufio` + `os` — keep `bufio`; `bytes` is likely now unused — remove it if so). Verify with `go build`.
|
||||
|
||||
- [ ] **Step 2: Wire streaming into the agent loop**
|
||||
|
||||
In `agent/internal/sync/sync.go`, the `cmd.RunStep != nil` goroutine currently calls `agentexec.RunStep(rc)` and sends one `StepResult` via `send()`. Change it to pass an `emit` closure that streams chunks, then send an eof chunk, then the terminal result — all through the existing mutex-guarded `send()`:
|
||||
|
||||
```go
|
||||
if cmd.RunStep != nil {
|
||||
go func(rc *pb.RunStepCmd, cid string) {
|
||||
emit := func(seq uint64, data []byte) {
|
||||
_ = send(&pb.AgentMessage{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
StepOutput: &pb.StepOutputChunk{CommandId: cid, Seq: seq, Data: data},
|
||||
})
|
||||
}
|
||||
res := agentexec.RunStep(rc, emit)
|
||||
res.CommandId = cid
|
||||
// Final eof marker so the server closes the log file.
|
||||
_ = send(&pb.AgentMessage{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
StepOutput: &pb.StepOutputChunk{CommandId: cid, Eof: true},
|
||||
})
|
||||
_ = send(&pb.AgentMessage{
|
||||
ServerId: cfg.ServerID,
|
||||
AgentToken: cfg.AgentToken,
|
||||
StepResult: res,
|
||||
})
|
||||
}(cmd.RunStep, cmd.CommandId)
|
||||
continue
|
||||
}
|
||||
```
|
||||
|
||||
(Match the exact field names already used by the existing `send()` calls in this function — `cfg.ServerID`, `cfg.AgentToken`, and the `send` closure. If the existing RunStep branch used different local names, keep those.)
|
||||
|
||||
- [ ] **Step 3: Verify build**
|
||||
|
||||
Run: `cd agent && go build ./... && go vet ./...`
|
||||
Expected: success. Resolve any leftover unused-import error from Step 1.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add agent/internal/exec/exec.go agent/internal/sync/sync.go
|
||||
git commit -m "feat(agent): stream step output chunks over CommandStream"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Server log-writer registry + retention sweeper
|
||||
|
||||
**Files:**
|
||||
- Create: `server/internal/services/steplogs.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `settings` service (retention), `db.Col("workflow_runs")` (sweeper), env `VANTAGE_WORKFLOW_LOG_DIR`.
|
||||
- Produces:
|
||||
- `WorkflowLogDir() string` — resolved base dir (env or default), created on first call.
|
||||
- `ServerRunLogPath(runID, serverID string) string` — `<logdir>/<runID>/<serverID>.log`.
|
||||
- `AppendMarker(runID, serverID, line string) (int64, error)` — appends a marker line, returns the byte offset **before** the write (the step's `log_offset`).
|
||||
- `var StepLogs *stepLogRegistry` with `Open(commandID, path string, secrets []string) error`, `Append(commandID string, data []byte)`, `Close(commandID string)`.
|
||||
- `StartLogSweeper()` — launches the hourly retention goroutine; also sweeps once immediately.
|
||||
|
||||
- [ ] **Step 1: Write the registry, paths, and sweeper**
|
||||
|
||||
```go
|
||||
package services
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mrhid6/vantage/server/internal/db"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
)
|
||||
|
||||
// WorkflowLogDir returns the base directory for workflow step logs, creating it.
|
||||
func WorkflowLogDir() string {
|
||||
dir := os.Getenv("VANTAGE_WORKFLOW_LOG_DIR")
|
||||
if dir == "" {
|
||||
dir = filepath.Join("data", "workflow-logs")
|
||||
}
|
||||
_ = os.MkdirAll(dir, 0700)
|
||||
return dir
|
||||
}
|
||||
|
||||
// ServerRunLogPath is the per-server-run log file path.
|
||||
func ServerRunLogPath(runID, serverID string) string {
|
||||
return filepath.Join(WorkflowLogDir(), runID, serverID+".log")
|
||||
}
|
||||
|
||||
// AppendMarker appends a line to the server-run log and returns the byte offset
|
||||
// at which the write began (used as a step's log_offset).
|
||||
func AppendMarker(runID, serverID, line string) (int64, error) {
|
||||
path := ServerRunLogPath(runID, serverID)
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
off, _ := f.Seek(0, 2) // current end = offset before write
|
||||
if _, err := f.WriteString(line); err != nil {
|
||||
return off, err
|
||||
}
|
||||
return off, nil
|
||||
}
|
||||
|
||||
// ---- streamed chunk writer, boundary-safe secret masking ----
|
||||
|
||||
type stepLogWriter struct {
|
||||
mu sync.Mutex
|
||||
f *os.File
|
||||
carry []byte
|
||||
secrets []string
|
||||
maxSecret int
|
||||
}
|
||||
|
||||
type stepLogRegistry struct {
|
||||
mu sync.Mutex
|
||||
writers map[string]*stepLogWriter
|
||||
}
|
||||
|
||||
var StepLogs = &stepLogRegistry{writers: make(map[string]*stepLogWriter)}
|
||||
|
||||
// Open opens (append) the server-run file for a step's streamed chunks.
|
||||
func (r *stepLogRegistry) Open(commandID, path string, secrets []string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
max := 0
|
||||
for _, s := range secrets {
|
||||
if len(s) > max {
|
||||
max = len(s)
|
||||
}
|
||||
}
|
||||
w := &stepLogWriter{f: f, secrets: secrets, maxSecret: max}
|
||||
r.mu.Lock()
|
||||
r.writers[commandID] = w
|
||||
r.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *stepLogRegistry) get(commandID string) *stepLogWriter {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.writers[commandID]
|
||||
}
|
||||
|
||||
// Append masks and writes a chunk, holding back the last maxSecret-1 bytes so a
|
||||
// secret split across a chunk boundary is still masked on the next append/close.
|
||||
func (r *stepLogRegistry) Append(commandID string, data []byte) {
|
||||
w := r.get(commandID)
|
||||
if w == nil {
|
||||
return
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if len(w.secrets) == 0 || w.maxSecret <= 1 {
|
||||
_, _ = w.f.Write(data)
|
||||
return
|
||||
}
|
||||
buf := append(w.carry, data...)
|
||||
hold := w.maxSecret - 1
|
||||
if len(buf) <= hold {
|
||||
w.carry = buf
|
||||
return
|
||||
}
|
||||
flush := buf[:len(buf)-hold]
|
||||
w.carry = append([]byte{}, buf[len(buf)-hold:]...)
|
||||
_, _ = w.f.Write(maskBytes(flush, w.secrets))
|
||||
}
|
||||
|
||||
// Close flushes the carry (masked) and closes the file.
|
||||
func (r *stepLogRegistry) Close(commandID string) {
|
||||
r.mu.Lock()
|
||||
w := r.writers[commandID]
|
||||
delete(r.writers, commandID)
|
||||
r.mu.Unlock()
|
||||
if w == nil {
|
||||
return
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if len(w.carry) > 0 {
|
||||
_, _ = w.f.Write(maskBytes(w.carry, w.secrets))
|
||||
w.carry = nil
|
||||
}
|
||||
_ = w.f.Close()
|
||||
}
|
||||
|
||||
func maskBytes(b []byte, secrets []string) []byte {
|
||||
s := string(b)
|
||||
for _, v := range secrets {
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
s = strings.ReplaceAll(s, v, "***")
|
||||
}
|
||||
return []byte(s)
|
||||
}
|
||||
|
||||
// ---- retention sweeper ----
|
||||
|
||||
// StartLogSweeper sweeps expired run-log dirs hourly (and once now).
|
||||
func StartLogSweeper() {
|
||||
go func() {
|
||||
sweepLogs()
|
||||
t := time.NewTicker(time.Hour)
|
||||
defer t.Stop()
|
||||
for range t.C {
|
||||
sweepLogs()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func sweepLogs() {
|
||||
days := retentionDays()
|
||||
if days <= 0 {
|
||||
return
|
||||
}
|
||||
cutoff := time.Now().AddDate(0, 0, -days)
|
||||
base := WorkflowLogDir()
|
||||
entries, err := os.ReadDir(base)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() {
|
||||
continue
|
||||
}
|
||||
runID := e.Name()
|
||||
dir := filepath.Join(base, runID)
|
||||
if runExpired(runID, dir, cutoff) {
|
||||
_ = os.RemoveAll(dir)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// runExpired is true when the run finished before cutoff (falling back to dir
|
||||
// mtime when the run doc is gone).
|
||||
func runExpired(runID, dir string, cutoff time.Time) bool {
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
var run struct {
|
||||
FinishedAt *time.Time `bson:"finished_at"`
|
||||
}
|
||||
err := db.Col("workflow_runs").FindOne(ctx, bson.M{"run_id": runID}).Decode(&run)
|
||||
if err == nil {
|
||||
if run.FinishedAt == nil {
|
||||
return false // still running / never finished — keep
|
||||
}
|
||||
return run.FinishedAt.Before(cutoff)
|
||||
}
|
||||
// run doc gone: use dir mtime
|
||||
if fi, e := os.Stat(dir); e == nil {
|
||||
return fi.ModTime().Before(cutoff)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func retentionDays() int {
|
||||
if v, err := GetWorkflowLogRetentionDays(); err == nil {
|
||||
return v
|
||||
}
|
||||
return 30
|
||||
}
|
||||
```
|
||||
|
||||
Note: `wfCtx` is defined in `workflows.go` (same package) — reuse it. `GetWorkflowLogRetentionDays` is added in Task 4; this file references it (same package, compiles together).
|
||||
|
||||
- [ ] **Step 2: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./...`
|
||||
Expected: FAIL — `GetWorkflowLogRetentionDays` undefined until Task 4. This is expected; proceed to commit the file so Task 4 completes it. (If you prefer a green build, do Task 4's settings accessor first, then return — but committing here is fine since Task 4 immediately follows.)
|
||||
|
||||
Actually to keep every commit buildable: **temporarily** add a local stub at the bottom of this file and remove it in Task 4:
|
||||
|
||||
```go
|
||||
// TEMP stub, replaced in Task 4.
|
||||
func GetWorkflowLogRetentionDays() (int, error) { return 30, nil }
|
||||
```
|
||||
|
||||
Then `cd server && go build ./... && go vet ./...` must succeed.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add server/internal/services/steplogs.go
|
||||
git commit -m "feat(server): workflow log-writer registry, paths, retention sweeper"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Settings — retention accessor + startup wiring
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/internal/services/settings.go` (or wherever settings get/set lives — search `settings` collection usage)
|
||||
- Modify: `server/internal/services/steplogs.go` (remove the temp stub)
|
||||
- Modify: `server/cmd/main.go` (start the sweeper)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `GetWorkflowLogRetentionDays() (int, error)` (default 30 when unset), `SetWorkflowLogRetentionDays(int) error`. If settings are exposed as a single document/struct, add the field there and derive these accessors.
|
||||
|
||||
- [ ] **Step 1: Inspect the settings service**
|
||||
|
||||
Read the existing settings service (search for the `settings` collection: `grep -rn "\"settings\"" server/internal/services`). Determine whether settings are a typed struct document or key/value. Match that pattern.
|
||||
|
||||
- [ ] **Step 2: Add the retention accessor**
|
||||
|
||||
If settings are a **typed document** (e.g. a `GetSettings()/UpdateSettings()`), add a field `WorkflowLogRetentionDays int `bson:"workflow_log_retention_days" json:"workflow_log_retention_days"`` to the settings struct and implement:
|
||||
|
||||
```go
|
||||
func GetWorkflowLogRetentionDays() (int, error) {
|
||||
s, err := GetSettings() // use the real accessor name
|
||||
if err != nil {
|
||||
return 30, err
|
||||
}
|
||||
if s.WorkflowLogRetentionDays == 0 && /* unset sentinel */ !s.WorkflowLogRetentionSet {
|
||||
return 30, nil
|
||||
}
|
||||
return s.WorkflowLogRetentionDays, nil
|
||||
}
|
||||
```
|
||||
|
||||
Simplify to match reality: if the settings doc uses zero-value-means-unset and you cannot distinguish "0 = keep forever" from "unset", store the retention as a pointer `*int` or default at read: **treat a missing field as 30, an explicit 0 as keep-forever.** Prefer `*int` in the struct so the three states (unset→30, 0→forever, N→N) are representable. Implement `GetWorkflowLogRetentionDays` to return 30 when the pointer is nil, else its value. `SetWorkflowLogRetentionDays(n int)` sets the pointer.
|
||||
|
||||
If settings are **key/value**, implement both accessors against that store with the same nil→30 / 0→forever semantics (store empty/absent = 30).
|
||||
|
||||
- [ ] **Step 3: Remove the temp stub from `steplogs.go`**
|
||||
|
||||
Delete the `// TEMP stub` `GetWorkflowLogRetentionDays` added in Task 3 so the real one is used.
|
||||
|
||||
- [ ] **Step 4: Start the sweeper at boot**
|
||||
|
||||
In `server/cmd/main.go`, next to `EnsureWorkflowIndexes()`, add `services.StartLogSweeper()`.
|
||||
|
||||
- [ ] **Step 5: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./...`
|
||||
Expected: success (real accessor now resolves the reference from Task 3).
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add server/internal/services/settings.go server/internal/services/steplogs.go server/cmd/main.go
|
||||
git commit -m "feat(server): workflow log retention setting + sweeper startup"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Runner + model — write to files, drop log bodies from Mongo
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/internal/models/workflow.go` (`StepRun`)
|
||||
- Modify: `server/internal/services/workflow_runner.go`
|
||||
- Modify: `server/internal/grpc/server.go` (stream delivery of `StepOutput`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `StepLogs`, `AppendMarker`, `ServerRunLogPath` (Task 3), `pb.StepOutputChunk` (Task 1).
|
||||
- Produces: runner writes markers + streams chunks to files; `StepRun.LogOffset` persisted; `StepRun.Stdout/Stderr` removed.
|
||||
|
||||
- [ ] **Step 1: Update the `StepRun` model**
|
||||
|
||||
In `server/internal/models/workflow.go`, in `type StepRun struct`:
|
||||
- Remove the `Stdout` and `Stderr` fields.
|
||||
- Add: `LogOffset int64 `bson:"log_offset" json:"log_offset"``
|
||||
|
||||
- [ ] **Step 2: Deliver StepOutput chunks in the gRPC receive loop**
|
||||
|
||||
In `server/internal/grpc/server.go`, after the existing `if m.StepResult != nil { services.StepResults.Deliver(m.StepResult) }` block, add:
|
||||
|
||||
```go
|
||||
if m.StepOutput != nil {
|
||||
if m.StepOutput.Eof {
|
||||
services.StepLogs.Close(m.StepOutput.CommandId)
|
||||
} else {
|
||||
services.StepLogs.Append(m.StepOutput.CommandId, m.StepOutput.Data)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Rework `runServer` to open logs + write markers, drop persisted bodies**
|
||||
|
||||
In `server/internal/services/workflow_runner.go`, `runServer`:
|
||||
|
||||
Inside the per-step loop, **before** `dispatchAndWait`, add marker + open (compute `secretVals` first, which already exists in the loop):
|
||||
|
||||
```go
|
||||
// Write the step marker and remember the offset for later slicing.
|
||||
marker := fmt.Sprintf("\n===== step %d: %s =====\n", step.Order, step.Name)
|
||||
offset, _ := AppendMarker(runID, serverID, marker)
|
||||
logPath := ServerRunLogPath(runID, serverID)
|
||||
_ = StepLogs.Open(commandID_placeholder, logPath, secretsSlice(secretVals))
|
||||
```
|
||||
|
||||
There is a chicken-and-egg with `commandID`: today `dispatchAndWait` generates the `commandID` internally. Refactor so the runner owns the `commandID`:
|
||||
|
||||
1. Change `dispatchAndWait(serverID string, cmd *pb.RunStepCmd)` to `dispatchAndWait(serverID, commandID string, cmd *pb.RunStepCmd)` and remove its internal `commandID := uuid.New().String()` (use the passed one).
|
||||
2. In `runServer`, generate `commandID := uuid.New().String()` at the top of each attempt-group (before the marker/open), open the log with it, then call `dispatchAndWait(serverID, commandID, cmd)`.
|
||||
3. After the step completes (result received), call `StepLogs.Close(commandID)` defensively (idempotent — the agent's eof usually closed it already; Close on a missing key is a no-op).
|
||||
|
||||
Add a helper to convert the `secretVals map[string]string` to a `[]string` of values:
|
||||
|
||||
```go
|
||||
func secretsSlice(m map[string]string) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for _, v := range m {
|
||||
out = append(out, v)
|
||||
}
|
||||
return out
|
||||
}
|
||||
```
|
||||
|
||||
Update `finishStep(...)` call + signature: **remove** the `stdout, stderr string` params and the `output_env` masking stays. Persist `log_offset` instead. New `finishStep`:
|
||||
|
||||
```go
|
||||
func finishStep(runID, serverID string, order int, status string, attempts, exit int, logOffset int64, outEnv map[string]string) {
|
||||
now := time.Now()
|
||||
updateStep(runID, serverID, order, bson.M{
|
||||
"server_runs.$[s].steps.$[t].status": status,
|
||||
"server_runs.$[s].steps.$[t].attempts": attempts,
|
||||
"server_runs.$[s].steps.$[t].exit_code": exit,
|
||||
"server_runs.$[s].steps.$[t].log_offset": logOffset,
|
||||
"server_runs.$[s].steps.$[t].output_env": outEnv,
|
||||
"server_runs.$[s].steps.$[t].finished_at": now,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
In the loop, after receiving `res`, drop the `stdout, stderr := ...` masking of `res.Stdout/res.Stderr` (those are now streamed to file). Keep the `outEnv` build **with existing masking** (`maskSecrets(v, allSecrets)` per the merged secret-leak fix) — `output_env`/`run_env` masking is unchanged. Call:
|
||||
|
||||
```go
|
||||
finishStep(runID, serverID, i, status, attempts, exit, offset, outEnv)
|
||||
```
|
||||
|
||||
where `offset` is the marker offset captured before dispatch. If `res == nil`, still write a short note to the file so failures are visible:
|
||||
|
||||
```go
|
||||
if res == nil {
|
||||
_, _ = AppendMarker(runID, serverID, "[vantage] agent did not return a result\n")
|
||||
}
|
||||
```
|
||||
|
||||
Remove the initial `StepRun{... Status:"queued"}` `Stdout/Stderr` references if any (the model no longer has them — the queued StepRun in `TriggerWorkflow` set only `Order/Name/Status/OutputEnv`, so no change needed there; verify).
|
||||
|
||||
Ensure `fmt` is imported (it already is).
|
||||
|
||||
- [ ] **Step 4: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./...`
|
||||
Expected: success. Fix any remaining references to the removed `Stdout`/`Stderr` fields or the old `finishStep`/`dispatchAndWait` signatures.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add server/internal/models/workflow.go server/internal/services/workflow_runner.go server/internal/grpc/server.go
|
||||
git commit -m "feat(server): stream step logs to files, drop log bodies from run docs"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: REST — log fetch + SSE stream endpoints
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/internal/api/workflows.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `ServerRunLogPath`, `GetRun` (existing).
|
||||
- Produces: `GET /api/runs/:runId/servers/:serverId/logs` and `GET /api/runs/:runId/servers/:serverId/logs/stream` (SSE).
|
||||
|
||||
- [ ] **Step 1: Add the two handlers + routes**
|
||||
|
||||
In `registerWorkflowRoutes`, add:
|
||||
|
||||
```go
|
||||
g.GET("/runs/:runId/servers/:serverId/logs", getServerRunLog)
|
||||
g.GET("/runs/:runId/servers/:serverId/logs/stream", streamServerRunLog)
|
||||
```
|
||||
|
||||
Add a UUID-ish validator and the handlers:
|
||||
|
||||
```go
|
||||
var uuidLike = regexp.MustCompile(`^[a-zA-Z0-9-]{1,64}$`)
|
||||
|
||||
func getServerRunLog(c *gin.Context) {
|
||||
runID, serverID := c.Param("runId"), c.Param("serverId")
|
||||
if !uuidLike.MatchString(runID) || !uuidLike.MatchString(serverID) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid id"})
|
||||
return
|
||||
}
|
||||
path := services.ServerRunLogPath(runID, serverID)
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": "no logs"})
|
||||
return
|
||||
}
|
||||
c.Data(http.StatusOK, "text/plain; charset=utf-8", b)
|
||||
}
|
||||
|
||||
func streamServerRunLog(c *gin.Context) {
|
||||
runID, serverID := c.Param("runId"), c.Param("serverId")
|
||||
if !uuidLike.MatchString(runID) || !uuidLike.MatchString(serverID) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid id"})
|
||||
return
|
||||
}
|
||||
path := services.ServerRunLogPath(runID, serverID)
|
||||
|
||||
c.Writer.Header().Set("Content-Type", "text/event-stream")
|
||||
c.Writer.Header().Set("Cache-Control", "no-cache")
|
||||
c.Writer.Header().Set("Connection", "keep-alive")
|
||||
c.Writer.Header().Set("X-Accel-Buffering", "no")
|
||||
|
||||
flusher, ok := c.Writer.(http.Flusher)
|
||||
if !ok {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": "stream unsupported"})
|
||||
return
|
||||
}
|
||||
|
||||
var offset int64
|
||||
sendNew := func() bool {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return true // file may not exist yet; keep waiting
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := f.Seek(offset, 0); err != nil {
|
||||
return true
|
||||
}
|
||||
buf := make([]byte, 32*1024)
|
||||
for {
|
||||
n, _ := f.Read(buf)
|
||||
if n <= 0 {
|
||||
break
|
||||
}
|
||||
offset += int64(n)
|
||||
// SSE data frame; split on newlines to keep frames well-formed.
|
||||
for _, line := range splitSSE(buf[:n]) {
|
||||
_, _ = c.Writer.WriteString("data: " + line + "\n")
|
||||
}
|
||||
_, _ = c.Writer.WriteString("\n")
|
||||
flusher.Flush()
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
ctx := c.Request.Context()
|
||||
ticker := time.NewTicker(500 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
sendNew()
|
||||
if serverRunTerminal(runID, serverID) {
|
||||
sendNew() // final drain
|
||||
_, _ = c.Writer.WriteString("event: done\ndata: end\n\n")
|
||||
flusher.Flush()
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// serverRunTerminal reports whether the given server-run has reached a terminal status.
|
||||
func serverRunTerminal(runID, serverID string) bool {
|
||||
r, err := services.GetRun(runID)
|
||||
if err != nil {
|
||||
return true
|
||||
}
|
||||
for _, sr := range r.ServerRuns {
|
||||
if sr.ServerID == serverID {
|
||||
switch sr.Status {
|
||||
case "success", "failed", "skipped", "cancelled":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// splitSSE turns a raw byte slice into SSE-safe payload lines (newlines become
|
||||
// separate data lines; carriage returns stripped).
|
||||
func splitSSE(b []byte) []string {
|
||||
s := strings.ReplaceAll(string(b), "\r", "")
|
||||
return strings.Split(s, "\n")
|
||||
}
|
||||
```
|
||||
|
||||
Add imports: `"os"`, `"regexp"`, `"strings"`, `"time"`, `"net/http"` (already present). Confirm `services.GetRun` and `ServerRun.Status`/`ServerID` fields exist (they do from the Workflows feature).
|
||||
|
||||
- [ ] **Step 2: Verify build**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./...`
|
||||
Expected: success.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add server/internal/api/workflows.go
|
||||
git commit -m "feat(api): server-run log fetch and SSE stream endpoints"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Frontend — live SSE tail + retention setting
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/lib/api.ts`
|
||||
- Modify: `web/app/workflows/[id]/runs/[runId]/page.tsx`
|
||||
- Modify: `web/app/settings/page.tsx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: SSE endpoint, logs endpoint, settings mutation.
|
||||
|
||||
- [ ] **Step 1: Update API types + helpers**
|
||||
|
||||
In `web/lib/api.ts`:
|
||||
- In `StepRun`, remove `stdout` and `stderr`; add `log_offset: number`.
|
||||
- Add: `getServerRunLog: (runId: string, serverId: string) => request<string>(...)` — but the logs endpoint returns `text/plain`, so add a dedicated fetch that reads text. If `request<T>` assumes JSON, add a sibling:
|
||||
|
||||
```ts
|
||||
async getServerRunLog(runId: string, serverId: string): Promise<string> {
|
||||
const res = await fetch(`${API_BASE}/api/runs/${runId}/servers/${serverId}/logs`, { credentials: "include" });
|
||||
if (!res.ok) throw new Error("no logs");
|
||||
return res.text();
|
||||
},
|
||||
```
|
||||
|
||||
(Use the file's real base-URL constant / credentials pattern — inspect how `request` builds URLs and mirror it. If the app is same-origin with a rewrite, a relative `/api/...` fetch is fine.)
|
||||
- Export a helper to build the SSE URL: `serverRunLogStreamUrl(runId, serverId)` returning the `/api/runs/:runId/servers/:serverId/logs/stream` URL against the same base.
|
||||
- In the Settings type, add `workflow_log_retention_days?: number | null`.
|
||||
|
||||
- [ ] **Step 2: Live tail in the run detail page**
|
||||
|
||||
In `web/app/workflows/[id]/runs/[runId]/page.tsx`:
|
||||
- Remove all use of `st.stdout` / `st.stderr` (fields gone). Step `<details>` now show status/exit/attempts pills only.
|
||||
- Add a per-server live terminal. For each `server_run`, render a `<pre>` and, while `sr.status === "running"`, subscribe via `EventSource`:
|
||||
|
||||
```tsx
|
||||
function ServerLog({ runId, serverId, status }: { runId: string; serverId: string; status: string }) {
|
||||
const [text, setText] = useState("");
|
||||
const preRef = useRef<HTMLPreElement>(null);
|
||||
const running = status === "running";
|
||||
|
||||
useEffect(() => {
|
||||
if (running) {
|
||||
const es = new EventSource(api.serverRunLogStreamUrl(runId, serverId), { withCredentials: true });
|
||||
es.onmessage = (e) => setText((t) => t + e.data + "\n");
|
||||
es.addEventListener("done", () => es.close());
|
||||
es.onerror = () => es.close();
|
||||
return () => es.close();
|
||||
}
|
||||
// terminal: fetch the whole file once
|
||||
api.getServerRunLog(runId, serverId).then(setText).catch(() => setText(""));
|
||||
}, [running, runId, serverId]);
|
||||
|
||||
useEffect(() => { preRef.current?.scrollTo(0, preRef.current.scrollHeight); }, [text]);
|
||||
|
||||
return (
|
||||
<pre ref={preRef} className="mt-2 max-h-80 overflow-auto rounded bg-black/40 p-2 font-mono text-xs text-text-secondary whitespace-pre-wrap">
|
||||
{text || (running ? "Waiting for output…" : "No output.")}
|
||||
</pre>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
Render `<ServerLog runId={run.run_id} serverId={sr.server_id} status={sr.status} />` inside each server card, below the step pills. Keep the existing react-query `refetchInterval` on the run (drives status pills); the SSE handles live text.
|
||||
|
||||
- [ ] **Step 3: Retention field in Settings**
|
||||
|
||||
In `web/app/settings/page.tsx`, add a "Workflow log retention (days)" number input bound to `workflow_log_retention_days`, saved through the existing settings save mutation. Add helper text: "0 = keep forever." Match the page's existing input styling.
|
||||
|
||||
- [ ] **Step 4: Verify build**
|
||||
|
||||
Run: `cd web && npm run build`
|
||||
Expected: type-checks and builds. Fix any lingering `st.stdout`/`st.stderr` references.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add web/lib/api.ts web/app/workflows/[id]/runs/[runId]/page.tsx web/app/settings/page.tsx
|
||||
git commit -m "feat(web): live SSE log tail and log retention setting"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: End-to-end verification
|
||||
|
||||
**Files:** none (verification only).
|
||||
|
||||
- [ ] **Step 1: Build everything**
|
||||
|
||||
Run: `cd server && go build ./... && go vet ./... && cd ../agent && go build ./... && go vet ./... && cd ../web && npm run build`
|
||||
Expected: all succeed.
|
||||
|
||||
- [ ] **Step 2: Manual smoke (documented, run if an environment is available)**
|
||||
|
||||
With server + MongoDB + a connected agent:
|
||||
1. Run a workflow with a step that emits output slowly (e.g. `for i in $(seq 1 10); do echo "line $i"; sleep 1; done`). Open the run detail page while running; confirm lines appear live (SSE), not only at the end.
|
||||
2. Confirm `<logdir>/<run_id>/<server_id>.log` exists on the server with step markers and the output.
|
||||
3. Confirm `workflow_runs` doc no longer stores stdout/stderr bodies; `steps[].log_offset` is set.
|
||||
4. Add a secret ref and echo it; confirm the file shows `***`, including when the secret would straddle a chunk boundary.
|
||||
5. Set retention to 0 in Settings → confirm sweeper keeps files; set to a small value and backdate a run's `finished_at` → confirm the dir is removed within the hour (or call `sweepLogs` path manually).
|
||||
|
||||
- [ ] **Step 3: Commit any fixes found**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: workflow log streaming e2e fixes"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- **Spec coverage:** §3 proto → T1; §4 agent streaming → T2; §5.1 registry + §7 sweeper → T3; §7.1 setting + startup → T4; §5.3/§5.4 runner+model → T5; §6 REST/SSE → T6; §8 frontend → T7. Tests omitted per Global Constraints.
|
||||
- **Masking** boundary-safe carry buffer in `StepLogs.Append`, flushed in `Close` (T3); `output_env`/`run_env` masking unchanged (T5 keeps the merged fix).
|
||||
- **commandID ownership** moved to the runner so the log file can be opened before dispatch (T5) — mirrors the `StepResults.Await`-before-dispatch ordering.
|
||||
- **Buildable commits:** T3 adds a temp stub for `GetWorkflowLogRetentionDays`, removed in T4.
|
||||
- **Removed fields** `StepRun.Stdout/Stderr` — every reader updated in T5 (runner) and T7 (frontend).
|
||||
- **Open follow-ups (out of scope):** per-step SSE channels, log download/zip, compression, pre-existing runs have no files.
|
||||
```
|
||||
Reference in New Issue
Block a user