Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6c5472760b | ||
|
|
7c4a676742 | ||
|
|
fbda26a188 | ||
|
|
90ce7af769 | ||
|
|
b543cd1b3d | ||
|
|
15c9da1b01 | ||
|
|
baa7bb239d | ||
|
|
7342c46d99 | ||
|
|
813f9e6fef | ||
|
|
434f14ae3a | ||
|
|
8398fd2279 | ||
|
|
56f06b9eaf | ||
|
|
d9d241f83b | ||
|
|
aee910c1f8 |
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,267 @@
|
||||
# Ad-hoc Steps, Step Import/Export, and Default Steps — Design
|
||||
|
||||
Date: 2026-07-21
|
||||
|
||||
## Summary
|
||||
|
||||
Four related additions to the workflow step system:
|
||||
|
||||
1. **Ad-hoc steps** — steps defined inline in a single workflow, not written to the
|
||||
shared step library.
|
||||
2. **Import/Export** — single-step portable JSON (`vantage.step/v1`). Import can
|
||||
target the shared library or a workflow as an inline ad-hoc step.
|
||||
3. **Default steps** — JSON files in a bind-mounted directory, seeded into the
|
||||
library on boot and re-syncable on demand. Org-ready for a future SaaS plan.
|
||||
4. **Auto-derived outputs** — `declared_outputs` is scanned from the script
|
||||
(writes to `$WORKFLOW_ENV`) instead of being entered by hand.
|
||||
|
||||
Existing model: shared steps live in the `workflow_steps` collection; a
|
||||
`Workflow.Steps[]` is a list of `WorkflowStepRef` that reference a library step by
|
||||
`step_id` and may carry `Overrides` + `Inputs`. `resolveSteps` freezes each ref
|
||||
into a `ResolvedStep` snapshot at run time.
|
||||
|
||||
## 1. Data model
|
||||
|
||||
`server/internal/models/workflow.go`.
|
||||
|
||||
### WorkflowStep
|
||||
|
||||
Add a provenance field:
|
||||
|
||||
```go
|
||||
Source string `bson:"source" json:"source"` // "user" | "default"
|
||||
Slug string `bson:"slug" json:"slug"` // kebab of name; stable key for default seeding
|
||||
```
|
||||
|
||||
`Slug` is set for `source="default"` steps (used as the upsert key by the seeder).
|
||||
For `source="user"` steps it may be empty. Existing steps default to
|
||||
`source="user"` (absent field decodes to "").
|
||||
|
||||
### WorkflowStepRef
|
||||
|
||||
Add an inline definition. A ref is **either** a library ref (`StepID` set) **or**
|
||||
ad-hoc (`Inline` set). Never both.
|
||||
|
||||
```go
|
||||
type WorkflowStepRef struct {
|
||||
StepID string `bson:"step_id,omitempty" json:"step_id"`
|
||||
Inline *WorkflowStep `bson:"inline,omitempty" json:"inline,omitempty"`
|
||||
Order int `bson:"order" json:"order"`
|
||||
OnFailure string `bson:"on_failure" json:"on_failure"`
|
||||
MaxRetries int `bson:"max_retries" json:"max_retries"`
|
||||
Overrides *StepOverride `bson:"overrides,omitempty" json:"overrides,omitempty"` // library-ref only
|
||||
Inputs map[string]string `bson:"inputs,omitempty" json:"inputs,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
`Inline` reuses `WorkflowStep` (name, description, interpreter, script,
|
||||
declared_inputs, declared_outputs, secret_refs). Its `ID`, `StepID`, `Slug`,
|
||||
`Source`, and timestamps stay empty and are never persisted to `workflow_steps`.
|
||||
|
||||
Validation on workflow create/update: for each ref, exactly one of `StepID` /
|
||||
`Inline` must be set. `Overrides` is ignored when `Inline` is set.
|
||||
|
||||
## 2. Resolve at run time
|
||||
|
||||
`server/internal/services/workflow_runner.go`, `resolveSteps`.
|
||||
|
||||
For each ref:
|
||||
|
||||
- If `ref.Inline != nil`: build `ResolvedStep` from `ref.Inline` directly
|
||||
(name/interpreter/script/secret_refs), apply `ref.Inputs` against
|
||||
`Inline.DeclaredInputs` defaults. Skip `getStep`, skip `Overrides`.
|
||||
- Else: current library path unchanged (load step, apply overrides).
|
||||
|
||||
`ResolvedStep` output shape and the run snapshot are unchanged, so the runner and
|
||||
the run-history UI need no changes.
|
||||
|
||||
`DeleteStep` cascade is unaffected — ad-hoc refs carry no `step_id`, so they never
|
||||
match the cascade query.
|
||||
|
||||
## 3. Import / Export
|
||||
|
||||
Portable single-step JSON, `kind: "vantage.step/v1"`:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "vantage.step/v1",
|
||||
"name": "...",
|
||||
"description": "...",
|
||||
"interpreter": "bash",
|
||||
"script": "...",
|
||||
"declared_inputs": [ { "name": "...", "default": "...", "description": "..." } ],
|
||||
"declared_outputs": ["..."],
|
||||
"secret_refs": ["NAME"]
|
||||
}
|
||||
```
|
||||
|
||||
Export strips `_id`, `step_id`, `slug`, `source`, and timestamps.
|
||||
Secret refs are exported as names only. On import, dangling secret refs are kept
|
||||
verbatim (not auto-created).
|
||||
|
||||
### Service functions (`services/workflows.go`)
|
||||
|
||||
- `ExportStep(stepID string) ([]byte, error)` — load library step, marshal to the
|
||||
v1 shape.
|
||||
- `ParseStepDoc(b []byte) (models.WorkflowStep, error)` — validate `kind`, decode
|
||||
into a `WorkflowStep` (no id/source). Shared by both import targets.
|
||||
- `ImportStepToLibrary(b []byte) (*models.WorkflowStep, error)` — `ParseStepDoc`
|
||||
then `CreateStep` (fresh `step_id`, `source="user"`).
|
||||
|
||||
Import-to-inline needs no new service fn: the web editor calls `ParseStepDoc`'s
|
||||
API equivalent (see routes) and drops the returned step object into a new
|
||||
`WorkflowStepRef.Inline` in the workflow it's editing, then saves the workflow
|
||||
normally.
|
||||
|
||||
### Routes (`api/workflows.go`)
|
||||
|
||||
- `GET /api/steps/:id/export` — returns JSON as a downloadable attachment
|
||||
(`Content-Disposition`).
|
||||
- `POST /api/steps/import` — body is the v1 JSON; imports to library; returns the
|
||||
created step. (Used by the "import to library" flow.)
|
||||
- `POST /api/steps/parse` — body is the v1 JSON; validates and returns the
|
||||
normalized step object **without** persisting. Used by "import to inline" so the
|
||||
editor can insert it as an ad-hoc ref. (Keeps parsing/validation server-side.)
|
||||
|
||||
Audit: `workflow.step_imported` logged on library import.
|
||||
|
||||
## 4. Default steps (seed + admin re-sync)
|
||||
|
||||
Mirrors the existing `WorkflowLogDir` pattern — a bind-mounted directory, no Go
|
||||
`embed`.
|
||||
|
||||
### Directory
|
||||
|
||||
```go
|
||||
// DefaultStepsDir returns the directory holding default step JSON files, creating it.
|
||||
func DefaultStepsDir() string {
|
||||
dir := os.Getenv("VANTAGE_DEFAULT_STEPS_DIR")
|
||||
if dir == "" {
|
||||
dir = filepath.Join("data", "default-steps")
|
||||
}
|
||||
_ = os.MkdirAll(dir, 0700)
|
||||
return dir
|
||||
}
|
||||
```
|
||||
|
||||
Compose already bind-mounts `./data:/data`. Set
|
||||
`VANTAGE_DEFAULT_STEPS_DIR=/data/default-steps` for explicitness (optional).
|
||||
Operator drops `*.json` (`vantage.step/v1`) files into that folder.
|
||||
|
||||
### Seeder
|
||||
|
||||
`SeedDefaultSteps() (created, updated int, err error)`:
|
||||
|
||||
1. Glob `DefaultStepsDir()/*.json`.
|
||||
2. For each file: `ParseStepDoc`, compute `slug = kebab(name)`.
|
||||
3. Upsert into `workflow_steps` keyed on `{ slug, source: "default" }`:
|
||||
- absent → insert with fresh `step_id`, `source="default"`, `slug`. (`created++`)
|
||||
- present → `$set` name/description/interpreter/script/declared_*/secret_refs +
|
||||
`updated_at`. (`updated++`)
|
||||
|
||||
**Override rule (confirmed):** re-sync is authoritative for `source="default"`
|
||||
steps and overwrites their content, reverting any user edits to those steps.
|
||||
`source="user"` steps are never touched by the seeder, even on a slug collision
|
||||
(the seeder query is scoped to `source: "default"`).
|
||||
|
||||
Add a partial unique index on `slug` where `source == "default"` (or enforce
|
||||
uniqueness in the seeder loop) to keep default slugs unambiguous.
|
||||
|
||||
### Boot
|
||||
|
||||
Call `SeedDefaultSteps()` from server startup after `EnsureWorkflowIndexes()`
|
||||
(alongside index setup in `server/cmd/main.go`). Log the created/updated counts;
|
||||
a seed error is logged but non-fatal (server still boots).
|
||||
|
||||
### Route
|
||||
|
||||
- `POST /api/steps/seed-defaults` (admin) — runs `SeedDefaultSteps()`, returns
|
||||
`{ "created": n, "updated": m }`. Audit `workflow.defaults_synced`.
|
||||
|
||||
### Org readiness
|
||||
|
||||
Signature stays global today. When Orgs land, `SeedDefaultSteps(orgID)` seeds
|
||||
per-org and the upsert key becomes `{ org_id, slug, source }`. No schema churn
|
||||
blocks that later change.
|
||||
|
||||
## 5. Auto-derived outputs
|
||||
|
||||
Today `WorkflowStep.DeclaredOutputs` is entered by hand and consumed only by the
|
||||
UI (no runtime reads it — outputs are captured at run time by `parseEnvFile` on
|
||||
the agent). Replace manual entry with a server-side scan of the script.
|
||||
|
||||
At run time the agent exposes an env file path in `$WORKFLOW_ENV` (bash) /
|
||||
`$env:WORKFLOW_ENV` (powershell); a step emits an output by appending a
|
||||
`KEY=value` line to it, e.g. `echo "test=123" >> $WORKFLOW_ENV`.
|
||||
|
||||
### Scanner
|
||||
|
||||
`services.DeriveOutputs(script string) []string`:
|
||||
|
||||
- Scan line by line. For each line that references `WORKFLOW_ENV`, extract every
|
||||
`KEY=` assignment target on that line, where `KEY` matches
|
||||
`[A-Za-z_][A-Za-z0-9_]*`.
|
||||
- Covers the common forms across both interpreters (line mentions `WORKFLOW_ENV`
|
||||
and contains `KEY=...`):
|
||||
- `echo "test=123" >> $WORKFLOW_ENV`
|
||||
- `echo "test=123" >> "$WORKFLOW_ENV"`
|
||||
- `printf 'k=v\n' >> $WORKFLOW_ENV`
|
||||
- `"k=v" >> $env:WORKFLOW_ENV` / `Add-Content $env:WORKFLOW_ENV "k=v"`
|
||||
- Deduplicate, preserve first-seen order. Best-effort heuristic — false positives
|
||||
are acceptable (they only widen the documented output list); it never affects
|
||||
what the agent actually captures.
|
||||
|
||||
### Wiring
|
||||
|
||||
- `CreateStep` and `UpdateStep` set `DeclaredOutputs = DeriveOutputs(s.Script)`,
|
||||
ignoring any client-sent value.
|
||||
- Inline ad-hoc steps: `DeriveOutputs` is applied when the workflow is saved (for
|
||||
each `ref.Inline`), so inline outputs are derived too.
|
||||
- `ParseStepDoc` (import) also derives outputs, so `declared_outputs` in an
|
||||
imported/exported file is informational and always recomputed on import.
|
||||
- `SeedDefaultSteps` derives outputs the same way when upserting.
|
||||
|
||||
`DeclaredOutputs` stays in the model and JSON (still shown in the UI and used to
|
||||
wire step-to-step input references), it is just no longer user-authored.
|
||||
|
||||
### Web
|
||||
|
||||
The step editor's "declared outputs" input becomes a read-only, auto-populated
|
||||
display (derived from the script, refreshed on save / on script edit). No manual
|
||||
add/remove.
|
||||
|
||||
## 6. Web
|
||||
|
||||
`web/app/workflows/[id]/page.tsx` and the steps list page.
|
||||
|
||||
- **Steps list:** per-row **Export** (downloads JSON) and top-level **Import**
|
||||
(file picker → `POST /api/steps/import` → library). **Sync defaults** admin
|
||||
button → `POST /api/steps/seed-defaults`, toast the counts. `source="default"`
|
||||
rows get a "default" badge.
|
||||
- **Workflow editor — Add step:** existing "add from library" plus **Add ad-hoc
|
||||
step** (inline mini-form: name, interpreter, script, optional inputs) stored as
|
||||
a `WorkflowStepRef.Inline`. Also **Import ad-hoc from file** → `POST
|
||||
/api/steps/parse` → inserts the returned step as a new inline ref.
|
||||
- Ad-hoc rows in the editor are visually distinguished from library refs (badge)
|
||||
and are editable in place; library refs keep the existing override UI.
|
||||
|
||||
## Testing
|
||||
|
||||
- `resolveSteps`: inline ref resolves without touching the library; inputs apply
|
||||
from `Inline.DeclaredInputs` defaults and ref overrides; library path unchanged.
|
||||
- Workflow validation: rejects a ref with both `StepID` and `Inline`, and one with
|
||||
neither.
|
||||
- Export → import round-trips to an equivalent library step with a new `step_id`.
|
||||
- `ParseStepDoc` rejects a wrong/missing `kind`.
|
||||
- `SeedDefaultSteps`: insert-then-update idempotency; user steps untouched;
|
||||
user-edited default step reverted on re-sync; counts correct.
|
||||
- `DeleteStep` cascade ignores ad-hoc refs.
|
||||
- `DeriveOutputs`: extracts keys from each interpreter form above, dedupes,
|
||||
preserves order, ignores lines not referencing `WORKFLOW_ENV`; create/update/
|
||||
import/seed all populate `declared_outputs` from it and ignore client input.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Whole-workflow export/import.
|
||||
- Auto-creating secret refs on import.
|
||||
- Multi-tenant Org model (design is forward-compatible only).
|
||||
@@ -31,6 +31,12 @@ func main() {
|
||||
log.Printf("warning: failed to ensure workflow indexes: %v", err)
|
||||
}
|
||||
|
||||
if created, updated, err := services.SeedDefaultSteps(); err != nil {
|
||||
log.Printf("warning: failed to seed default steps: %v", err)
|
||||
} else {
|
||||
log.Printf("default steps seeded: %d created, %d updated", created, updated)
|
||||
}
|
||||
|
||||
services.StartLogSweeper()
|
||||
|
||||
redisAddr := getEnv("REDIS_ADDR", "localhost:6379")
|
||||
|
||||
@@ -2,6 +2,7 @@ package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"regexp"
|
||||
@@ -19,6 +20,10 @@ func registerWorkflowRoutes(g *gin.RouterGroup) {
|
||||
g.POST("/steps", createStep)
|
||||
g.PUT("/steps/:id", updateStep)
|
||||
g.DELETE("/steps/:id", deleteStep)
|
||||
g.GET("/steps/:id/export", exportStep)
|
||||
g.POST("/steps/import", importStep)
|
||||
g.POST("/steps/seed-defaults", seedDefaults)
|
||||
g.POST("/steps/parse", parseStep)
|
||||
|
||||
g.GET("/workflows", listWorkflows)
|
||||
g.POST("/workflows", createWorkflow)
|
||||
@@ -187,6 +192,61 @@ func deleteStep(c *gin.Context) {
|
||||
c.JSON(http.StatusOK, gin.H{"deleted": true})
|
||||
}
|
||||
|
||||
func exportStep(c *gin.Context) {
|
||||
b, err := services.ExportStep(c.Param("id"))
|
||||
if err != nil {
|
||||
c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.Header("Content-Disposition", fmt.Sprintf("attachment; filename=step-%s.json", c.Param("id")))
|
||||
c.Data(http.StatusOK, "application/json", b)
|
||||
}
|
||||
|
||||
func seedDefaults(c *gin.Context) {
|
||||
created, updated, err := services.SeedDefaultSteps()
|
||||
if err != nil {
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
services.LogEvent("workflow.defaults_synced", actorFromCtx(c), "", "", fmt.Sprintf("default steps synced: %d created, %d updated", created, updated))
|
||||
c.JSON(http.StatusOK, gin.H{"created": created, "updated": updated})
|
||||
}
|
||||
|
||||
const maxStepBodyBytes = 1 << 20 // 1 MiB
|
||||
|
||||
func importStep(c *gin.Context) {
|
||||
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxStepBodyBytes)
|
||||
body, err := io.ReadAll(c.Request.Body)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
out, err := services.ImportStepToLibrary(body)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
services.LogEvent("workflow.step_imported", actorFromCtx(c), "", out.StepID, fmt.Sprintf("step '%s' imported", out.Name))
|
||||
c.JSON(http.StatusCreated, out)
|
||||
}
|
||||
|
||||
// parseStep validates a step doc and returns the normalized step WITHOUT
|
||||
// persisting — used by the editor to insert an imported ad-hoc (inline) step.
|
||||
func parseStep(c *gin.Context) {
|
||||
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxStepBodyBytes)
|
||||
body, err := io.ReadAll(c.Request.Body)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
s, err := services.ParseStepDoc(body)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, s)
|
||||
}
|
||||
|
||||
func listWorkflows(c *gin.Context) {
|
||||
wfs, err := services.ListWorkflows()
|
||||
if err != nil {
|
||||
|
||||
@@ -22,12 +22,15 @@ type WorkflowStep struct {
|
||||
DeclaredOutputs []string `bson:"declared_outputs" json:"declared_outputs"`
|
||||
DeclaredInputs []InputParam `bson:"declared_inputs" json:"declared_inputs"`
|
||||
SecretRefs []string `bson:"secret_refs" json:"secret_refs"`
|
||||
Source string `bson:"source" json:"source"` // "user" | "default"
|
||||
Slug string `bson:"slug,omitempty" json:"slug,omitempty"`
|
||||
CreatedAt time.Time `bson:"created_at" json:"created_at"`
|
||||
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
|
||||
}
|
||||
|
||||
type WorkflowStepRef struct {
|
||||
StepID string `bson:"step_id" json:"step_id"`
|
||||
StepID string `bson:"step_id,omitempty" json:"step_id,omitempty"`
|
||||
Inline *WorkflowStep `bson:"inline,omitempty" json:"inline,omitempty"`
|
||||
Order int `bson:"order" json:"order"`
|
||||
OnFailure string `bson:"on_failure" json:"on_failure"` // "stop" | "continue" | "retry"
|
||||
MaxRetries int `bson:"max_retries" json:"max_retries"`
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/mrhid6/vantage/server/internal/db"
|
||||
"github.com/mrhid6/vantage/server/internal/models"
|
||||
"go.mongodb.org/mongo-driver/v2/bson"
|
||||
"go.mongodb.org/mongo-driver/v2/mongo/options"
|
||||
)
|
||||
|
||||
// DefaultStepsDir returns the directory holding default step JSON files.
|
||||
func DefaultStepsDir() string {
|
||||
dir := os.Getenv("VANTAGE_DEFAULT_STEPS_DIR")
|
||||
if dir == "" {
|
||||
dir = filepath.Join("data", "default-steps")
|
||||
}
|
||||
_ = os.MkdirAll(dir, 0700)
|
||||
return dir
|
||||
}
|
||||
|
||||
// readDefaultStepFiles parses every *.json in the defaults dir into
|
||||
// source=default library steps (with slug set). Non-json and invalid files are
|
||||
// skipped silently; a slug is derived from the step name.
|
||||
func readDefaultStepFiles() ([]models.WorkflowStep, error) {
|
||||
matches, err := filepath.Glob(filepath.Join(DefaultStepsDir(), "*.json"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := []models.WorkflowStep{}
|
||||
for _, path := range matches {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s, err := ParseStepDoc(b)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s.Source = "default"
|
||||
s.Slug = Slugify(s.Name)
|
||||
if s.Slug == "" {
|
||||
continue
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// SeedDefaultSteps upserts default steps from disk keyed on {slug, source}.
|
||||
// Re-sync overwrites default-step content; user steps are never touched.
|
||||
func SeedDefaultSteps() (created, updated int, err error) {
|
||||
steps, err := readDefaultStepFiles()
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
col := db.Col("workflow_steps")
|
||||
for _, s := range steps {
|
||||
filter := bson.M{"slug": s.Slug, "source": "default"}
|
||||
set := bson.M{
|
||||
"name": s.Name,
|
||||
"description": s.Description,
|
||||
"interpreter": s.Interpreter,
|
||||
"script": s.Script,
|
||||
"declared_outputs": s.DeclaredOutputs,
|
||||
"declared_inputs": s.DeclaredInputs,
|
||||
"secret_refs": s.SecretRefs,
|
||||
"updated_at": time.Now(),
|
||||
}
|
||||
res, uerr := col.UpdateOne(ctx, filter, bson.M{
|
||||
"$set": set,
|
||||
"$setOnInsert": bson.M{
|
||||
"step_id": uuid.New().String(),
|
||||
"slug": s.Slug,
|
||||
"source": "default",
|
||||
"created_at": time.Now(),
|
||||
},
|
||||
}, options.UpdateOne().SetUpsert(true))
|
||||
if uerr != nil {
|
||||
return created, updated, uerr
|
||||
}
|
||||
if res.UpsertedCount > 0 {
|
||||
created++
|
||||
} else if res.ModifiedCount > 0 {
|
||||
updated++
|
||||
}
|
||||
}
|
||||
return created, updated, nil
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDefaultStepsDirEnv(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "ds")
|
||||
t.Setenv("VANTAGE_DEFAULT_STEPS_DIR", dir)
|
||||
got := DefaultStepsDir()
|
||||
if got != dir {
|
||||
t.Fatalf("got %q want %q", got, dir)
|
||||
}
|
||||
if _, err := os.Stat(dir); err != nil {
|
||||
t.Fatalf("dir not created: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadDefaultStepFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("VANTAGE_DEFAULT_STEPS_DIR", dir)
|
||||
good := `{"kind":"vantage.step/v1","name":"Ping Host","interpreter":"bash","script":"ping -c1 x=1 >> $WORKFLOW_ENV"}`
|
||||
os.WriteFile(filepath.Join(dir, "ping.json"), []byte(good), 0600)
|
||||
os.WriteFile(filepath.Join(dir, "notes.txt"), []byte("ignore me"), 0600)
|
||||
|
||||
steps, err := readDefaultStepFiles()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(steps) != 1 {
|
||||
t.Fatalf("want 1 step, got %d", len(steps))
|
||||
}
|
||||
if steps[0].Slug != "ping-host" || steps[0].Source != "default" {
|
||||
t.Fatalf("bad seed step: %+v", steps[0])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
func TestResolveInlineStep(t *testing.T) {
|
||||
ref := models.WorkflowStepRef{
|
||||
Order: 2,
|
||||
OnFailure: "",
|
||||
Inline: &models.WorkflowStep{
|
||||
Name: "adhoc",
|
||||
Interpreter: "bash",
|
||||
Script: "echo hi",
|
||||
SecretRefs: []string{"TOKEN"},
|
||||
DeclaredInputs: []models.InputParam{
|
||||
{Name: "REGION", Default: "eu"},
|
||||
},
|
||||
},
|
||||
Inputs: map[string]string{"REGION": "us"},
|
||||
}
|
||||
rs := resolveInlineStep(ref)
|
||||
if rs.Name != "adhoc" || rs.Script != "echo hi" || rs.Order != 2 {
|
||||
t.Fatalf("bad resolve: %+v", rs)
|
||||
}
|
||||
if rs.OnFailure != "stop" {
|
||||
t.Fatalf("want default on_failure=stop, got %q", rs.OnFailure)
|
||||
}
|
||||
if rs.Inputs["REGION"] != "us" {
|
||||
t.Fatalf("want input override us, got %q", rs.Inputs["REGION"])
|
||||
}
|
||||
if len(rs.SecretRefs) != 1 || rs.SecretRefs[0] != "TOKEN" {
|
||||
t.Fatalf("bad secret refs: %v", rs.SecretRefs)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
const StepDocKind = "vantage.step/v1"
|
||||
|
||||
// StepDoc is the portable, id-free representation of a step.
|
||||
type StepDoc struct {
|
||||
Kind string `json:"kind"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Interpreter string `json:"interpreter"`
|
||||
Script string `json:"script"`
|
||||
DeclaredOutputs []string `json:"declared_outputs"`
|
||||
DeclaredInputs []models.InputParam `json:"declared_inputs"`
|
||||
SecretRefs []string `json:"secret_refs"`
|
||||
}
|
||||
|
||||
// ExportStepDoc builds a portable doc from a library step (ids/source stripped).
|
||||
func ExportStepDoc(s models.WorkflowStep) StepDoc {
|
||||
return StepDoc{
|
||||
Kind: StepDocKind,
|
||||
Name: s.Name,
|
||||
Description: s.Description,
|
||||
Interpreter: s.Interpreter,
|
||||
Script: s.Script,
|
||||
DeclaredOutputs: s.DeclaredOutputs,
|
||||
DeclaredInputs: s.DeclaredInputs,
|
||||
SecretRefs: s.SecretRefs,
|
||||
}
|
||||
}
|
||||
|
||||
// ParseStepDoc validates a v1 doc and returns a normalized (id-free) step with
|
||||
// declared_outputs recomputed from the script.
|
||||
func ParseStepDoc(b []byte) (models.WorkflowStep, error) {
|
||||
var d StepDoc
|
||||
if err := json.Unmarshal(b, &d); err != nil {
|
||||
return models.WorkflowStep{}, fmt.Errorf("invalid step JSON: %w", err)
|
||||
}
|
||||
if d.Kind != StepDocKind {
|
||||
return models.WorkflowStep{}, fmt.Errorf("unsupported kind %q (want %q)", d.Kind, StepDocKind)
|
||||
}
|
||||
if d.Name == "" || d.Interpreter == "" {
|
||||
return models.WorkflowStep{}, fmt.Errorf("step name and interpreter are required")
|
||||
}
|
||||
if d.SecretRefs == nil {
|
||||
d.SecretRefs = []string{}
|
||||
}
|
||||
if d.DeclaredInputs == nil {
|
||||
d.DeclaredInputs = []models.InputParam{}
|
||||
}
|
||||
return models.WorkflowStep{
|
||||
Name: d.Name,
|
||||
Description: d.Description,
|
||||
Interpreter: d.Interpreter,
|
||||
Script: d.Script,
|
||||
DeclaredOutputs: DeriveOutputs(d.Script),
|
||||
DeclaredInputs: d.DeclaredInputs,
|
||||
SecretRefs: d.SecretRefs,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// ImportStepToLibrary parses a doc and persists it as a new user library step.
|
||||
func ImportStepToLibrary(b []byte) (*models.WorkflowStep, error) {
|
||||
s, err := ParseStepDoc(b)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return CreateStep(s)
|
||||
}
|
||||
|
||||
// ExportStep loads a library step and marshals it to a portable doc.
|
||||
func ExportStep(stepID string) ([]byte, error) {
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
s, err := getStep(ctx, stepID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return json.MarshalIndent(ExportStepDoc(*s), "", " ")
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
func mkStep() models.WorkflowStep {
|
||||
return models.WorkflowStep{
|
||||
StepID: "should-not-export", Source: "default", Name: "Restart",
|
||||
Interpreter: "bash", Script: "echo x=1 >> $WORKFLOW_ENV",
|
||||
SecretRefs: []string{"TOK"},
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStepDocValid(t *testing.T) {
|
||||
raw := `{"kind":"vantage.step/v1","name":"Restart","interpreter":"bash",
|
||||
"script":"echo x=1 >> $WORKFLOW_ENV","declared_outputs":["stale"],
|
||||
"declared_inputs":[{"name":"A","default":"1"}],"secret_refs":["TOK"]}`
|
||||
s, err := ParseStepDoc([]byte(raw))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s.Name != "Restart" || s.Interpreter != "bash" {
|
||||
t.Fatalf("bad parse: %+v", s)
|
||||
}
|
||||
// declared_outputs recomputed from script, ignoring the file's ["stale"].
|
||||
if len(s.DeclaredOutputs) != 1 || s.DeclaredOutputs[0] != "x" {
|
||||
t.Fatalf("outputs should be derived, got %v", s.DeclaredOutputs)
|
||||
}
|
||||
if s.StepID != "" || s.Source != "" {
|
||||
t.Fatalf("parse must not set id/source")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStepDocBadKind(t *testing.T) {
|
||||
if _, err := ParseStepDoc([]byte(`{"kind":"nope","name":"x"}`)); err == nil {
|
||||
t.Fatal("want error for bad kind")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStepDocBadJSON(t *testing.T) {
|
||||
if _, err := ParseStepDoc([]byte(`{`)); err == nil {
|
||||
t.Fatal("want error for bad json")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExportStepDocRoundTrip(t *testing.T) {
|
||||
doc := ExportStepDoc(mkStep())
|
||||
b, _ := json.Marshal(doc)
|
||||
s, err := ParseStepDoc(b)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s.Name != "Restart" || s.Interpreter != "bash" {
|
||||
t.Fatalf("round trip lost data: %+v", s)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// keyAssign matches an env-var assignment target: KEY= (captures KEY).
|
||||
var keyAssign = regexp.MustCompile(`([A-Za-z_][A-Za-z0-9_]*)=`)
|
||||
|
||||
// DeriveOutputs scans a step script and returns the output keys it writes to
|
||||
// $WORKFLOW_ENV. Best-effort: only lines that reference WORKFLOW_ENV are
|
||||
// considered. Deduplicated, first-seen order preserved.
|
||||
func DeriveOutputs(script string) []string {
|
||||
out := []string{}
|
||||
seen := map[string]bool{}
|
||||
for _, line := range strings.Split(script, "\n") {
|
||||
if !strings.Contains(line, "WORKFLOW_ENV") {
|
||||
continue
|
||||
}
|
||||
for _, m := range keyAssign.FindAllStringSubmatch(line, -1) {
|
||||
key := m[1]
|
||||
// Skip the sentinel itself (e.g. "WORKFLOW_ENV=..." assignments).
|
||||
if key == "WORKFLOW_ENV" || key == "env" {
|
||||
continue
|
||||
}
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, key)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var slugStrip = regexp.MustCompile(`[^a-z0-9]+`)
|
||||
|
||||
// Slugify converts a step name into a stable kebab-case slug.
|
||||
func Slugify(name string) string {
|
||||
s := strings.ToLower(name)
|
||||
s = slugStrip.ReplaceAllString(s, "-")
|
||||
return strings.Trim(s, "-")
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDeriveOutputs(t *testing.T) {
|
||||
script := `#!/bin/bash
|
||||
echo "test=123" >> $WORKFLOW_ENV
|
||||
echo "other=hi" >> "$WORKFLOW_ENV"
|
||||
printf 'third=1\n' >> $WORKFLOW_ENV
|
||||
echo "test=456" >> $WORKFLOW_ENV
|
||||
echo "ignored=nope"
|
||||
NORMAL=assignment
|
||||
`
|
||||
got := DeriveOutputs(script)
|
||||
want := []string{"test", "other", "third"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %v want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeriveOutputsPowershell(t *testing.T) {
|
||||
script := `"result=ok" >> $env:WORKFLOW_ENV
|
||||
Add-Content $env:WORKFLOW_ENV "count=5"`
|
||||
got := DeriveOutputs(script)
|
||||
want := []string{"result", "count"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %v want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeriveOutputsNone(t *testing.T) {
|
||||
got := DeriveOutputs("echo hello\nNOPE=1")
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("got %v want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlugify(t *testing.T) {
|
||||
if got := Slugify("Restart NGINX Service!"); got != "restart-nginx-service" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
// ValidateWorkflow checks each step ref sets exactly one of step_id / inline.
|
||||
func ValidateWorkflow(w models.Workflow) error {
|
||||
for i, ref := range w.Steps {
|
||||
hasLib := ref.StepID != ""
|
||||
hasInline := ref.Inline != nil
|
||||
if hasLib == hasInline {
|
||||
return fmt.Errorf("step %d: exactly one of step_id or inline must be set", i)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mrhid6/vantage/server/internal/models"
|
||||
)
|
||||
|
||||
func TestValidateWorkflow(t *testing.T) {
|
||||
inline := &models.WorkflowStep{Name: "x", Interpreter: "bash", Script: "echo hi"}
|
||||
cases := []struct {
|
||||
name string
|
||||
ref models.WorkflowStepRef
|
||||
wantErr bool
|
||||
}{
|
||||
{"library only", models.WorkflowStepRef{StepID: "abc"}, false},
|
||||
{"inline only", models.WorkflowStepRef{Inline: inline}, false},
|
||||
{"both set", models.WorkflowStepRef{StepID: "abc", Inline: inline}, true},
|
||||
{"neither set", models.WorkflowStepRef{}, true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
err := ValidateWorkflow(models.Workflow{Steps: []models.WorkflowStepRef{tc.ref}})
|
||||
if (err != nil) != tc.wantErr {
|
||||
t.Fatalf("got err=%v want wantErr=%v", err, tc.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package services
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -82,6 +83,10 @@ func resolveSteps(wf *models.Workflow) ([]models.ResolvedStep, error) {
|
||||
defer cancel()
|
||||
out := make([]models.ResolvedStep, 0, len(wf.Steps))
|
||||
for _, ref := range wf.Steps {
|
||||
if ref.Inline != nil {
|
||||
out = append(out, resolveInlineStep(ref))
|
||||
continue
|
||||
}
|
||||
lib, err := getStep(ctx, ref.StepID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -122,6 +127,35 @@ func resolveSteps(wf *models.Workflow) ([]models.ResolvedStep, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// resolveInlineStep freezes an ad-hoc (inline) step ref into a ResolvedStep.
|
||||
func resolveInlineStep(ref models.WorkflowStepRef) models.ResolvedStep {
|
||||
in := ref.Inline
|
||||
inputs := map[string]string{}
|
||||
for _, p := range in.DeclaredInputs {
|
||||
if ref.Inputs != nil {
|
||||
if v, ok := ref.Inputs[p.Name]; ok {
|
||||
inputs[p.Name] = v
|
||||
continue
|
||||
}
|
||||
}
|
||||
inputs[p.Name] = p.Default
|
||||
}
|
||||
onFailure := ref.OnFailure
|
||||
if onFailure == "" {
|
||||
onFailure = "stop"
|
||||
}
|
||||
return models.ResolvedStep{
|
||||
Order: ref.Order,
|
||||
Name: in.Name,
|
||||
Interpreter: in.Interpreter,
|
||||
Script: in.Script,
|
||||
SecretRefs: in.SecretRefs,
|
||||
OnFailure: onFailure,
|
||||
MaxRetries: ref.MaxRetries,
|
||||
Inputs: inputs,
|
||||
}
|
||||
}
|
||||
|
||||
// executeRun fans out one goroutine per server run and waits for all to finish.
|
||||
func executeRun(runID string) {
|
||||
run, err := GetRun(runID)
|
||||
@@ -188,9 +222,19 @@ func runServer(runID string, srvIdx int, steps []models.ResolvedStep, serverID s
|
||||
for k, v := range secretVals {
|
||||
allSecrets[k] = v
|
||||
}
|
||||
// Input values may template earlier step outputs and secrets, e.g.
|
||||
// URL="http://example.com/$VersionNumber". Expand against runEnv (outputs
|
||||
// threaded from prior steps) and this step's secrets before dispatch.
|
||||
subst := map[string]string{}
|
||||
for k, v := range runEnv {
|
||||
subst[k] = v
|
||||
}
|
||||
for k, v := range secretVals {
|
||||
subst[k] = v
|
||||
}
|
||||
cmdEnv := map[string]string{}
|
||||
for k, v := range step.Inputs {
|
||||
cmdEnv[k] = v
|
||||
cmdEnv[k] = expandVars(v, subst)
|
||||
}
|
||||
for k, v := range runEnv {
|
||||
cmdEnv[k] = v
|
||||
@@ -310,6 +354,18 @@ func dispatchAndWait(serverID, commandID string, cmd *pb.RunStepCmd) *pb.StepRes
|
||||
}
|
||||
}
|
||||
|
||||
// expandVars substitutes $VAR and ${VAR} references in an input value from the
|
||||
// given lookup (prior step outputs and secrets). Unknown references expand to
|
||||
// empty, matching shell behaviour; a literal "$" is written as "$$".
|
||||
func expandVars(v string, lookup map[string]string) string {
|
||||
return os.Expand(v, func(name string) string {
|
||||
if name == "$" {
|
||||
return "$"
|
||||
}
|
||||
return lookup[name]
|
||||
})
|
||||
}
|
||||
|
||||
func resolveSecrets(refs []string) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, ref := range refs {
|
||||
|
||||
@@ -25,6 +25,13 @@ func EnsureWorkflowIndexes() error {
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := db.Col("workflow_steps").Indexes().CreateOne(ctx, mongo.IndexModel{
|
||||
Keys: bson.D{{Key: "slug", Value: 1}},
|
||||
Options: options.Index().SetUnique(true).
|
||||
SetPartialFilterExpression(bson.M{"source": "default"}),
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := db.Col("workflows").Indexes().CreateOne(ctx, mongo.IndexModel{
|
||||
Keys: bson.D{{Key: "workflow_id", Value: 1}}, Options: options.Index().SetUnique(true),
|
||||
}); err != nil {
|
||||
@@ -60,8 +67,9 @@ func CreateStep(s models.WorkflowStep) (*models.WorkflowStep, error) {
|
||||
s.StepID = uuid.New().String()
|
||||
s.CreatedAt = time.Now()
|
||||
s.UpdatedAt = s.CreatedAt
|
||||
if s.DeclaredOutputs == nil {
|
||||
s.DeclaredOutputs = []string{}
|
||||
s.DeclaredOutputs = DeriveOutputs(s.Script)
|
||||
if s.Source == "" {
|
||||
s.Source = "user"
|
||||
}
|
||||
if s.SecretRefs == nil {
|
||||
s.SecretRefs = []string{}
|
||||
@@ -83,7 +91,7 @@ func UpdateStep(stepID string, s models.WorkflowStep) error {
|
||||
"description": s.Description,
|
||||
"interpreter": s.Interpreter,
|
||||
"script": s.Script,
|
||||
"declared_outputs": s.DeclaredOutputs,
|
||||
"declared_outputs": DeriveOutputs(s.Script),
|
||||
"declared_inputs": s.DeclaredInputs,
|
||||
"secret_refs": s.SecretRefs,
|
||||
"updated_at": time.Now(),
|
||||
@@ -178,6 +186,10 @@ func CreateWorkflow(w models.Workflow) (*models.Workflow, error) {
|
||||
if w.Steps == nil {
|
||||
w.Steps = []models.WorkflowStepRef{}
|
||||
}
|
||||
if err := ValidateWorkflow(w); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
normalizeInlineSteps(&w)
|
||||
if _, err := db.Col("workflows").InsertOne(ctx, w); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -187,6 +199,10 @@ func CreateWorkflow(w models.Workflow) (*models.Workflow, error) {
|
||||
func UpdateWorkflow(id string, w models.Workflow) error {
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
if err := ValidateWorkflow(w); err != nil {
|
||||
return err
|
||||
}
|
||||
normalizeInlineSteps(&w)
|
||||
_, err := db.Col("workflows").UpdateOne(ctx, bson.M{"workflow_id": id}, bson.M{"$set": bson.M{
|
||||
"name": w.Name,
|
||||
"target_server_ids": w.TargetServerIDs,
|
||||
@@ -196,6 +212,29 @@ func UpdateWorkflow(id string, w models.Workflow) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// normalizeInlineSteps derives outputs for inline steps and strips fields that
|
||||
// only belong to library steps.
|
||||
func normalizeInlineSteps(w *models.Workflow) {
|
||||
for i := range w.Steps {
|
||||
in := w.Steps[i].Inline
|
||||
if in == nil {
|
||||
continue
|
||||
}
|
||||
in.DeclaredOutputs = DeriveOutputs(in.Script)
|
||||
in.StepID = ""
|
||||
in.Slug = ""
|
||||
in.Source = ""
|
||||
in.CreatedAt = time.Time{}
|
||||
in.UpdatedAt = time.Time{}
|
||||
if in.SecretRefs == nil {
|
||||
in.SecretRefs = []string{}
|
||||
}
|
||||
if in.DeclaredInputs == nil {
|
||||
in.DeclaredInputs = []models.InputParam{}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func DeleteWorkflow(id string) error {
|
||||
ctx, cancel := wfCtx()
|
||||
defer cancel()
|
||||
|
||||
+242
-30
@@ -1,6 +1,6 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useState } from "react";
|
||||
import { useEffect, useRef, useState } from "react";
|
||||
import Link from "next/link";
|
||||
import { useParams, useRouter } from "next/navigation";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
@@ -27,6 +27,22 @@ function ShellBadge({ interpreter }: { interpreter: "bash" | "powershell" }) {
|
||||
);
|
||||
}
|
||||
|
||||
function DefaultBadge() {
|
||||
return (
|
||||
<span className="rounded px-1.5 py-0.5 font-mono text-[10px] uppercase bg-surface-2 text-text-secondary">
|
||||
default
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
function AdhocBadge() {
|
||||
return (
|
||||
<span className="rounded px-1.5 py-0.5 font-mono text-[10px] uppercase bg-signal/15 text-signal">
|
||||
ad-hoc
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
export default function WorkflowBuilder() {
|
||||
const params = useParams<{ id: string }>();
|
||||
const id = params.id;
|
||||
@@ -39,6 +55,12 @@ export default function WorkflowBuilder() {
|
||||
const [saving, setSaving] = useState(false);
|
||||
const [running, setRunning] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [notice, setNotice] = useState<string | null>(null);
|
||||
const [importing, setImporting] = useState(false);
|
||||
const [importingInline, setImportingInline] = useState(false);
|
||||
const [syncing, setSyncing] = useState(false);
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const inlineFileInputRef = useRef<HTMLInputElement>(null);
|
||||
const [groupKeys, setGroupKeys] = useState<Record<string, string[]>>({});
|
||||
const [editWorkflowOpen, setEditWorkflowOpen] = useState(false);
|
||||
const [editingStep, setEditingStep] = useState<WorkflowStep | null>(null);
|
||||
@@ -84,7 +106,7 @@ export default function WorkflowBuilder() {
|
||||
return <div className="p-8 text-text-secondary">Loading…</div>;
|
||||
}
|
||||
|
||||
const libById = (sid: string) => library?.find((l) => l.step_id === sid);
|
||||
const libById = (sid?: string) => (sid ? library?.find((l) => l.step_id === sid) : undefined);
|
||||
|
||||
const sortedSteps = [...wf.steps].sort((a, b) => a.order - b.order);
|
||||
const selectedRef = selected !== null ? sortedSteps[selected] : null;
|
||||
@@ -128,6 +150,45 @@ export default function WorkflowBuilder() {
|
||||
setWf({ ...wf, steps: resequence(next) });
|
||||
};
|
||||
|
||||
const appendRef = (ref: WorkflowStepRef) => {
|
||||
setWf({ ...wf, steps: resequence([...sortedSteps, ref]) });
|
||||
};
|
||||
|
||||
const addAdhocStep = () => {
|
||||
appendRef({
|
||||
inline: {
|
||||
step_id: "",
|
||||
name: "New ad-hoc step",
|
||||
description: "",
|
||||
interpreter: "bash",
|
||||
script: "",
|
||||
declared_outputs: [],
|
||||
declared_inputs: [],
|
||||
secret_refs: [],
|
||||
},
|
||||
order: wf.steps.length,
|
||||
on_failure: "stop",
|
||||
max_retries: 0,
|
||||
});
|
||||
};
|
||||
|
||||
const onImportInline = async (e: React.ChangeEvent<HTMLInputElement>) => {
|
||||
const file = e.target.files?.[0];
|
||||
if (!file) return;
|
||||
setImportingInline(true);
|
||||
setError(null);
|
||||
try {
|
||||
const doc = JSON.parse(await file.text());
|
||||
const step = await api.parseStep(doc);
|
||||
appendRef({ inline: step, order: wf.steps.length, on_failure: "stop", max_retries: 0 });
|
||||
} catch (err) {
|
||||
setError((err as Error).message);
|
||||
} finally {
|
||||
setImportingInline(false);
|
||||
e.target.value = "";
|
||||
}
|
||||
};
|
||||
|
||||
const moveStep = (from: number, pos: number) => {
|
||||
const next = [...sortedSteps];
|
||||
const [item] = next.splice(from, 1);
|
||||
@@ -165,6 +226,12 @@ export default function WorkflowBuilder() {
|
||||
steps: wf.steps.map((r, i) => (i === idx ? { ...r, ...patch } : r)),
|
||||
});
|
||||
|
||||
const updateInline = (idx: number, patch: Partial<WorkflowStep>) =>
|
||||
setWf({
|
||||
...wf,
|
||||
steps: wf.steps.map((r, i) => (i === idx && r.inline ? { ...r, inline: { ...r.inline, ...patch } } : r)),
|
||||
});
|
||||
|
||||
const removeStep = (idx: number) => {
|
||||
const remaining = resequence(wf.steps.filter((_, i) => i !== idx));
|
||||
setWf({ ...wf, steps: remaining });
|
||||
@@ -173,17 +240,61 @@ export default function WorkflowBuilder() {
|
||||
|
||||
const toggleSecretRef = (ref: string) => {
|
||||
if (selectedIdxInWf === -1 || !selectedRef) return;
|
||||
if (selectedRef.inline) {
|
||||
const current = selectedRef.inline.secret_refs ?? [];
|
||||
const next = current.includes(ref) ? current.filter((r) => r !== ref) : [...current, ref];
|
||||
updateInline(selectedIdxInWf, { secret_refs: next });
|
||||
return;
|
||||
}
|
||||
const current = selectedRef.overrides?.secret_refs ?? [];
|
||||
const next = current.includes(ref) ? current.filter((r) => r !== ref) : [...current, ref];
|
||||
updateRef(selectedIdxInWf, { overrides: { ...selectedRef.overrides, secret_refs: next } });
|
||||
};
|
||||
|
||||
const onImportFile = async (e: React.ChangeEvent<HTMLInputElement>) => {
|
||||
const file = e.target.files?.[0];
|
||||
if (!file) return;
|
||||
setImporting(true);
|
||||
setError(null);
|
||||
try {
|
||||
const doc = JSON.parse(await file.text());
|
||||
await api.importStep(doc);
|
||||
await queryClient.invalidateQueries({ queryKey: ["steps"] });
|
||||
setNotice("Step imported.");
|
||||
} catch (err) {
|
||||
setError((err as Error).message);
|
||||
} finally {
|
||||
setImporting(false);
|
||||
e.target.value = "";
|
||||
}
|
||||
};
|
||||
|
||||
const onSyncDefaults = async () => {
|
||||
setSyncing(true);
|
||||
setError(null);
|
||||
try {
|
||||
const { created, updated } = await api.seedDefaults();
|
||||
await queryClient.invalidateQueries({ queryKey: ["steps"] });
|
||||
setNotice(`${created} created, ${updated} updated`);
|
||||
} catch (err) {
|
||||
setError((err as Error).message);
|
||||
} finally {
|
||||
setSyncing(false);
|
||||
}
|
||||
};
|
||||
|
||||
const filteredLibrary = (library ?? []).filter((s) => s.name.toLowerCase().includes(search.toLowerCase()));
|
||||
const bashSteps = filteredLibrary.filter((s) => s.interpreter === "bash");
|
||||
const pwshSteps = filteredLibrary.filter((s) => s.interpreter === "powershell");
|
||||
|
||||
const upstreamOutputsFor = (i: number) =>
|
||||
Array.from(new Set(sortedSteps.slice(0, i).flatMap((r) => libById(r.step_id)?.declared_outputs ?? [])));
|
||||
Array.from(
|
||||
new Set(
|
||||
sortedSteps
|
||||
.slice(0, i)
|
||||
.flatMap((r) => r.inline?.declared_outputs ?? libById(r.step_id)?.declared_outputs ?? []),
|
||||
),
|
||||
);
|
||||
|
||||
const DropZone = ({ pos }: { pos: number }) => (
|
||||
<div
|
||||
@@ -236,6 +347,9 @@ export default function WorkflowBuilder() {
|
||||
{error && (
|
||||
<div className="border-b border-danger/30 bg-danger/10 px-4 py-2 text-sm text-danger">{error}</div>
|
||||
)}
|
||||
{notice && (
|
||||
<div className="border-b border-signal/30 bg-signal/10 px-4 py-2 text-sm text-signal">{notice}</div>
|
||||
)}
|
||||
|
||||
<div className="grid h-[calc(100vh-53px)] grid-cols-[264px_1fr_320px]">
|
||||
{/* LEFT: library */}
|
||||
@@ -253,6 +367,46 @@ export default function WorkflowBuilder() {
|
||||
+
|
||||
</Button>
|
||||
</div>
|
||||
<div className="mb-2 flex items-center gap-2">
|
||||
<input
|
||||
ref={fileInputRef}
|
||||
type="file"
|
||||
accept="application/json"
|
||||
className="hidden"
|
||||
onChange={onImportFile}
|
||||
/>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
loading={importing}
|
||||
onClick={() => fileInputRef.current?.click()}
|
||||
>
|
||||
Import
|
||||
</Button>
|
||||
<Button variant="ghost" size="sm" loading={syncing} onClick={onSyncDefaults}>
|
||||
Sync defaults
|
||||
</Button>
|
||||
</div>
|
||||
<div className="mb-3 flex items-center gap-2">
|
||||
<input
|
||||
ref={inlineFileInputRef}
|
||||
type="file"
|
||||
accept="application/json"
|
||||
className="hidden"
|
||||
onChange={onImportInline}
|
||||
/>
|
||||
<Button variant="ghost" size="sm" onClick={addAdhocStep}>
|
||||
+ Add ad-hoc step
|
||||
</Button>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
loading={importingInline}
|
||||
onClick={() => inlineFileInputRef.current?.click()}
|
||||
>
|
||||
Import ad-hoc
|
||||
</Button>
|
||||
</div>
|
||||
<input
|
||||
className={`${inputClass} mb-3`}
|
||||
placeholder="Search steps…"
|
||||
@@ -300,7 +454,7 @@ export default function WorkflowBuilder() {
|
||||
{sortedSteps.map((ref, i) => {
|
||||
const lib = libById(ref.step_id);
|
||||
const outs = upstreamOutputsFor(i);
|
||||
const script = ref.overrides?.script ?? lib?.script ?? "";
|
||||
const script = ref.inline?.script ?? ref.overrides?.script ?? lib?.script ?? "";
|
||||
const wfIdx = wf.steps.indexOf(ref);
|
||||
const isSelected = selected === i;
|
||||
return (
|
||||
@@ -338,8 +492,10 @@ export default function WorkflowBuilder() {
|
||||
<span className="grid h-5 w-5 place-items-center rounded border border-border font-mono text-[10px] text-text-secondary">
|
||||
{i + 1}
|
||||
</span>
|
||||
<span className="text-sm font-medium text-text-primary">{lib?.name ?? ref.step_id}</span>
|
||||
{lib && <ShellBadge interpreter={lib.interpreter} />}
|
||||
<span className="text-sm font-medium text-text-primary">{ref.inline?.name ?? lib?.name ?? ref.step_id}</span>
|
||||
{ref.inline && <ShellBadge interpreter={ref.inline.interpreter} />}
|
||||
{lib && !ref.inline && <ShellBadge interpreter={lib.interpreter} />}
|
||||
{ref.inline && <AdhocBadge />}
|
||||
</div>
|
||||
<pre className="max-h-16 overflow-hidden text-ellipsis whitespace-pre-wrap rounded border border-border bg-surface-2 p-2 font-mono text-xs text-text-secondary">
|
||||
{script.slice(0, 200)}
|
||||
@@ -372,33 +528,75 @@ export default function WorkflowBuilder() {
|
||||
Step {selected + 1} · Inspector
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
{selectedLib && <ShellBadge interpreter={selectedLib.interpreter} />}
|
||||
<h2 className="text-sm font-bold text-text-primary">{selectedLib?.name ?? selectedRef.step_id}</h2>
|
||||
{selectedRef.inline && <ShellBadge interpreter={selectedRef.inline.interpreter} />}
|
||||
{selectedLib && !selectedRef.inline && <ShellBadge interpreter={selectedLib.interpreter} />}
|
||||
{selectedRef.inline && <AdhocBadge />}
|
||||
<h2 className="text-sm font-bold text-text-primary">{selectedRef.inline?.name ?? selectedLib?.name ?? selectedRef.step_id}</h2>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="border-b border-border pb-4">
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Command</label>
|
||||
<textarea
|
||||
className={`${inputClass} h-32 font-mono text-xs`}
|
||||
value={selectedRef.overrides?.script ?? selectedLib?.script ?? ""}
|
||||
onChange={(e) =>
|
||||
updateRef(selectedIdxInWf, {
|
||||
overrides: { ...selectedRef.overrides, script: e.target.value },
|
||||
})
|
||||
}
|
||||
/>
|
||||
<p className="mt-1 text-xs text-text-secondary">
|
||||
Write <code className="text-signal">KEY=value</code> to <code className="text-signal">$WORKFLOW_ENV</code> to expose
|
||||
it to later steps.
|
||||
</p>
|
||||
</div>
|
||||
{selectedRef.inline ? (
|
||||
<div className="space-y-4 border-b border-border pb-4">
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Name</label>
|
||||
<input
|
||||
className={inputClass}
|
||||
value={selectedRef.inline.name}
|
||||
onChange={(e) => updateInline(selectedIdxInWf, { name: e.target.value })}
|
||||
/>
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Interpreter</label>
|
||||
<select
|
||||
className={inputClass}
|
||||
value={selectedRef.inline.interpreter}
|
||||
onChange={(e) =>
|
||||
updateInline(selectedIdxInWf, {
|
||||
interpreter: e.target.value as WorkflowStep["interpreter"],
|
||||
})
|
||||
}
|
||||
>
|
||||
<option value="bash">bash</option>
|
||||
<option value="powershell">powershell</option>
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Command</label>
|
||||
<textarea
|
||||
className={`${inputClass} h-32 font-mono text-xs`}
|
||||
value={selectedRef.inline.script}
|
||||
onChange={(e) => updateInline(selectedIdxInWf, { script: e.target.value })}
|
||||
/>
|
||||
<p className="mt-1 text-xs text-text-secondary">
|
||||
Write <code className="text-signal">KEY=value</code> to <code className="text-signal">$WORKFLOW_ENV</code> to
|
||||
expose it to later steps. Outputs are derived automatically on save.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div className="border-b border-border pb-4">
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Command</label>
|
||||
<textarea
|
||||
className={`${inputClass} h-32 font-mono text-xs`}
|
||||
value={selectedRef.overrides?.script ?? selectedLib?.script ?? ""}
|
||||
onChange={(e) =>
|
||||
updateRef(selectedIdxInWf, {
|
||||
overrides: { ...selectedRef.overrides, script: e.target.value },
|
||||
})
|
||||
}
|
||||
/>
|
||||
<p className="mt-1 text-xs text-text-secondary">
|
||||
Write <code className="text-signal">KEY=value</code> to <code className="text-signal">$WORKFLOW_ENV</code> to expose
|
||||
it to later steps.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{(selectedLib?.declared_inputs ?? []).length > 0 && (
|
||||
{(selectedRef.inline?.declared_inputs ?? selectedLib?.declared_inputs ?? []).length > 0 && (
|
||||
<div className="border-b border-border pb-4">
|
||||
<label className="mb-2 block text-xs uppercase text-text-secondary">Inputs</label>
|
||||
<div className="space-y-2">
|
||||
{selectedLib?.declared_inputs.map((param) => (
|
||||
{(selectedRef.inline?.declared_inputs ?? selectedLib?.declared_inputs ?? []).map((param) => (
|
||||
<div key={param.name}>
|
||||
<div className="mb-1 font-mono text-xs text-text-primary">{param.name}</div>
|
||||
{param.description && (
|
||||
@@ -438,10 +636,10 @@ export default function WorkflowBuilder() {
|
||||
<div className="border-b border-border pb-4">
|
||||
<label className="mb-2 block text-xs uppercase text-text-secondary">Outputs · to $WORKFLOW_ENV</label>
|
||||
<div className="flex flex-wrap gap-1">
|
||||
{(selectedLib?.declared_outputs ?? []).length === 0 && (
|
||||
{(selectedRef.inline?.declared_outputs ?? selectedLib?.declared_outputs ?? []).length === 0 && (
|
||||
<p className="text-xs text-text-secondary">No declared outputs.</p>
|
||||
)}
|
||||
{(selectedLib?.declared_outputs ?? []).map((o) => (
|
||||
{(selectedRef.inline?.declared_outputs ?? selectedLib?.declared_outputs ?? []).map((o) => (
|
||||
<span key={o} className="flex items-center gap-1 rounded bg-signal px-2 py-0.5 font-mono text-[11px] text-signal-ink">
|
||||
<span className="text-[9px] uppercase">out</span>
|
||||
{o}
|
||||
@@ -458,7 +656,11 @@ export default function WorkflowBuilder() {
|
||||
<div className="font-mono text-[11px] font-semibold text-text-secondary">{g.group}</div>
|
||||
{(groupKeys[g.group] ?? []).map((key) => {
|
||||
const ref = `${g.group}/${key}`;
|
||||
const checked = (selectedRef.overrides?.secret_refs ?? []).includes(ref);
|
||||
const checked = (
|
||||
selectedRef.inline
|
||||
? (selectedRef.inline.secret_refs ?? [])
|
||||
: (selectedRef.overrides?.secret_refs ?? [])
|
||||
).includes(ref);
|
||||
return (
|
||||
<label key={ref} className="ml-2 flex cursor-pointer items-center gap-2 text-xs text-text-primary">
|
||||
<input
|
||||
@@ -542,13 +744,23 @@ function LibraryCard({ step, onAdd, onEdit }: { step: WorkflowStep; onAdd: () =>
|
||||
<div className="mb-1 flex items-center gap-2">
|
||||
<span className="text-text-secondary">⠿</span>
|
||||
<ShellBadge interpreter={step.interpreter} />
|
||||
{step.source === "default" && <DefaultBadge />}
|
||||
<span className="text-sm font-medium text-text-primary">{step.name}</span>
|
||||
<a
|
||||
href={api.exportStepUrl(step.step_id)}
|
||||
download
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
className="ml-auto hidden text-text-secondary hover:text-text-primary group-hover:block"
|
||||
title="Export step"
|
||||
>
|
||||
⬇
|
||||
</a>
|
||||
<button
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
onEdit();
|
||||
}}
|
||||
className="ml-auto hidden text-text-secondary hover:text-text-primary group-hover:block"
|
||||
className="hidden text-text-secondary hover:text-text-primary group-hover:block"
|
||||
title="Edit step"
|
||||
>
|
||||
✎
|
||||
|
||||
@@ -13,9 +13,7 @@ export function EditStepModal({ open, step, onClose }: { open: boolean; step: Wo
|
||||
const [name, setName] = useState(step?.name ?? "");
|
||||
const [interpreter, setInterpreter] = useState<"bash" | "powershell">(step?.interpreter ?? "bash");
|
||||
const [script, setScript] = useState(step?.script ?? "");
|
||||
const [outputs, setOutputs] = useState<string[]>(step?.declared_outputs ?? []);
|
||||
const [inputs, setInputs] = useState<InputParam[]>(step?.declared_inputs ?? []);
|
||||
const [newOut, setNewOut] = useState("");
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
@@ -27,7 +25,7 @@ export function EditStepModal({ open, step, onClose }: { open: boolean; step: Wo
|
||||
try {
|
||||
const payload: Partial<WorkflowStep> = {
|
||||
name: name.trim(), description: step?.description ?? "", interpreter, script,
|
||||
declared_outputs: outputs, declared_inputs: inputs.filter((i) => i.name.trim() !== ""),
|
||||
declared_inputs: inputs.filter((i) => i.name.trim() !== ""),
|
||||
secret_refs: step?.secret_refs ?? [],
|
||||
};
|
||||
if (step) await api.updateStep(step.step_id, payload);
|
||||
@@ -71,17 +69,17 @@ export function EditStepModal({ open, step, onClose }: { open: boolean; step: Wo
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Outputs</label>
|
||||
<div className="mb-2 flex flex-wrap gap-1">
|
||||
{outputs.map((o) => (
|
||||
<div className="mb-1 flex flex-wrap gap-1">
|
||||
{(step?.declared_outputs ?? []).length === 0 && (
|
||||
<p className="text-xs text-text-secondary">No declared outputs.</p>
|
||||
)}
|
||||
{(step?.declared_outputs ?? []).map((o) => (
|
||||
<span key={o} className="flex items-center gap-1 rounded bg-signal px-2 py-0.5 font-mono text-[11px] text-signal-ink">
|
||||
{o}<button onClick={() => setOutputs(outputs.filter((x) => x !== o))}>✕</button>
|
||||
{o}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<input className={inputClass} placeholder="OUTPUT_NAME" value={newOut} onChange={(e) => setNewOut(e.target.value)} />
|
||||
<Button variant="ghost" size="sm" onClick={() => { if (newOut.trim()) { setOutputs([...outputs, newOut.trim()]); setNewOut(""); } }}>Add</Button>
|
||||
</div>
|
||||
<p className="text-xs text-text-secondary">Outputs are detected automatically from lines writing to $WORKFLOW_ENV.</p>
|
||||
</div>
|
||||
<div>
|
||||
<label className="mb-1 block text-xs uppercase text-text-secondary">Inputs</label>
|
||||
|
||||
+28
-1
@@ -148,10 +148,13 @@ export interface WorkflowStep {
|
||||
declared_outputs: string[];
|
||||
declared_inputs: InputParam[];
|
||||
secret_refs: string[];
|
||||
source?: "user" | "default";
|
||||
slug?: string;
|
||||
}
|
||||
|
||||
export interface WorkflowStepRef {
|
||||
step_id: string;
|
||||
step_id?: string;
|
||||
inline?: WorkflowStep;
|
||||
order: number;
|
||||
on_failure: "stop" | "continue" | "retry";
|
||||
max_retries: number;
|
||||
@@ -419,6 +422,30 @@ export const api = {
|
||||
return request<void>(`/steps/${stepId}`, { method: "DELETE" });
|
||||
},
|
||||
|
||||
exportStepUrl(stepId: string): string {
|
||||
return `/api/steps/${stepId}/export`;
|
||||
},
|
||||
|
||||
importStep(doc: unknown): Promise<WorkflowStep> {
|
||||
return request<WorkflowStep>("/steps/import", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(doc),
|
||||
});
|
||||
},
|
||||
|
||||
parseStep(doc: unknown): Promise<WorkflowStep> {
|
||||
return request<WorkflowStep>("/steps/parse", {
|
||||
method: "POST",
|
||||
body: JSON.stringify(doc),
|
||||
});
|
||||
},
|
||||
|
||||
seedDefaults(): Promise<{ created: number; updated: number }> {
|
||||
return request<{ created: number; updated: number }>("/steps/seed-defaults", {
|
||||
method: "POST",
|
||||
});
|
||||
},
|
||||
|
||||
// Workflows
|
||||
listWorkflows(): Promise<Workflow[]> {
|
||||
return request<Workflow[]>("/workflows");
|
||||
|
||||
Reference in New Issue
Block a user