vantage/custom-secrets-vault-gin-guide.md

Custom Secrets Vault + ESO Webhook Integration

Self-hosted secrets API at https://keymanager.hostxtra.co.uk/secrets backed by MongoDB with Gin, exposed to K3s via ESO's Webhook provider.

---

Architecture

MongoDB (encrypted at rest)
  └── secrets collection: { group, key, encryptedValue, updatedAt }
        ↑ CRUD via Go API (Gin)
Go API (keymanager secrets service)
  └── GET    /secrets/:group        ← ESO webhook calls this
  └── PUT    /secrets/:group        ← admin writes secrets to a group
  └── DELETE /secrets/:group        ← admin deletes entire group
  └── DELETE /secrets/:group/:key   ← admin deletes one key from a group
        ↓ bearer token auth (read token for ESO, admin token for writes)
ESO Webhook ClusterSecretStore
  └── ExternalSecret (per namespace)
        └── K8s Secret
              └── Deployment env vars

Data model: A "group" is a logical namespace for a set of related secrets — e.g. myapp-prod, postgres, infra. Each group contains one or more key/value pairs stored individually as encrypted documents in MongoDB.

Encryption: AES-256-GCM per value, random nonce per write, master key loaded from the MASTER_KEY environment variable (32-byte hex string).

---

Part 1 — The Go API

1.1 — Project structure

keymanager/
├── cmd/
│   └── server/
│       └── main.go
├── internal/
│   ├── crypto/
│   │   └── crypto.go
│   ├── store/
│   │   └── store.go
│   └── api/
│       └── api.go
├── go.mod
└── Dockerfile

1.2 — go.mod

module github.com/yourusername/keymanager

go 1.23

require (
    go.mongodb.org/mongo-driver v1.17.0
    github.com/gin-gonic/gin v1.10.0
)

1.3 — internal/crypto/crypto.go

AES-256-GCM encryption. Each value gets a unique random nonce so identical plaintext values produce different ciphertext on every write.

package crypto

import (
	"crypto/aes"
	"crypto/cipher"
	"crypto/rand"
	"encoding/base64"
	"errors"
	"io"
)

// Encrypt encrypts plaintext using AES-256-GCM.
// Returns base64(nonce + ciphertext).
func Encrypt(key []byte, plaintext string) (string, error) {
	block, err := aes.NewCipher(key)
	if err != nil {
		return "", err
	}
	gcm, err := cipher.NewGCM(block)
	if err != nil {
		return "", err
	}
	nonce := make([]byte, gcm.NonceSize())
	if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
		return "", err
	}
	ciphertext := gcm.Seal(nonce, nonce, []byte(plaintext), nil)
	return base64.StdEncoding.EncodeToString(ciphertext), nil
}

// Decrypt decrypts a base64(nonce + ciphertext) produced by Encrypt.
func Decrypt(key []byte, encoded string) (string, error) {
	data, err := base64.StdEncoding.DecodeString(encoded)
	if err != nil {
		return "", err
	}
	block, err := aes.NewCipher(key)
	if err != nil {
		return "", err
	}
	gcm, err := cipher.NewGCM(block)
	if err != nil {
		return "", err
	}
	if len(data) < gcm.NonceSize() {
		return "", errors.New("ciphertext too short")
	}
	nonce, ciphertext := data[:gcm.NonceSize()], data[gcm.NonceSize():]
	plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
	if err != nil {
		return "", err
	}
	return string(plaintext), nil
}

1.4 — internal/store/store.go

MongoDB storage. Each document represents one key within a group.

package store

import (
	"context"
	"time"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)

type SecretDoc struct {
	Group          string    `bson:"group"`
	Key            string    `bson:"key"`
	EncryptedValue string    `bson:"encryptedValue"`
	UpdatedAt      time.Time `bson:"updatedAt"`
}

type Store struct {
	col *mongo.Collection
}

func New(client *mongo.Client, dbName string) (*Store, error) {
	col := client.Database(dbName).Collection("secrets")

	// Unique compound index on (group, key)
	_, err := col.Indexes().CreateOne(context.Background(), mongo.IndexModel{
		Keys:    bson.D{{Key: "group", Value: 1}, {Key: "key", Value: 1}},
		Options: options.Index().SetUnique(true),
	})
	if err != nil {
		return nil, err
	}

	return &Store{col: col}, nil
}

// GetGroup returns all SecretDocs belonging to the given group.
func (s *Store) GetGroup(ctx context.Context, group string) ([]SecretDoc, error) {
	cursor, err := s.col.Find(ctx, bson.M{"group": group})
	if err != nil {
		return nil, err
	}
	var docs []SecretDoc
	if err := cursor.All(ctx, &docs); err != nil {
		return nil, err
	}
	return docs, nil
}

// Upsert writes or updates a single key within a group.
func (s *Store) Upsert(ctx context.Context, group, key, encryptedValue string) error {
	filter := bson.M{"group": group, "key": key}
	update := bson.M{"$set": bson.M{
		"encryptedValue": encryptedValue,
		"updatedAt":      time.Now(),
	}}
	_, err := s.col.UpdateOne(ctx, filter, update, options.Update().SetUpsert(true))
	return err
}

// DeleteGroup removes all keys belonging to a group.
func (s *Store) DeleteGroup(ctx context.Context, group string) error {
	_, err := s.col.DeleteMany(ctx, bson.M{"group": group})
	return err
}

// DeleteKey removes a single key from a group.
func (s *Store) DeleteKey(ctx context.Context, group, key string) error {
	_, err := s.col.DeleteOne(ctx, bson.M{"group": group, "key": key})
	return err
}

1.5 — internal/api/api.go

Gin handlers. Two token tiers: ESO gets a read-only token, admins get a write token.

package api

import (
	"encoding/hex"
	"net/http"
	"os"

	"github.com/gin-gonic/gin"
	"github.com/yourusername/keymanager/internal/crypto"
	"github.com/yourusername/keymanager/internal/store"
)

type API struct {
	store      *store.Store
	masterKey  []byte
	esoToken   string
	adminToken string
}

func New(s *store.Store) *API {
	keyHex := os.Getenv("MASTER_KEY")
	key, err := hex.DecodeString(keyHex)
	if err != nil || len(key) != 32 {
		panic("MASTER_KEY must be a 64-character hex string (32 bytes)")
	}
	return &API{
		store:      s,
		masterKey:  key,
		esoToken:   os.Getenv("ESO_TOKEN"),
		adminToken: os.Getenv("ADMIN_TOKEN"),
	}
}

func (a *API) RegisterRoutes(r *gin.Engine) {
	secrets := r.Group("/secrets")

	// Read routes — ESO token
	secrets.GET("/:group", a.bearerAuth(a.esoToken), a.getGroup)

	// Write routes — admin token
	secrets.PUT("/:group", a.bearerAuth(a.adminToken), a.putGroup)
	secrets.DELETE("/:group", a.bearerAuth(a.adminToken), a.deleteGroup)
	secrets.DELETE("/:group/:key", a.bearerAuth(a.adminToken), a.deleteKey)
}

// bearerAuth returns a Gin middleware that validates a Bearer token.
func (a *API) bearerAuth(expected string) gin.HandlerFunc {
	return func(c *gin.Context) {
		auth := c.GetHeader("Authorization")
		const prefix = "Bearer "
		if len(auth) <= len(prefix) || auth[:len(prefix)] != prefix {
			c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "missing bearer token"})
			return
		}
		token := auth[len(prefix):]
		if token != expected {
			c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "invalid token"})
			return
		}
		c.Next()
	}
}

// getGroup handles GET /secrets/:group
// Returns a flat JSON object { "KEY": "value", ... } for ESO to consume.
// Returns 404 if the group has no secrets — ESO treats 404 as "deleted".
func (a *API) getGroup(c *gin.Context) {
	group := c.Param("group")

	docs, err := a.store.GetGroup(c.Request.Context(), group)
	if err != nil {
		c.JSON(http.StatusInternalServerError, gin.H{"error": "store error"})
		return
	}
	if len(docs) == 0 {
		c.JSON(http.StatusNotFound, gin.H{"error": "group not found"})
		return
	}

	result := make(map[string]string, len(docs))
	for _, doc := range docs {
		val, err := crypto.Decrypt(a.masterKey, doc.EncryptedValue)
		if err != nil {
			c.JSON(http.StatusInternalServerError, gin.H{"error": "decrypt error"})
			return
		}
		result[doc.Key] = val
	}

	c.JSON(http.StatusOK, result)
}

// putGroup handles PUT /secrets/:group
// Body: { "KEY": "value", ... } — upserts each key in the group.
func (a *API) putGroup(c *gin.Context) {
	group := c.Param("group")

	var payload map[string]string
	if err := c.ShouldBindJSON(&payload); err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": "invalid JSON body"})
		return
	}
	if len(payload) == 0 {
		c.JSON(http.StatusBadRequest, gin.H{"error": "body must contain at least one key"})
		return
	}

	for key, val := range payload {
		encrypted, err := crypto.Encrypt(a.masterKey, val)
		if err != nil {
			c.JSON(http.StatusInternalServerError, gin.H{"error": "encrypt error"})
			return
		}
		if err := a.store.Upsert(c.Request.Context(), group, key, encrypted); err != nil {
			c.JSON(http.StatusInternalServerError, gin.H{"error": "store error"})
			return
		}
	}

	c.Status(http.StatusNoContent)
}

// deleteGroup handles DELETE /secrets/:group
// Removes the entire group and all its keys.
func (a *API) deleteGroup(c *gin.Context) {
	group := c.Param("group")

	if err := a.store.DeleteGroup(c.Request.Context(), group); err != nil {
		c.JSON(http.StatusInternalServerError, gin.H{"error": "store error"})
		return
	}
	c.Status(http.StatusNoContent)
}

// deleteKey handles DELETE /secrets/:group/:key
// Removes a single key from a group.
func (a *API) deleteKey(c *gin.Context) {
	group := c.Param("group")
	key := c.Param("key")

	if err := a.store.DeleteKey(c.Request.Context(), group, key); err != nil {
		c.JSON(http.StatusInternalServerError, gin.H{"error": "store error"})
		return
	}
	c.Status(http.StatusNoContent)
}

1.6 — cmd/server/main.go

package main

import (
	"context"
	"log"
	"os"
	"time"

	"github.com/gin-gonic/gin"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"github.com/yourusername/keymanager/internal/api"
	"github.com/yourusername/keymanager/internal/store"
)

func main() {
	mongoURI := os.Getenv("MONGODB_URI")
	if mongoURI == "" {
		mongoURI = "mongodb://localhost:27017"
	}

	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	client, err := mongo.Connect(ctx, options.Client().ApplyURI(mongoURI))
	if err != nil {
		log.Fatalf("MongoDB connect: %v", err)
	}
	if err := client.Ping(ctx, nil); err != nil {
		log.Fatalf("MongoDB ping: %v", err)
	}
	log.Println("Connected to MongoDB")

	s, err := store.New(client, "keymanager")
	if err != nil {
		log.Fatalf("Store init: %v", err)
	}

	gin.SetMode(gin.ReleaseMode)
	r := gin.New()
	r.Use(gin.Logger(), gin.Recovery())

	a := api.New(s)
	a.RegisterRoutes(r)

	log.Println("Secrets API listening on :8080")
	if err := r.Run(":8080"); err != nil {
		log.Fatalf("Server error: %v", err)
	}
}

1.7 — Dockerfile

FROM golang:1.23-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o secrets-api ./cmd/server

FROM alpine:3.20
RUN apk add --no-cache ca-certificates
WORKDIR /app
COPY --from=builder /app/secrets-api .
EXPOSE 8080
CMD ["./secrets-api"]

---

Part 2 — Deploying the API

2.1 — Generate your secrets

# 32-byte master key — back this up in your password manager
openssl rand -hex 32

# ESO read token
openssl rand -hex 32

# Admin token
openssl rand -hex 32

2.2 — Docker Compose

services:
  secrets-api:
    image: ghcr.io/yourusername/keymanager-secrets:latest
    restart: unless-stopped
    environment:
      MONGODB_URI: mongodb://mongo:27017
      MASTER_KEY: "<your-32-byte-hex-key>"
      ESO_TOKEN: "<your-eso-token>"
      ADMIN_TOKEN: "<your-admin-token>"
    ports:
      - "127.0.0.1:8082:8080"

2.3 — Caddyfile

keymanager.hostxtra.co.uk {
    # ... existing KeyManager routes ...

    handle /secrets* {
        reverse_proxy localhost:8082
    }
}

caddy reload --config /etc/caddy/Caddyfile

2.4 — Smoke test

export ADMIN="<your-admin-token>"
export ESO="<your-eso-token>"
export BASE="https://keymanager.hostxtra.co.uk/secrets"

# Write a group
curl -s -X PUT $BASE/myapp-prod \
  -H "Authorization: Bearer $ADMIN" \
  -H "Content-Type: application/json" \
  -d '{"DB_PASSWORD": "supersecret123", "API_KEY": "myapikey456"}'
# → 204 No Content

# Read back (as ESO would)
curl -s $BASE/myapp-prod \
  -H "Authorization: Bearer $ESO"
# → {"API_KEY":"myapikey456","DB_PASSWORD":"supersecret123"}

# Delete a single key
curl -s -X DELETE $BASE/myapp-prod/API_KEY \
  -H "Authorization: Bearer $ADMIN"
# → 204 No Content

# Confirm it's gone
curl -s $BASE/myapp-prod \
  -H "Authorization: Bearer $ESO"
# → {"DB_PASSWORD":"supersecret123"}

# Delete the whole group
curl -s -X DELETE $BASE/myapp-prod \
  -H "Authorization: Bearer $ADMIN"
# → 204 No Content

# Confirm 404
curl -s -o /dev/null -w "%{http_code}" $BASE/myapp-prod \
  -H "Authorization: Bearer $ESO"
# → 404

---

Part 3 — ESO Webhook Integration

3.1 — Install ESO

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true \
  --wait

3.2 — Store ESO token as a K8s Secret

The external-secrets.io/type=webhook label is required — without it the webhook provider is not permitted to read the secret.

kubectl create secret generic keymanager-eso-token \
  --namespace external-secrets \
  --from-literal=token="<your-eso-token>"

kubectl label secret keymanager-eso-token \
  --namespace external-secrets \
  external-secrets.io/type=webhook

3.3 — ClusterSecretStore

# cluster-secret-store.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: keymanager-store
spec:
  provider:
    webhook:
      url: "https://keymanager.hostxtra.co.uk/secrets/{{ .remoteRef.key }}"
      method: GET
      result:
        jsonPath: "$"
      headers:
        Content-Type: "application/json"
        Authorization: "Bearer {{ .auth.token }}"
      secrets:
        - name: auth
          secretRef:
            name: keymanager-eso-token
            namespace: external-secrets

kubectl apply -f cluster-secret-store.yaml
kubectl get clustersecretstore keymanager-store

3.4 — ExternalSecret

The remoteRef.key value is the group name — ESO substitutes it into the URL template, calling GET /secrets/myapp-prod.

# external-secret.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myapp
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: myapp
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: keymanager-store
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: myapp-prod        # group name → GET /secrets/myapp-prod

Or to pull specific keys from a group:

  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: myapp-prod        # group name
        property: DB_PASSWORD  # key within the group's JSON response

kubectl apply -f external-secret.yaml

kubectl get externalsecret myapp-secrets -n myapp
# STATUS: SecretSynced

# Decode and verify values
kubectl get secret myapp-secrets -n myapp -o json | \
  jq '.data | map_values(@base64d)'

3.5 — Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        reloader.stakater.com/auto: "true"   # optional: auto-restart on secret change
    spec:
      containers:
        - name: myapp
          image: your-image:latest
          envFrom:
            - secretRef:
                name: myapp-secrets

---

Part 4 — Day-to-Day Secret Management

export ADMIN="<your-admin-token>"
export BASE="https://keymanager.hostxtra.co.uk/secrets"

# Create or update a group (upserts — safe to re-run)
curl -s -X PUT $BASE/postgres \
  -H "Authorization: Bearer $ADMIN" \
  -H "Content-Type: application/json" \
  -d '{"POSTGRES_PASSWORD": "dbpass", "POSTGRES_USER": "app"}'

# Add a new key to an existing group (existing keys are untouched)
curl -s -X PUT $BASE/postgres \
  -H "Authorization: Bearer $ADMIN" \
  -H "Content-Type: application/json" \
  -d '{"POSTGRES_DB": "mydb"}'

# Remove a single key from a group
curl -s -X DELETE $BASE/postgres/POSTGRES_USER \
  -H "Authorization: Bearer $ADMIN"

# Remove an entire group
curl -s -X DELETE $BASE/postgres \
  -H "Authorization: Bearer $ADMIN"

# Force ESO to re-sync immediately after a rotation
kubectl annotate externalsecret myapp-secrets -n myapp \
  force-sync=$(date +%s) --overwrite

---

Quick Reference

Endpoint	Token	Description
GET /secrets/:group	ESO token	Returns all keys in group as JSON
PUT /secrets/:group	Admin token	Upserts keys into group
DELETE /secrets/:group	Admin token	Deletes entire group
DELETE /secrets/:group/:key	Admin token	Deletes one key from group