Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b5b9775d2b | ||
|
|
ecd703d502 | ||
|
|
db75d7dcbd | ||
|
|
a4745834de | ||
|
|
51b58fef21 | ||
|
|
5f5e19e23e | ||
|
|
3aa24bb938 | ||
|
|
b777ffcf58 | ||
|
|
1ad46dfda2 |
@@ -30,24 +30,24 @@ vantage-agent/
|
||||
| Repository | Relationship |
|
||||
| ---------------- | ------------------------------------------------------------------------------------------------------ |
|
||||
| `vantage-shared` | a private Go module. `grpc/pb` and `grpc/codec` are the wire contract; `proto/` there documents them |
|
||||
| `vantage` | the control plane. **No import in either direction** — the coupling is the gRPC wire, and it is entirely mediated by `vantage-shared` |
|
||||
| `vantage` | the control plane. **No import in either direction** - the coupling is the gRPC wire, and it is entirely mediated by `vantage-shared` |
|
||||
|
||||
**`vantage-shared` is private**, so every Go build needs
|
||||
`GOPRIVATE=gitea.hostxtra.co.uk/*` plus a credential. CI writes a netrc from
|
||||
`REGISTRY_USER` + `RELEASE_TOKEN` (**that token needs read access to the
|
||||
`vantage` org**) — twice, because the `msi` job is Windows and Go looks for
|
||||
`vantage` org**) - twice, because the `msi` job is Windows and Go looks for
|
||||
`_netrc` in the profile directory there, not `.netrc`. Locally, either a netrc
|
||||
or `git config --global url."git@gitea.hostxtra.co.uk:".insteadOf https://gitea.hostxtra.co.uk/`.
|
||||
|
||||
### A wire change is three steps, in order
|
||||
|
||||
`shared/grpc/pb` is hand-written and shared by both sides, so a new message is a
|
||||
compile error rather than a silent disagreement — but only once each side moves:
|
||||
compile error rather than a silent disagreement - but only once each side moves:
|
||||
|
||||
1. release `vantage-shared` (and change `proto/vantage/v1/vantage.proto` in the
|
||||
same commit as the Go types)
|
||||
2. bump the pin in `vantage`'s `server/go.mod` — live at the next push to main
|
||||
3. bump the pin here — live only at the next `agent/v*` tag
|
||||
2. bump the pin in `vantage`'s `server/go.mod` - live at the next push to main
|
||||
3. bump the pin here - live only at the next `agent/v*` tag
|
||||
|
||||
The control plane runs ahead of the fleet in between. That was true before the
|
||||
split too; it is now explicit in two `go.mod` files rather than implicit in a
|
||||
@@ -68,7 +68,7 @@ git tag agent/v1.2.0 && git push origin agent/v1.2.0
|
||||
Builds `linux/amd64`, `linux/arm64` and `windows/amd64`, writes `checksums.txt`,
|
||||
creates the Gitea release. A second `msi` job on `windows-2022` builds the exe
|
||||
again, packages it with WiX and appends the MSI to the same release **through
|
||||
the API** — `gitea-release-action` cannot find a tag with a slash in it.
|
||||
the API** - `gitea-release-action` cannot find a tag with a slash in it.
|
||||
|
||||
### The self-update path, and what the move broke
|
||||
|
||||
@@ -77,7 +77,7 @@ the API** — `gitea-release-action` cannot find a tag with a slash in it.
|
||||
SHA-256 from `checksums.txt` before swapping itself. That path is **compiled
|
||||
into the binary**.
|
||||
|
||||
**Every agent built before this move has the old path — `mrhid6/vantage` —
|
||||
**Every agent built before this move has the old path - `mrhid6/vantage` -
|
||||
baked in, and releases are no longer published there.** For those agents the
|
||||
push-button update in the UI will fail: the download 404s. They are not
|
||||
stranded, because `/update` and `/update.ps1` are generated by the control plane
|
||||
@@ -85,18 +85,23 @@ at request time and point wherever the current server says, so re-running the
|
||||
update one-liner on a host moves it onto a build that knows the new address.
|
||||
After that, self-update works again permanently.
|
||||
|
||||
This was a deliberate choice — the alternative was publishing releases to a
|
||||
repository that no longer holds the source — but it means **the fleet needs one
|
||||
This was a deliberate choice - the alternative was publishing releases to a
|
||||
repository that no longer holds the source - but it means **the fleet needs one
|
||||
pass of the update one-liner**, and the control plane must be redeployed with
|
||||
the new release paths *first*, or the one-liner points at the old repository
|
||||
too.
|
||||
|
||||
## What the agent will not do
|
||||
|
||||
- **It never reboots a host.** `ApplyUpdatesCmd` installs and stops there;
|
||||
`inventory.reboot_required` reports that one is owed.
|
||||
- **It reboots a host only when told to and only when owed.** An
|
||||
`ApplyUpdatesCmd` with `reboot_if_required` set, on a host whose OS reports a
|
||||
reboot is owed, with at least 5 minutes left before `deadline_unix`, reboots
|
||||
after a one-minute grace period (`shutdown -r +1`, `shutdown /r /t 60`), and
|
||||
only once the `PatchResult` announcing it has been sent. Anything else
|
||||
installs and stops there, and `inventory.reboot_required` reports what is
|
||||
owed.
|
||||
- **It decides what it will not touch.** The protected workload set is computed
|
||||
and enforced agent-side — `vantage-agent.service`, `VantageAgent` on Windows,
|
||||
and enforced agent-side - `vantage-agent.service`, `VantageAgent` on Windows,
|
||||
and its own container ID from `/proc/self/cgroup`. The control plane may name
|
||||
a target; the agent decides what it will do to itself. A server-side denylist
|
||||
alone would be bypassed by the next dispatch path someone adds, and the
|
||||
@@ -105,12 +110,12 @@ too.
|
||||
the control plane can name a port and nothing else.
|
||||
- **No `authorized_keys` management on Windows**, and no package inventory: a
|
||||
Windows agent never calls `ReportPackages`, so no `server_packages` document
|
||||
exists for it at all — a different, earlier state than the `unsupported` a
|
||||
exists for it at all - a different, earlier state than the `unsupported` a
|
||||
Linux distribution reaches when its family has no security feed.
|
||||
|
||||
## Platform split
|
||||
|
||||
Windows support is build tags, not runtime branches — `systemd_linux.go` /
|
||||
Windows support is build tags, not runtime branches - `systemd_linux.go` /
|
||||
`services_windows.go` and the matching `control_` and `logs_` pairs. Windows
|
||||
collection runs PowerShell through `internal/winexec`, and **every script that
|
||||
reports data emits JSON that a build-tag-free parser reads**, so those parsers
|
||||
@@ -124,10 +129,38 @@ need a PowerShell Gallery install on every host and fails on an air-gapped
|
||||
fleet. `CurrentVersion` is empty on Windows and `NewVersion` carries the KB
|
||||
article ID: a Windows update is not a version bump of a named package.
|
||||
|
||||
## Patching
|
||||
|
||||
`updates.Apply` takes a scope and a deadline and answers with the tail of the
|
||||
package manager's output. Security-only uses `--security` on dnf/yum,
|
||||
`zypper patch --category security`, the Security and Critical classifications
|
||||
on Windows, and for apt a temporary `SourceParts` directory holding only the
|
||||
`-security` suites (with `APT::Get::List-Cleanup=0`, or the reduced update
|
||||
deletes every other list file). apk and pacman have no security metadata and
|
||||
report `unsupported`; security-only never falls back to installing
|
||||
everything. One run at a time: a second command answers `busy`.
|
||||
|
||||
The deadline (`deadline_unix`, the window end) only gates the **start** of each
|
||||
phase: the apt index refresh, the upgrade command, the Windows install script.
|
||||
A phase that has not started by then is refused with "the maintenance window
|
||||
ended before <phase> could start" (`canStart` in `internal/updates/phase.go`).
|
||||
A started upgrade is never killed by the window: it runs under a 2 hour
|
||||
backstop from its own start (`defaultApplyCap`), which on Linux sends SIGTERM
|
||||
and waits 5 minutes before a kill. Interrupting a package manager mid-transaction
|
||||
is worse than letting it finish late.
|
||||
|
||||
A `PatchResult` whose send fails (the command stream reconnected while the
|
||||
patch ran) is kept in a bounded queue (32, oldest dropped) and flushed on the
|
||||
next stream right after `AgentReady`. A queued result that announced a reboot
|
||||
has the reboot decision taken again at flush time, and the host still reboots
|
||||
only once the result is delivered. The startup static inventory report is
|
||||
retried every 30 seconds, up to 10 attempts: it carries the boot time the
|
||||
control plane uses to prove a patch reboot.
|
||||
|
||||
## Two constants that mirror the control plane
|
||||
|
||||
Neither can be shared — this is a separate module and the control plane's are
|
||||
under `internal/` — so both must change in step, by hand:
|
||||
Neither can be shared - this is a separate module and the control plane's are
|
||||
under `internal/` - so both must change in step, by hand:
|
||||
|
||||
- the workload log cap, **500 lines and 256KB whichever binds first**, mirrored
|
||||
in the control plane's `services.MaxWorkloadLogLines`
|
||||
@@ -135,3 +168,7 @@ under `internal/` — so both must change in step, by hand:
|
||||
control plane's 20s `PingCmd`. The watchdog arms only **after** a first ping
|
||||
has been seen, so an older server that sends none is treated as working rather
|
||||
than put into a reconnect loop.
|
||||
|
||||
## Writing style
|
||||
|
||||
Never use em dashes (the long dash character) anywhere: code, comments, UI copy, docs, commit messages. Use a plain hyphen ` - `, a comma, a colon, or split the sentence instead.
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
module gitea.hostxtra.co.uk/vantage/vantage-agent
|
||||
|
||||
go 1.26
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
golang.org/x/sys v0.47.0
|
||||
google.golang.org/grpc v1.64.0
|
||||
golang.org/x/sys v0.48.0
|
||||
google.golang.org/grpc v1.83.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
gitea.hostxtra.co.uk/vantage/vantage-shared v0.1.0
|
||||
golang.org/x/net v0.56.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e // indirect
|
||||
google.golang.org/protobuf v1.34.1 // indirect
|
||||
gitea.hostxtra.co.uk/vantage/vantage-shared v0.5.0
|
||||
golang.org/x/net v0.58.0 // indirect
|
||||
golang.org/x/text v0.41.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260908043556-f8649ddbbfe6 // indirect
|
||||
google.golang.org/protobuf v1.36.12 // indirect
|
||||
)
|
||||
|
||||
@@ -1,19 +1,43 @@
|
||||
gitea.hostxtra.co.uk/vantage/vantage-shared v0.1.0 h1:H6PCb8JHucrRiqPe9kGOhXUjBD66tKFHCP3qz5TjdZc=
|
||||
gitea.hostxtra.co.uk/vantage/vantage-shared v0.1.0/go.mod h1:dWjeOFLltQ8sv9Pnn1xRxGfWGgqa2fkG0esuaJLoPXQ=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
|
||||
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e h1:Elxv5MwEkCI9f5SkoL6afed6NTdxaGoAo39eANBwHL8=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e/go.mod h1:EfXuqaE1J41VCDicxHzUDm+8rk+7ZdXzHV0IhO/I6s0=
|
||||
google.golang.org/grpc v1.64.0 h1:KH3VH9y/MgNQg1dE7b3XfVK0GsPSIzJwdF617gUSbvY=
|
||||
google.golang.org/grpc v1.64.0/go.mod h1:oxjF8E3FBnjp+/gVFYdWacaLDx9na1aqy9oovLpxQYg=
|
||||
google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg=
|
||||
google.golang.org/protobuf v1.34.1/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
|
||||
gitea.hostxtra.co.uk/vantage/vantage-shared v0.5.0 h1:xwSIEkQKTd4Qk+BYHvoGN+h84Isr2h5qqnitUWF1m2w=
|
||||
gitea.hostxtra.co.uk/vantage/vantage-shared v0.5.0/go.mod h1:Zo66XhqF8No3dveIowLCepvMxVg8KnhsNMz0k0Xpuck=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
|
||||
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
|
||||
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
||||
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
|
||||
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260908043556-f8649ddbbfe6 h1:ieEbjQ6lzbvntOXUB9nMx9uH+yIU/HbgkNDjnk/mJuk=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260908043556-f8649ddbbfe6/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
|
||||
google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU=
|
||||
google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
package inventory
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// parseBtime reads the boot time, in Unix seconds, from /proc/stat. The
|
||||
// control plane compares it with the moment it asked for a reboot: a report
|
||||
// is only proof the host restarted if its boot time is later.
|
||||
func parseBtime(stat string) int64 {
|
||||
for _, line := range strings.Split(stat, "\n") {
|
||||
if rest, ok := strings.CutPrefix(line, "btime "); ok {
|
||||
v, _ := strconv.ParseInt(strings.TrimSpace(rest), 10, 64)
|
||||
return v
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package inventory
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseBtime(t *testing.T) {
|
||||
stat := "cpu 1 2 3 4\ncpu0 1 2 3 4\nintr 1\nctxt 99\nbtime 1757800000\nprocesses 5\n"
|
||||
if got := parseBtime(stat); got != 1757800000 {
|
||||
t.Fatalf("got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBtimeMissing(t *testing.T) {
|
||||
if got := parseBtime("cpu 1 2 3\n"); got != 0 {
|
||||
t.Fatalf("got %d, want 0", got)
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
)
|
||||
|
||||
func collect(r *pb.InventoryReport, includeStatic bool) {
|
||||
r.BootTimeUnix = parseBtime(readProc("/proc/stat"))
|
||||
r.CPU.UsagePct = cpuUsage()
|
||||
r.CPU.Load1 = load1()
|
||||
memTotal, memAvail, swapTotal, swapFree := meminfo()
|
||||
|
||||
@@ -20,6 +20,8 @@ var (
|
||||
)
|
||||
|
||||
func collect(r *pb.InventoryReport, includeStatic bool) {
|
||||
// DurationSinceBoot returns milliseconds since boot and does not wrap.
|
||||
r.BootTimeUnix = time.Now().Add(-windows.DurationSinceBoot()).Unix()
|
||||
r.CPU.UsagePct = cpuUsage()
|
||||
// Windows has no load average. Left at zero; the UI already treats it as
|
||||
// optional because it is omitempty on the wire.
|
||||
@@ -28,7 +30,7 @@ func collect(r *pb.InventoryReport, includeStatic bool) {
|
||||
if m.TotalPhys > m.AvailPhys {
|
||||
r.Memory.UsedBytes = m.TotalPhys - m.AvailPhys
|
||||
}
|
||||
// TotalPageFile is the commit limit — physical memory plus the pagefile —
|
||||
// TotalPageFile is the commit limit - physical memory plus the pagefile -
|
||||
// so the pagefile alone is the difference.
|
||||
swapTotal := sub(m.TotalPageFile, m.TotalPhys)
|
||||
swapUsed := sub(sub(m.TotalPageFile, m.AvailPageFile), sub(m.TotalPhys, m.AvailPhys))
|
||||
|
||||
@@ -16,7 +16,7 @@ const collectTimeout = 2 * time.Minute
|
||||
//
|
||||
// The format strings below are raw string literals on purpose. The "\t" and
|
||||
// "\n" reach dpkg-query and rpm as two characters each, and those tools do the
|
||||
// interpreting themselves — Go must not consume the escapes first.
|
||||
// interpreting themselves - Go must not consume the escapes first.
|
||||
func Collect() (OSRelease, []Package, error) {
|
||||
if runtime.GOOS != "linux" {
|
||||
return OSRelease{}, nil, fmt.Errorf("package collection is linux-only, got %s", runtime.GOOS)
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
)
|
||||
|
||||
// Package is one installed package as the distribution reports it. Version is
|
||||
// the distribution's own version string, verbatim — never normalised, because
|
||||
// the distribution's own version string, verbatim - never normalised, because
|
||||
// the advisory feeds are keyed on exactly this form.
|
||||
type Package struct {
|
||||
Name string
|
||||
@@ -29,7 +29,7 @@ type Package struct {
|
||||
//
|
||||
// The fifth column is why "rc" packages do not appear. dpkg-query -W lists
|
||||
// every package dpkg knows about, including ones removed with their config
|
||||
// files left behind — a host that has upgraded its kernel a dozen times reports
|
||||
// files left behind - a host that has upgraded its kernel a dozen times reports
|
||||
// a dozen old linux-modules versions that are not on disk, and the oldest of
|
||||
// them sorts first and reads as the installed version. Only "installed" is
|
||||
// installed. An empty status means dpkg did not understand the field, in which
|
||||
@@ -143,7 +143,7 @@ func splitAPK(s string) (name, version string) {
|
||||
//
|
||||
// It sorts first: the ordering of dpkg or rpm output is not guaranteed stable,
|
||||
// and an ordering-sensitive hash would resend the full ~150KB list every hour
|
||||
// for no reason — a cost visible only as traffic.
|
||||
// for no reason - a cost visible only as traffic.
|
||||
func Hash(pkgs []Package) string {
|
||||
lines := make([]string, 0, len(pkgs))
|
||||
for _, p := range pkgs {
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
)
|
||||
|
||||
// collectPackagesFlag is written by the 30s key poll and read by the hourly
|
||||
// package loop — two different goroutines, hence the atomic.
|
||||
// package loop - two different goroutines, hence the atomic.
|
||||
//
|
||||
// It defaults to false, so an agent that has not yet completed a poll, or is
|
||||
// talking to a server too old to send the field, collects nothing. Off is the
|
||||
@@ -27,7 +27,7 @@ var collectPackagesFlag atomic.Bool
|
||||
//
|
||||
// Without it the boot-time package report loses a race it can only lose: the
|
||||
// hourly loop starts before the first poll, reads a flag that is still false by
|
||||
// construction, and skips — so a freshly installed agent reports no packages for
|
||||
// construction, and skips - so a freshly installed agent reports no packages for
|
||||
// an hour and the server shows nothing to scan.
|
||||
// How long the boot package report waits for that first poll. Two poll
|
||||
// intervals plus slack: long enough to cover one failed attempt, short enough
|
||||
@@ -43,7 +43,7 @@ func markFirstPoll() { firstPollOnce.Do(func() { close(firstPoll) }) }
|
||||
|
||||
// waitFirstPoll blocks until the flag is known, or gives up. The wait is
|
||||
// bounded because this loop also reports OS updates, which do not depend on the
|
||||
// flag at all — a control plane that cannot be polled must not silence those too.
|
||||
// flag at all - a control plane that cannot be polled must not silence those too.
|
||||
func waitFirstPoll(ctx context.Context, limit time.Duration) {
|
||||
t := time.NewTimer(limit)
|
||||
defer t.Stop()
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/vantage/vantage-agent/internal/config"
|
||||
grpcclient "gitea.hostxtra.co.uk/vantage/vantage-agent/internal/grpc"
|
||||
"gitea.hostxtra.co.uk/vantage/vantage-agent/internal/updates"
|
||||
"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
|
||||
)
|
||||
|
||||
// minRebootLeeway is the least time that must remain in the window for the
|
||||
// agent to start a reboot. A reboot that lands after the window closes is the
|
||||
// outage the window existed to prevent.
|
||||
const minRebootLeeway = 5 * time.Minute
|
||||
|
||||
// shouldReboot is the whole reboot decision. The agent reboots a host only
|
||||
// when the command asked, the OS reports a reboot is owed, and the window
|
||||
// still has room for it. A command with no deadline is a manual run, which
|
||||
// never reboots.
|
||||
func shouldReboot(requested, owed bool, now, deadline time.Time) bool {
|
||||
if !requested || !owed || deadline.IsZero() {
|
||||
return false
|
||||
}
|
||||
return deadline.Sub(now) >= minRebootLeeway
|
||||
}
|
||||
|
||||
func handleApplyUpdates(send func(*pb.AgentMessage) error, cfg *config.Config, cmd *pb.ServerCommand) {
|
||||
c := cmd.ApplyUpdates
|
||||
var deadline time.Time
|
||||
if c.DeadlineUnix > 0 {
|
||||
deadline = time.Unix(c.DeadlineUnix, 0)
|
||||
}
|
||||
log.Printf("applying OS updates (cmd=%s scope=%q reboot=%v)", cmd.CommandId, c.Scope, c.RebootIfRequired)
|
||||
|
||||
res, err := updates.Apply(updates.ApplyOptions{SecurityOnly: c.Scope == pb.PatchScopeSecurity, Deadline: deadline})
|
||||
pr := &pb.PatchResult{CommandId: cmd.CommandId, OutputTail: res.Output, PendingAfter: -1}
|
||||
switch {
|
||||
case errors.Is(err, updates.ErrBusy):
|
||||
pr.Status, pr.Message = pb.PatchStatusBusy, err.Error()
|
||||
case err != nil:
|
||||
pr.Status, pr.Message = pb.PatchStatusFailed, err.Error()
|
||||
case res.Unsupported:
|
||||
pr.Status, pr.Message = pb.PatchStatusUnsupported, res.Reason
|
||||
default:
|
||||
pr.Status = pb.PatchStatusOK
|
||||
}
|
||||
|
||||
// Refresh the pending list whether the run succeeded or not, so the counts
|
||||
// the operator sees are this host's real state. A busy refusal changed
|
||||
// nothing, and the run in progress will report for itself.
|
||||
if pr.Status != pb.PatchStatusBusy {
|
||||
pr.PendingAfter = int32(reportPendingUpdates(cfg))
|
||||
}
|
||||
pr.RebootRequired = updates.RebootRequired()
|
||||
pr.Rebooting = pr.Status == pb.PatchStatusOK && shouldReboot(c.RebootIfRequired, pr.RebootRequired, time.Now(), deadline)
|
||||
|
||||
msg := &pb.AgentMessage{ServerId: cfg.ServerID, AgentToken: cfg.AgentToken, PatchResult: pr}
|
||||
if err := send(msg); err != nil {
|
||||
log.Printf("send patch result (cmd=%s): %v; keeping it for the next command stream", cmd.CommandId, err)
|
||||
// A reboot nobody was told about looks like a crash. Without a
|
||||
// delivered result, do not reboot: the queued result retakes the
|
||||
// decision when it is finally sent.
|
||||
if pendingResults.push(pendingResult{msg: msg, requested: c.RebootIfRequired, deadline: deadline}) {
|
||||
log.Printf("patch result queue full: dropped the oldest undelivered result")
|
||||
}
|
||||
return
|
||||
}
|
||||
log.Printf("patch result sent (cmd=%s status=%s pending_after=%d rebooting=%v)", cmd.CommandId, pr.Status, pr.PendingAfter, pr.Rebooting)
|
||||
if pr.Rebooting {
|
||||
if err := updates.ScheduleReboot(); err != nil {
|
||||
log.Printf("schedule reboot (cmd=%s): %v", cmd.CommandId, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pendingResults holds PatchResults whose send failed, typically because the
|
||||
// command stream reconnected while a patch ran. The server guards result
|
||||
// writes on status and command id, so a late result is safe to deliver.
|
||||
var pendingResults = &resultQueue{max: maxPendingResults}
|
||||
|
||||
// flushPendingResults delivers queued results on a newly established stream.
|
||||
// A queued result that announced a reboot has that decision taken again now:
|
||||
// time has passed, so the window may be too close to its end, or the reboot
|
||||
// may no longer be owed. Rebooting is cleared before sending when it no
|
||||
// longer holds, and the host reboots only once the result is delivered.
|
||||
func flushPendingResults(send func(*pb.AgentMessage) error) {
|
||||
pendingResults.flush(func(p pendingResult) error {
|
||||
pr := p.msg.PatchResult
|
||||
if pr.Rebooting {
|
||||
owed := updates.RebootRequired()
|
||||
pr.RebootRequired = owed
|
||||
if !shouldReboot(p.requested, owed, time.Now(), p.deadline) {
|
||||
pr.Rebooting = false
|
||||
}
|
||||
}
|
||||
if err := send(p.msg); err != nil {
|
||||
log.Printf("send queued patch result (cmd=%s): %v", pr.CommandId, err)
|
||||
return err
|
||||
}
|
||||
log.Printf("queued patch result sent (cmd=%s status=%s rebooting=%v)", pr.CommandId, pr.Status, pr.Rebooting)
|
||||
if pr.Rebooting {
|
||||
if err := updates.ScheduleReboot(); err != nil {
|
||||
log.Printf("schedule reboot (cmd=%s): %v", pr.CommandId, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// reportPendingUpdates re-checks pending updates and reports them, returning
|
||||
// the count, or -1 if either step failed.
|
||||
func reportPendingUpdates(cfg *config.Config) int {
|
||||
pkgs, err := updates.CheckAvailable()
|
||||
if err != nil {
|
||||
log.Printf("post-apply update check: %v", err)
|
||||
return -1
|
||||
}
|
||||
list := make([]pb.PackageUpdate, len(pkgs))
|
||||
for i, p := range pkgs {
|
||||
list[i] = pb.PackageUpdate{Name: p.Name, CurrentVersion: p.CurrentVersion, NewVersion: p.NewVersion}
|
||||
}
|
||||
client, err := grpcclient.New(cfg.ServerURL, cfg.TLS)
|
||||
if err != nil {
|
||||
log.Printf("post-apply report dial: %v", err)
|
||||
return len(pkgs)
|
||||
}
|
||||
defer client.Close()
|
||||
if err := client.ReportUpdates(cfg.ServerID, cfg.AgentToken, list); err != nil {
|
||||
log.Printf("post-apply ReportUpdates: %v", err)
|
||||
}
|
||||
return len(pkgs)
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestShouldReboot(t *testing.T) {
|
||||
now := time.Date(2026, 9, 20, 2, 30, 0, 0, time.UTC)
|
||||
cases := []struct {
|
||||
name string
|
||||
requested, owed bool
|
||||
deadline time.Time
|
||||
want bool
|
||||
}{
|
||||
{"asked, owed, plenty of time", true, true, now.Add(time.Hour), true},
|
||||
{"not asked", false, true, now.Add(time.Hour), false},
|
||||
{"nothing owed", true, false, now.Add(time.Hour), false},
|
||||
{"exactly five minutes left", true, true, now.Add(5 * time.Minute), true},
|
||||
{"under five minutes left", true, true, now.Add(4*time.Minute + 59*time.Second), false},
|
||||
{"no deadline (manual run)", true, true, time.Time{}, false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := shouldReboot(c.requested, c.owed, now, c.deadline); got != c.want {
|
||||
t.Errorf("%s: got %v, want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
|
||||
)
|
||||
|
||||
// maxPendingResults bounds the results kept for a later stream. A host that
|
||||
// cannot reach the control plane for a very long time drops its oldest
|
||||
// results rather than growing without limit.
|
||||
const maxPendingResults = 32
|
||||
|
||||
// pendingResult is a PatchResult that could not be sent, with what the reboot
|
||||
// decision needs to be taken again when it finally can be.
|
||||
type pendingResult struct {
|
||||
msg *pb.AgentMessage
|
||||
requested bool // the command's RebootIfRequired
|
||||
deadline time.Time // the command's deadline, zero for a manual run
|
||||
}
|
||||
|
||||
// resultQueue holds undelivered PatchResults until the next command stream.
|
||||
type resultQueue struct {
|
||||
mu sync.Mutex
|
||||
items []pendingResult
|
||||
max int
|
||||
}
|
||||
|
||||
// push appends p, dropping the oldest entry when the queue is full. It
|
||||
// reports whether an entry was dropped.
|
||||
func (q *resultQueue) push(p pendingResult) bool {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
q.items = append(q.items, p)
|
||||
if len(q.items) > q.max {
|
||||
q.items = q.items[len(q.items)-q.max:]
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// flush sends the queued results oldest first, removing each once sent. It
|
||||
// stops at the first failure and keeps that entry and everything after it
|
||||
// for the next stream. The lock is held throughout so two streams never
|
||||
// deliver the same result.
|
||||
func (q *resultQueue) flush(send func(pendingResult) error) {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
for len(q.items) > 0 {
|
||||
if err := send(q.items[0]); err != nil {
|
||||
return
|
||||
}
|
||||
q.items = q.items[1:]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package agentsync
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.hostxtra.co.uk/vantage/vantage-shared/grpc/pb"
|
||||
)
|
||||
|
||||
func pending(id string) pendingResult {
|
||||
return pendingResult{msg: &pb.AgentMessage{PatchResult: &pb.PatchResult{CommandId: id}}}
|
||||
}
|
||||
|
||||
func ids(q *resultQueue) []string {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
var out []string
|
||||
for _, p := range q.items {
|
||||
out = append(out, p.msg.PatchResult.CommandId)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestResultQueueEnqueue(t *testing.T) {
|
||||
q := &resultQueue{max: 3}
|
||||
q.push(pending("a"))
|
||||
q.push(pending("b"))
|
||||
if got := ids(q); len(got) != 2 || got[0] != "a" || got[1] != "b" {
|
||||
t.Fatalf("got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResultQueueBoundDropsOldest(t *testing.T) {
|
||||
q := &resultQueue{max: 2}
|
||||
q.push(pending("a"))
|
||||
q.push(pending("b"))
|
||||
if dropped := q.push(pending("c")); !dropped {
|
||||
t.Fatal("push over the bound must report a drop")
|
||||
}
|
||||
if got := ids(q); len(got) != 2 || got[0] != "b" || got[1] != "c" {
|
||||
t.Fatalf("got %v, want [b c]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResultQueueDefaultBound(t *testing.T) {
|
||||
if maxPendingResults != 32 {
|
||||
t.Fatalf("maxPendingResults = %d, want 32", maxPendingResults)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResultQueueFlushOrderAndRemoval(t *testing.T) {
|
||||
q := &resultQueue{max: 8}
|
||||
for _, id := range []string{"a", "b", "c"} {
|
||||
q.push(pending(id))
|
||||
}
|
||||
var sent []string
|
||||
q.flush(func(p pendingResult) error {
|
||||
sent = append(sent, p.msg.PatchResult.CommandId)
|
||||
return nil
|
||||
})
|
||||
if len(sent) != 3 || sent[0] != "a" || sent[1] != "b" || sent[2] != "c" {
|
||||
t.Fatalf("flush order %v, want [a b c]", sent)
|
||||
}
|
||||
if got := ids(q); len(got) != 0 {
|
||||
t.Fatalf("sent entries must be removed, left %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResultQueueFlushRetainsOnFailure(t *testing.T) {
|
||||
q := &resultQueue{max: 8}
|
||||
for _, id := range []string{"a", "b", "c"} {
|
||||
q.push(pending(id))
|
||||
}
|
||||
var sent []string
|
||||
q.flush(func(p pendingResult) error {
|
||||
if p.msg.PatchResult.CommandId == "b" {
|
||||
return errors.New("stream gone")
|
||||
}
|
||||
sent = append(sent, p.msg.PatchResult.CommandId)
|
||||
return nil
|
||||
})
|
||||
if len(sent) != 1 || sent[0] != "a" {
|
||||
t.Fatalf("sent %v, want [a]", sent)
|
||||
}
|
||||
if got := ids(q); len(got) != 2 || got[0] != "b" || got[1] != "c" {
|
||||
t.Fatalf("retained %v, want [b c]", got)
|
||||
}
|
||||
}
|
||||
+28
-25
@@ -134,7 +134,7 @@ func poll(client *grpcclient.Client, cfg *config.Config, version string) error {
|
||||
const streamHealthyAfter = time.Minute
|
||||
|
||||
// Stream staleness. The server beats every 20s, so 70s tolerates three missed
|
||||
// beats before the stream is written off — high enough that a slow network or a
|
||||
// beats before the stream is written off - high enough that a slow network or a
|
||||
// briefly busy server does not cost a reconnect, low enough that an agent is
|
||||
// not uncommandable for minutes after a control-plane restart.
|
||||
const (
|
||||
@@ -142,7 +142,7 @@ const (
|
||||
streamStaleCheck = 10 * time.Second
|
||||
|
||||
// How often a healthy stream reports itself. Also the interval at which an
|
||||
// agent talking to a control plane too old to send heartbeats says so —
|
||||
// agent talking to a control plane too old to send heartbeats says so -
|
||||
// that agent is running without a watchdog, and the journal should not be
|
||||
// silent about it.
|
||||
pingSummaryInterval = 5 * time.Minute
|
||||
@@ -184,7 +184,7 @@ func runCommandStream(ctx context.Context, cfg *config.Config) {
|
||||
|
||||
// The uptime is in the line because it is what distinguishes a stream
|
||||
// that never worked from one that ran for hours and was dropped by a
|
||||
// deploy — and it is the same measure that decides whether the backoff
|
||||
// deploy - and it is the same measure that decides whether the backoff
|
||||
// resets, so a reader can see why the delay is what it is.
|
||||
up := time.Since(started).Truncate(time.Second)
|
||||
if err != nil {
|
||||
@@ -243,6 +243,11 @@ func connectAndHandleStream(ctx context.Context, cfg *config.Config) error {
|
||||
return stream.Send(msg)
|
||||
}
|
||||
|
||||
// Results that could not be sent on an earlier stream go out first. In
|
||||
// its own goroutine: a queued reboot re-check can take a while on
|
||||
// Windows, and the receive loop below must start promptly.
|
||||
go flushPendingResults(send)
|
||||
|
||||
// Stream liveness, tracked here rather than left to gRPC keepalive.
|
||||
//
|
||||
// Keepalive operates on the transport, and behind an L7 proxy the transport
|
||||
@@ -253,7 +258,7 @@ func connectAndHandleStream(ctx context.Context, cfg *config.Config) error {
|
||||
// the operator watches nothing happen.
|
||||
//
|
||||
// The watchdog only arms once a ping has actually been seen. A server too
|
||||
// old to send them must not be treated as dead — that would put the agent
|
||||
// old to send them must not be treated as dead - that would put the agent
|
||||
// in a reconnect loop against a control plane that is working perfectly.
|
||||
var (
|
||||
lastMu sync.Mutex
|
||||
@@ -283,7 +288,7 @@ func connectAndHandleStream(ctx context.Context, cfg *config.Config) error {
|
||||
|
||||
// Reported periodically rather than per beat: at one every 20s the
|
||||
// journal would be nothing else. The count is what makes a partial
|
||||
// failure visible — beats arriving but fewer than expected is a
|
||||
// failure visible - beats arriving but fewer than expected is a
|
||||
// different problem from beats stopping altogether.
|
||||
summary := time.NewTicker(pingSummaryInterval)
|
||||
defer summary.Stop()
|
||||
@@ -341,7 +346,7 @@ func connectAndHandleStream(ctx context.Context, cfg *config.Config) error {
|
||||
go handleUpdateAgent(cmd)
|
||||
}
|
||||
if cmd.ApplyUpdates != nil {
|
||||
go handleApplyUpdates(cfg, cmd)
|
||||
go handleApplyUpdates(send, cfg, cmd)
|
||||
}
|
||||
if cmd.CleanupWorkspace != nil {
|
||||
go handleCleanupWorkspace(cmd)
|
||||
@@ -446,11 +451,11 @@ func runInventory(ctx context.Context, cfg *config.Config) {
|
||||
}
|
||||
defer client.Close()
|
||||
|
||||
report := func(static bool) {
|
||||
report := func(static bool) error {
|
||||
r := inventory.Collect(static)
|
||||
r.ServerId = cfg.ServerID
|
||||
r.AgentToken = cfg.AgentToken
|
||||
// Static snapshots only — every 15 minutes, not every 30 seconds. On
|
||||
// Static snapshots only - every 15 minutes, not every 30 seconds. On
|
||||
// Windows this spawns a PowerShell process, which is not something to
|
||||
// do twice a minute forever, and a host rebooted by hand clearing the
|
||||
// flag within a quarter of an hour is soon enough.
|
||||
@@ -462,10 +467,18 @@ func runInventory(ctx context.Context, cfg *config.Config) {
|
||||
}
|
||||
if err := client.ReportInventory(r); err != nil {
|
||||
log.Printf("report inventory: %v", err)
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
report(true)
|
||||
// The startup static report carries the boot time the server uses to
|
||||
// prove a patch reboot happened, so a failure is retried on the next
|
||||
// ticks (every 30 seconds, up to startupStaticAttempts in total) instead
|
||||
// of waiting a quarter of an hour for the next static snapshot.
|
||||
const startupStaticAttempts = 10
|
||||
attempts := 1
|
||||
startupPending := report(true) != nil
|
||||
ticker := time.NewTicker(30 * time.Second)
|
||||
defer ticker.Stop()
|
||||
tick := 0
|
||||
@@ -475,27 +488,17 @@ func runInventory(ctx context.Context, cfg *config.Config) {
|
||||
return
|
||||
case <-ticker.C:
|
||||
tick++
|
||||
if startupPending && attempts < startupStaticAttempts {
|
||||
attempts++
|
||||
startupPending = report(true) != nil
|
||||
continue
|
||||
}
|
||||
startupPending = false
|
||||
report(tick%30 == 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func handleApplyUpdates(cfg *config.Config, cmd *pb.ServerCommand) {
|
||||
log.Printf("applying OS updates (cmd=%s)…", cmd.CommandId)
|
||||
if err := updates.ApplyAll(); err != nil {
|
||||
log.Printf("OS upgrade failed (cmd=%s): %v", cmd.CommandId, err)
|
||||
return
|
||||
}
|
||||
log.Printf("OS updates applied successfully (cmd=%s)", cmd.CommandId)
|
||||
|
||||
client, err := grpcclient.New(cfg.ServerURL, cfg.TLS)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer client.Close()
|
||||
_ = client.ReportUpdates(cfg.ServerID, cfg.AgentToken, nil)
|
||||
}
|
||||
|
||||
func handleCleanupWorkspace(cmd *pb.ServerCommand) {
|
||||
id := cmd.CleanupWorkspace.WorkspaceId
|
||||
dir := agentexec.WorkspacePath(id)
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Two package managers running at once corrupt each other's locks. The second
|
||||
// caller must be told, not queued.
|
||||
func TestApplyRefusesWhileBusy(t *testing.T) {
|
||||
applyMu.Lock()
|
||||
defer applyMu.Unlock()
|
||||
_, err := Apply(ApplyOptions{Deadline: time.Now().Add(time.Minute)})
|
||||
if !errors.Is(err, ErrBusy) {
|
||||
t.Fatalf("err = %v, want ErrBusy", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// isSecuritySuite reports whether an apt suite carries security fixes. Debian
|
||||
// 11+ and every supported Ubuntu name them "<codename>-security".
|
||||
func isSecuritySuite(s string) bool { return strings.HasSuffix(s, "-security") }
|
||||
|
||||
// securitySources reduces a host's apt source files to only the entries that
|
||||
// point at a security suite, so an upgrade run against them installs security
|
||||
// fixes and nothing else.
|
||||
//
|
||||
// It is a pure function of file contents so it is tested on any platform. The
|
||||
// caller writes list to a *.list file and deb822 to a *.sources file in a
|
||||
// temporary SourceParts directory: keeping deb822 paragraphs as deb822 means
|
||||
// an inline Signed-By key block survives verbatim, which a conversion to
|
||||
// one-line format could not carry.
|
||||
//
|
||||
// ok is false when no security suite exists at all. The caller must then
|
||||
// report unsupported, never fall back to installing everything.
|
||||
func securitySources(files map[string]string) (list string, deb822 string, ok bool) {
|
||||
paths := make([]string, 0, len(files))
|
||||
for p := range files {
|
||||
paths = append(paths, p)
|
||||
}
|
||||
sort.Strings(paths) // deterministic output
|
||||
|
||||
var lb, db strings.Builder
|
||||
for _, p := range paths {
|
||||
if strings.HasSuffix(p, ".sources") {
|
||||
db.WriteString(filterDeb822(files[p]))
|
||||
} else {
|
||||
lb.WriteString(filterOneLine(files[p]))
|
||||
}
|
||||
}
|
||||
list, deb822 = lb.String(), db.String()
|
||||
return list, deb822, list != "" || deb822 != ""
|
||||
}
|
||||
|
||||
func filterOneLine(content string) string {
|
||||
var b strings.Builder
|
||||
for _, raw := range strings.Split(content, "\n") {
|
||||
line := strings.TrimSpace(raw)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 3 || fields[0] != "deb" {
|
||||
continue
|
||||
}
|
||||
i := 1
|
||||
if strings.HasPrefix(fields[i], "[") {
|
||||
// Options run until the token that closes the bracket.
|
||||
for i < len(fields) && !strings.HasSuffix(fields[i], "]") {
|
||||
i++
|
||||
}
|
||||
i++
|
||||
}
|
||||
// fields[i] is the URI, fields[i+1] the suite.
|
||||
if i+1 < len(fields) && isSecuritySuite(fields[i+1]) {
|
||||
b.WriteString(line)
|
||||
b.WriteString("\n")
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// deb822Fields groups a paragraph's lines into fields. A line that starts
|
||||
// with a space or tab continues the field above it (a folded or multi-line
|
||||
// value, such as a long Suites list or an inline Signed-By key block).
|
||||
func deb822Fields(para string) [][]string {
|
||||
var fields [][]string
|
||||
for _, l := range strings.Split(strings.Trim(para, "\n"), "\n") {
|
||||
if (strings.HasPrefix(l, " ") || strings.HasPrefix(l, "\t")) && len(fields) > 0 {
|
||||
fields[len(fields)-1] = append(fields[len(fields)-1], l)
|
||||
continue
|
||||
}
|
||||
fields = append(fields, []string{l})
|
||||
}
|
||||
return fields
|
||||
}
|
||||
|
||||
func filterDeb822(content string) string {
|
||||
var b strings.Builder
|
||||
for _, para := range strings.Split(strings.ReplaceAll(content, "\r\n", "\n"), "\n\n") {
|
||||
var out []string
|
||||
isDeb, enabled, kept := false, true, false
|
||||
for _, field := range deb822Fields(para) {
|
||||
key, val, found := strings.Cut(field[0], ":")
|
||||
k := strings.ToLower(strings.TrimSpace(key))
|
||||
v := strings.TrimSpace(val)
|
||||
switch {
|
||||
case found && k == "types":
|
||||
for _, t := range strings.Fields(v) {
|
||||
if t == "deb" {
|
||||
isDeb = true
|
||||
}
|
||||
}
|
||||
case found && k == "enabled":
|
||||
enabled = strings.ToLower(v) != "no"
|
||||
case found && k == "suites":
|
||||
// The value runs across every continuation line. The filtered
|
||||
// result is written back as one line and the continuation
|
||||
// lines are dropped with the rest of the original field.
|
||||
all := strings.Fields(strings.Join(append([]string{v}, field[1:]...), " "))
|
||||
var sec []string
|
||||
for _, s := range all {
|
||||
if isSecuritySuite(s) {
|
||||
sec = append(sec, s)
|
||||
}
|
||||
}
|
||||
if len(sec) == 0 {
|
||||
continue // drop the field; the paragraph is dropped below
|
||||
}
|
||||
kept = true
|
||||
out = append(out, "Suites: "+strings.Join(sec, " "))
|
||||
continue
|
||||
}
|
||||
// Every other field, continuation lines included, stays verbatim.
|
||||
out = append(out, field...)
|
||||
}
|
||||
if isDeb && enabled && kept {
|
||||
b.WriteString(strings.Join(out, "\n"))
|
||||
b.WriteString("\n\n")
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSecuritySourcesOneLine(t *testing.T) {
|
||||
files := map[string]string{
|
||||
"/etc/apt/sources.list": `# comment
|
||||
deb http://deb.debian.org/debian bookworm main
|
||||
deb http://deb.debian.org/debian bookworm-updates main
|
||||
deb http://security.debian.org/debian-security bookworm-security main contrib
|
||||
deb [arch=amd64 signed-by=/usr/share/keyrings/x.gpg] http://archive.ubuntu.com/ubuntu jammy-security main
|
||||
deb-src http://security.debian.org/debian-security bookworm-security main
|
||||
`,
|
||||
}
|
||||
list, d822, ok := securitySources(files)
|
||||
if !ok {
|
||||
t.Fatal("ok = false, want true")
|
||||
}
|
||||
if d822 != "" {
|
||||
t.Fatalf("deb822 = %q, want empty", d822)
|
||||
}
|
||||
want := "deb http://security.debian.org/debian-security bookworm-security main contrib\n" +
|
||||
"deb [arch=amd64 signed-by=/usr/share/keyrings/x.gpg] http://archive.ubuntu.com/ubuntu jammy-security main\n"
|
||||
if list != want {
|
||||
t.Fatalf("list =\n%s\nwant\n%s", list, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSecuritySourcesDeb822(t *testing.T) {
|
||||
files := map[string]string{
|
||||
"/etc/apt/sources.list.d/ubuntu.sources": `Types: deb
|
||||
URIs: http://archive.ubuntu.com/ubuntu/
|
||||
Suites: noble noble-updates noble-backports
|
||||
Components: main restricted
|
||||
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
|
||||
|
||||
Types: deb
|
||||
URIs: http://security.ubuntu.com/ubuntu/
|
||||
Suites: noble-security
|
||||
Components: main restricted
|
||||
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
|
||||
`,
|
||||
}
|
||||
list, d822, ok := securitySources(files)
|
||||
if !ok || list != "" {
|
||||
t.Fatalf("ok=%v list=%q", ok, list)
|
||||
}
|
||||
if !strings.Contains(d822, "Suites: noble-security") || strings.Contains(d822, "noble-updates") {
|
||||
t.Fatalf("deb822 wrong:\n%s", d822)
|
||||
}
|
||||
if !strings.Contains(d822, "Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg") {
|
||||
t.Fatalf("Signed-By must be kept verbatim:\n%s", d822)
|
||||
}
|
||||
}
|
||||
|
||||
// A paragraph listing several suites keeps only the security ones.
|
||||
func TestSecuritySourcesDeb822MixedSuites(t *testing.T) {
|
||||
files := map[string]string{"/x.sources": "Types: deb\nURIs: http://a/\nSuites: noble noble-security\nComponents: main\n"}
|
||||
_, d822, ok := securitySources(files)
|
||||
if !ok || !strings.Contains(d822, "Suites: noble-security\n") || strings.Contains(d822, "Suites: noble noble") {
|
||||
t.Fatalf("got ok=%v\n%s", ok, d822)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSecuritySourcesDisabledParagraphIgnored(t *testing.T) {
|
||||
files := map[string]string{"/x.sources": "Types: deb\nURIs: http://a/\nSuites: noble-security\nComponents: main\nEnabled: no\n"}
|
||||
if _, _, ok := securitySources(files); ok {
|
||||
t.Fatal("a disabled paragraph must not count")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSecuritySourcesNone(t *testing.T) {
|
||||
files := map[string]string{"/etc/apt/sources.list": "deb http://mirror/debian bookworm main\n"}
|
||||
if _, _, ok := securitySources(files); ok {
|
||||
t.Fatal("ok = true with no security suite, want false")
|
||||
}
|
||||
}
|
||||
|
||||
// A folded Suites field continues on lines that start with whitespace. Those
|
||||
// lines belong to Suites and must be filtered with it, not copied verbatim.
|
||||
func TestSecuritySourcesDeb822FoldedSuites(t *testing.T) {
|
||||
files := map[string]string{"/x.sources": "Types: deb\nURIs: http://a/\nSuites: noble\n noble-security\nComponents: main\n"}
|
||||
_, d822, ok := securitySources(files)
|
||||
if !ok || !strings.Contains(d822, "Suites: noble-security\n") {
|
||||
t.Fatalf("got ok=%v\n%s", ok, d822)
|
||||
}
|
||||
if strings.Contains(d822, "\n noble") {
|
||||
t.Fatalf("the folded continuation line must be dropped:\n%s", d822)
|
||||
}
|
||||
}
|
||||
|
||||
// A folded Suites field with no security suite on any line drops the paragraph.
|
||||
func TestSecuritySourcesDeb822FoldedSuitesNoSecurity(t *testing.T) {
|
||||
files := map[string]string{"/x.sources": "Types: deb\nURIs: http://a/\nSuites: noble\n\tnoble-updates\nComponents: main\n"}
|
||||
if _, _, ok := securitySources(files); ok {
|
||||
t.Fatal("no security suite across the folded lines, want ok=false")
|
||||
}
|
||||
}
|
||||
|
||||
// An inline Signed-By key block is a multi-line field of its own. Its
|
||||
// continuation lines stay verbatim, including the "." blank-line marker.
|
||||
func TestSecuritySourcesDeb822InlineSignedBy(t *testing.T) {
|
||||
key := "Signed-By: -----BEGIN PGP PUBLIC KEY BLOCK-----\n .\n mQINBGRkZXYBEAC\n -----END PGP PUBLIC KEY BLOCK-----\n"
|
||||
files := map[string]string{"/x.sources": "Types: deb\nURIs: http://a/\nSuites: noble noble-security\nComponents: main\n" + key}
|
||||
_, d822, ok := securitySources(files)
|
||||
if !ok || !strings.Contains(d822, "Suites: noble-security\n") {
|
||||
t.Fatalf("got ok=%v\n%s", ok, d822)
|
||||
}
|
||||
if !strings.Contains(d822, key) {
|
||||
t.Fatalf("inline key block must be kept verbatim:\n%s", d822)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// canStart is the whole "may this phase begin" decision. The maintenance
|
||||
// window deadline only gates the start of a phase: a package manager that is
|
||||
// already running is never interrupted by it, because killing apt, dnf or
|
||||
// Windows Update partway through a transaction is worse than letting it
|
||||
// finish late. A zero deadline is a manual run, which always may start.
|
||||
func canStart(now, deadline time.Time) bool {
|
||||
return deadline.IsZero() || now.Before(deadline)
|
||||
}
|
||||
|
||||
// startGate returns the error reported when a phase is refused.
|
||||
func startGate(deadline time.Time, phase string) error {
|
||||
if canStart(time.Now(), deadline) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("the maintenance window ended before %s could start", phase)
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestCanStart(t *testing.T) {
|
||||
now := time.Date(2026, 9, 20, 2, 30, 0, 0, time.UTC)
|
||||
cases := []struct {
|
||||
name string
|
||||
deadline time.Time
|
||||
want bool
|
||||
}{
|
||||
{"no deadline (manual run)", time.Time{}, true},
|
||||
{"deadline ahead", now.Add(time.Second), true},
|
||||
{"deadline exactly now", now, false},
|
||||
{"deadline passed", now.Add(-time.Minute), false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := canStart(now, c.deadline); got != c.want {
|
||||
t.Errorf("%s: got %v, want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStartGate(t *testing.T) {
|
||||
if err := startGate(time.Time{}, "the upgrade"); err != nil {
|
||||
t.Fatalf("zero deadline: %v", err)
|
||||
}
|
||||
err := startGate(time.Now().Add(-time.Minute), "the upgrade")
|
||||
if err == nil || !strings.Contains(err.Error(), "the maintenance window ended before the upgrade could start") {
|
||||
t.Fatalf("passed deadline: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package updates
|
||||
|
||||
import "sync"
|
||||
|
||||
// outputTailMax bounds the package-manager output carried back in a
|
||||
// PatchResult. The end of the output is where apt and dnf say what failed, so
|
||||
// the newest bytes are the ones kept.
|
||||
const outputTailMax = 64 << 10
|
||||
|
||||
// tailBuffer is an io.Writer that retains only the last max bytes written.
|
||||
// Stdout and stderr are both pointed at one, so it is safe for concurrent use.
|
||||
type tailBuffer struct {
|
||||
mu sync.Mutex
|
||||
max int
|
||||
buf []byte
|
||||
}
|
||||
|
||||
func newTailBuffer(max int) *tailBuffer { return &tailBuffer{max: max} }
|
||||
|
||||
func (t *tailBuffer) Write(p []byte) (int, error) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.buf = append(t.buf, p...)
|
||||
if over := len(t.buf) - t.max; over > 0 {
|
||||
t.buf = append([]byte(nil), t.buf[over:]...)
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (t *tailBuffer) String() string {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return string(t.buf)
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTailBufferKeepsNewestBytes(t *testing.T) {
|
||||
b := newTailBuffer(10)
|
||||
_, _ = b.Write([]byte("0123456789"))
|
||||
_, _ = b.Write([]byte("abcde"))
|
||||
if got := b.String(); got != "56789abcde" {
|
||||
t.Fatalf("got %q, want %q", got, "56789abcde")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTailBufferSingleWriteLargerThanMax(t *testing.T) {
|
||||
b := newTailBuffer(4)
|
||||
n, err := b.Write([]byte("abcdefgh"))
|
||||
if err != nil || n != 8 {
|
||||
t.Fatalf("Write must report the full length consumed, got %d, %v", n, err)
|
||||
}
|
||||
if got := b.String(); got != "efgh" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTailBufferDefaultCap(t *testing.T) {
|
||||
b := newTailBuffer(outputTailMax)
|
||||
_, _ = b.Write([]byte(strings.Repeat("x", outputTailMax+100)))
|
||||
if len(b.String()) != outputTailMax {
|
||||
t.Fatalf("len %d, want %d", len(b.String()), outputTailMax)
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,11 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PackageUpdate is one pending update. On Linux it is a package with a version
|
||||
// on each side. On Windows CurrentVersion is empty and NewVersion carries the
|
||||
// KB article ID: a Windows update is not a version bump of a named package,
|
||||
@@ -14,11 +20,51 @@ type PackageUpdate struct {
|
||||
// CheckAvailable lists pending OS updates.
|
||||
func CheckAvailable() ([]PackageUpdate, error) { return checkAvailable() }
|
||||
|
||||
// ApplyAll installs every pending update. It never reboots: a control plane
|
||||
// silently restarting a production server is unrecoverable from the UI, so the
|
||||
// reboot stays a decision a person or a workflow makes. RebootRequired reports
|
||||
// when one is owed.
|
||||
func ApplyAll() error { return applyAll() }
|
||||
// ApplyOptions selects what an Apply run installs and when it must stop.
|
||||
type ApplyOptions struct {
|
||||
// SecurityOnly installs security fixes only. A host with no security
|
||||
// metadata reports Unsupported and installs nothing: it never falls back
|
||||
// to installing everything.
|
||||
SecurityOnly bool
|
||||
// Deadline is the end of the maintenance window. It only gates the start
|
||||
// of each phase (index refresh, upgrade, Windows install): a phase that
|
||||
// has not started by then is not started, and one already running is
|
||||
// allowed to finish, bounded by defaultApplyCap from its own start. Zero
|
||||
// means a manual run with no window.
|
||||
Deadline time.Time
|
||||
}
|
||||
|
||||
// Result is what one Apply run did. Output is the tail of the package
|
||||
// manager's combined output, for the operator to read when something failed.
|
||||
type Result struct {
|
||||
Output string
|
||||
Unsupported bool
|
||||
Reason string // why Unsupported, in words for the run page
|
||||
}
|
||||
|
||||
// ErrBusy means another Apply is already running on this host.
|
||||
var ErrBusy = errors.New("an update run is already in progress on this host")
|
||||
|
||||
// defaultApplyCap is the backstop for one started upgrade command, counted
|
||||
// from that command's own start. It exists for a package manager that hangs,
|
||||
// not to enforce the window.
|
||||
const defaultApplyCap = 2 * time.Hour
|
||||
|
||||
var applyMu sync.Mutex
|
||||
|
||||
// Apply installs pending updates. It never reboots: ScheduleReboot is a
|
||||
// separate decision taken by the caller, and only when the command asked.
|
||||
func Apply(opts ApplyOptions) (Result, error) {
|
||||
if !applyMu.TryLock() {
|
||||
return Result{}, ErrBusy
|
||||
}
|
||||
defer applyMu.Unlock()
|
||||
return apply(opts.SecurityOnly, opts.Deadline)
|
||||
}
|
||||
|
||||
// ScheduleReboot restarts the host after a short grace period, so a result
|
||||
// sent just before it has time to leave.
|
||||
func ScheduleReboot() error { return scheduleReboot() }
|
||||
|
||||
// RebootRequired reports whether this host is waiting on a restart.
|
||||
func RebootRequired() bool { return rebootRequired() }
|
||||
|
||||
@@ -4,9 +4,14 @@ import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -22,8 +27,6 @@ func detectPM() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
|
||||
|
||||
func checkAvailable() ([]PackageUpdate, error) {
|
||||
switch detectPM() {
|
||||
case "apt":
|
||||
@@ -43,30 +46,160 @@ func checkAvailable() ([]PackageUpdate, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// aptRefreshTimeout bounds the index refresh alone. The upgrade itself runs
|
||||
// under defaultApplyCap: one shared five-minute limit used to kill large
|
||||
// upgrades partway through.
|
||||
const aptRefreshTimeout = 5 * time.Minute
|
||||
|
||||
func applyAll() error {
|
||||
switch detectPM() {
|
||||
case "apt":
|
||||
// termGrace is how long a command has to exit after SIGTERM before it is
|
||||
// killed. Package managers finish or roll back their current step on TERM.
|
||||
const termGrace = 5 * time.Minute
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
if err := exec.CommandContext(ctx, "apt-get", "update", "-qq").Run(); err != nil {
|
||||
return err
|
||||
}
|
||||
return exec.CommandContext(ctx, "apt-get", "upgrade", "-y").Run()
|
||||
case "dnf":
|
||||
return exec.Command("dnf", "upgrade", "-y").Run()
|
||||
case "yum":
|
||||
return exec.Command("yum", "upgrade", "-y").Run()
|
||||
case "pacman":
|
||||
return exec.Command("pacman", "-Syu", "--noconfirm").Run()
|
||||
case "zypper":
|
||||
return exec.Command("zypper", "update", "-y").Run()
|
||||
case "apk":
|
||||
return exec.Command("apk", "upgrade").Run()
|
||||
default:
|
||||
return nil
|
||||
// phase is one package manager command, started only if the window deadline
|
||||
// has not passed and then bounded by its own limit from its own start.
|
||||
type phase struct {
|
||||
deadline time.Time
|
||||
out io.Writer
|
||||
env []string
|
||||
}
|
||||
|
||||
func (p phase) run(name string, limit time.Duration, args ...string) error {
|
||||
return p.runNamed("the upgrade", name, limit, args...)
|
||||
}
|
||||
|
||||
func (p phase) runNamed(label, name string, limit time.Duration, args ...string) error {
|
||||
if err := startGate(p.deadline, label); err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), limit)
|
||||
defer cancel()
|
||||
fmt.Fprintf(p.out, "$ %s %s\n", name, strings.Join(args, " "))
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
cmd.Stdout, cmd.Stderr = p.out, p.out
|
||||
cmd.Env = append(os.Environ(), p.env...)
|
||||
// On the backstop, ask the package manager to stop rather than killing it
|
||||
// outright, and give it time to leave its database consistent.
|
||||
cmd.Cancel = func() error { return cmd.Process.Signal(syscall.SIGTERM) }
|
||||
cmd.WaitDelay = termGrace
|
||||
return cmd.Run()
|
||||
}
|
||||
|
||||
func apply(securityOnly bool, deadline time.Time) (Result, error) {
|
||||
out := newTailBuffer(outputTailMax)
|
||||
ph := phase{deadline: deadline, out: out}
|
||||
|
||||
var err error
|
||||
switch pm := detectPM(); pm {
|
||||
case "apt":
|
||||
var res Result
|
||||
res, err = applyApt(deadline, out, securityOnly)
|
||||
if res.Unsupported {
|
||||
res.Output = out.String()
|
||||
return res, nil
|
||||
}
|
||||
case "dnf", "yum":
|
||||
args := []string{"upgrade", "-y"}
|
||||
if securityOnly {
|
||||
args = append(args, "--security")
|
||||
}
|
||||
err = ph.run(pm, defaultApplyCap, args...)
|
||||
case "zypper":
|
||||
if securityOnly {
|
||||
err = ph.run("zypper", defaultApplyCap, "--non-interactive", "patch", "--category", "security")
|
||||
} else {
|
||||
err = ph.run("zypper", defaultApplyCap, "--non-interactive", "update")
|
||||
}
|
||||
// 102 and 103 mean "installed, and a reboot or restart is now needed".
|
||||
// That is success; RebootRequired reports the rest.
|
||||
var ee *exec.ExitError
|
||||
if errors.As(err, &ee) && (ee.ExitCode() == 102 || ee.ExitCode() == 103) {
|
||||
err = nil
|
||||
}
|
||||
case "pacman":
|
||||
if securityOnly {
|
||||
return Result{Unsupported: true, Reason: "pacman publishes no security metadata"}, nil
|
||||
}
|
||||
err = ph.run("pacman", defaultApplyCap, "-Syu", "--noconfirm")
|
||||
case "apk":
|
||||
if securityOnly {
|
||||
return Result{Unsupported: true, Reason: "apk publishes no security metadata"}, nil
|
||||
}
|
||||
if err = ph.runNamed("the apk index refresh", "apk", aptRefreshTimeout, "update"); err == nil {
|
||||
err = ph.run("apk", defaultApplyCap, "upgrade")
|
||||
}
|
||||
default:
|
||||
return Result{Unsupported: true, Reason: "no supported package manager found"}, nil
|
||||
}
|
||||
return Result{Output: out.String()}, err
|
||||
}
|
||||
|
||||
func applyApt(deadline time.Time, out io.Writer, securityOnly bool) (Result, error) {
|
||||
ph := phase{deadline: deadline, out: out, env: []string{"DEBIAN_FRONTEND=noninteractive"}}
|
||||
var srcOpts []string
|
||||
if securityOnly {
|
||||
dir, res, err := writeSecuritySourceParts()
|
||||
if err != nil || res.Unsupported {
|
||||
return res, err
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
srcOpts = []string{
|
||||
"-o", "Dir::Etc::SourceList=/dev/null",
|
||||
"-o", "Dir::Etc::SourceParts=" + dir,
|
||||
// Without this, an update against the reduced source set deletes
|
||||
// every other list file and the next normal apt call sees nothing.
|
||||
"-o", "APT::Get::List-Cleanup=0",
|
||||
}
|
||||
}
|
||||
if err := ph.runNamed("the apt index refresh", "apt-get", aptRefreshTimeout, append([]string{"update", "-q"}, srcOpts...)...); err != nil {
|
||||
return Result{}, fmt.Errorf("apt-get update: %w", err)
|
||||
}
|
||||
args := []string{"upgrade", "-y", "-q",
|
||||
"-o", "Dpkg::Options::=--force-confdef",
|
||||
"-o", "Dpkg::Options::=--force-confold"}
|
||||
return Result{}, ph.runNamed("the upgrade", "apt-get", defaultApplyCap, append(args, srcOpts...)...)
|
||||
}
|
||||
|
||||
// writeSecuritySourceParts writes the security-only sources to a temporary
|
||||
// directory for Dir::Etc::SourceParts. The caller removes the directory.
|
||||
func writeSecuritySourceParts() (string, Result, error) {
|
||||
files := map[string]string{}
|
||||
paths := []string{"/etc/apt/sources.list"}
|
||||
for _, pat := range []string{"/etc/apt/sources.list.d/*.list", "/etc/apt/sources.list.d/*.sources"} {
|
||||
m, _ := filepath.Glob(pat)
|
||||
paths = append(paths, m...)
|
||||
}
|
||||
for _, p := range paths {
|
||||
if b, err := os.ReadFile(p); err == nil {
|
||||
files[p] = string(b)
|
||||
}
|
||||
}
|
||||
list, d822, ok := securitySources(files)
|
||||
if !ok {
|
||||
return "", Result{Unsupported: true, Reason: "no security suites found in apt sources"}, nil
|
||||
}
|
||||
dir, err := os.MkdirTemp("", "vantage-apt-security-")
|
||||
if err != nil {
|
||||
return "", Result{}, err
|
||||
}
|
||||
if list != "" {
|
||||
if err := os.WriteFile(filepath.Join(dir, "security.list"), []byte(list), 0o644); err != nil {
|
||||
os.RemoveAll(dir)
|
||||
return "", Result{}, err
|
||||
}
|
||||
}
|
||||
if d822 != "" {
|
||||
if err := os.WriteFile(filepath.Join(dir, "security.sources"), []byte(d822), 0o644); err != nil {
|
||||
os.RemoveAll(dir)
|
||||
return "", Result{}, err
|
||||
}
|
||||
}
|
||||
return dir, Result{}, nil
|
||||
}
|
||||
|
||||
// scheduleReboot gives the host one minute, so the PatchResult announcing the
|
||||
// reboot is on the wire before the network goes down.
|
||||
func scheduleReboot() error {
|
||||
return exec.Command("shutdown", "-r", "+1", "Vantage patch policy").Run()
|
||||
}
|
||||
|
||||
// rebootRequired reads what the distributions themselves record. Debian and
|
||||
|
||||
@@ -4,6 +4,17 @@
|
||||
// without it this file compiles on Linux too and collides with updates_linux.go.
|
||||
package updates
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
func checkAvailable() ([]PackageUpdate, error) { return nil, nil }
|
||||
func applyAll() error { return nil }
|
||||
func rebootRequired() bool { return false }
|
||||
|
||||
func apply(securityOnly bool, deadline time.Time) (Result, error) {
|
||||
return Result{Unsupported: true, Reason: "OS updates are not supported on this platform"}, nil
|
||||
}
|
||||
|
||||
func scheduleReboot() error { return errors.New("reboot is not supported on this platform") }
|
||||
|
||||
func rebootRequired() bool { return false }
|
||||
|
||||
@@ -3,6 +3,7 @@ package updates
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -14,10 +15,6 @@ const (
|
||||
// routinely slow. Ten minutes is not generous, it is realistic.
|
||||
searchTimeout = 10 * time.Minute
|
||||
|
||||
// A patch-Tuesday cumulative genuinely takes this long to download and
|
||||
// install on a modest server.
|
||||
applyTimeout = 60 * time.Minute
|
||||
|
||||
rebootTimeout = 2 * time.Minute
|
||||
)
|
||||
|
||||
@@ -39,37 +36,6 @@ foreach ($u in $result.Updates) {
|
||||
ConvertTo-Json -InputObject @($rows) -Depth 3 -Compress
|
||||
`
|
||||
|
||||
const applyScript = `
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$session = New-Object -ComObject Microsoft.Update.Session
|
||||
$result = $session.CreateUpdateSearcher().Search("IsInstalled=0 and Type='Software' and IsHidden=0")
|
||||
|
||||
$batch = New-Object -ComObject Microsoft.Update.UpdateColl
|
||||
foreach ($u in $result.Updates) {
|
||||
if ($u.InstallationBehavior.CanRequestUserInput) { continue }
|
||||
if (-not $u.EulaAccepted) {
|
||||
try { $u.AcceptEula() } catch { continue }
|
||||
}
|
||||
$null = $batch.Add($u)
|
||||
}
|
||||
|
||||
if ($batch.Count -eq 0) { Write-Output 'nothing-to-install'; exit 0 }
|
||||
|
||||
$downloader = $session.CreateUpdateDownloader()
|
||||
$downloader.Updates = $batch
|
||||
$null = $downloader.Download()
|
||||
|
||||
$installer = $session.CreateUpdateInstaller()
|
||||
$installer.Updates = $batch
|
||||
$r = $installer.Install()
|
||||
|
||||
Write-Output ('resultcode=' + $r.ResultCode)
|
||||
# 2 = succeeded, 3 = succeeded with errors. Anything else failed, and this
|
||||
# process must exit non-zero so the agent logs a failure rather than an ack.
|
||||
if ($r.ResultCode -ne 2 -and $r.ResultCode -ne 3) { exit 1 }
|
||||
exit 0
|
||||
`
|
||||
|
||||
const rebootScript = `
|
||||
$ErrorActionPreference = 'SilentlyContinue'
|
||||
$si = New-Object -ComObject Microsoft.Update.SystemInfo
|
||||
@@ -95,14 +61,29 @@ func checkAvailable() ([]PackageUpdate, error) {
|
||||
return parseUpdateSearch(out)
|
||||
}
|
||||
|
||||
func applyAll() error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), applyTimeout)
|
||||
func apply(securityOnly bool, deadline time.Time) (Result, error) {
|
||||
if err := startGate(deadline, "the Windows Update install"); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
// The window deadline only gates the start. Once running, the install is
|
||||
// bounded by defaultApplyCap from its own start, with the default kill:
|
||||
// Windows has no SIGTERM to offer PowerShell.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), defaultApplyCap)
|
||||
defer cancel()
|
||||
|
||||
if _, err := winexec.Run(ctx, applyScript); err != nil {
|
||||
return fmt.Errorf("windows update install: %w", err)
|
||||
out, err := winexec.Run(ctx, applyScriptFor(securityOnly))
|
||||
tail := newTailBuffer(outputTailMax)
|
||||
_, _ = tail.Write([]byte(out))
|
||||
if err != nil {
|
||||
return Result{Output: tail.String()}, fmt.Errorf("windows update install: %w", err)
|
||||
}
|
||||
return nil
|
||||
return Result{Output: tail.String()}, nil
|
||||
}
|
||||
|
||||
// scheduleReboot gives the host sixty seconds, so the PatchResult announcing
|
||||
// the reboot is sent before the service stops.
|
||||
func scheduleReboot() error {
|
||||
return exec.Command("shutdown", "/r", "/t", "60", "/c", "Vantage patch policy").Run()
|
||||
}
|
||||
|
||||
func rebootRequired() bool {
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package updates
|
||||
|
||||
// applyScriptFor builds the Windows Update install script. It lives in a file
|
||||
// with no build tag so its content is tested on Linux: this module has no
|
||||
// Windows CI.
|
||||
//
|
||||
// Security-only keeps updates in the Security Updates or Critical Updates
|
||||
// classifications. Those two GUIDs are fixed by Microsoft and identical on
|
||||
// every Windows Update and WSUS server.
|
||||
func applyScriptFor(securityOnly bool) string {
|
||||
flag := "$false"
|
||||
if securityOnly {
|
||||
flag = "$true"
|
||||
}
|
||||
return "$SecurityOnly = " + flag + "\n" + applyScriptBody
|
||||
}
|
||||
|
||||
const applyScriptBody = `
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$securityCats = @('0FA1201D-4330-4FA8-8AE9-B877473B6441', 'E6CF1350-C01B-414D-A61F-263D14D133B4')
|
||||
$session = New-Object -ComObject Microsoft.Update.Session
|
||||
$result = $session.CreateUpdateSearcher().Search("IsInstalled=0 and Type='Software' and IsHidden=0")
|
||||
|
||||
$batch = New-Object -ComObject Microsoft.Update.UpdateColl
|
||||
foreach ($u in $result.Updates) {
|
||||
if ($u.InstallationBehavior.CanRequestUserInput) { continue }
|
||||
if ($SecurityOnly) {
|
||||
$isSec = $false
|
||||
foreach ($c in $u.Categories) { if ($securityCats -contains $c.CategoryID.ToUpper()) { $isSec = $true } }
|
||||
if (-not $isSec) { continue }
|
||||
}
|
||||
if (-not $u.EulaAccepted) {
|
||||
try { $u.AcceptEula() } catch { continue }
|
||||
}
|
||||
Write-Output ('selected: ' + $u.Title)
|
||||
$null = $batch.Add($u)
|
||||
}
|
||||
|
||||
if ($batch.Count -eq 0) { Write-Output 'nothing-to-install'; exit 0 }
|
||||
|
||||
$downloader = $session.CreateUpdateDownloader()
|
||||
$downloader.Updates = $batch
|
||||
$null = $downloader.Download()
|
||||
|
||||
$installer = $session.CreateUpdateInstaller()
|
||||
$installer.Updates = $batch
|
||||
$r = $installer.Install()
|
||||
|
||||
Write-Output ('resultcode=' + $r.ResultCode)
|
||||
# 2 = succeeded, 3 = succeeded with errors. Anything else failed, and this
|
||||
# process must exit non-zero so the agent reports a failure rather than an ack.
|
||||
if ($r.ResultCode -ne 2 -and $r.ResultCode -ne 3) { exit 1 }
|
||||
exit 0
|
||||
`
|
||||
@@ -0,0 +1,29 @@
|
||||
package updates
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
catSecurity = "0FA1201D-4330-4FA8-8AE9-B877473B6441"
|
||||
catCritical = "E6CF1350-C01B-414D-A61F-263D14D133B4"
|
||||
)
|
||||
|
||||
func TestApplyScriptSecurityOnly(t *testing.T) {
|
||||
s := applyScriptFor(true)
|
||||
if !strings.HasPrefix(strings.TrimSpace(s), "$SecurityOnly = $true") {
|
||||
t.Fatalf("script must open with the flag set:\n%s", s)
|
||||
}
|
||||
for _, id := range []string{catSecurity, catCritical} {
|
||||
if !strings.Contains(s, id) {
|
||||
t.Errorf("script missing category %s", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyScriptAll(t *testing.T) {
|
||||
if !strings.HasPrefix(strings.TrimSpace(applyScriptFor(false)), "$SecurityOnly = $false") {
|
||||
t.Fatal("script must open with the flag cleared")
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
// Package winexec runs PowerShell on Windows hosts.
|
||||
//
|
||||
// It exists because three subsystems — updates, workload collection and
|
||||
// workload logs — all need the same invocation, and because getting a
|
||||
// It exists because three subsystems - updates, workload collection and
|
||||
// workload logs - all need the same invocation, and because getting a
|
||||
// multi-line script past Go quoting, cmd.exe quoting and PowerShell's own
|
||||
// parser is a problem worth solving once.
|
||||
package winexec
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"gitea.hostxtra.co.uk/vantage/vantage-agent/internal/winexec"
|
||||
)
|
||||
|
||||
// AgentUnit is the service this agent runs as — the NSSM service name written
|
||||
// AgentUnit is the service this agent runs as - the NSSM service name written
|
||||
// by installer/setup.ps1. Change one, change the other.
|
||||
const AgentUnit = "VantageAgent"
|
||||
|
||||
|
||||
@@ -30,8 +30,8 @@ const dockerTimeout = 30 * time.Second
|
||||
// dockerInspect is the subset of `docker inspect` output we read.
|
||||
//
|
||||
// We use inspect rather than `docker ps --format '{{json .}}'` because ps
|
||||
// reports health and uptime inside a human Status string — "Up 2 hours
|
||||
// (healthy)" — and anything built on that is parsing English that is
|
||||
// reports health and uptime inside a human Status string - "Up 2 hours
|
||||
// (healthy)" - and anything built on that is parsing English that is
|
||||
// localised, reworded between releases, and silently different for a paused or
|
||||
// restarting container. inspect gives typed fields instead.
|
||||
type dockerInspect struct {
|
||||
@@ -59,7 +59,7 @@ type dockerInspect struct {
|
||||
}
|
||||
|
||||
// collectDocker enumerates containers. It returns ok=false with an empty error
|
||||
// string when Docker is simply not installed — the common case on this fleet,
|
||||
// string when Docker is simply not installed - the common case on this fleet,
|
||||
// and not a fault.
|
||||
func collectDocker(ctx context.Context) ([]Workload, bool, string) {
|
||||
if _, err := exec.LookPath("docker"); err != nil {
|
||||
|
||||
@@ -25,7 +25,7 @@ func logsPlatform(ctx context.Context, kind, id string, tail int) (string, error
|
||||
|
||||
// Timestamps are formatted PowerShell-side rather than left to
|
||||
// ConvertTo-Json, whose DateTime rendering differs between PowerShell
|
||||
// versions — one of them emits /Date(1699...)/.
|
||||
// versions - one of them emits /Date(1699...)/.
|
||||
//
|
||||
// $ErrorActionPreference = 'SilentlyContinue' because Get-WinEvent
|
||||
// treats "no events matched" as a terminating error, and a quiet
|
||||
@@ -38,7 +38,7 @@ func logsPlatform(ctx context.Context, kind, id string, tail int) (string, error
|
||||
|
||||
// ProviderName includes the host-wide Service Control Manager, so a
|
||||
// -MaxEvents cap of exactly tail would apply to the combined stream
|
||||
// before parseEvents narrows SCM rows down to this service — on a
|
||||
// before parseEvents narrows SCM rows down to this service - on a
|
||||
// host with busy service churn the target's own events could be
|
||||
// squeezed out of the window entirely. Over-fetch instead, hard-capped
|
||||
// so a pathological host cannot pull an unbounded batch across the
|
||||
@@ -76,7 +76,7 @@ ConvertTo-Json -InputObject @($rows) -Depth 3 -Compress
|
||||
}
|
||||
|
||||
// serviceDisplayName resolves a service's display name, which is what Service
|
||||
// Control Manager events name it by. An empty answer is fine — the filter then
|
||||
// Control Manager events name it by. An empty answer is fine - the filter then
|
||||
// matches on the service name alone.
|
||||
func serviceDisplayName(ctx context.Context, id string) string {
|
||||
out, err := winexec.Run(ctx,
|
||||
|
||||
@@ -15,7 +15,7 @@ const systemdTimeout = 30 * time.Second
|
||||
var excludedPrefixes = []string{"systemd-", "user@", "user-", "session-", "init.scope"}
|
||||
|
||||
// collectUnits enumerates services in two passes, because "running or
|
||||
// failed" and "enabled but stopped" are different questions — and an enabled
|
||||
// failed" and "enabled but stopped" are different questions - and an enabled
|
||||
// unit that is not running is exactly the one worth seeing.
|
||||
func collectUnits(ctx context.Context) ([]Workload, bool, string) {
|
||||
if _, err := exec.LookPath("systemctl"); err != nil {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build !linux && !windows
|
||||
|
||||
// The build constraint is load-bearing — see updates_other.go.
|
||||
// The build constraint is load-bearing - see updates_other.go.
|
||||
package workloads
|
||||
|
||||
import (
|
||||
|
||||
@@ -19,7 +19,7 @@ type winService struct {
|
||||
}
|
||||
|
||||
// exitCodeNeverStarted is ERROR_SERVICE_NEVER_STARTED. A stopped service
|
||||
// carrying it has not failed — it has not run since boot — and painting that
|
||||
// carrying it has not failed - it has not run since boot - and painting that
|
||||
// red would cry wolf on every host.
|
||||
const exitCodeNeverStarted = 1077
|
||||
|
||||
@@ -52,8 +52,8 @@ func servicePath(pathName string) string {
|
||||
}
|
||||
|
||||
// exeBoundaryIndex finds the first ".exe" (case-insensitive) in s that
|
||||
// actually ends the executable name — followed by end-of-string, whitespace,
|
||||
// or a double quote — rather than continuing into a longer segment such as
|
||||
// actually ends the executable name - followed by end-of-string, whitespace,
|
||||
// or a double quote - rather than continuing into a longer segment such as
|
||||
// ".exec". It returns -1 when no such occurrence exists, so a path like
|
||||
// `C:\Program Files\Ad.exec\tool.com -flag` is not misparsed by matching the
|
||||
// ".exe" inside "Ad.exec" and silently dropping the real filename.
|
||||
@@ -113,8 +113,8 @@ func parseServices(jsonText, systemRoot string) ([]Workload, error) {
|
||||
continue
|
||||
}
|
||||
|
||||
// The wire shape is shared with the systemd collector — both report
|
||||
// under kind "unit" — so the state word has to be too, or the UI
|
||||
// The wire shape is shared with the systemd collector - both report
|
||||
// under kind "unit" - so the state word has to be too, or the UI
|
||||
// (which colours and filters on it, and does so before it knows
|
||||
// which platform sent the row) needs two vocabularies for one kind.
|
||||
// running/stopped/failed become active/inactive/failed to match.
|
||||
@@ -168,7 +168,7 @@ type winEvent struct {
|
||||
// applied before SCM rows are narrowed down to this service would squeeze the
|
||||
// target's own events out of the window on a host with busy service churn.
|
||||
// tail is therefore applied here, AFTER filtering and AFTER the oldest-first
|
||||
// reversal, keeping the last tail lines — the most recent lines are the ones
|
||||
// reversal, keeping the last tail lines - the most recent lines are the ones
|
||||
// worth keeping, matching capLog's front-trim reasoning in the shared
|
||||
// logs.go.
|
||||
func parseEvents(jsonText, serviceName, displayName string, tail int) (string, error) {
|
||||
|
||||
@@ -13,7 +13,7 @@ func TestServicePath(t *testing.T) {
|
||||
{`"C:\no\args.exe"`, `C:\no\args.exe`},
|
||||
{``, ``},
|
||||
// ".exe" appearing inside an earlier segment ("Ad.exec") must not be
|
||||
// treated as the end of the executable — that would drop the real
|
||||
// treated as the end of the executable - that would drop the real
|
||||
// filename and arguments.
|
||||
{`C:\Program Files\Ad.exec\tool.com -flag`, `C:\Program`},
|
||||
// An unterminated quote falls back to the unquoted handling on the
|
||||
@@ -72,7 +72,7 @@ func TestParseServicesFilters(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// 1077 means "no attempt to start since boot" — a clean stopped service, not a
|
||||
// 1077 means "no attempt to start since boot" - a clean stopped service, not a
|
||||
// failure, and reporting it red would cry wolf on every host.
|
||||
func TestParseServicesExitCode1077(t *testing.T) {
|
||||
in := `[{"Name":"Idle","DisplayName":"Idle","State":"Stopped","StartMode":"Auto","PathName":"C:\\Idle\\i.exe","ExitCode":1077}]`
|
||||
|
||||
@@ -37,7 +37,7 @@ func Collect(ctx context.Context) Result {
|
||||
//
|
||||
// It sorts first: `docker ps` output ordering is not stable, and an
|
||||
// ordering-sensitive hash would resend the full list every 60 seconds forever
|
||||
// — a cost visible only as traffic.
|
||||
// - a cost visible only as traffic.
|
||||
//
|
||||
// StartedAt is deliberately excluded: it does not change while a container
|
||||
// runs, and including it would add nothing. Restarts IS included, because a
|
||||
|
||||
Reference in New Issue
Block a user