feat(runner): production-ready tier 1 service management (#22) #27
merged
cole.christensen@gmail.com wants to merge
feat/runner-production-ready
into main
No CI
Summary
Closes #22. Makes the anvil runner production-deployable across the three tier-1 modes — standalone, user systemd, system systemd (and user launchd / system launchd on macOS) — with one coherent control surface and N parallel job slots first-class.
Single PR by user request — everything needed for production deployment in one release.
What lands
Runtime foundation
- PID file + RAII Guard at `~/.anvil-runner/runner.pid`. Single-instance check on start (live → refuse, stale → warn + overwrite). `–pid-file` override.
- SIGTERM + SIGINT both trigger graceful drain. SIGHUP + SIGPIPE absorbed so dropped terminals / broken pipes don’t kill the runner.
- Cancel propagation to the executor — child gets SIGTERM, 5s grace, SIGKILL. Exit code 130 to distinguish from a normal failure.
- N-slot drain — `–shutdown-timeout` (default 60s) is a single wall-clock budget for all slots to wind up.
- Heartbeat reports capacity — `{"parallel": N, "slots_busy": M}` so the server can route jobs intelligently. Atomic counter, RAII reservation panic-safe.
New CLI subcommands
- `anvil runner stop [–force] [–timeout]` — reads PID file, SIGTERM, polls for clean exit, optional SIGKILL fallback.
- `anvil runner restart` — stop + `exec` into a fresh `runner start` (handles install.sh binary-upgrade case cleanly).
Service install overhauled
- Explicit `–scope user|system` (default detected). No more brittle `getuid()` heuristic.
- `–scope=system` requires `–user-account ` — refuses to install a system service running as root.
- `~/.anvil-runner/service.json` records scope + unit path + config path + user account + parallel. `svc-*` commands read it; no more guessing.
- macOS `–scope=system` writes `/Library/LaunchDaemons/` with `UserName`/`GroupName`. Previously only user-LaunchAgents was supported.
Hardened unit files
- systemd: `TimeoutStopSec=90`, `Restart=always`, `RestartSec=5`, `KillSignal=SIGTERM`, `ProtectSystem=strict`, `ProtectHome=true`, `NoNewPrivileges=true`, `PrivateTmp=true`, `StandardOutput=journal`. System scope adds `User=`, `Group=`, `WorkingDirectory=/var/lib/anvil-runner`.
- launchd: `ProcessType=Background`, `ThrottleInterval=5`, log paths under `~/Library/Logs/anvil-runner/` (user) or `/var/log/anvil-runner/` (system).
Test plan
- `cargo test` — 102/102 pass
- `pid_file`: absent / stale / dead-pid / live-pid / Guard lifecycle
- `slot_counter`: increment + RAII decrement
- `service_mode`: JSON round-trip
- `commands/runner`: clap parsing for stop / restart / service-install flags
- `cargo clippy –all-targets – -D warnings` clean
- `cargo fmt` clean
- CLI surface verified live: `anvil runner –help`, `runner stop –help`, `runner start –help`, `runner service install –help` all show the new flags/verbs
Out of scope (filed as follow-ups)
These would be useful but each is independently shippable and none blocks production deployment. Operators can use `systemctl status anvil-runner` and `journalctl -u anvil-runner` for now:
- `runner status` runtime view + `runner logs` + `runner doctor`
- `runner start –detach` standalone background mode
- macOS modern launchctl `bootstrap`/`bootout` migration (current code uses `load`/`unload`, which still works as compatibility shim)
- Multiple distinct runner identities on one host via `–service-name`
- Server-side `runner_shutdown` cancel reason for jobs killed mid-run
Closes #22
Created Jun 05, 2026 at 01:14 UTC
| Merged Jun 05, 2026 at 01:26 UTC
by
cole.christensen@gmail.com