ref:main

runner: complete tier-1 production work (status/logs/doctor/detach/multi-instance/launchctl-modern/runner-shutdown) #23

closed Opened by cole.christensen@gmail.com

Links

No links yet.

Problem

PR #22 ("production-ready tier 1 service management") shipped the PID-file + SIGTERM + service-install foundation but deferred the rest of the tier-1 scope into a follow-up. The remaining gaps:

  1. No `runner status` — operators can’t quickly see config / install / process state without running three separate commands.
  2. No `runner logs` — the user has to remember whether the install is systemd / launchd / detach and use the matching tool by hand.
  3. No `runner doctor` — there’s no single command that runs config-readable + server-reachable + token-valid + PID-state + unit-on-disk checks and returns a non-zero exit for monitoring.
  4. No `runner start –detach` — running the runner under nohup is fragile (terminal SIGHUP race during startup, no rotation, no clean parent-side "started OK / died with this tail" feedback).
  5. launchctl `load`/`unload` are deprecated on modern macOS; bootstrap/bootout/kill is the supported path.
  6. No multi-instance — `–service-name ` lets one host run multiple distinct runner installs (e.g. `gpu1`, `gpu2`), each with its own unit, state dir, and PID file.
  7. No runner-shutdown signal to server — when the runner cancels a job mid-flight (drain / SIGTERM), the server has no way to distinguish "runner pulled this" from "the job failed."

Proposal

Land all seven in a single PR (per the "each PR is a release" convention) so the new install can be marketed as the tier-1 production-ready cut.

Acceptance criteria

  • `runner status` shows config + install + process state, JSON-mode compatible
  • `runner logs –follow –lines` auto-detects systemd / launchd / detach mode and routes to journalctl / tail accordingly
  • `runner doctor` returns non-zero on any failed check
  • `runner doctor` uses a read-only endpoint (no side effects)
  • `runner start –detach` daemonizes with new session + process group, redirects stdio to a size-rotated log file, waits for the PID file to appear, and reports the live PID or the log tail on startup failure
  • launchctl path uses bootstrap / bootout / print on modern macOS with graceful fallback to load / unload / list on older macOS
  • launchctl bootstrap is idempotent (case-folded "already loaded" detection + EEXIST(17) exit code)
  • launchctl gui/ falls back to user/ for headless ssh where no Aqua session exists
  • `–service-name` everywhere (install/uninstall/start/stop/ restart/svc-status/logs/status/doctor)
  • per-instance state at `~/.anvil-runner/services/.json` with legacy single-file migration
  • per-instance PID file at `runner-.pid` for named installs; default keeps the legacy `runner.pid` path
  • `runner service list` enumerates installed instances
  • systemd ExecStart embeds `–pid-file` and `–parallel` so multi-instance can’t collide on PIDs
  • systemd user-scope keeps `ProtectHome=read-only` so the runner can still read `~/.anvil-runner/config.json`; system-scope uses StateDirectory and full ProtectHome
  • job-status update carries `cancel_reason: "runner_shutdown"` when the runner yanked the job (only — not on timeout-fail)
  • tests cover EndReason → cancel_reason wiring, PID-file clobber-resistance, log rotation chain, instance-name validation
  • `cargo fmt && cargo clippy –all-targets – -D warnings && cargo test` clean

Origin

Follow-up to PR #22; user feedback: "I said all in one PR… then you proceeded to not do a big chunk of the work."