No way to inspect, prune, or force-rebuild prepared images #42
Links
No links yet.
Prepared images are the most expensive cached artifact a runner holds, and there is no operator interface to them at all.
Today, to answer “which prepared images does this worker have, how old are they, and which pipeline does each belong to”, you SSH to the box and run docker images anvil-prepared. The tag is a truncated sha256 with no back-reference to the pipeline that produced it, so even that only tells you how many there are.
To force a rebuild — after a base-image CVE, or when a curl | sh line has silently drifted (see fangorn/anvil#357) — the only supported move is to edit a command string so the hash changes. That is why the # bump 2 comment pattern shows up in CI configs.
Ask
CLI surface on the runner, alongside anvil runner status / logs / doctor:
anvil runner images— list prepared images: tag, base image, size, built-at, last-used (see #41), and the repo/pipeline that last used it.anvil runner images prune [--all｜--older-than <days>｜--keep <n>]— manual reclaim, and a way to exercise the retention policy without restarting the runner.anvil runner images rm <tag>— force one rebuild.
anvil runner doctor should probably also report prepared-image count and total disk, since “worker is out of disk” is otherwise diagnosed the hard way — cf. fangorn/anvil#334, where a full runner disk blocked every PR.
Part of fangorn/anvil#354.
cole.christensen@gmail.com
commented 2026-07-22 23:07
Deferred from epic fangorn/anvil#354 (rescoped to prepare adoption + same-ISA multi-arch). Operator convenience; revisit alongside fangorn/anvil-cli#41 when image management needs a human interface.
cole.christensen@gmail.com
commented 2026-07-30 19:05
Related but not the same defect — recording the relationship so neither gets closed as a duplicate of the other.
fangorn/anvil#374 was a compile job that ran on the runner host against the host’s Elixir 1.17.3 while .anvil.yml declared hexpm/elixir:1.20.2-…. The obvious suspicion was prepared-image lifecycle: stale image selected, or one never built. It was neither. The image path was never entered at all — the server built that run’s jobs from a stale DB-stored pipeline definition (the run’s commit was the all-zero SHA of a ref deletion, so .anvil.yml could not be read), and that definition declares no image:. The runner received image: null and ran the command on the host. No prepared image was involved in any direction.
So #42’s asks stand on their own. Two small effects from fangorn/anvil-cli#55, which is in flight:
Makes #42 slightly easier. Two of the three things #42 wants from anvil runner images now appear in the job log for the image a run actually used:
Using cached prepared image: anvil-prepared:9f2c1a04b7de (built from hexpm/elixir:1.20.2-erlang-29.0.3-debian-trixie-20260713)
That is the base-image back-reference the issue notes is missing from the tag — for the one image the job used, at the moment it used it. It is not the inventory anvil runner images should give you (nothing lists what a host holds, or its size or age), so it doesn’t close any part of this.
Neither helps nor hurts the rest. #55 does not touch tag computation, prune_prepared_images, the LRU markers, or image_exists_locally. anvil runner images rm <tag> as a force-rebuild lever works exactly as it does today.
One thing worth folding into anvil runner doctor while it is being specified here: with #55, a runner whose Docker daemon is unreachable now fails each claimed job with cannot obtain base image <image>: <docker's error> rather than a generic exit 1. That is the condition that took carl’s whole fleet down on 2026-07-30 (it claimed 4 jobs in 13 seconds while its socket was dying). doctor reporting daemon reachability alongside prepared-image count and disk would let an operator see it before the jobs do.