Anvil
fangorn/anvil-cli public
ref:main
← Linked to a private epic

Prepared-image pruning is a silent no-op (mis-parsed docker output); fix as last-used, not creation-time #41

closed Opened by cole.christensen@gmail.com

Links

Parent
  • 🔒 private issue

prune_prepared_images(max_age_days, max_count) (src/runner/prepare.rs:173) has two retention problems.

1. It runs only at runner startup

src/runner/loop_runner.rs:75 calls prepare::prune_prepared_images(30, 10) once, before the poller loop. A runner that stays up for weeks — which is the normal case, and the whole point of a stateful worker — never prunes again. Disk grows until someone restarts it, and the 30-day expiry never fires.

2. It evicts by creation date, not last use

images.sort_by(|a, b| b.1.cmp(a.1));       // CreatedAt, newest first

for (tag, _) in images.iter().skip(max_count) { /* docker rmi */ }

So “keep the 10 newest” is by build time. A prepared image built once and used every day for a month is evicted in favour of one built yesterday and never used since. That is backwards for a cache.

The cost is not uniform: anvil-cli’s cross-compilation toolchain (zig + cargo-zigbuild from source + macOS SDK + four rustup targets) takes roughly 25 minutes to rebuild. One unlucky eviction is a 25-minute stall on an otherwise 2-minute job.

Ask

  • Prune periodically (e.g. alongside the existing heartbeat), not just at boot.
  • Track last-used and evict LRU. Docker does not record this, so the runner needs its own marker — touching a file keyed by tag when image_exists_locally hits is enough, and survives restarts if it lives under the runner config dir.

Minor, same function

The age comparison does a lexicographic string compare of Docker’s {{.CreatedAt}} ("2026-07-22 10:06:54 +0000 UTC") against a %Y-%m-%d cutoff. It works while every timestamp shares a timezone and stays > 9 chars, but it is comparing formats that were never guaranteed to align. Parsing the date would be cheaper to reason about than the invariant.

Part of fangorn/anvil#354.

colechristensen cole.christensen@gmail.com commented 2026-07-22 23:07

Scoped within epic fangorn/anvil#354 to the LRU / last-used half: keep the expensive (~25-min) prepared image from being evicted by the current creation-time sort, since runners are shared across repos and image churn is real. The ‘prune more than once per process’ disk-hygiene half is worth doing but is not on the epic’s critical path — treat it as a separable follow-up.

colechristensen cole.christensen@gmail.com commented 2026-07-22 23:13

Correction + confirmed root cause (raised by a scoping review)

The “minor, same function” note below undersold this. The prune deletes nothing, ever — verified.

prune_prepared_images (src/runner/prepare.rs:173) formats with {{.CreatedAt}}, which Docker renders as a multi-token string — on a real host: 2026-07-22 02:57:58 -0500 CDT. The parser then does:

let parts: Vec<&str> = line.splitn(3, ' ').collect();   // line: "<ID> <date> <time> <tz> <TZ> anvil-prepared:<hash>"

Some((parts[2], parts[1]))                               // (tag, created_at)

For a real line abc123 2026-07-22 10:06:54 +0000 UTC anvil-prepared:b0f07b5ea862:

  • parts[1] (used as created_at) = "2026-07-22" — clean date, so the age comparison actually works.
  • parts[2] (passed to docker rmi) = "10:06:54 +0000 UTC anvil-prepared:b0f07b5ea862" — a ref containing spaces.

.args(["rmi", tag]) passes that whole string as one argv, docker rmi rejects it as an invalid reference, and let _ = swallows the error. Both prune loops (keep-N and age) hit the same garbage tag, so no prepared image is ever removed.

Consequence: unbounded disk growth on every runner, until the disk fills (cf fangorn/anvil#334 — a full runner disk blocked every PR). The 30-day / keep-10 policy has never actually run.

This reframes the fix priority

  1. Correctness (new, urgent): make prune work at all. Don’t string-split {{.CreatedAt}}. Use a delimiter-safe format — e.g. --format '{{.Repository}}:{{.Tag}}\t{{.ID}}' and get the timestamp from docker inspect --format '{{.Created}}' (RFC3339, no spaces) or {{.Metadata.LastTagTime}}, splitting on \t. Parse the tag as one field, never by whitespace.

  2. Scoping (the original ask): evict by last-used, not creation time. Content-addressing means one prepared image is legitimately shared across repos and steps, so retention must not be scoped per repo/step/arch — that would defeat the sharing. It must be scoped by use: keep an image iff something still uses it, regardless of which workload built it or how old it is. Docker doesn’t record last-use, so the runner needs its own marker — touch a file keyed by tag on each image_exists_locally hit (under the runner config dir, survives restarts), evict LRU.

  3. Cadence: prune periodically, not once at boot (loop_runner.rs:75). A long-lived runner currently never re-prunes — though today that’s moot since the prune is a no-op anyway.

Namespace scoping is already correct: docker images anvil-prepared only ever considers anvil-prepared:*, so base/service/user images are never at risk.

(Everything below predates this correction.)

colechristensen cole.christensen@gmail.com commented 2026-07-26 20:08

Fixed by #43 (merged).

Verified on main (src/runner/prepare.rs, src/runner/loop_runner.rs):

  • Periodic, not boot-only — prune_prepared_images(30, 10) runs at startup and on a 6h interval via spawn_blocking, so a long-lived runner keeps reclaiming disk.
  • LRU, not creation-time — touch_prepared_image(&tag) records use (a marker file under the runner dir, surviving restarts) on every image_exists_locally hit and after a build; last_used(tag) feeds the eviction sort, which is now sort_by_key(|(_, used)| Reverse(*used)). An image built once and used daily now survives.
  • The lexicographic date compare is gone — retention operates on SystemTime/Duration (now.duration_since(*used) > max_age), and the policy is unit-tested without touching Docker or the clock.
Back to issues