ref:main

Stop the CI runner from dropping log lines #36

merged colechristensen cole.christensen@gmail.com wants to merge fix/runner-log-drop into main

Summary

Failing CI jobs were showing a truncated log — the compiler/error block (the part that explains the failure) was missing, so anvil ci job-view jumped from the last Compiling … line straight to Uploading N artifact(s)... / Job failed. This made two real build failures undiagnosable from CI.

Proof it was dropped, not absent

I reproduced the identical build in Docker (same runner image + pinned toolchain). Locally cargo prints a ~15-line error block ending in error: could not compile \anvil-cli``. The CI log for the same build contained none of it. So the runner captured the lines but never persisted them.

Root cause (runner log shipper, src/runner/)

Storage/read paths have no caps — the loss was fire-and-forget shipping:

  • log_reporter.rs: flush() did mem::take on the buffer, then send_lines silently dropped the batch after 2 failed retries. The executor’s final flush() carries the build’s error tail, so one failed POST lost it permanently while later small flushes (Uploading...) still landed — exactly the observed symptom.
  • loop_runner.rs: flush_handle.abort() with no final flush stranded whatever the 1 s timer hadn’t yet picked up.

Fix

  • flush() re-queues lines at the front of the buffer on send failure instead of dropping them, so a later flush retries (order preserved).
  • New bounded drain() — flush until the buffer empties (capped at MAX_DRAIN_PASSES so an unreachable server can’t hang completion) — called at every job-completion path (normal + workspace/service/prepare failures) after aborting the timer, so the job’s final output is delivered before the job is marked terminal.

Tests (wiremock)

  • does_not_drop_lines_when_a_flush_fails_then_server_recovers — server 503s through a full flush’s retries then recovers; asserts the lines are still delivered and a send happened after the failure window. Fails on the old code (only 3 requests, “dropping lines”), passes now.
  • drain_delivers_buffered_taildrain() delivers a buffered line with no timer involved.

147 tests pass; clippy -D warnings and fmt clean.

Closes #27

Created Jul 13, 2026 at 02:35 UTC | Merged Jul 13, 2026 at 02:59 UTC by colechristensen cole.christensen@gmail.com