Stop the CI runner from dropping log lines #36
fix/runner-log-drop
into main
Summary
Failing CI jobs were showing a truncated log — the compiler/error block (the part that explains the failure) was missing, so anvil ci job-view jumped from the last Compiling … line straight to Uploading N artifact(s)... / Job failed. This made two real build failures undiagnosable from CI.
Proof it was dropped, not absent
I reproduced the identical build in Docker (same runner image + pinned toolchain). Locally cargo prints a ~15-line error block ending in error: could not compile \anvil-cli``. The CI log for the same build contained none of it. So the runner captured the lines but never persisted them.
Root cause (runner log shipper, src/runner/)
Storage/read paths have no caps — the loss was fire-and-forget shipping:
log_reporter.rs:flush()didmem::takeon the buffer, thensend_linessilently dropped the batch after 2 failed retries. The executor’s finalflush()carries the build’s error tail, so one failed POST lost it permanently while later small flushes (Uploading...) still landed — exactly the observed symptom.loop_runner.rs:flush_handle.abort()with no final flush stranded whatever the 1 s timer hadn’t yet picked up.
Fix
flush()re-queues lines at the front of the buffer on send failure instead of dropping them, so a later flush retries (order preserved).- New bounded
drain()— flush until the buffer empties (capped atMAX_DRAIN_PASSESso an unreachable server can’t hang completion) — called at every job-completion path (normal + workspace/service/prepare failures) after aborting the timer, so the job’s final output is delivered before the job is marked terminal.
Tests (wiremock)
does_not_drop_lines_when_a_flush_fails_then_server_recovers— server 503s through a full flush’s retries then recovers; asserts the lines are still delivered and a send happened after the failure window. Fails on the old code (only 3 requests, “dropping lines”), passes now.drain_delivers_buffered_tail—drain()delivers a buffered line with no timer involved.
147 tests pass; clippy -D warnings and fmt clean.
Closes #27