CI runner silently drops log lines, hiding job errors #27
Links
No links yet.
Symptom
A failing CI job’s log is missing its tail — the actual error. anvil ci job-view shows the compile progress, then jumps straight to Uploading N artifact(s)... / Job failed, with the entire cargo/compiler error block gone. This made two fix/macos-release-builds runs (jobs 2c887282fa, f314fc3d49) undiagnosable from CI alone.
Proof it’s dropped, not absent
Reproduced the identical build in Docker (same runner image, pinned zig/cargo-zigbuild/SDK). Locally cargo prints a ~15-line error block ending in error: could not compile \anvil-cli`. The CI log for the same build contains **none** of it — it ends at Compiling reqwest→Uploading 4 artifact(s)…`. So the runner captured the lines but never persisted them.
Root cause (runner, src/runner/)
Storage and read paths have no caps/truncation — the loss is in the runner’s log shipper:
-
PRIMARY —
log_reporter.rs:flush()doesstd::mem::take(&buffer)before the POST (line 47), andsend_linessilently drops the batch after 2 failed retries (lines 70-75). A failed final flush (executor.rs:390, which carries the build’s tail incl. the error block) discards those lines permanently, while later small flushes (Uploading...) still land — exactly the observed symptom. -
SECONDARY —
loop_runner.rs:411:flush_handle.abort()with no finalflush().awaitafterward, so any line the 1s timer hasn’t picked up is never sent.
Fix
The runner must not lose captured log lines: re-queue on send failure instead of dropping, and do a reliable blocking drain at job completion before the job is marked terminal.
Acceptance
- On transient server failure, no log lines are lost (delivered once the server recovers).
- A job’s final output (error block +
Uploading/Uploaded) always appears inci job-view. - Regression test with a failing-then-recovering mock server.