ref:main

CI runner silently drops log lines, hiding job errors #27

closed Opened by cole.christensen@gmail.com

Links

No links yet.

Symptom

A failing CI job’s log is missing its tail — the actual error. anvil ci job-view shows the compile progress, then jumps straight to Uploading N artifact(s)... / Job failed, with the entire cargo/compiler error block gone. This made two fix/macos-release-builds runs (jobs 2c887282fa, f314fc3d49) undiagnosable from CI alone.

Proof it’s dropped, not absent

Reproduced the identical build in Docker (same runner image, pinned zig/cargo-zigbuild/SDK). Locally cargo prints a ~15-line error block ending in error: could not compile \anvil-cli`. The CI log for the same build contains **none** of it — it ends at Compiling reqwestUploading 4 artifact(s)…`. So the runner captured the lines but never persisted them.

Root cause (runner, src/runner/)

Storage and read paths have no caps/truncation — the loss is in the runner’s log shipper:

  1. PRIMARY — log_reporter.rs: flush() does std::mem::take(&buffer) before the POST (line 47), and send_lines silently drops the batch after 2 failed retries (lines 70-75). A failed final flush (executor.rs:390, which carries the build’s tail incl. the error block) discards those lines permanently, while later small flushes (Uploading...) still land — exactly the observed symptom.

  2. SECONDARY — loop_runner.rs:411: flush_handle.abort() with no final flush().await afterward, so any line the 1s timer hasn’t picked up is never sent.

Fix

The runner must not lose captured log lines: re-queue on send failure instead of dropping, and do a reliable blocking drain at job completion before the job is marked terminal.

Acceptance

  • On transient server failure, no log lines are lost (delivered once the server recovers).
  • A job’s final output (error block + Uploading/Uploaded) always appears in ci job-view.
  • Regression test with a failing-then-recovering mock server.