ref:main

Runner log re-queue loops forever on permanent (4xx) upload errors #28

closed Opened by cole.christensen@gmail.com

Links

No links yet.

Problem

The #27 fix made LogReporter.flush() re-queue log lines on send failure instead of dropping them (correct for transient failures). But it re-queues on ANY non-2xx, including permanent 4xx errors. When the server returns a permanent error for a batch (e.g. 422 for a malformed/rejected batch), the batch is re-queued → retried → 422 → re-queued forever.

Observed on runner carl: infinite log upload failed (status 422 Unprocessable Entity), retrying... / will re-queue, spinning two job slots and wedging the runner so it stopped picking up jobs. (The 422 root cause is fixed server-side in anvil#338, but the runner must never wedge on a permanent error regardless.)

Fix

Distinguish transient vs permanent send failures:

  • 2xx → ok.
  • 5xx / 408 / 429 / network/timeout → transient → re-queue (retry).
  • other 4xx → permanent → DROP the batch with a loud error (re-queueing can’t help).

Acceptance

  • On a permanent 4xx, the batch is dropped (not re-queued) and the reporter makes forward progress — no infinite loop.
  • On a transient 5xx-then-recover, lines are still delivered (the #27 behavior is preserved).
  • Regression tests for both.