Two runs of this repo started one second apart landed on the same runner and one failed on a source file that is provably present in its own commit.

What happened

Both on runner carl, overlapping for ~1 minute.

The failure:

error: couldn't read `tests/api_paths.rs`: No such file or directory (os error 2)
error: could not compile `anvil-cli` (test "api_paths") due to 1 previous error

tests/api_paths.rs is in commit eec9bab4:

$ git ls-tree -r --name-only origin/fix/registry-token-api-path | grep tests/
tests/api_paths.rs
tests/json_output.rs

and is not in a71a9a73, the commit the concurrent run was building. So the tree the compiler saw was the other run’s checkout.

Re-running the identical commit with nothing else in flight passes: run bdfca164ed, check/test/clippy/fmt/build all green.

Why this is more than flakiness

src/runner/workspace.rs:17 derives the workspace from the repo and slot only:

let workspace = work_dir.join(&owner).join(&repo).join(slot.to_string());

then git checkout --force <sha> (:39), with no git clean by design (:41, preserving build caches). So anything that maps two concurrent jobs onto the same path will silently rewrite one job’s source tree mid-build — which matches the symptom exactly.

Within a single runner process this should not happen: loop_runner.rs:89 gives each poller a fixed slot 1..=parallel. But the path contains no instance identifier, so two runner service instances sharing a work_dir collide — instance A slot 1 and instance B slot 1 resolve to the same directory. anvil runner service list exists precisely because multiple instances per host are supported.

I could not confirm the mechanism from the outside; cache restore across runs keyed on Cargo.lock is a second candidate. The runner logs for carl around 07:57–07:59 should settle it.

Impact

Silent and misleading: CI fails with an error implicating the developer’s commit, and passes on re-run, so the natural read is “flaky test” rather than “the runner served the wrong source”. Any two branches pushed together can hit it.

Suggested fix

Include the runner instance (and ideally the run id) in the workspace path, or take a lock on work_dir/owner/repo/<slot> for the duration of a job.

Found while pushing two PRs at once.