CI service health check passes prematurely: pg_isready probes the socket, not TCP #29
Links
No links yet.
Symptom
Jobs with a postgres service intermittently fail at DB setup with:
tcp connect (localhost:5432): connection refused — :econnrefused
** (Mix) The database for <Repo> couldn't be created: killed
Reproduced deterministically with a postgres:18 service (fangorn/reader test job); a postgres:16 service (fangorn/mail) usually wins the race and passes. Not memory, and not pg18-specific startup — a bare postgres:18 container starts and accepts connections normally.
Root cause
src/runner/service_manager.rs:
health_check_cmdfor postgres runsdocker exec <c> pg_isready -q— no-h, so pg_isready checks the UNIX socket.- The official postgres image’s first boot runs
initdb, which starts a temporary server listening only on the Unix socket (listen_addresses='', TCP disabled) to create the DB and run init scripts. - During that window
pg_isready(socket) returns success →wait_for_healthreturns → the job starts → but the job connects over TCP to localhost:5432, where the real listener isn’t up yet →econnrefused.
Secondary: wait_for_health prints warning: health check timed out and returns anyway on a real timeout, so a service that never becomes healthy yields a confusing downstream error instead of failing the job.
Fix
- Probe the TCP endpoint the job actually uses:
pg_isready -q -h 127.0.0.1(during initdb’s socket-only temp server, TCP is refused → the wait correctly continues until the real listener is up). - Make
wait_for_healthfail the service start on timeout (clear error + cleanup) instead of proceeding silently. Bump the window to 60s for slower first-boot inits.
This is a shared-primitive fix — it makes readiness honest for every postgres CI service, not just the one that hit the race.
Acceptance
- postgres health check probes TCP (127.0.0.1), verified by a unit test on
health_check_cmd - timeout is a hard failure with a clear message, not a silent proceed
- reader’s test job (postgres:18) goes green without a per-repo readiness workaround