ref:main

CI service health check passes prematurely: pg_isready probes the socket, not TCP #29

closed Opened by cole.christensen@gmail.com

Links

No links yet.

Symptom

Jobs with a postgres service intermittently fail at DB setup with:

tcp connect (localhost:5432): connection refused — :econnrefused
** (Mix) The database for <Repo> couldn't be created: killed

Reproduced deterministically with a postgres:18 service (fangorn/reader test job); a postgres:16 service (fangorn/mail) usually wins the race and passes. Not memory, and not pg18-specific startup — a bare postgres:18 container starts and accepts connections normally.

Root cause

src/runner/service_manager.rs:

  • health_check_cmd for postgres runs docker exec <c> pg_isready -qno -h, so pg_isready checks the UNIX socket.
  • The official postgres image’s first boot runs initdb, which starts a temporary server listening only on the Unix socket (listen_addresses='', TCP disabled) to create the DB and run init scripts.
  • During that window pg_isready (socket) returns success → wait_for_health returns → the job starts → but the job connects over TCP to localhost:5432, where the real listener isn’t up yet → econnrefused.

Secondary: wait_for_health prints warning: health check timed out and returns anyway on a real timeout, so a service that never becomes healthy yields a confusing downstream error instead of failing the job.

Fix

  1. Probe the TCP endpoint the job actually uses: pg_isready -q -h 127.0.0.1 (during initdb’s socket-only temp server, TCP is refused → the wait correctly continues until the real listener is up).
  2. Make wait_for_health fail the service start on timeout (clear error + cleanup) instead of proceeding silently. Bump the window to 60s for slower first-boot inits.

This is a shared-primitive fix — it makes readiness honest for every postgres CI service, not just the one that hit the race.

Acceptance

  • postgres health check probes TCP (127.0.0.1), verified by a unit test on health_check_cmd
  • timeout is a hard failure with a clear message, not a silent proceed
  • reader’s test job (postgres:18) goes green without a per-repo readiness workaround