ref:656f31b0648236594b491a3ef10bceeae9716349

Make CI service readiness honest: TCP health probe + fail on timeout (#38)

## Why CI jobs with a `postgres` service intermittently failed at DB setup: ``` tcp connect (localhost:5432): connection refused — :econnrefused ** (Mix) The database for <Repo> couldn't be created: killed ``` Deterministic with `postgres:18` (fangorn/reader test job); `postgres:16` (fangorn/mail) usually won the race. Not memory, not pg18 startup — a bare `postgres:18` starts fine. **Root cause:** `service_manager.rs` ran `pg_isready -q` with **no `-h`**, probing the **Unix socket**. The postgres image's first-boot `initdb` runs a temporary server that listens on the socket ONLY (`listen_addresses=''`); the socket check passes before the real **TCP** listener is up, so the job connects over TCP and gets `econnrefused`. Secondary: `wait_for_health` warned and **proceeded anyway** on a real timeout. ## What - **postgres health check probes TCP** (`pg_isready -q -h 127.0.0.1`) — during initdb's socket-only phase TCP is refused, so the wait correctly continues until the real listener is accepting the connections the job will make. - **`wait_for_health` returns `bool`; timeout is now a hard failure** (cleanup + clear `did not become healthy within 60s` error) instead of silently proceeding. Window 30s → 60s for slower first-boot inits. - Regression test: `postgres_health_check_probes_tcp_not_socket`. Shared-primitive fix — makes readiness honest for **every** postgres CI service, not a per-repo workaround. Verified: `cargo fmt/clippy` clean, 148+23 tests pass. ## Follow-up (not in this PR) Needs an anvil-cli **release + runner update on carl/xps** to take effect. Once deployed, fangorn/reader's test job (postgres:18) should go green without any per-repo readiness hack. Closes #29 🤖 Generated with [Claude Code](https://claude.com/claude-code)
SHA: 656f31b0648236594b491a3ef10bceeae9716349
Author: Anvil <noreply@anvil.fangorn.io>
Date: 2026-07-17 06:31
Parents: 450d86e
1 files changed +41 -6
Type
src/runner/service_manager.rs +41 −6
@@ -29,12 +29,21 @@
fn health_check_cmd(image: &str, container: &str) -> Option<Vec<String>> {
let lower = image.to_lowercase();
if lower.contains("postgres") {
// Probe over TCP (-h 127.0.0.1), NOT the default Unix socket. During
// first-boot initdb the postgres image runs a temporary server that
// listens ONLY on the socket (listen_addresses=''), so a socket
// pg_isready reports ready before the real TCP listener is up — the
// job then connects to localhost:5432 over TCP and gets econnrefused.
// Checking TCP makes readiness honest: it only passes once the real
// server is accepting the connections the job will actually make.
Some(vec![
"docker".into(),
"exec".into(),
container.into(),
"pg_isready".into(),
"-q".into(),
"-h".into(),
"127.0.0.1".into(),
])
} else if lower.contains("redis") {
Some(vec![
@@ -124,9 +133,17 @@
containers.push(container_name.clone());
// Wait for health check (up to 30s)
// Wait for the service to become ready before starting the job. A
// service that never becomes healthy is a hard failure — proceeding
// would surface later as a confusing econnrefused mid-job.
if let Some(check_cmd) = health_check_cmd(image, &container_name) {
if !wait_for_health(&check_cmd, 60).await {
wait_for_health(&check_cmd, 30).await;
cleanup_services(&containers, &network);
return Err(format!(
"service '{svc_name}' ({image}) did not become healthy within 60s"
)
.into());
}
} else {
// Brief pause for unknown services
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
@@ -171,19 +188,22 @@
})
}
/// Wait for a health check command to succeed. Returns `true` once it passes,
/// or `false` if it never succeeds within `timeout_secs` (the caller treats a
/// `false` as a hard failure rather than proceeding against a service that
/// Wait for a health check command to succeed.
async fn wait_for_health(cmd: &[String], timeout_secs: u64) {
/// isn't ready).
async fn wait_for_health(cmd: &[String], timeout_secs: u64) -> bool {
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(timeout_secs);
while tokio::time::Instant::now() < deadline {
if let Ok(output) = Command::new(&cmd[0]).args(&cmd[1..]).output() {
if output.status.success() {
return true;
return;
}
}
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
}
eprintln!("warning: health check timed out after {timeout_secs}s");
false
}
/// Build the `docker run` argument list for one service. Pure function —
@@ -322,5 +342,20 @@
let args = build_run_args("alias-name", &svc, "c", "net").unwrap();
let alias_pos = args.iter().position(|a| a == "--network-alias").unwrap();
assert_eq!(args.get(alias_pos + 1), Some(&"alias-name".to_string()));
}
#[test]
fn postgres_health_check_probes_tcp_not_socket() {
// Regression: `pg_isready` with no -h checks the Unix socket, which is
// up during initdb's temporary server before the real TCP listener is
// — so the job connects over TCP and gets econnrefused. The probe must
// force TCP (-h 127.0.0.1).
let cmd = health_check_cmd("postgres:18", "anvil-ci-svc-X-postgres").unwrap();
assert!(cmd.contains(&"pg_isready".to_string()));
let h_pos = cmd
.iter()
.position(|a| a == "-h")
.expect("postgres health check must pass -h for a TCP probe");
assert_eq!(cmd.get(h_pos + 1), Some(&"127.0.0.1".to_string()));
}
}