ref:54347b0462b8b88d8dab5565cebbb10572dc570f

ci: bake the runner-build toolchain into a prepared image (prepare adoption) (#44)

Delivers pillar 1 of the prepare epic (fangorn/anvil#354 → fangorn/anvil-cli#43): anvil-cli's own release build now uses Anvil's headline `prepare` feature. ## Why `build-runner` took ~1860s, of which **~1750s was re-acquiring a toolchain every run** — downloading zig, compiling `cargo-zigbuild` from source (~130 crates), fetching the macOS SDK, adding four rustup targets. The four cross-compiles are ~110s. That re-acquisition is also what pushed the job past its 1800s timeout and shipped release 2026.07.6 with half its assets missing (#40). ## What - **Toolchain → step-level `prepare`.** Baked into a cached image once, reused across runs. `ci/build-runner.sh` drops to build + verify + publish. Version pins move into the `.anvil.yml` prepare commands, because the prepared-image cache key hashes those strings — a bump in the script wouldn't invalidate the image (the correctness constraint I flagged). - **Atomic publish (#40).** Create the release as a `--draft`, upload all five assets, then `release publish`. Confirmed server-side: `latest_release()` filters `draft: false`, so `/runner/version` never serves a draft — a mid-upload failure leaves an invisible draft, not a broken public release. prepare removing the timeout is the primary fix; the draft is the backstop. - **Delete the `cache:` blocks.** `cache` is validated but does nothing (fangorn/anvil#358); the six blocks cached nothing and misrepresented the pipeline. ## Deferred to a follow-up (blocked on deploy) The same-ISA multi-arch split — one linux/arm64 + one linux/amd64 runner, each building its own ISA to avoid slow cross-ISA `ring` codegen — is the other half of #43. It needs the affinity fix (fangorn/anvil#209 / #361) **deployed** first: two per-ISA jobs on the current server would deadlock, since affinity pins the whole run to one worker and the other arch's job hangs. Shipping it now would hang this repo's CI. ## Validation note The **first** CI run on this branch builds the prepared image (~25 min, once); subsequent runs are fast. I can't build the prepared image locally (no Docker, 25-min build) — CI on this PR is the first real exercise of the prepare commands. They are the same install steps `ci/build-runner.sh` ran before, relocated verbatim with pins intact, so the risk is low, but watch the first run.
SHA: 54347b0462b8b88d8dab5565cebbb10572dc570f
Author: Anvil <noreply@anvil.fangorn.io>
Date: 2026-07-23 03:30
Parents: 7c350b4
2 files changed +94 -110
Type
.anvil.yml +26 −28
@@ -1,21 +1,18 @@
image: rust:1.95-trixie
# NOTE: no `cache:` blocks. `cache` is validated but not implemented
# (fangorn/anvil#358) — the blocks that were here cached nothing and only
# misrepresented what the pipeline does. Build state is kept warm by the
# runner's persistent workspace; the runner build's heavy toolchain is baked
# into a reused image via the build-runner `prepare` block below.
steps:
- name: check
run: cargo check --all-targets 2>&1
cache:
key: Cargo.lock
paths:
- target/
- /usr/local/cargo/registry/
- name: test
run: cargo test 2>&1
depends_on: [check]
cache:
key: Cargo.lock
paths:
- target/
- name: coverage
timeout_seconds: 1800
@@ -24,10 +21,6 @@
cargo install cargo-tarpaulin --version 0.32.7 2>&1
cargo tarpaulin --engine llvm --out lcov --output-dir coverage/ 2>&1
depends_on: [check]
cache:
key: Cargo.lock
paths:
- target/
artifacts:
- name: lcov.info
path: coverage/lcov.info
@@ -37,10 +30,6 @@
rustup component add clippy
cargo clippy --all-targets -- -D warnings 2>&1
depends_on: [check]
cache:
key: Cargo.lock
paths:
- target/
- name: fmt
run: |
@@ -50,26 +39,35 @@
- name: build
run: cargo build --release 2>&1
depends_on: [test, clippy, fmt]
cache:
key: Cargo.lock
paths:
- target/
artifacts:
- name: anvil-binary
path: target/release/anvil
- name: build-runner
timeout_seconds: 1800
# The cross-compilation toolchain is baked into a cached prepared image
# (fangorn/anvil#354), so it is installed once and reused across runs rather
# than re-downloaded and re-compiled every build (~1750s → seconds).
#
# Version pins live HERE, in the prepare commands, on purpose: the prepared
# image's cache key hashes these command strings. A bump must change a
# command to invalidate the image — bumping a value in ci/build-runner.sh
# would not. See ci/build-runner.sh for why each tool is pinned as it is.
prepare:
- apt-get update && apt-get install -y --no-install-recommends bash curl jq xz-utils ca-certificates
# zig 0.15.2 — 0.14.x has a macho-linker regression that can't resolve
# -liconv/-lcharset for apple-darwin under rust >= 1.82.
- curl -sSL "https://ziglang.org/download/0.15.2/zig-$(uname -m)-linux-0.15.2.tar.xz" -o /tmp/zig.tar.xz && mkdir -p /opt/zig && tar -xJf /tmp/zig.tar.xz -C /opt/zig --strip-components=1 && rm /tmp/zig.tar.xz
# cargo-zigbuild 0.23.0 — pinned in lockstep with the zig above.
- cargo install --locked cargo-zigbuild --version 0.23.0
- rustup target add aarch64-unknown-linux-gnu x86_64-unknown-linux-gnu aarch64-apple-darwin x86_64-apple-darwin
# macOS SDK 12.3 — libSystem + frameworks zig does not bundle. Unpacks to
# /opt/MacOSX12.3.sdk; ci/build-runner.sh globs /opt/MacOSX*.sdk for it.
- curl -sSL "https://github.com/joseluisq/macosx-sdks/releases/download/12.3/MacOSX12.3.sdk.tar.xz" -o /tmp/macos-sdk.tar.xz && mkdir -p /opt && tar -xJf /tmp/macos-sdk.tar.xz -C /opt && rm /tmp/macos-sdk.tar.xz
# Delegated to a bash script because the runner invokes /bin/sh (dash)
# and we need bash features (pipefail, trap ERR) for publish rollback.
run: exec bash ci/build-runner.sh
run: |
apt-get update && apt-get install -y bash curl jq xz-utils 2>&1
exec bash ci/build-runner.sh
depends_on: [test, clippy, fmt]
cache:
key: Cargo.lock
paths:
- target/
artifacts:
- name: anvil_linux_arm64
path: runner-dist/anvil_linux_arm64
ci/build-runner.sh +68 −82
@@ -1,13 +1,26 @@
#!/usr/bin/env bash
set -euo pipefail
# Build and (on main) publish the runner binaries for all four targets.
#
# The cross-compilation toolchain — zig, cargo-zigbuild, the macOS SDK, and the
# rustup targets — is NOT installed here. It is baked into a cached prepared
# image by the `prepare:` block on the build-runner step in .anvil.yml, so it is
# not reinstalled on every run (fangorn/anvil-cli#43, fangorn/anvil#354). This
# script assumes that image: zig on PATH under /opt/zig, cargo-zigbuild in
# CARGO_HOME, the targets added, and the SDK unpacked under /opt.
#
# IMPORTANT: the toolchain *version pins* live in .anvil.yml's prepare commands,
# because the prepared-image cache key hashes those command strings, not this
# script. Bumping a version here would not rebuild the image. Keep pins there.
# The CI runner bind-mounts the workspace owned by a non-root host user
# into a root-run container. Git's safe-directory check rejects that,
# exiting 128 when we later try to read log history for the changelog.
git config --global --add safe.directory /workspace
# Fail fast if we're about to publish a release but the secret is missing —
# don't waste 10+ minutes on a build that will error at the end.
# don't waste minutes on a build that will error at the end.
PUBLISH=0
if [ "${ANVIL_BRANCH:-}" = "main" ] || [ "${ANVIL_BRANCH:-}" = "refs/heads/main" ]; then
if [ -z "${ANVIL_TOKEN:-}" ]; then
@@ -19,8 +32,8 @@
fi
# Compute the CalVer version and tag HEAD locally so that build.rs can
# read it via 'git describe --tags'. The tag is local-only — it never
# gets pushed. The server-side tag is created by 'anvil release create'.
# read it via 'git describe --tags'. The tag is local-only — it never
# gets pushed. The server-side tag is created by 'anvil release create'.
if [ "$PUBLISH" = "1" ]; then
VERSION=$(bash ci/release.sh)
if [ -z "$VERSION" ]; then
@@ -31,67 +44,41 @@
git tag -f "$VERSION"
fi
# ── Pinned glibc floor (anvil-cli#20) ───────────────────────────────────
# ── Toolchain env (binaries come from the prepared image) ────────────────
#
# Building natively on the trixie image bakes a GLIBC_2.39 requirement into
# the binary, which then fails to load on older runtimes (Debian 12 bookworm
# / glibc 2.36, Ubuntu 22.04 / 2.35, ...). cargo-zigbuild links against an
# older glibc supplied by zig, so the binaries run on glibc >= GLIBC_FLOOR.
# zig cc also cross-links amd64, so no separate gnu cross-compiler is needed.
# The prepare block installs the toolchain but a `docker commit` snapshots the
# filesystem, not exported env vars — so set PATH/SDKROOT here at build time.
export PATH="/opt/zig:$PATH"
# Pinned, mutually-compatible pair (cargo-zigbuild is sensitive to the zig
# version it drives). Bump both together and re-check the glibc floor.
# zig 0.14.0 has a macho-linker regression that can't resolve -liconv/-lcharset
# for apple-darwin under rust >= 1.82 (rust-lang/rust#128370, cargo-zigbuild#316)
# — the macOS cross-link fails with "unable to find dynamic system library
# 'iconv'". zig 0.15.2 resolves it. Bumped cargo-zigbuild in lockstep.
ZIG_VERSION=0.15.2
ZIGBUILD_VERSION=0.23.0
GLIBC_FLOOR=2.31
if ! command -v zig >/dev/null 2>&1; then
echo "==> Installing zig $ZIG_VERSION"
# macOS targets: no glibc floor (Apple libSystem). zig bundles glibc/musl but
# NOT Apple's non-redistributable libSystem + frameworks, so cross-linking
# against Darwin needs a macOS SDK, found via SDKROOT. Glob the SDK dir so this
# script carries no SDK version (the version pin lives in .anvil.yml).
SDKROOT=$(echo /opt/MacOSX*.sdk)
if [ ! -d "$SDKROOT" ]; then
echo "ERROR: macOS SDK not found under /opt (expected the prepared image to unpack it)" >&2
exit 1
# zig flipped its archive naming to zig-<arch>-<os>-<ver> as of 0.14.1.
curl -sSL "https://ziglang.org/download/${ZIG_VERSION}/zig-$(uname -m)-linux-${ZIG_VERSION}.tar.xz" \
-o /tmp/zig.tar.xz
mkdir -p /opt/zig
tar -xJf /tmp/zig.tar.xz -C /opt/zig --strip-components=1
export PATH="/opt/zig:$PATH"
fi
export SDKROOT
# Link against an older Darwin so the binaries run on macOS 11+ (Big Sur).
export MACOSX_DEPLOYMENT_TARGET=11.0
command -v cargo-zigbuild >/dev/null 2>&1 || \
cargo install --locked cargo-zigbuild --version "$ZIGBUILD_VERSION" 2>&1
# ── Pinned glibc floor (anvil-cli#20) ───────────────────────────────────
#
# Building natively on the trixie image bakes a GLIBC_2.39 requirement into the
rustup target add \
aarch64-unknown-linux-gnu x86_64-unknown-linux-gnu \
aarch64-apple-darwin x86_64-apple-darwin 2>&1
# binary, which then fails to load on older runtimes. cargo-zigbuild links
# against an older glibc supplied by zig, so the binaries run on
# glibc >= GLIBC_FLOOR. This floor is a build/verify constant (not an install
# version), so it stays here.
GLIBC_FLOOR=2.31
echo "==> Building linux arm64 (glibc $GLIBC_FLOOR floor)..."
cargo zigbuild --release --target "aarch64-unknown-linux-gnu.${GLIBC_FLOOR}" 2>&1
echo "==> Building linux amd64 (glibc $GLIBC_FLOOR floor)..."
cargo zigbuild --release --target "x86_64-unknown-linux-gnu.${GLIBC_FLOOR}" 2>&1
# macOS targets: no glibc floor (Apple libSystem, not glibc). zig bundles
# glibc/musl but NOT Apple's (non-redistributable) libSystem + frameworks, so
# cross-linking against Darwin frameworks needs a macOS SDK — cargo-zigbuild
# finds it via SDKROOT (auto-detected on macOS, but this runner is Linux).
# Dropping native-tls for rustls removed the Security framework, but std +
# iana-time-zone (via chrono) still link CoreFoundation, so the SDK is required.
# Pinned, immutable SDK release asset; extracts to /opt/MacOSX${VER}.sdk.
MACOS_SDK_VERSION=12.3
MACOS_SDK_DIR="/opt/MacOSX${MACOS_SDK_VERSION}.sdk"
if [ ! -d "$MACOS_SDK_DIR" ]; then
echo "==> Fetching macOS SDK $MACOS_SDK_VERSION"
curl -sSL "https://github.com/joseluisq/macosx-sdks/releases/download/${MACOS_SDK_VERSION}/MacOSX${MACOS_SDK_VERSION}.sdk.tar.xz" \
-o /tmp/macos-sdk.tar.xz
mkdir -p /opt
echo "==> Building macos arm64..."
tar -xJf /tmp/macos-sdk.tar.xz -C /opt
fi
export SDKROOT="$MACOS_SDK_DIR"
# Link against an older Darwin so the binaries run on macOS 11+ (Big Sur), not
# just the SDK's own version.
export MACOSX_DEPLOYMENT_TARGET=11.0
echo "==> Building macos arm64 (SDK $MACOS_SDK_VERSION)..."
cargo zigbuild --release --target aarch64-apple-darwin 2>&1
echo "==> Building macos amd64 (SDK $MACOS_SDK_VERSION)..."
echo "==> Building macos amd64..."
cargo zigbuild --release --target x86_64-apple-darwin 2>&1
ARM64_BIN="target/aarch64-unknown-linux-gnu/release/anvil"
@@ -99,11 +86,10 @@
MACOS_ARM64_BIN="target/aarch64-apple-darwin/release/anvil"
MACOS_AMD64_BIN="target/x86_64-apple-darwin/release/anvil"
# Hard-gate the glibc floor: fail the build if either binary references a
# GLIBC symbol newer than the floor. This is the exact failure we're fixing
# (GLIBC_2.39 on a glibc-2.36 runner); without the gate a future dep or
# toolchain bump silently raises it again. The version tags live as ASCII in
# the dynamic string table, so grep finds them without binutils.
# Hard-gate the glibc floor: fail the build if either linux binary references a
# GLIBC symbol newer than the floor. Without the gate a future dep or toolchain
# bump silently raises it again. The version tags live as ASCII in the dynamic
# string table, so grep finds them without binutils.
assert_glibc_floor() {
local bin="$1" max
max=$(grep -aoE 'GLIBC_[0-9]+\.[0-9]+' "$bin" | sort -V | tail -1)
@@ -134,8 +120,8 @@
# Use the binary matching THIS runner's architecture to drive `release
# list`/`release create` — the publish job can land on either an arm64 (carl)
# or amd64 (xps) runner. Hardcoding the arm64 build here made publishing fail
# or amd64 (xps) runner. Hardcoding one made publishing fail with "Exec format
# error" whenever the job ran on the other runner.
# with "Exec format error" whenever the job ran on the amd64 runner.
case "$(uname -m)" in
aarch64 | arm64) ANVIL_CLI="$PWD/$ARM64_BIN" ;;
x86_64 | amd64) ANVIL_CLI="$PWD/$AMD64_BIN" ;;
@@ -152,13 +138,11 @@
cp "$MACOS_AMD64_BIN" "runner-dist/anvil_macos_amd64_${VERSION}"
# Compute SHA256 checksums for every versioned asset — published as
# SHA256SUMS_${VERSION} alongside the binaries so install scripts can verify
# SHA256SUMS_${VERSION} alongside the binaries so install scripts can
# verify what they downloaded. Format matches `shasum -a 256` /
# `sha256sum` output: `<hash> <filename>`.
# what they downloaded. Format matches `sha256sum` output: `<hash> <filename>`.
echo "==> Computing SHA256 checksums..."
(
cd runner-dist
# Use whichever is available; macOS has shasum, Linux has sha256sum.
ASSETS="anvil_linux_arm64_${VERSION} anvil_linux_amd64_${VERSION} anvil_macos_arm64_${VERSION} anvil_macos_amd64_${VERSION}"
if command -v sha256sum >/dev/null 2>&1; then
sha256sum $ASSETS > "SHA256SUMS_${VERSION}"
@@ -180,35 +164,37 @@
fi
BODY=$(printf 'Runner binaries for linux/{amd64,arm64} and macos/{amd64,arm64}.\n\n## Changes\n\n%s\n' "$CHANGELOG")
# Publish atomically: create the release as a DRAFT, upload every asset, and
# only publish once they are all present. A failure mid-upload leaves a draft
# (invisible to `anvil update` and to the version bump), never a half-populated
# public release — the failure mode that shipped 2026.07.6 broken (#40).
"$ANVIL_CLI" release create \
--tag "$VERSION" \
--title "anvil-cli $VERSION" \
--body "$BODY" \
--draft \
--repo fangorn/anvil-cli
# Roll back the draft if any subsequent step fails — don't leave orphaned
# drafts lying around. A SIGKILL (e.g. step timeout) can't run this trap, but
# Roll back the release if any subsequent step fails — don't leave
# a killed run leaves only an unpublished draft, which is harmless.
# orphaned empty/partial releases lying around.
cleanup_release() {
echo "==> Publish failed — rolling back release $VERSION" >&2
"$ANVIL_CLI" release delete "$VERSION" --repo fangorn/anvil-cli >&2 || true
}
trap cleanup_release ERR
for asset in \
"anvil_linux_arm64_${VERSION}" \
"anvil_linux_amd64_${VERSION}" \
"anvil_macos_arm64_${VERSION}" \
"anvil_macos_amd64_${VERSION}" \
"SHA256SUMS_${VERSION}"; do
"$ANVIL_CLI" release upload "$VERSION" "runner-dist/${asset}" --repo fangorn/anvil-cli
done
"$ANVIL_CLI" release upload "$VERSION" \
"runner-dist/anvil_linux_arm64_${VERSION}" \
--repo fangorn/anvil-cli
"$ANVIL_CLI" release upload "$VERSION" \
"runner-dist/anvil_linux_amd64_${VERSION}" \
--repo fangorn/anvil-cli
"$ANVIL_CLI" release upload "$VERSION" \
"runner-dist/anvil_macos_arm64_${VERSION}" \
--repo fangorn/anvil-cli
"$ANVIL_CLI" release upload "$VERSION" \
"runner-dist/anvil_macos_amd64_${VERSION}" \
--repo fangorn/anvil-cli
"$ANVIL_CLI" release upload "$VERSION" \
"runner-dist/SHA256SUMS_${VERSION}" \
--repo fangorn/anvil-cli
# Everything is present — flip the draft to published in one step.
"$ANVIL_CLI" release publish "$VERSION" --repo fangorn/anvil-cli
trap - ERR
echo "==> Published release $VERSION"