Symptom

An ordinary git push of a new branch from stock git is rejected:

error: remote unpack failed: ref_delta_base_not_found: unresolvable REF_DELTA at offset 1370
 ! [remote rejected]   fix/375-ci-retry-semantics -> fix/375-ci-retry-semantics
    (ref_delta_base_not_found: unresolvable REF_DELTA at offset 1370)

The identical commit pushed with --no-thin succeeds. Single commit, 12 files, 760 insertions — nothing exotic. git fsck --no-dangling on the client is clean and the commit object resolves, so the client repo is fine.

Root cause

ExGitObjectstore.Protocol.ReceivePack.build_external_resolver/1 — lib/ex_git_objectstore/protocol/receive_pack.ex:542-552:

  # Build a function that resolves object SHAs from the existing repo.
  # Returns {type_atom, raw_content} matching the format Pack.Reader expects.
  defp build_external_resolver(repo) do
    fn sha ->
      case Repo.storage_call(repo, :get_object, [sha]) do
        {:ok, compressed} ->
          decompress_and_parse_object(compressed, sha)

        {:error, _} = err ->
          err
      end
    end
  end

storage_call(repo, :get_object, [sha]) is the loose-object-only path. On the Filesystem backend it is a bare File.read/1 against objects/xx/yyyy… (storage/filesystem.ex:40-48), returning {:error, :not_found} when the object is not a loose file. S3 and Memory are the same shape. None of them look inside packfiles.

The pack-aware reader is ExGitObjectstore.ObjectResolver.read/2, whose own moduledoc states the contract:

Resolves git objects by checking loose objects first, then searching packs.

So this resolver only sees one of the two places objects live.

Why that breaks push

Git sends a thin pack by default. Objects may be encoded as REF_DELTA against a base the sender knows the receiver already has, with the base referenced by SHA and deliberately not included. Completing them against existing repository objects is the receiver’s job — git index-pack --fix-thin is exactly this step.

If the delta base happens to be a loose object, resolution succeeds and the push works. If it lives in a packfile, this resolver returns :not_found and the whole push is rejected. That is precisely the observed error, and it is why --no-thin is a clean workaround: it forces complete objects, so there is no REF_DELTA to resolve at all.

It also explains the confusing pattern of pushes to fangorn/anvil failing while fangorn/ex_git_objectstore succeeds — different repositories with different pack-to-loose ratios, which reads as flakiness rather than a deterministic bug.

This is about to get much worse

Anvil PR #228 merged Anvil.Git.Maintenance to main: a scheduled repack, default interval 6 hours (lib/anvil/git/maintenance.ex:52, @default_interval_ms :timer.hours(6)), whose whole purpose is to move loose objects into packs — its own stats are packed / deleted. It is the fix for Anvil #324 (unbounded loose objects blocking efficient S3 storage).

The moment that deploys, repositories become predominantly packed, delta bases stop being loose objects, and git push starts failing across the org. Prod has not deployed recently, which is the only reason this is still latent rather than an outage. The repack landing turns a rare, confusing rejection into the normal case.

Suggested fix

Route the resolver through ObjectResolver.read/2 (pack-aware) instead of storage_call(:get_object, …), adapting the return shape to the {type_atom, raw_content} that Pack.Reader expects. I have not implemented or tested this — treat it as the likely fix, not a verified one.

A regression test needs a repository whose delta base is packed, not loose; a fixture built only from loose objects will pass against the current broken code and prove nothing.

Audit of the other loose-only call sites

There are three storage_call(repo, :get_object, …) sites:

Cross-references

• fangorn/ex_git_objectstore #77 — perf audit found receive_pack explodes packs into loose objects. Same code path, opposite direction: one writes loose when it should pack, the other reads only loose when it should read both. Likely worth fixing together.
• fangorn/anvil #324 (closed) — the server-side repack this collides with.
• fangorn/anvil #377 — where I first reported the symptom, before the root cause was known. Should probably be closed in favour of this.

Observed 2026-07-30 against anvil.fangorn.io. Prod pack/loose counts for fangorn/anvil (3 packs, 3,792 loose objects) were measured by a colleague; I did not reproduce that measurement myself, since prod git storage is on S3 rather than container disk.