perf(s3): ranged pack reads + adaptive ObjectResolver (no whole-pack fetch for point reads) #44

Why

Investigating anvil’s Filesystem→S3 git cutover (anvil #84/#325), a single-object read on a packed repo downloaded the entire pack and parsed the whole index. Cold read on S3 (134k-object repo, 36 MiB pack): ~1300 ms — making the common web workload (view a file/commit/PR) unusable on S3.

What

Index.lookup_in_raw/2 — O(log n) point lookup from raw .idx bytes (fanout + binary search), no full parse. Cross-checked vs parse+lookup for every SHA of a real index.
get_pack_range/5 optional storage callback — S3 HTTP Range GET, Filesystem pread, Memory slice.
Reader.read_object_ranged/3 — read one object via windowed ranged fetches, resolving OFS/REF delta bases with more ranged fetches; truncation detected via the header’s declared uncompressed size. Cross-checked vs whole-pack read for every object incl. delta chains + a >window object.
Adaptive ObjectResolver — point reads go ranged (O(1) round-trips, no whole-pack); a pack escalates to whole-pack after 64 reads/process so clone/walk still pay one fetch.
Fixed clear_pack_cache/0 (was a no-op for 3-tuple-keyed caches).

Results (S3-packed via MinIO, 134k-object repo)

workload	before	after
single-object cold read	1304 ms	38 ms (matches S3-loose)
tree HEAD cold	107 ms	18 ms
full log walk (20,797)	10.8 s	4.6 s
clone (11k objects)	14.0 s	4.2 s

Tests

Full suite 979 passed; the Memory backend now advertises get_pack_range, so the whole suite (graph/diff/walk/upload-pack) exercises the ranged path. New: index/reader cross-checks + window growth + S3 Range-GET against MinIO.

Follow-up

On real S3 the per-pack .idx download becomes the cold-read long pole; a cross-request index cache would close it.

🤖 Generated with Claude Code