perf(pack): streaming parse-and-store with bounded LRU base cache #31

merged colechristensen cole.christensen@gmail.com wants to merge stream-pack-resolve into main
No CI

Replaces Reader.parse/2’s collect-everything-into-a-list shape with a callback-driven Reader.parse_stream/3. Each resolved object is handed to the caller as it’s decoded, then dropped — the BEAM is free to GC the binary before the next entry arrives.

The receive-pack flow uses this to write each object to storage immediately. Base lookups for in-pack OFS_DELTA / REF_DELTA chains hit a bounded LRU (~4096 entries) first; on a miss, we fall back to an external_resolver that reads the just-written object back from storage. Memory usage becomes O(LRU + 1 in-flight object), independent of pack size.

Measured on the openvswitch/ovs pack (96 MB, 134k objects, 108k deltas):

peak RSS post-GC
before 3.4–3.5 GB (OOM at 3.82 GiB) n/a
after 622 MB 63 MB

Memory stays bounded throughout the run (oscillates 200–470 MB) instead of growing with pack size. Parse time ~54 s including the simulated storage write per object.

Reader.parse/2 is kept as a thin wrapper over parse_stream/3 for legacy callers — same buffered shape, same buffered cost. Only the streaming caller (receive_pack) gets the new memory profile.

mix test 928/928 passes, mix format and mix credo –strict clean.

Final piece of fangorn/anvil#153 — “pushing huge repos sucks”. Builds on #27 (streaming receive), #28 (no SHA throttle), #29 (offset cache), #30 (delta unroll).

Created May 06, 2026 at 18:12 UTC | Merged May 06, 2026 at 19:05 UTC by colechristensen cole.christensen@gmail.com