perf(pack): streaming parse-and-store with bounded LRU base cache #31
stream-pack-resolve
into main
Replaces Reader.parse/2’s collect-everything-into-a-list shape with a callback-driven Reader.parse_stream/3. Each resolved object is handed to the caller as it’s decoded, then dropped — the BEAM is free to GC the binary before the next entry arrives.
The receive-pack flow uses this to write each object to storage immediately. Base lookups for in-pack OFS_DELTA / REF_DELTA chains hit a bounded LRU (~4096 entries) first; on a miss, we fall back to an external_resolver that reads the just-written object back from storage. Memory usage becomes O(LRU + 1 in-flight object), independent of pack size.
Measured on the openvswitch/ovs pack (96 MB, 134k objects, 108k deltas):
| peak RSS | post-GC | |
|---|---|---|
| before | 3.4–3.5 GB (OOM at 3.82 GiB) | n/a |
| after | 622 MB | 63 MB |
Memory stays bounded throughout the run (oscillates 200–470 MB) instead of growing with pack size. Parse time ~54 s including the simulated storage write per object.
Reader.parse/2 is kept as a thin wrapper over parse_stream/3 for legacy callers — same buffered shape, same buffered cost. Only the streaming caller (receive_pack) gets the new memory profile.
mix test 928/928 passes, mix format and mix credo –strict clean.
Final piece of fangorn/anvil#153 — “pushing huge repos sucks”. Builds on #27 (streaming receive), #28 (no SHA throttle), #29 (offset cache), #30 (delta unroll).