fangorn/ex_git_objectstore
public
ref:44a17299073eb06719b0ccbf12678b75b33525d2
feat(upload-pack): streaming v2 fetch response — no full-pack buffering
Adds a streaming variant for protocol v2 fetches so the entire packfile
no longer materializes in BEAM process heap before being sent.
Before, building a 161 MB pack response held three full-pack-sized
binaries (raw pack + sideband-wrapped pack + concatenated final
response) plus the object list, peaking at ~1.8 GB transient heap and
contributing to OOM-kills on the host (Anvil prod, 2026-05-11 02:50).
What's new (additive, no API breaks):
* Writer.generate_stream/2 and /3 — invoke a write callback for each
pack chunk (header, one entry per object, trailing SHA-1). SHA-1 is
computed incrementally via :crypto.hash_update/2 so no intermediate
full-pack binary exists. /3 threads an accumulator through each
write for stateful sinks.
* PktLine.encode_sideband_frame/2 + max_sideband_data/0 — single-frame
encoder for streaming callers. encode_sideband/2 is unchanged.
* Protocol.SidebandWriter — re-chunks arbitrary-sized writes into
spec-compliant sideband-1 frames (≤ 65515 bytes per frame). Buffer
held as iodata for O(1) amortized appends.
* UploadPackV2.feed/3 — streaming counterpart to feed/2. ls-refs and
multi-round ack responses still arrive as a single write_fn call;
packfile responses stream through the sideband chunker.
Tests:
* Writer: streamed bytes are byte-identical to Writer.generate/1 for
small and many-mixed-object pack inputs. /3 accumulator threading
verified.
* SidebandWriter: small writes coalesce, oversized writes split into
spec-compliant frames, exact-boundary writes drain cleanly.
* UploadPackV2: ls-refs, multi-round acks, and full clone responses
are byte-identical via feed/3 vs feed/2. A 70 KiB-blob fetch emits
multiple chunks (not one giant binary) with every chunk ≤ 65520.
Existing feed/2 path is untouched; consumers migrate at their own pace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SHA:
44a17299073eb06719b0ccbf12678b75b33525d2
Author:
CI <ci@anvil.test>
Date:
2026-05-11 14:53
Parents:
0ac0140
7 files changed
+703
-1
| Type | ||
|---|---|---|
|
|
lib/ex_git_objectstore/pack/writer.ex | +54 −0 |
|
||