fangorn/ex_git_objectstore
public
ref:main
perf(pack): implement delta and deflated byte reuse in pack generation (pack-objects) #82
open
Opened by cole.christensen@gmail.com
Links
No links yet.
Problem Description
During upload-pack negotiation on git clone or large git fetch, ExGitObjectstore.Pack.Writer generates packfiles by reading each reachable object, decompressing it into memory, and running :zlib.compress individually on every single object.
For repositories with tens of thousands of objects (e.g. hephaestus with ~30,000 objects), packfile generation is orders of magnitude slower than native git pack-objects (Git CLI finishes in ~30s whereas EGOS times out or takes several minutes).
Proposed Solution
Implement pre-compressed and delta reuse matching Git’s native pack-objects architecture (builtin/pack-objects.c):
- Raw Byte Copy: When an object is already stored compressed in a packfile or loose file, copy the compressed byte slice directly into the output pack stream without decompressing to plaintext and re-compressing with zlib.
- Delta Chain Preservation: Reuse existing
OBJ_OFS_DELTA/OBJ_REF_DELTAstreams directly when all base objects in the delta chain are included in the pack being sent. - Chunked Streaming: Pipe compressed slices directly to the client sideband socket to bound peak memory usage during large clone transfers.