ref:main

perf(pack): implement delta and deflated byte reuse in pack generation (pack-objects) #82

open Opened by cole.christensen@gmail.com

Links

No links yet.

Problem Description

During upload-pack negotiation on git clone or large git fetch, ExGitObjectstore.Pack.Writer generates packfiles by reading each reachable object, decompressing it into memory, and running :zlib.compress individually on every single object.

For repositories with tens of thousands of objects (e.g. hephaestus with ~30,000 objects), packfile generation is orders of magnitude slower than native git pack-objects (Git CLI finishes in ~30s whereas EGOS times out or takes several minutes).

Proposed Solution

Implement pre-compressed and delta reuse matching Git’s native pack-objects architecture (builtin/pack-objects.c):

  1. Raw Byte Copy: When an object is already stored compressed in a packfile or loose file, copy the compressed byte slice directly into the output pack stream without decompressing to plaintext and re-compressing with zlib.
  2. Delta Chain Preservation: Reuse existing OBJ_OFS_DELTA / OBJ_REF_DELTA streams directly when all base objects in the delta chain are included in the pack being sent.
  3. Chunked Streaming: Pipe compressed slices directly to the client sideband socket to bound peak memory usage during large clone transfers.