ref:main

Add streaming pack reader for large repositories #2

open Opened by cole.christensen@gmail.com

Links

No links yet.

Problem

The pack reader currently loads the entire packfile into memory before parsing. Repositories with packfiles >500MB will cause excessive memory usage or OOM.

Impact

  • Cannot serve or consume large repositories
  • Memory usage is proportional to largest packfile, not working set
  • Limits production viability

Acceptance Criteria

  • Pack reader supports offset-based reads (seek + read range)
  • Memory usage stays bounded regardless of packfile size
  • Storage behaviour extended with stream_pack / random-access read support
  • Works with all three backends (filesystem, S3, memory)
  • Existing pack reader tests continue to pass
  • Added benchmark comparing memory usage for large packs