fangorn/ex_git_objectstore
public
ref:main
Add streaming pack reader for large repositories #2
open
Opened by cole.christensen@gmail.com
Links
No links yet.
Problem
The pack reader currently loads the entire packfile into memory before parsing. Repositories with packfiles >500MB will cause excessive memory usage or OOM.
Impact
- Cannot serve or consume large repositories
- Memory usage is proportional to largest packfile, not working set
- Limits production viability
Acceptance Criteria
- Pack reader supports offset-based reads (seek + read range)
- Memory usage stays bounded regardless of packfile size
- Storage behaviour extended with
stream_pack/ random-access read support - Works with all three backends (filesystem, S3, memory)
- Existing pack reader tests continue to pass
- Added benchmark comparing memory usage for large packs