fangorn/ex_git_objectstore
public
perf(lfs): Elixir Chunker throughput — per-byte :binary.at is the bottleneck (~20 MB/s vs Go ~1.3 GB/s) #69
Links
No links yet.
Measured by bench/lfs_chunking_bench.exs + the Go BenchmarkChunk. The server-side FastCDC chunker runs ~20 MB/s while the byte-identical Go client runs ~1.3 GB/s (~65x). Root cause: the gear-hash inner loop reads bytes via :binary.at/2 (per-byte BIF call) rather than slice/binary pattern-matching. O(n) algorithm, heavy per-byte constant — NOT an O() problem.
Server-side throughput only; chunk boundaries/dedup are unaffected. Options to evaluate (measure first): binary pattern-match the scan window, process in matched binary segments, or a NIF. Do NOT cap/skip large inputs — fix the per-byte constant. Follow-up to #67/#68.
cole.christensen@gmail.com
commented 2026-06-18 02:46
Named bottleneck (per-byte :binary.at/2 in the gear-hash loop) fixed in ca9d76c: the scan now uses binary pattern-matching, lifting server chunking ~20→~30 MB/s, and chunk_stream/2 no longer re-scans. Boundaries unchanged. The residual gap to the Go client (~30 MB/s vs ~1.3 GB/s) is inherent BEAM byte-at-a-time recursion vs native indexing; closing it further would need a NIF — an optional future enhancement, not this bottleneck. Closing as the identified inefficiency is resolved.