feat(lfs): content-defined chunking (FastCDC) for dedup + delta transfer #43

open colechristensen cole.christensen@gmail.com wants to merge feat/lfs-chunking into main

Closes #67.

Summary

Adds a chunk-aware LFS transfer (chunked) alongside the untouched basic path. Files are split at content-defined boundaries (FastCDC), each unique chunk is stored once (dedup), and a file is represented by a manifest of its chunks. A one-byte edit re-chunks only the affected region, so the client transfers just the changed chunks and the backend dedups the rest.

Server (this repo)

  • Lfs.Chunker — FastCDC (NC=2), compile-time splitmix64 gear table, paper 8 KiB masks, streaming chunk_stream/2 in bounded memory. (REQ-LFS-009)
  • Lfs.Manifest — versioned parse/emit + integrity. (REQ-LFS-010)
  • Lfs.Chunk.Store + new optional Lfs.Store callbacks exists_many/3, put_meta/4, get_meta/3 across Memory / Filesystem / S3. (REQ-LFS-011)
  • Lfs.Chunk (put/get/negotiate/commit/missing_chunks/get_manifest, telemetry-wrapped) + Lfs.Chunk.Batch advertising chunked. (REQ-LFS-012)

Client (forked git-lfs)

Forked cole-christensen/git-lfs branch feat/chunked-transfer: native chunked tq adapter + tools/fastcdc chunker byte-identical to the server (frozen gear table), gated on lfs.chunkedtransfers.

  • Cross-impl agreement pinned by a shared golden vector reproduced on both sides. (REQ-LFS-014)
  • ChunkedInteropTest drives the real forked binary through git lfs push/smudge against this server — missing-only upload, dedup, exact reconstruction. (REQ-LFS-013)

Verification

  • mix test: 1057 passed, 58 excluded (incl. 2 interop tests vs the forked binary).
  • mix format --check-formatted, mix compile --warnings-as-errors, mix credo --strict clean on all new files.
  • anvil requirement status exits 0 (REQ-LFS-009..014).
  • git-lfs fork: go test ./tq/ ./tools/fastcdc/ green; go build ./... clean.

🤖 Generated with Claude Code

Created Jun 17, 2026 at 16:44 UTC