ref:main
feat(lfs): content-defined chunking (FastCDC) for dedup + delta transfer #43
open
cole.christensen@gmail.com wants to merge
feat/lfs-chunking
into main
No CI
Mergeable
Closes #67.
Summary
Adds a chunk-aware LFS transfer (chunked) alongside the untouched basic path. Files are split at content-defined boundaries (FastCDC), each unique chunk is stored once (dedup), and a file is represented by a manifest of its chunks. A one-byte edit re-chunks only the affected region, so the client transfers just the changed chunks and the backend dedups the rest.
Server (this repo)
Lfs.Chunker— FastCDC (NC=2), compile-time splitmix64 gear table, paper 8 KiB masks, streamingchunk_stream/2in bounded memory. (REQ-LFS-009)Lfs.Manifest— versioned parse/emit + integrity. (REQ-LFS-010)Lfs.Chunk.Store+ new optionalLfs.Storecallbacksexists_many/3,put_meta/4,get_meta/3across Memory / Filesystem / S3. (REQ-LFS-011)Lfs.Chunk(put/get/negotiate/commit/missing_chunks/get_manifest, telemetry-wrapped) +Lfs.Chunk.Batchadvertisingchunked. (REQ-LFS-012)
Client (forked git-lfs)
Forked cole-christensen/git-lfs branch feat/chunked-transfer: native chunked tq adapter + tools/fastcdc chunker byte-identical to the server (frozen gear table), gated on lfs.chunkedtransfers.
- Cross-impl agreement pinned by a shared golden vector reproduced on both sides. (REQ-LFS-014)
ChunkedInteropTestdrives the real forked binary throughgit lfs push/smudge against this server — missing-only upload, dedup, exact reconstruction. (REQ-LFS-013)
Verification
mix test: 1057 passed, 58 excluded (incl. 2 interop tests vs the forked binary).mix format --check-formatted,mix compile --warnings-as-errors,mix credo --strictclean on all new files.anvil requirement statusexits 0 (REQ-LFS-009..014).- git-lfs fork:
go test ./tq/ ./tools/fastcdc/green;go build ./...clean.
🤖 Generated with Claude Code
Created Jun 17, 2026 at 16:44 UTC