ref:dbd335ebd51f52f204b60eca61aef1ef1f911ebb

feat: git lfs — pointer, store, batch, transfer, locks (#22)

## Summary Full Git LFS (spec v1) support, closes #38. Exposed as pure request/response modules that mirror the existing \`UploadPack\`/\`ReceivePack\` style — no HTTP server is introduced; consumers wire the handlers into their own routing. ## What's in - **\`ExGitObjectstore.Lfs.Pointer\`** — parser/emitter with strict spec-v1 validation (version-first, alphabetical keys, sha256-only OIDs, canonical decimal size, trailing LF). - **\`ExGitObjectstore.Lfs.Store\`** — behaviour parallel to \`Storage\`, keyed by SHA256. Streaming \`put/4\` verifies the observed SHA256 matches the claimed OID; on mismatch the write is discarded. Backends: Memory, Filesystem (2+2 fanout, atomic rename), and S3 (multipart upload with optional presigned URLs). Shared conformance macro at \`ExGitObjectstore.Test.LfsStoreConformance\` exercises every backend identically. - **\`ExGitObjectstore.Lfs.Batch\`** — Batch API returning spec-compliant upload/download/verify actions. Uses presigned URLs when the backend supports them; falls back to library-served URLs for Filesystem/Memory. - **\`ExGitObjectstore.Lfs.Transfer\`** — basic-transfer handlers for \`GET/PUT /objects/:oid\` and \`POST /objects/:oid/verify\`. - **\`ExGitObjectstore.Lfs.Locks\`** — v1 locks API (create, list, verify, unlock with owner/force). - **Telemetry** at \`[:ex_git_objectstore, :lfs, :batch | :transfer | :lock]\`. - **\`Repo\`** gains optional \`:lfs_storage\` alongside \`:storage\`. - CHANGELOG entry under \`[Unreleased]\`. - Anvil requirements REQ-LFS-001…006 created; tests carry \`@moduletag requirements: [...]\`. ## Test plan - [x] 91 new LFS tests pass (14 S3 tests excluded without MinIO) - [x] Full suite: 891 tests, 0 failures - [x] \`mix credo --strict\` clean for all new code - [x] \`mix format\` clean - [x] \`anvil requirement status\` passes - [ ] Manual interop against \`git lfs\` client (follow-up, not in PR) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
SHA: dbd335ebd51f52f204b60eca61aef1ef1f911ebb
Author: Anvil <noreply@anvil.fangorn.io>
Date: 2026-04-20 13:33
Parents: 6f15b9b
23 files changed +4040 -2
Type
CHANGELOG.md +54 −0
@@ -7,6 +7,60 @@
## [Unreleased]
### Added
- **Git LFS support (spec v1).** Full Large File Storage implementation
exposed as pure request/response modules, matching the existing
`UploadPack`/`ReceivePack` style (no HTTP server in-tree).
- `ExGitObjectstore.Lfs.Pointer` — parse and emit spec-compliant
pointer blobs with strict validation (version-first, alphabetical
key order, sha256-only OIDs, canonical decimal size, trailing LF).
- `ExGitObjectstore.Lfs.Store` — behaviour parallel to `Storage`,
keyed by SHA256. Streaming `put/4` verifies the observed SHA256
matches the claimed OID and discards the write on mismatch.
Shared conformance test suite at
`ExGitObjectstore.Test.LfsStoreConformance`.
- Backends: `Lfs.Store.Memory`, `Lfs.Store.Filesystem`,
`Lfs.Store.S3`. S3 uses multipart upload for streaming PUT and
exposes optional `presigned_upload/5` / `presigned_download/4`
callbacks for direct-to-S3 client transfers.
- `ExGitObjectstore.Lfs.Batch` — Batch API handler
(`POST /objects/batch`) returning spec-compliant upload / download /
verify actions. Uses presigned URLs when the backend supports them;
falls back to library-served URLs for Filesystem/Memory.
- `ExGitObjectstore.Lfs.Transfer` — basic-transfer handlers for
`GET/PUT /objects/:oid` and `POST /objects/:oid/verify`, with
streaming downloads and SHA256-verified uploads.
- `ExGitObjectstore.Lfs.Locks` — Locks API v1: create, list, verify,
unlock (with `force` for admin override). Lock metadata stored on
the repo's `Storage` backend under `lfs/locks/*.json`.
- Telemetry spans emitted at
`[:ex_git_objectstore, :lfs, :batch | :transfer | :lock]`.
- `Repo` gains optional `:lfs_storage` option alongside `:storage`.
- End-to-end interop coverage against the real `git lfs` binary via
a Bandit-backed test HTTP adapter. 11 scenarios exercise push,
smudge (download), idempotent re-push, edge-case payload sizes
(0-byte via direct HTTP, 1-byte via the CLI), multi-file batch
pushes, mixed-state batches (present + absent), concurrent
parallel uploads (10 files with `lfs.concurrenttransfers=8`),
direct-HTTP OID tampering (server must 422 and leave nothing on
disk), lock create/list/verify/unlock with conflict and 403-by
-non-owner paths, and a full end-to-end `git push` → `git clone`
→ `git lfs pull` roundtrip over smart-http. Found and fixed one
real bug: `Batch.handle/3` was double-prefixing `repos/<id>/lfs`
into action URLs — the `:base_url` is now the LFS root itself
and the module emits `<base_url>/objects/<oid>` and
`<base_url>/verify`. The test adapter also wires the existing
`UploadPack`/`ReceivePack` modules to the git smart-http v0
endpoints (`GET /info/refs`, `POST /git-upload-pack`,
`POST /git-receive-pack`) so a real `git clone` can complete
the full clone-then-lfs-pull flow.
- S3 backend interop coverage: 14 conformance tests against real
MinIO plus 2 end-to-end `git lfs push`/`smudge` scenarios that
exercise the presigned-URL path — client uploads directly to
MinIO via presigned PUTs and downloads via presigned GETs, with
the library only mediating batch + verify.
### Fixed
- **UploadPackV2: omit `acknowledgments` section when the client sends
mix.exs +3 −1
@@ -99,7 +99,9 @@
{:telemetry, "~> 1.0"},
{:ex_doc, "~> 0.34", only: :dev, runtime: false},
{:dialyxir, "~> 1.4", only: [:dev, :test], runtime: false},
{:credo, "~> 1.7", only: [:dev, :test], runtime: false}
{:credo, "~> 1.7", only: [:dev, :test], runtime: false},
{:plug, "~> 1.16", only: :test},
{:bandit, "~> 1.5", only: :test}
]
end
end
mix.lock +6 −0
@@ -1,4 +1,5 @@
%{
"bandit": {:hex, :bandit, "1.10.4", "02b9734c67c5916a008e7eb7e2ba68aaea6f8177094a5f8d95f1fb99069aac17", [:mix], [{:hpax, "~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}, {:plug, "~> 1.18", [hex: :plug, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:thousand_island, "~> 1.0", [hex: :thousand_island, repo: "hexpm", optional: false]}, {:websock, "~> 0.5", [hex: :websock, repo: "hexpm", optional: false]}], "hexpm", "a5faf501042ac1f31d736d9d4a813b3db4ef812e634583b6a457b0928798a51d"},
"bunt": {:hex, :bunt, "1.0.0", "081c2c665f086849e6d57900292b3a161727ab40431219529f13c4ddcf3e7a44", [:mix], [], "hexpm", "dc5f86aa08a5f6fa6b8096f0735c4e76d54ae5c9fa2c143e5a1fc7c1cd9bb6b5"},
"certifi": {:hex, :certifi, "2.15.0", "0e6e882fcdaaa0a5a9f2b3db55b1394dba07e8d6d9bcad08318fb604c6839712", [:rebar3], [], "hexpm", "b147ed22ce71d72eafdad94f055165c1c182f61a2ff49df28bcc71d1d5b94a60"},
"credo": {:hex, :credo, "1.7.16", "a9f1389d13d19c631cb123c77a813dbf16449a2aebf602f590defa08953309d4", [:mix], [{:bunt, "~> 0.2.1 or ~> 1.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:file_system, "~> 0.2 or ~> 1.0", [hex: :file_system, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "d0562af33756b21f248f066a9119e3890722031b6d199f22e3cf95550e4f1579"},
@@ -10,6 +11,7 @@
"ex_doc": {:hex, :ex_doc, "0.40.1", "67542e4b6dde74811cfd580e2c0149b78010fd13001fda7cfeb2b2c2ffb1344d", [:mix], [{:earmark_parser, "~> 1.4.44", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "bcef0e2d360d93ac19f01a85d58f91752d930c0a30e2681145feea6bd3516e00"},
"file_system": {:hex, :file_system, "1.1.1", "31864f4685b0148f25bd3fbef2b1228457c0c89024ad67f7a81a3ffbc0bbad3a", [:mix], [], "hexpm", "7a15ff97dfe526aeefb090a7a9d3d03aa907e100e262a0f8f7746b78f8f87a5d"},
"hackney": {:hex, :hackney, "1.25.0", "390e9b83f31e5b325b9f43b76e1a785cbdb69b5b6cd4e079aa67835ded046867", [:rebar3], [{:certifi, "~> 2.15.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~> 6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~> 1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~> 1.4", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.4.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~> 0.7.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "7209bfd75fd1f42467211ff8f59ea74d6f2a9e81cbcee95a56711ee79fd6b1d4"},
"hpax": {:hex, :hpax, "1.0.3", "ed67ef51ad4df91e75cc6a1494f851850c0bd98ebc0be6e81b026e765ee535aa", [:mix], [], "hexpm", "8eab6e1cfa8d5918c2ce4ba43588e894af35dbd8e91e6e55c817bca5847df34a"},
"idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"},
"jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"},
"makeup": {:hex, :makeup, "1.2.1", "e90ac1c65589ef354378def3ba19d401e739ee7ee06fb47f94c687016e3713d1", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "d36484867b0bae0fea568d10131197a4c2e47056a6fbe84922bf6ba71c8d17ce"},
@@ -20,8 +22,12 @@
"mimerl": {:hex, :mimerl, "1.4.0", "3882a5ca67fbbe7117ba8947f27643557adec38fa2307490c4c4207624cb213b", [:rebar3], [], "hexpm", "13af15f9f68c65884ecca3a3891d50a7b57d82152792f3e19d88650aa126b144"},
"nimble_parsec": {:hex, :nimble_parsec, "1.4.2", "8efba0122db06df95bfaa78f791344a89352ba04baedd3849593bfce4d0dc1c6", [:mix], [], "hexpm", "4b21398942dda052b403bbe1da991ccd03a053668d147d53fb8c4e0efe09c973"},
"parse_trans": {:hex, :parse_trans, "3.4.1", "6e6aa8167cb44cc8f39441d05193be6e6f4e7c2946cb2759f015f8c56b76e5ff", [:rebar3], [], "hexpm", "620a406ce75dada827b82e453c19cf06776be266f5a67cff34e1ef2cbb60e49a"},
"plug": {:hex, :plug, "1.19.1", "09bac17ae7a001a68ae393658aa23c7e38782be5c5c00c80be82901262c394c0", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "560a0017a8f6d5d30146916862aaf9300b7280063651dd7e532b8be168511e62"},
"plug_crypto": {:hex, :plug_crypto, "2.1.1", "19bda8184399cb24afa10be734f84a16ea0a2bc65054e23a62bb10f06bc89491", [:mix], [], "hexpm", "6470bce6ffe41c8bd497612ffde1a7e4af67f36a15eea5f921af71cf3e11247c"},
"ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"},
"sweet_xml": {:hex, :sweet_xml, "0.7.5", "803a563113981aaac202a1dbd39771562d0ad31004ddbfc9b5090bdcd5605277", [:mix], [], "hexpm", "193b28a9b12891cae351d81a0cead165ffe67df1b73fe5866d10629f4faefb12"},
"telemetry": {:hex, :telemetry, "1.3.0", "fedebbae410d715cf8e7062c96a1ef32ec22e764197f70cda73d82778d61e7a2", [:rebar3], [], "hexpm", "7015fc8919dbe63764f4b4b87a95b7c0996bd539e0d499be6ec9d7f3875b79e6"},
"thousand_island": {:hex, :thousand_island, "1.4.3", "2158209580f633be38d43ec4e3ce0a01079592b9657afff9080d5d8ca149a3af", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "6e4ce09b0fd761a58594d02814d40f77daff460c48a7354a15ab353bb998ea0b"},
"unicode_util_compat": {:hex, :unicode_util_compat, "0.7.1", "a48703a25c170eedadca83b11e88985af08d35f37c6f664d6dcfb106a97782fc", [:rebar3], [], "hexpm", "b3a917854ce3ae233619744ad1e0102e05673136776fb2fa76234f3e03b23642"},
"websock": {:hex, :websock, "0.5.3", "2f69a6ebe810328555b6fe5c831a851f485e303a7c8ce6c5f675abeb20ebdadc", [:mix], [], "hexpm", "6105453d7fac22c712ad66fab1d45abdf049868f253cf719b625151460b8b453"},
}