ref:ed878d981a4f22a333de680e275b1a6f7b9dd184

epic #215 Phase 1 + S1: walker property test + lazy object streaming (#37)

Bundles the library-side work for Anvil epic #215 into one PR per repo. Two commits, two REQs, both backed by failure-injection proof. ## What's in here ### T1 — Walker invariant property test (REQ-GIT-076) \`stream_data\` generator produces random commit graphs (1-3 chains of length 1-6, optional merges, optional mode-160000 gitlinks whose SHA targets either the root commit or the current tip). For every generated graph: assert \`MapSet.new(walker_pack_shas) == bfs_reachable(repo, wants)\`. Catches the entire class of walker bugs that hit production in May 2026 (the gitlink-reachability regression — 1043 reachable objects silently pruned from \`fangorn/hephaestus\`'s pack). **Failure-injection proof.** Temporarily reverting the \`mode: \"160000\"\` head in \`collect_single_tree_entry\` makes the property fail on iteration 0: \`\`\` Failed with generated values (after 0 successful runs): Clause: spec <- graph_spec() Generated: %{..., inject_gitlink?: true, gitlink_target: :self, ...} walker missed reachable SHAs: [\"087267660a...\", \"209f7076a8...\", ... 8 SHAs total ...] \`\`\` Restored: 80 random iterations pass in ~100 ms. Adds \`{:stream_data, \"~> 1.1\", only: :test}\` to deps. ### S1 — Stream walker objects through writer (REQ-GIT-080) After \`collect_objects_maybe_shallow\` returns the walker's \`[{type, content, sha}, ...]\` list, project it down to \`[{type, sha}, ...]\` and let the original content list go out of scope (so it's GC'd). Pipe the SHA list through a \`Stream.map(fn {type, sha} -> ... ObjectResolver.read(repo, sha) ... end)\` into a new \`Writer.generate_stream_enum/4\`. Pack-write phase peak heap is bounded by one object at a time. Surface changes: - **New** \`Writer.generate_stream_enum/4\` — accepts \`Enumerable.t/0\` + explicit count. The pack format needs the count in the hashed header, so it cannot be deferred. Same byte output as the existing \`generate_stream/3\`. - \`generate_stream/3\` becomes a thin wrapper that supplies \`length/1\` for list inputs. Same public contract. - \`UploadPackV2.stream_packfile_response\` uses \`drop_content/1\` + \`stream_object_contents/2\` helpers to set up the lazy stream. **Measured impact** (Anvil REQ-GIT-077 budget test, 10 MiB random-blob fetch, avg peak BEAM heap delta over 3 runs): | Library version | Peak heap delta | |---|---| | main (eager) | 14.7 MB | | **this branch** | **3.5 MB** | ~4× reduction on the test fixture. Scaled to prod hephaestus (~210 MB pack), absolute savings should exceed 200 MB. The reduction is in the pack-write phase; the walk phase still allocates content (will be addressed by S4 / pass-through packed object reuse in a follow-up). ## Test plan - [x] \`mix test\` — 945 tests, 0 failures (51 excluded) - [x] \`mix format --check-formatted\` clean - [x] \`mix credo --strict\` clean - [x] Property test demonstrated to fail on the gitlink shape when the fix is reverted - [x] S1 byte output verified equivalent to non-streaming via existing v2 byte-equivalence tests ## Tracks Epic Anvil #215 (REQ-GIT-076, REQ-GIT-080).
SHA: ed878d981a4f22a333de680e275b1a6f7b9dd184
Author: Anvil <noreply@anvil.fangorn.io>
Date: 2026-05-14 18:10
Parents: 81b6f84
5 files changed +450 -10
Type
mix.exs +2 −1
@@ -101,7 +101,8 @@
{:dialyxir, "~> 1.4", only: [:dev, :test], runtime: false},
{:credo, "~> 1.7", only: [:dev, :test], runtime: false},
{:plug, "~> 1.16", only: :test},
{:bandit, "~> 1.5", only: :test}
{:bandit, "~> 1.5", only: :test},
{:stream_data, "~> 1.1", only: :test}
]
end
end
mix.lock +1 −0
@@ -25,6 +25,7 @@
"plug": {:hex, :plug, "1.19.1", "09bac17ae7a001a68ae393658aa23c7e38782be5c5c00c80be82901262c394c0", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "560a0017a8f6d5d30146916862aaf9300b7280063651dd7e532b8be168511e62"},
"plug_crypto": {:hex, :plug_crypto, "2.1.1", "19bda8184399cb24afa10be734f84a16ea0a2bc65054e23a62bb10f06bc89491", [:mix], [], "hexpm", "6470bce6ffe41c8bd497612ffde1a7e4af67f36a15eea5f921af71cf3e11247c"},
"ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"},
"stream_data": {:hex, :stream_data, "1.3.0", "bde37905530aff386dea1ddd86ecbf00e6642dc074ceffc10b7d4e41dfd6aac9", [:mix], [], "hexpm", "3cc552e286e817dca43c98044c706eec9318083a1480c52ae2688b08e2936e3c"},
"sweet_xml": {:hex, :sweet_xml, "0.7.5", "803a563113981aaac202a1dbd39771562d0ad31004ddbfc9b5090bdcd5605277", [:mix], [], "hexpm", "193b28a9b12891cae351d81a0cead165ffe67df1b73fe5866d10629f4faefb12"},
"telemetry": {:hex, :telemetry, "1.3.0", "fedebbae410d715cf8e7062c96a1ef32ec22e764197f70cda73d82778d61e7a2", [:rebar3], [], "hexpm", "7015fc8919dbe63764f4b4b87a95b7c0996bd539e0d499be6ec9d7f3875b79e6"},
"thousand_island": {:hex, :thousand_island, "1.4.3", "2158209580f633be38d43ec4e3ce0a01079592b9657afff9080d5d8ca149a3af", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "6e4ce09b0fd761a58594d02814d40f77daff460c48a7354a15ab353bb998ea0b"},