Measured performance audit of ex_git_objectstore, companion to Anvil #372. Same method: measure first, root-cause in code, rank fixes by how deep in the stack the cost lives.

Ranking used throughout (standing protocol):

1. per-unit cost in a shared primitive
2. algorithmic class of a named operation (cite the published bound)
3. avoid redundant work
4. cache results — avoid
5. schedule around it — last resort

Harness

Everything below is reproducible. Nothing was run against production; production is read-only.

• Packed corpus — fangorn/hephaestus .git (6 packs, 1.07 GB, 31,449 objects; 21,844 reachable from main), mounted read-only as an egos Repo via a symlink under a Filesystem storage root.
• Loose corpus — a bare clone of fangorn/anvil with every pack exploded via git unpack-objects: 12,766 loose objects, 61 MB. This models the layout production actually has (236,756 loose objects, 4 packs across 39 repos).
• Single-pack corpus — the same repo after git repack -adq. Models the post-#324 world.
• Reference implementation — git rev-list --objects <ref> | git pack-objects --stdout --revs on the identical object set, timed with /usr/bin/time -l. This is the bound to beat, not “faster than before”.
• Sample profiler: 5 ms interval, backtrace_depth 100, one line per sample carrying :erlang.memory() alongside the stack. Call counts via :erlang.trace_pattern(mfa, true, [:call_count]).

Headline: a clone is 12–33× slower than git and holds 6.7× the memory

Same 21,844 objects, same repo, same machine:

	wall	peak memory	pack produced
git pack-objects (default: reuse on)	1.25 s	272 MB RSS	195 MB
git pack-objects --no-reuse-object --no-reuse-delta	13.8 s	606 MB RSS	195 MB
UploadPackV2 streaming fetch	18.2 s	1,819 MB BEAM total	316 MB

On the single-pack corpus (12,460 objects) the gap is wider, because git can use its bitmap and reuse the pack wholesale: git 0.06 s / 10.3 MB vs egos 1.96 s / 22.4 MB — 33× slower, 2.2× larger output.

Where the 18.2 s goes, by sample (2,998 samples):

:zlib.append_iolist/2                              1763   58.8%   deflate, writing the output pack
ExGitObjectstore.Pack.Reader.safe_inflate_all/5     530   17.7%   inflate, reading the source packs
                                                          ------
                                                           76.5%  zlib
String.Unicode.upcase/3                             144    4.8%
ExGitObjectstore.Pack.Delta.apply_instructions/3     86    2.9%
:crypto.hash_update/4                                52    1.7%

76.5% of a clone is decompressing bytes out of a packfile and immediately recompressing them into another packfile.

---

F1 — Every File.* in the Filesystem backend goes through the singleton file_server_2; read throughput collapses under concurrency

Rank 1 — per-unit cost in a shared primitive.

Storage.Filesystem makes 38 File.* calls. File.read/1 is :file.read_file/1, which is a gen_server:call to the single file_server_2 process. Every loose-object read in the system funnels through one Erlang process.

storage/filesystem.ex:43:

def get_object(config, prefix, sha) do
  path = object_path(config, prefix, sha)
  case File.read(path) do

Measured, 6,000 loose object files, N concurrent reader processes:

concurrency	File.read/1	:prim_file.read_file/1	ratio
1	37.8 µs/read	35.3 µs/read	1.07×
2	32.6 µs/read	20.3 µs/read	1.61×
4	35.2 µs/read	15.0 µs/read	2.35×
8	107.0 µs/read	10.6 µs/read	10.11×

File.read gets worse as concurrency rises (37.8 → 107.0 µs) because the requests queue behind one process. :prim_file.read_file gets better (35.3 → 10.6 µs) because it runs on dirty IO schedulers and the reads overlap.

The profile of a clone against the loose corpus shows this directly — 475 of 617 samples (77% of wall time) are:

Storage.Filesystem.get_object/3 ; :file.call/2 ; :gen_server.call/3 ; :gen.do_call/4

Fix: :prim_file.read_file/1 for get_object, and raw-mode :file / :prim_file for the rest. The pattern is already established in the same module — get_pack_range/5 at storage/filesystem.ex:156 correctly uses :file.open(path, [:read, :binary, :raw]).

Honest scope. Substituting :prim_file.read_file in a full clone gave only 5,435 → 4,690 ms at concurrency 4 (−14%), because a clone is zlib-bound, not read-bound (see F3). The 10× is a read-throughput number, so it pays off on the read-heavy concurrent paths — LiveView diff/log/tree browsing, CI checkouts, ls-refs — rather than on clones. It is still rank 1: it is a per-unit cost in a primitive every consumer sits on, and it is a mechanical fix.

Adjacent: object_path/3 → safe_path/2 (storage/filesystem.ex:467) calls Path.expand/1 twice per object read — once on the joined path, once on the root. Measured 49,842 Path.expand/1 calls for a 12,460-object clone; safe_path + Path.join costs 10.9 µs per object. Path.expand(root) is loop-invariant.

---

F2 — upload-pack reads, decompresses and delta-resolves every object exactly twice

Rank 3 — avoid redundant work.

collect_objects/4 (protocol/upload_pack_v2.ex:914) walks reachability and materializes {type, content, sha} for every object including blobs. stream_generated_pack/7 then calls drop_content/1 (:762) — which exists specifically to throw that content away —

Enum.reduce(objects, {[], 0}, fn {type, _content, sha}, {acc, n} ->
  {[{type, sha} | acc], n + 1}
end)

— and stream_object_contents/2 (:775) reads every object from storage a second time.

Measured call counts, one clone:

	packed corpus (21,844 objects)	loose corpus (12,460 objects)
ObjectResolver.read/2	43,689 (2.00× per object)	24,921 (2.00× per object)
Storage.Filesystem.get_object/3	—	24,921
Pack.Index.lookup_in_raw/2	244,664	—
Pack.Delta.apply/2	158,832	—
Tree.encode_content/1	22,196	13,508
Path.expand/1	1,302	49,842

The fix is not to hold the content — that is what causes the memory peak. It is to keep the content the walk already had to decode for commits and trees (small, and needed for traversal anyway) and mark blobs nil so they are read once, lazily, at pack-write time. I implemented this as a measured experiment, together with F5:

	before	after	change
packed: ObjectResolver.read/2	43,689	21,845	−50% (exactly 1× per object)
packed: Index.lookup_in_raw/2	244,664	122,335	−50%
packed: Delta.apply/2	158,832	79,416	−50%
packed: Tree.encode_content/1	22,196	11,098	−50%
packed: String.upcase/1	385,732	0	−100%
packed: wall	18,034 ms	15,112–16,559 ms	−11%
packed: peak BEAM total	1,819 MB	1,317 MB	−502 MB (−28%)
loose: get_object/3	24,921	12,461	−50%
loose: Path.expand/1	49,842	24,922	−50%
loose: wall	4,627 ms	3,894 ms	−16%

mix test test/ex_git_objectstore/protocol/ — 112 passed with the patch applied.

This also halves the work in Index.lookup_in_raw and Delta.apply, which are downstream of the read. Both were doing 11.2× and 7.3× per object respectively; the residual multiplier is the per-pack linear scan in find_in_packs/3 (6 packs → ~5.6 index probes per read) and delta chain depth (~3.7 applications per read).

---

F3 — No pack-data reuse, no delta reuse, no deltification at all

Rank 2 — algorithmic class.

pack/writer.ex:26 states it plainly:

Objects are stored without delta compression (full objects only).

Every object served is inflated out of its source pack and deflated again from scratch into the output pack, and the output is fully undeltified.

The cost is not a guess — git measures it for us on the identical object set:

• git pack-objects with reuse: 1.25 s
• git pack-objects --no-reuse-object --no-reuse-delta: 13.8 s

Reuse is worth 11× in git’s own implementation. egos is at 18.2 s, i.e. slower than git’s deliberately-pessimised mode, while emitting a 62% larger pack (316 MB vs 195 MB) because it never deltifies. On the single-pack corpus the ratio is 33× (0.06 s vs 1.96 s) and 2.2× on output size.

Output size is a production cost in its own right: every clone and fetch pushes 1.6–2.2× the bytes it needs to, out of a 2-vCPU droplet.

The mechanism to copy is builtin/pack-objects.c — check_object() marks an object reusable when it lives in a pack in a form the output can take verbatim, and write_reuse_object() copies the compressed bytes without ever calling inflate. Deltas are reused by re-emitting REF_DELTA when the base is also in the output set.

Staging:

1. Verbatim non-delta reuse. Object is a non-delta entry in a source pack → copy its compressed bytes straight into the output. Removes both the inflate and the deflate. This alone should take the bulk of the 76.5% zlib share.
2. Delta reuse. Source entry is OFS_DELTA/REF_DELTA and its base is in the output set → re-emit as REF_DELTA with the same compressed delta bytes. Closes the output-size gap.
3. Delta search for objects with no packed form (the loose case). Bigger project; relates to #46 (bitmaps) and #47 (midx).

Note the ordering interaction: on the loose layout production has today, (1) and (2) have nothing to reuse. They become the dominant win the moment server-side repack (Anvil PR #228 / #324) lands, and F7 below is what stops the repo from going loose again.

---

F4 — The whole-pack reuse fast path costs a full reachability walk and essentially never fires

Rank 3 — avoid redundant work.

attempt_pack_reuse/2 (protocol/upload_pack_v2.ex:651):

with {:ok, pack_sha} <- single_pack(repo),
     {:ok, pack_shas} <- pack_sha_set(repo, pack_sha),
     {:ok, reachable} <- reachable_sha_set(repo, wants),
     true <- MapSet.equal?(reachable, pack_shas) do

reachable_sha_set/2 is a full commit+tree walk. When MapSet.equal? returns false the entire walk is discarded and collect_objects/4 immediately re-walks the same graph.

Measured on the single-pack corpus, freshly git repack -ad’d — the most favourable case that exists:

reuse MISSED.  ObjectResolver.read/2 = 19,957 for a 12,460-object clone
               (12,460 real walk + 7,497 discarded probe walk)
               Tree.encode_content/1 = 13,508 (6,754 × 2 — the probe encodes every tree too)

It misses because the pack holds objects reachable from all refs while wants is one branch tip. Any repo with more than one branch fails the equality test permanently. Adding one unreachable loose object (simulating a push landing after a repack) changes nothing — it was already missing.

MapSet.equal? is the wrong predicate. Completeness needs MapSet.subset?(reachable, pack_shas); the “don’t leak unreachable objects” property that motivated equality is satisfied properly by per-object reuse (F3), not by whole-pack reuse. Failing that, the probe should be compared against the reachable set of all refs, computed once.

---

F5 — Tree.encode_content/1 Unicode-upcases every SHA it re-encodes

Rank 1 — per-unit cost in a shared primitive.

object/tree.ex:100:

{:ok, sha_bin} = Base.decode16(String.upcase(entry.sha))

SHAs on a Tree struct always come from parse_content/1 (object/tree.ex:126), which produces them via Base.encode16(sha_bin, case: :lower). They are lowercase hex by construction. String.upcase/1 dispatches to String.Unicode.upcase/3, a per-codepoint table walk over 40 characters, to produce a string that Base.decode16(sha, case: :mixed) would have accepted unchanged.

• 385,732 String.upcase/1 calls per hephaestus clone
• 144 of 2,964 profile samples = 4.9% of total clone CPU, split across both encode sites (73 in the walk, 65 in the pack writer)
• Removing it alone: 18,034 → 17,455–17,582 ms (−3%)

The deeper redundancy: parse_content/1 hex-encodes all 20-byte SHAs into strings, then encode_content/1 hex-decodes them back and rebuilds the identical binary the object was read from. Retaining the raw content on the Tree struct removes the round trip entirely for the common read-then-serve path.

---

F6 — ls-refs reads every ref in the repo before applying ref-prefix

Rank 2 — work scaled to the whole dataset when only a slice was asked for.

collect_ref_entries/2 (protocol/upload_pack_v2.ex:294) unconditionally lists and reads all of refs/heads/ and refs/tags/. filter_by_prefixes/2 (:327) filters afterwards.

Measured, synthetic repo with 5,001 refs:

ls-refs ref-prefix=refs/heads/main  ->  209.1 ms   for a 65-byte response
ls-refs (no prefix, all 5,001 refs) ->  214.2 ms   for a 304,005-byte response

The client asked for one ref and paid for five thousand. Every git fetch sends ref-prefix args, so this is on the hot path of the most common operation.

Compounding with F1: list_files_recursive/2 (storage/filesystem.ex:558) does a File.lstat per directory entry (list_entry/2, :574) and then read_loose_ref/3 does a File.read per ref — two file_server_2 round trips per ref, 10,002 of them for the 65-byte answer above.

Fix: push the prefix list down into Ref.list/2 and the storage backend so only matching subtrees are walked.

---

F7 — receive-pack explodes every pushed packfile into loose objects

Rank 2 — algorithmic class. This is the origin of the production layout that makes F1 and F3 hurt.

store_pack_objects/2 → store_single_entry/2 (protocol/receive_pack.ex:513) does, per object: inflate (in the reader), zlib_compress(raw) — recompress — then put_object, which is atomic_write/2 (storage/filesystem.ex:479) = File.mkdir_p! + File.write (temp) + File.rename, i.e. three file_server_2 round trips.

Measured, 12,766 objects, 24.9 MB of already-compressed data:

put_object × 12,766  (explode to loose)  =  3,557 ms   (278.6 µs/object)
put_pack   × 1       (same bytes)        =     60 ms
                                            ------
                                             59× faster

That excludes the per-object recompression, which the pack path does not need at all.

The comment at receive_pack.ex:504 references a real push of ~134k objects. At the measured rate that is ~37 s of file writes alone, and it is what leaves production at 236,756 loose objects with 4 packs — which is what makes git-upload-pack on fangorn/hephaestus take 18–70 s.

git’s behaviour is receive.unpackLimit (default 100): above the threshold, receive-pack runs index-pack and keeps the received pack as a pack. Doing the same removes the deflate and three file operations per object from the push path, and keeps repos in the layout F3’s reuse needs.

---

F8 — The per-process whole-pack cache holds every pack file for the lifetime of the request

Extends #76 with numbers from the fetch path. Rank 5 as a standalone change — the real fix is F3.

Memory trace of one hephaestus clone (:erlang.memory/0 sampled every 5 ms):

t=0.0s     64 MB total /     0 MB binary
t=1.2s  1,421 MB total / 1,352 MB binary      <- collect phase; packs loaded
t=3.7s  1,145 MB total / 1,075 MB binary
...     ~1,145 MB total / ~1,075 MB binary    <- flat for the remaining 14 s
peak    1,819 MB total / 1,673 MB binary

ObjectResolver escalates each pack to the whole-pack path after 64 reads (object_resolver.ex:70) and cached_pack_data/2 (:300) File.reads the entire .pack into the process dictionary. 4 get_pack calls loaded 1.07 GB — the repo’s whole pack set — and it stayed resident for the full 18 s.

Attribution measured by raising @escalate_threshold to disable escalation:

	wall	peak BEAM total
escalation on (current)	18.0 s	1,819 MB
escalation off (ranged reads only)	30.0 s	765 MB

So today the cache trades 1.05 GB of resident memory for 12 s of wall time. Production is a 3 GB droplet: two concurrent clones of hephaestus do not fit. Neither branch of that trade is acceptable, which is the argument for F3 — verbatim reuse means streaming the pack rather than decoding it, and the reason to hold it disappears.

---

Measured and REJECTED

Negative results, so nobody re-runs them.

1. :prim_file.read_file/1 vs File.read/1 single-threaded — no difference. 12,000 loose objects, warm page cache: 34.2 µs/obj vs 33.6 µs/obj (1.02×). The entire F1 effect is a concurrency effect. A single-request benchmark of this change concludes “no bug”, which is the trap.
2. Quadratic list handling in collect_commit_objects/6 — real in the code, not measurable. objs ++ acc at upload_pack_v2.ex:966 concatenates the parent’s entire ancestry list, and tree_objects ++ Enum.reverse(parent_objects) reverses it again per level. On the 1,907-commit corpus it accounted for 16 of 2,998 samples (0.5%) as a leaf, and the deepest recursion observed was 43 nested frames (under the 100-frame sampling cap). Not worth restructuring for its own sake.
3. Walk.log/3 pagination is correct. max_count: 30 against hephaestus: 31 ObjectResolver.read/2 calls, 9 ms. No full-history walk, no whole-pack load. Hazard worth a separate one-line fix: unknown option keys are silently ignored, so a caller typo (limit: 30 instead of max_count: 30) walks all 1,907 commits — 147 ms, 1,907 reads, and it trips the whole-pack cache. Keyword.validate!/2 would turn that into a visible error.
4. Diff.diff_commits/4 on a single-file commit is clean. HEAD~1..HEAD on hephaestus: 6 object reads, 1 Myers.diff/2 call, 5 ms, no measurable allocation. No repeated tree walking or blob re-reads on this shape. (This does not clear the large-diff path, which #48 already covers.)
5. ReceivePack.feed/2 chunk buffering is not quadratic. The :pack phase uses an iolist accumulator (absorb/2) with O(1) append per chunk — the per-chunk SHA-1 rescan from PR #27 is gone. The remaining gap is pack_acc_to_binary/1 materialising the whole pack once, already tracked as #153.
6. Index.lookup_in_raw/2 hex-decoding its argument per call is real but small. 244,664 calls per clone, but Base.decode16mixed! is only 42 of 2,964 samples (1.4%). Hoisting decode_sha/1 out of the per-pack loop in find_in_packs/3 is a cheap follow-up, not a headline.

---

Suggested order

#	finding	rank	measured effect
F7	receive-pack keeps packs instead of exploding them	2	59× on the push store path; stops production regenerating the loose layout
F3	verbatim pack-entry + delta reuse in Pack.Writer	2	11× by git’s own measurement; removes 76.5% of clone CPU; −38% pack bytes
F2	walk carries content once, blobs read lazily	3	−50% object reads / index lookups / delta applies; −502 MB peak; −11% wall
F5	drop String.upcase from Tree.encode_content/1	1	−4.9% of clone CPU, one line
F1	raw file I/O in Storage.Filesystem	1	10.1× read throughput at concurrency 8
F6	push ref-prefix into Ref.list/2	2	209 ms → proportional, on every git fetch
F4	fix the pack-reuse predicate, or delete the fast path	3	removes a discarded full graph walk per clone
F8	drop the whole-pack cache once F3 lands	5	−1.05 GB resident per concurrent fetch

F7 and F3 are a pair: F7 puts repos into a layout where F3 has something to reuse, and F3 is what makes that layout pay off. Neither is worth much without the other.

The F2 + F5 experiment is a working patch that passes the 112 protocol tests; it is on hand and can be raised as a PR on request. Everything else is a proposal, not code.