zit 0.7.5's import-git imports the first 29,792 commits of the Linux
kernel in about 120 seconds, where 0.7.4 took 597. The memory the whole
kernel history needs is projected at about 3.3 GB, where it was about
6.2 GB. This note records what changed, how it was measured, and what
stays true.
How it was measured
The benchmark is the first 29,792 commits of torvalds/linux: a 2.5 GB
fast-export stream holding 97,015 blobs. Every stage was measured as a
back-to-back pair against trunk just before it, on the same machine, an
Apple M4 Pro with 14 cores. That machine was shared with other jobs, so
the two numbers in a pair compare; numbers from different pairs do not.
Every stage was checked to write the same store at 1 and 8 threads.
For reference, on the same slice, git fast-export alone takes 12 s,
git fast-import and git repack -adf take 175 s for a 105 MB pack,
and Fossil 2.28's import --git took 1 h 29 m on a 4-core ARM64 host.
Where the time went, and what changed
0.7.4 spent about 455 s ingesting, 190 s packing and 175 s verifying and publishing.
| Stage | Before | After |
|---|---|---|
| The tree editor keeps decoded nodes across commits and names each node once | 763 s | 572 s |
| Verify reconstructs only the drop set; pack leaves the published sweep to the verifier | 602 s | 569 s |
| One pass: no raw generation and no repack | 530–543 s | 314–384 s |
| Delta search and compression on worker threads, one writer in fixed order | 308 s | 203 s |
| The verifier on the worker pool, reads in bounded windows | 201 s | 172 s |
| The tree editor remembers each filename's partition hash | 199 s | 183 s |
| Parallel blob readers, overlapped blob writes, warm-up on the pool | ~155 s | ~120 s |
The largest step is the single pass. 0.7.4 wrote a raw generation of the store, then repacked it. 0.7.5 holds each blob until the commit that names its path, deltas it against the previous blob at that path, and deltas each tree node against the node it replaces, so the first write is the final one. The verifier then checks a tally of names rather than reading a raw generation back. The store is 177 MB where it was 201 MB, with the same artifacts and the same content.
The worker threads change who computes, never what is written: one
writer appends frames in a fixed order, so --threads N writes the
same bytes for every N.
Ingest is now bound by the main thread editing trees. The next large gain would mean editing trees in parallel, which is a project of its own.
Memory
On the slice the import peaks under 1 GB, but three tables grew with every commit and blob:
| Fix | Measured | At kernel scale |
|---|---|---|
| The cache warm-up writes its body to the cache file as it builds it | same speed | about −4 GB |
| Ingest tables: an array indexed by mark, one record per blob | 2,165 → 1,324 bytes a commit | about −2.8 GB |
| The verifier reads its witness and the import map back a window at a time | 4.07 → 2.07 KiB a commit | about 6.2 → 3.3 GB at the verify peak |
The kernel figures are projections from the growth measured on 10,000- and 30,000-commit slices. Two more savings are known and deferred: the repository view the verifier builds for one of its checks (about 2.5 GB), and binary names in the per-blob record (about 640 MB).
The verifier's witness is spilled to a file in the staging directory. It is authenticated with SipHash under a key chosen for the run, and each record is checked as it is read back, so a file changed during the run fails verification instead of passing it.
A full-kernel run
A build from before the memory work imported the whole history, 1,503,488 commits, on an 8-core x86-64 Linux server with 15 GB, at the lowest priority with 4 threads, under an 8 GB soft and 10 GB hard limit. It reached 543,744 commits in 2 h 19 m and 1,500,992 (99.8%) in about 9 hours, throttled at the limit. After ingest it stalled re-reading the store with no memory left to cache it, and was stopped at 13 h 38 m with a peak of 8,496 MB. The import's memory also squeezed the host enough that the out-of-memory killer took processes belonging to another job.
A full-kernel run with 0.7.5 has not been made yet. It belongs on a host with headroom beyond the import's limit.
What stays true
- The store holds the same artifacts with the same content as 0.7.4 writes for the same history.
- Every thread count writes the same bytes.
- The checkpoint file an import resumes from is still read whole and is limited to 4 GB.