Skip to content
zit zit 0.3.18
Login

zit 0.3.18

zit 0.3.18 builds the file-history index the way fossil builds mlink: at ingest, as each check-in is derived, rather than on demand by the first query. A file's history is an index lookup from a repository's first query onward, the whole-repository cost lands where clone and import already pay, and the build can no longer fail to converge on a restarting forge.

No artifact, tag namespace, sync message, or on-disk repository format changed. The index format moved to version 4; a version-3 index is rebuilt once.

The index is built by the derive

fossil answers finfo from a table it fills as each check-in artifact arrives: one row per changed file, indexed by name, so the query is a lookup. zit's derive — the pass that turns the log into leaves, tags and events, run once when a store is first opened and extended over the tail on every append — was already computing the identical fact for each check-in and writing it into the checkpoint's record stream. It just never kept it as an index. The index was instead rebuilt from scratch by a separate full pass on the first log PATH.

The derive now hands each check-in's change set, inverted by path, to an index builder as it links the check-in, and stores the result beside the checkpoint. Trees are diffed by S card there like everywhere else. On a fresh clone or import the index exists before the first query; on append, the tail-extend path adopts the stored index and extends it over the new check-ins; a cache hit computes nothing, so the zero-parse open is untouched.

pkgsrc submanifest mirror, 218,444 check-ins, fresh clone to first file-history answer 0.3.17 0.3.18
wall clock 11.7 min (derive, then the index pass) 7.1 min (derive with the index inline) + 1.15s
peak memory 1.21 GB 1.61 GB
where the index cost lands first query, in one non-resumable pass clone, checkpointed with the derive
every later query ~1s ~1s

The derive alone was about five minutes; building the index inside it adds about two, because both share one read of every manifest, and retires the separate 6.8-minute pass. The price is 0.4 GB of peak memory for the index builder during that one derive.

The on-query build is now the fallback, and it resumes

A store whose cache predates this release still builds its index on the first log PATH. That build now checkpoints every 8,192 entries, so a build a restart interrupts resumes from the last chunk instead of from zero. A forge that restarts more often than the whole build takes now converges across restarts; before, every restart spent the full duration and left nothing.

Measured on the pkgsrc mirror carrying a 0.3.17 index: the new binary refused that index and began the rebuild; killed at 90 seconds, it had saved 909,312 of 1,249,347 entries, marked incomplete; the next run walked only the remaining check-ins, 87 seconds, and finished with a complete index; the query after that walked nothing and answered in 1.06s.

A partial index is recorded as partial: version 4 stores how many entries existed when it was written, an index is complete only when it covers all of them, and the derive's tail-extend path adopts only a complete one. Without that, a partial index whose key matched the checkpoint's could be adopted as whole and claim coverage over the gap — an omission from file history, found in review before it shipped.

One limit of the fallback, stated rather than buried: it covers by entry count, so a phantom filled after a build is never revisited by a later resume. The derive walks by frame offset and re-covers it, which is one more reason the derive-built index is primary.

The crosslink stream used to say files unknown for a tree check-in, because reporting them meant flattening the tree and the derive runs on every cold open. With the S-card diff a tree costs about what changed, so the stream now carries its files like any other check-in's. files unknown still means what it always meant — a parent the store does not hold, or a sub-manifest that is absent or unreadable — and never a tree as such.

Compatibility

Qualification

zig build test, e2e, smoke (Fossil interop against a real fossil), muscle (41 differential steps), git-import-e2e, git-export-e2e, dst, and cross, plus the three-runner CI battery. The pkgsrc figures are release builds against the submanifest mirror in the benchmark corpus with the cache removed before each cold run; both columns ran alone on the same machine.