0.3.14 gave zit log CHECKIN PATH an index. One file's history on a mirror
of SQLite went from over a minute to under two seconds, and the release
notes had a table to prove it.
The table was true and the index was nearly useless.
What was missing
The index is a fact the derive already computes and throws away: for each
check-in, the set of paths whose content moved against the primary parent.
Inverted by path, that is a file's history. It is stored as a .zit-cache
member, keyed — like the cluster and disposition maps — to the log tail and
the shun fingerprint, so a store that has moved on is never answered from a
stale map.
Keyed to the tail is the problem. A key that says this index describes a log ending at T has exactly one useful answer to give: yes, or no. Commit once and the answer is no, and the only thing the code could do with no was throw the file away and walk all 36,702 check-ins again.
So the index helped precisely when nothing was happening. On a repository anyone was actually working in — which is the only kind worth indexing — the first PATH query after every commit paid the full rebuild. The benchmark measured a case the benchmark had created.
What changed
The stored index now records one more number: n_rids, how many log
entries it was built from. Validation moved from an exact tail match to a
prefix match, which is the whole idea in one line — an index built over a
shorter log is still a correct index over that prefix. It does not have
to be discarded. It has to be extended.
A query now adopts what is on disk, walks only the entries past n_rids,
folds the two together, and stores the union. Three cases, one path
through the code:
| on disk | walked |
|---|---|
| nothing | every check-in |
| an index over the whole log | none |
| an index over the log minus your last five commits | five |
On the SQLite mirror (131,558 artifacts), log --long -n 5 README.md:
no index 36,702 check-ins walked 55.4s
index, unchanged 0 check-ins walked 0.52s
a different path 0 check-ins walked 0.51s
The third line is the one that matters more than it looks. The index is per-repository, not per-path: asking about a second file costs nothing, because the work was never about the file you asked for.
The test that would have caught it
There wasn't one, and that is the honest lesson here. Every 0.3.14 test asserted the answer — byte-identical output against reconstruction, for every tracked path. All of them passed. None of them asserted the cost, so the version that rebuilt from scratch on every commit looked exactly like the version that didn't.
ZIT_FILEINDEX_WALK=1 now prints how many check-ins the walk covered, and
an end-to-end test commits into a repository with a warm index and asserts
that number is exactly the number of check-ins added. A regression to
rebuild-everything fails the gate instead of showing up as a slow
afternoon six weeks later.
What this is not
The index narrows the walk. It does not decide the answer. captureFileFacts
diffs at F-card level, so a delta manifest restating a path with the hash it
already had reads as a change there while the path's effective content did
not move — an over-report, never an omission. Every candidate the index
names is still confirmed by the change predicate, which is what keeps the
answer exact rather than merely fast. ZIT_NO_FILE_INDEX=1 answers by
reconstruction alone, and the test comparing the two for every tracked path
is unchanged.
The cache stays advisory: delete it and you pay one rebuild. A v1 file from 0.3.14 is ignored rather than misread.
What is still slow
About half a second of every figure above is opening the repository. With the walk at zero that is now the entire cost of a warm PATH query, and it is paid again by every single CLI invocation. It is the next thing, and it is tracked on its own.