C An\sindex\sthat\ssurvives\sa\scommit
D 2026-09-14T16:50:47.236
E 2026-09-14T16:50:47.236 3f58202e745aaa7453cb5fa090f31d624a24dbb8
N text/x-markdown
U dmestas
W 3971
# An index that survives a commit

0.3.14 gave `zit log CHECKIN PATH` an index. One file's history on a mirror
of SQLite went from over a minute to under two seconds, and the release
notes had a table to prove it.

The table was true and the index was nearly useless.

## What was missing

The index is a fact the derive already computes and throws away: for each
check-in, the set of paths whose content moved against the primary parent.
Inverted by path, that is a file's history. It is stored as a `.zit-cache`
member, keyed — like the cluster and disposition maps — to the log tail and
the shun fingerprint, so a store that has moved on is never answered from a
stale map.

Keyed to the tail is the problem. A key that says *this index describes a
log ending at T* has exactly one useful answer to give: yes, or no. Commit
once and the answer is no, and the only thing the code could do with no was
throw the file away and walk all 36,702 check-ins again.

So the index helped precisely when nothing was happening. On a repository
anyone was actually working in — which is the only kind worth indexing —
the first PATH query after every commit paid the full rebuild. The
benchmark measured a case the benchmark had created.

## What changed

The stored index now records one more number: `n_rids`, how many log
entries it was built from. Validation moved from an exact tail match to a
prefix match, which is the whole idea in one line — an index built over a
shorter log is still a correct index *over that prefix*. It does not have
to be discarded. It has to be extended.

A query now adopts what is on disk, walks only the entries past `n_rids`,
folds the two together, and stores the union. Three cases, one path
through the code:

| on disk | walked |
|---|---|
| nothing | every check-in |
| an index over the whole log | none |
| an index over the log minus your last five commits | five |

On the SQLite mirror (131,558 artifacts), `log --long -n 5 README.md`:

```
no index          36,702 check-ins walked    55.4s
index, unchanged       0 check-ins walked     0.52s
a different path       0 check-ins walked     0.51s
```

The third line is the one that matters more than it looks. The index is
per-repository, not per-path: asking about a second file costs nothing,
because the work was never about the file you asked for.

## The test that would have caught it

There wasn't one, and that is the honest lesson here. Every 0.3.14 test
asserted the answer — byte-identical output against reconstruction, for
every tracked path. All of them passed. None of them asserted the *cost*,
so the version that rebuilt from scratch on every commit looked exactly
like the version that didn't.

`ZIT_FILEINDEX_WALK=1` now prints how many check-ins the walk covered, and
an end-to-end test commits into a repository with a warm index and asserts
that number is exactly the number of check-ins added. A regression to
rebuild-everything fails the gate instead of showing up as a slow
afternoon six weeks later.

## What this is not

The index narrows the walk. It does not decide the answer. `captureFileFacts`
diffs at F-card level, so a delta manifest restating a path with the hash it
already had reads as a change there while the path's effective content did
not move — an over-report, never an omission. Every candidate the index
names is still confirmed by the change predicate, which is what keeps the
answer exact rather than merely fast. `ZIT_NO_FILE_INDEX=1` answers by
reconstruction alone, and the test comparing the two for every tracked path
is unchanged.

The cache stays advisory: delete it and you pay one rebuild. A v1 file from
0.3.14 is ignored rather than misread.

## What is still slow

About half a second of every figure above is opening the repository. With
the walk at zero that is now the entire cost of a warm PATH query, and it is
paid again by every single CLI invocation. It is the next thing, and it is
tracked on its own.

Z cea3f97608e6e62441fe5ca157801f65