C The\sderive\slearns\swhat\sto\sskip
D 2026-09-09T22:54:54.805
E 2026-09-09T21:30:00 b3652db4e08bc2cdc46b0e39b1fd9ccc444e74f5
N text/x-markdown
U dmestas
W 3319
# The derive learns what to skip

A zit repository's timeline, branches, tags, and leaves are not stored — they
are derived from the immutable log on demand, memoised in a disposable
`.zit-cache`. Steady state, that cache serves an open in milliseconds. But
three events throw it away and force the full derive: the first open of a
fresh clone, a lost or deleted cache, and a record-format upgrade. On a wide
repository each of those cost minutes, and 0.3.10 is about making them not.

The full derive walked every artifact in the store, reconstructed each
through its whole delta chain, parsed it, and — for roughly four in five —
threw the result away, because a file blob and (on a wide store) a
sub-manifest tree node contribute nothing to the timeline. On the pkgsrc
corpus that is more than a million reconstructions to keep about two hundred
thousand records.

The instinct is to guess which artifacts matter. We measured instead. A
throwaway per-class timer in the walk said the dominant cost on the sanctioned
sub-manifest shape was not the file blobs, as a first guess had it, but the
tree nodes: 434,000 of them, six gigabytes of plaintext inflated and parsed
and discarded. That one number redrew the fix. A naive "skip blobs" rule would
have reached about 235 seconds; the rule the measurement actually demanded —
"skip everything that is not a check-in or a control artifact" — reaches 22,
from 302.

So zit now records one bit per stored artifact in an advisory `.zit-cache`
member: is this artifact crosslink-relevant? The map is distilled during the
one derive that has to touch everything regardless, and every later full
derive consults it and reconstructs only the frames that can produce a record.
A clone distills the map on the walk it already makes and consumes it on the
reopen a moment later.

The map is advisory, and it earns that word carefully. It is self-checksummed,
keyed to a validated log prefix, and stamped with the shun fingerprint it was
distilled under — so lifting a shun discards the map rather than letting a
revived check-in vanish from a stale bit. Any doubt at all reads as absent,
and the from-scratch derive is always the fallback. A derive that consumes the
map produces a byte-identical record body to one that does not; the map only
removes work, and a test pins that it removes exactly the right work and
changes nothing else.

Underneath the same commit, a smaller correction with a wider reach: both
cache writers used to concatenate their header, record body, and image into
one resident buffer before writing a single byte — a full second copy of the
whole derived state, hundreds of megabytes, at the exact moment memory was
already highest. They stream the pieces to the file now, checksum folded in as
they go. It changes no output and it is the kind of thing that is invisible
until you measure the peak.

Prefix-sniffing — reject a blob by its first bytes without a full parse — was
on the table and the measurement killed it: a sub-manifest tree node is a
well-formed card-structured artifact with no prefix that distinguishes it from
a check-in, so a sniff could not have reached the larger half of the win even
in principle. The bit knows what the sniff could only guess, because it is
written by the pass that already parsed the truth.

Z 3ad5362c12d70ea583e13b23e98bdfda