C Where\sfossil\spays\sfor\sfile\shistory,\sand\swhere\szit\snow\sdoes
D 2026-09-16T22:13:22.572
E 2026-09-16T22:13:22.572 1891492270dccec4cc3173655fc5902518f63dd7
N text/x-markdown
U dmestas
W 4429
# Where fossil pays for file history, and where zit now does
`fossil finfo README` is fast on any repository, and it is worth being exact
about why, because for three releases zit had the same fact in hand and kept
throwing it away.
## fossil's answer is a lookup
fossil never computes a file's history when you ask for it. Every time a
check-in artifact arrives — by clone, sync, or commit — fossil diffs it against
its parent right then and inserts one row per changed file into a table called
`mlink`, indexed by file name. `finfo` is one query against that table joined
to the check-in table: find the rows for this name, order by date, limit. An
index lookup, milliseconds, regardless of size.
So the whole-repository cost is spread across the clone, once, and every later
check-in costs only its own diff. There is no "build the file index" step.
It can never be stale, and it cannot fail to converge, because it is part of
writing the artifact.
## zit had the row and dropped it
zit's derive — the pass that turns the log into leaves, tags and events, run
once when a store is first opened and extended over the tail on every append —
already computed each check-in's changed files against its parent. The code
comment even said "this is fossil's mlink row." It wrote them into the
checkpoint's record stream for the `crosslink` verb to hand to an external
consumer. It did not keep them as an index.
The file index was instead built by a separate full pass over the log on the
first `log PATH`. On a pkgsrc mirror that pass was 6.8 minutes in 0.3.17, and
it persisted nothing until it finished. A forge that warms the index on
startup and restarts more often than that never converged: every restart spent
the full duration and left nothing behind. Quarry deployed fifteen times in a
day, burned two hours of CPU on that mirror, and still had no index.
## The change is small, because the work was already being done
The derive now hands each check-in's change set, inverted by path, to an index
builder as it links the check-in, and stores the result beside the checkpoint.
Trees are diffed by S card there like everywhere else since 0.3.17. On a fresh
clone or import the index exists before the first query. On append the
tail-extend path adopts the stored index and extends it over the new check-ins.
A cache hit computes nothing, so opening a repository is untouched.
| pkgsrc, fresh clone to first file-history answer | 0.3.17 | 0.3.18 |
|---|---|---|
| wall clock | 11.7 min | 7.1 min + 1.15s |
| peak memory | 1.21 GB | 1.61 GB |
The derive alone was about five minutes. Building the index inside it adds
about two and retires the separate pass, because both now share one read of
every manifest. The cost is 0.4 GB of peak memory for the builder during that
one derive, paid once, at clone, which is exactly where fossil pays it.
## Two things the review caught
The on-query build is now the fallback, for a store whose cache predates
this, and it checkpoints every 8,192 entries so an interrupted build resumes.
That resumability is what quarry's existing mirror needs. It also opened a
hole: a partial checkpoint is written under the live log's key, and the
derive's tail-extend path adopted any index whose key matched the checkpoint's
— including a partial one — and extended it as if whole, claiming coverage
over the gap. That would have silently omitted history. The index now records
how many entries existed when it was written, is complete only when it covers
all of them, and only a complete index is ever extended.
The other was the kind of bug that only a test suite finds. The view struct
starts as undefined memory and its load paths write fields one at a time. A
field I added and pre-set from the caller was garbage on every path that
bypassed the caller — the git-import verifier among them — and the derive
called a function pointer made of `0xaa`. Fourteen tests crashed at once. The
field is now set by the load itself, from an explicit argument, and nulled
when the load ends. That hazard was already documented on a neighbouring
field, which is a lesson about reading the neighbours.
## What this means for a page
A page showing one file's history no longer needs anything warmed. The index
is there from the first open, and a query is a second on pkgsrc. The
whole-repository cost moved to the one place a repository is already paying
for everything else.
Z 657e6aeac004704b7711f8102bc7f105