fossil finfo README is fast on any repository, and it is worth being exact
about why, because for three releases zit had the same fact in hand and kept
throwing it away.
fossil's answer is a lookup
fossil never computes a file's history when you ask for it. Every time a
check-in artifact arrives — by clone, sync, or commit — fossil diffs it against
its parent right then and inserts one row per changed file into a table called
mlink, indexed by file name. finfo is one query against that table joined
to the check-in table: find the rows for this name, order by date, limit. An
index lookup, milliseconds, regardless of size.
So the whole-repository cost is spread across the clone, once, and every later check-in costs only its own diff. There is no "build the file index" step. It can never be stale, and it cannot fail to converge, because it is part of writing the artifact.
zit had the row and dropped it
zit's derive — the pass that turns the log into leaves, tags and events, run
once when a store is first opened and extended over the tail on every append —
already computed each check-in's changed files against its parent. The code
comment even said "this is fossil's mlink row." It wrote them into the
checkpoint's record stream for the crosslink verb to hand to an external
consumer. It did not keep them as an index.
The file index was instead built by a separate full pass over the log on the
first log PATH. On a pkgsrc mirror that pass was 6.8 minutes in 0.3.17, and
it persisted nothing until it finished. A forge that warms the index on
startup and restarts more often than that never converged: every restart spent
the full duration and left nothing behind. Quarry deployed fifteen times in a
day, burned two hours of CPU on that mirror, and still had no index.
The change is small, because the work was already being done
The derive now hands each check-in's change set, inverted by path, to an index builder as it links the check-in, and stores the result beside the checkpoint. Trees are diffed by S card there like everywhere else since 0.3.17. On a fresh clone or import the index exists before the first query. On append the tail-extend path adopts the stored index and extends it over the new check-ins. A cache hit computes nothing, so opening a repository is untouched.
| pkgsrc, fresh clone to first file-history answer | 0.3.17 | 0.3.18 |
|---|---|---|
| wall clock | 11.7 min | 7.1 min + 1.15s |
| peak memory | 1.21 GB | 1.61 GB |
The derive alone was about five minutes. Building the index inside it adds about two and retires the separate pass, because both now share one read of every manifest. The cost is 0.4 GB of peak memory for the builder during that one derive, paid once, at clone, which is exactly where fossil pays it.
Two things the review caught
The on-query build is now the fallback, for a store whose cache predates this, and it checkpoints every 8,192 entries so an interrupted build resumes. That resumability is what quarry's existing mirror needs. It also opened a hole: a partial checkpoint is written under the live log's key, and the derive's tail-extend path adopted any index whose key matched the checkpoint's — including a partial one — and extended it as if whole, claiming coverage over the gap. That would have silently omitted history. The index now records how many entries existed when it was written, is complete only when it covers all of them, and only a complete index is ever extended.
The other was the kind of bug that only a test suite finds. The view struct
starts as undefined memory and its load paths write fields one at a time. A
field I added and pre-set from the caller was garbage on every path that
bypassed the caller — the git-import verifier among them — and the derive
called a function pointer made of 0xaa. Fourteen tests crashed at once. The
field is now set by the load itself, from an explicit argument, and nulled
when the load ends. That hazard was already documented on a neighbouring
field, which is a lesson about reading the neighbours.
What this means for a page
A page showing one file's history no longer needs anything warmed. The index is there from the first open, and a query is a second on pkgsrc. The whole-repository cost moved to the one place a repository is already paying for everything else.