Artifact a14836f72ef947b47c15115ec74ac6aac54f7cc3a53c787fdcbc5fceeaab78b7:
- File
www/caching.html
— part of check-in
[269e24b3c3]
at
2026-08-18 16:04:59
on branch trunk
— www: embed the docs, pinned so the reference cannot drift
The command reference and the caching write-up move into the tree as embedded documentation, which quarry serves on fossil's own /doc/VERSION/PATH route, already implemented in its api/register.go.
Fossil's www/embeddeddoc.wiki gives the reason to prefer this over the wiki, and it is the reason the pin below can exist:
The documentation files are versioned together with the source code files so it is always clear what version of the documentation goes with a particular release.
A wiki page cannot be pinned by a test; nothing ties it to a checkout. A tracked file can. reference_test.zig checks both directions -- a dispatched verb with no section, and a section naming no verb -- with a reviewed-exception list for sections that document an absence on purpose, which is what the export-fossil section is. The reference had drifted for weeks: grep, purge, config and trust all shipped undocumented, and nothing noticed because noticing needed a human to compare two lists.
The caching page's ladder is corrected while embedding it. It reported 641.9s / 58.3s / 0.31s; those were Debug numbers. Measured in the mode that ships: 29.63s full derive, 0.34s record-body replay, 0.04s image. Having a cache is worth 87x and the image a further 8.5x -- both real, neither 188x. Its footnote about a separate environment reporting 0.38s 'unreconciled' is resolved: same machine, optimized build. info also moved to the image since it was written, leaving branch and tag on the record-body rung. (user: dmestas, size: 25269) [annotate] [blame] [check-ins using]
<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width,initial-scale=1"> <title>Disposable by Construction</title> <style> :root { --bg: #f3f5f6; --panel: #ffffff; --panel-2: #eaeef0; --ink: #15202a; --muted: #5c6a72; --line: #dbe2e5; --warm: #a8620d; --warm-soft: #f6ead9; --cold: #2c6b7a; --cold-soft: #e2eef1; --code-bg: #edf1f2; --display: ui-sans-serif, -apple-system, "Segoe UI", system-ui, sans-serif; --body: "Iowan Old Style", Charter, "Bitstream Charter", Georgia, serif; --mono: ui-monospace, "SF Mono", SFMono-Regular, Menlo, Consolas, monospace; } @media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) { --bg: #10161a; --panel: #171f24; --panel-2: #1d262c; --ink: #dfe7ea; --muted: #8b9aa2; --line: #263137; --warm: #dfa057; --warm-soft: #2a2016; --cold: #6bb6c8; --cold-soft: #13262b; --code-bg: #0e1418; } } :root[data-theme="dark"] { --bg: #10161a; --panel: #171f24; --panel-2: #1d262c; --ink: #dfe7ea; --muted: #8b9aa2; --line: #263137; --warm: #dfa057; --warm-soft: #2a2016; --cold: #6bb6c8; --cold-soft: #13262b; --code-bg: #0e1418; } * { box-sizing: border-box; } body { background: var(--bg); color: var(--ink); font: 17px/1.65 var(--body); margin: 0; padding: 0; -webkit-font-smoothing: antialiased; } .wrap { max-width: 62rem; margin: 0 auto; padding: 3.5rem 1.5rem 6rem; } h1, h2, h3, .tier h3, .ladder-l { font-family: var(--display); } h1 { font-size: clamp(2.1rem, 5vw, 3.1rem); line-height: 1.05; margin: 0 0 1rem; letter-spacing: -0.033em; font-weight: 700; text-wrap: balance; } .thesis { font-size: 1.2rem; max-width: 40ch; color: var(--muted); margin: 0; } .thesis b { color: var(--ink); font-weight: 600; } .eyebrow { font-family: var(--display); text-transform: uppercase; letter-spacing: 0.14em; font-size: 0.7rem; font-weight: 650; color: var(--warm); margin: 0 0 0.9rem; } h2 { font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.13em; font-weight: 650; color: var(--muted); margin: 0; padding-bottom: 0.5rem; border-bottom: 1px solid var(--line); } section { margin-top: 3.6rem; display: flex; flex-direction: column; gap: 1.1rem; } header.page { display: flex; flex-direction: column; gap: 0; } p { margin: 0; max-width: 66ch; } .stack { display: flex; flex-direction: column; gap: 1rem; } code, .mono { font-family: var(--mono); font-size: 0.86em; } code { background: var(--code-bg); padding: 0.1em 0.36em; border-radius: 3px; } .ladder { display: flex; flex-direction: column; gap: 0.5rem; margin: 0; } .rung { display: grid; grid-template-columns: 7.5rem 1fr auto; gap: 1rem; align-items: center; background: var(--panel); border: 1px solid var(--line); border-radius: 7px; padding: 0.75rem 1rem; } .ladder-t { font-family: var(--mono); font-size: 1.02rem; font-variant-numeric: tabular-nums; font-weight: 600; } .rung.cold .ladder-t { color: var(--cold); } .rung.warm .ladder-t { color: var(--warm); } .ladder-l { font-size: 0.93rem; font-weight: 600; } .ladder-d { color: var(--muted); font-size: 0.88rem; font-family: var(--body); } .bar { height: 6px; border-radius: 3px; background: var(--cold); opacity: 0.85; } .rung.warm .bar { background: var(--warm); } .barwrap { min-width: 4rem; } .tier { background: var(--panel); border: 1px solid var(--line); border-radius: 9px; padding: 1.15rem 1.3rem 1.25rem; display: flex; flex-direction: column; gap: 0.8rem; } .tier > header { display: flex; flex-wrap: wrap; align-items: baseline; gap: 0.6rem; } .tier h3 { margin: 0; font-size: 1.06rem; letter-spacing: -0.012em; font-weight: 680; } .path { font-family: var(--mono); font-size: 0.78rem; color: var(--muted); background: var(--panel-2); padding: 0.13em 0.5em; border-radius: 4px; } .pill { font-family: var(--display); font-size: 0.65rem; font-weight: 650; text-transform: uppercase; letter-spacing: 0.09em; padding: 0.2em 0.55em; border-radius: 20px; margin-left: auto; } .pill.warm { background: var(--warm-soft); color: var(--warm); } .pill.cold { background: var(--cold-soft); color: var(--cold); } .spec { display: grid; grid-template-columns: 8.5rem 1fr; gap: 0.35rem 1rem; margin: 0; } .spec dt { font-family: var(--display); font-size: 0.65rem; text-transform: uppercase; letter-spacing: 0.1em; font-weight: 650; color: var(--muted); padding-top: 0.32em; } .spec dd { margin: 0; font-size: 0.94rem; } .note { font-size: 0.93rem; color: var(--muted); } .note b { color: var(--ink); font-weight: 600; } .callout { border-left: 3px solid var(--warm); background: var(--panel); padding: 1rem 1.2rem; border-radius: 0 7px 7px 0; display: flex; flex-direction: column; gap: 0.6rem; } .callout.cold { border-left-color: var(--cold); } .callout h3 { margin: 0; font-size: 0.98rem; font-weight: 680; letter-spacing: -0.01em; } ul { margin: 0; padding-left: 1.15rem; display: flex; flex-direction: column; gap: 0.45rem; } li { max-width: 64ch; } .tablewrap { overflow-x: auto; } table { border-collapse: collapse; width: 100%; font-size: 0.9rem; } th, td { text-align: left; padding: 0.5rem 0.9rem 0.5rem 0; border-bottom: 1px solid var(--line); vertical-align: top; } th { font-family: var(--display); font-size: 0.66rem; text-transform: uppercase; letter-spacing: 0.1em; color: var(--muted); font-weight: 650; } td.m { font-family: var(--mono); font-size: 0.82rem; white-space: nowrap; } .credits { font-size: 0.92rem; color: var(--muted); } .credits b { color: var(--ink); font-weight: 600; font-family: var(--display); } footer { margin-top: 4rem; padding-top: 1.2rem; border-top: 1px solid var(--line); color: var(--muted); font-size: 0.86rem; } @media (max-width: 640px) { .rung { grid-template-columns: 5.5rem 1fr; } .barwrap { display: none; } .spec { grid-template-columns: 1fr; gap: 0.15rem 0; } .spec dd { margin-bottom: 0.5rem; } } @media (prefers-reduced-motion: no-preference) { html { scroll-behavior: smooth; } } </style> </head> <body> <div class="wrap"> <header class="page"> <p class="eyebrow">zit — optimization machinery</p> <h1>Disposable by Construction</h1> <p class="thesis">Nine caching layers sit between a zit repository and an answer. <b>Deleting any of them changes no answer</b> — that constraint is what makes the rest of the design possible.</p> </header> <section> <h2>The cost of one listing, three ways</h2> <p>Measured on the same store — tcl, 300 MB, 2,193 paths at tip — on one machine, through <code>-R</code>, in the mode zit ships (<code>--release=fast</code>). These are the same work; only which layer answered differs.</p> <div class="ladder"> <div class="rung cold"> <span class="ladder-t">29.63s</span> <span><span class="ladder-l">Full log derive</span><br><span class="ladder-d">Every frame read and CRC’d, all derived state rebuilt. What <code>--no-cache</code> forces.</span></span> <span class="barwrap"><div class="bar" style="width:11rem"></div></span> </div> <div class="rung cold"> <span class="ladder-t">0.34s</span> <span><span class="ladder-l">Record-body replay</span><br><span class="ladder-d">A validated cache hit that still reconstructs ~14 MB of state on every process start. Where <code>branch</code> and <code>tag</code> still land.</span></span> <span class="barwrap"><div class="bar" style="width:0.4rem"></div></span> </div> <div class="rung warm"> <span class="ladder-t">0.04s</span> <span><span class="ladder-l">Zero-parse image</span><br><span class="ladder-d">Columns viewed in place. No hashmap fill, no propagation, no adjacency walk.</span></span> <span class="barwrap"><div class="bar" style="width:0.1rem"></div></span> </div> </div> <p class="note">Two different gaps, worth separating. Having a cache at all is worth <b>87×</b> — that is the top rung against the middle, and it is the one that would hurt to lose. The image is a further <b>8.5×</b> on top. Both are cache <i>hits</i>; a hit that still rebuilds something is a different animal from one that does not, and <code>ls</code> took the middle rung for years because the verb asked for it, not because it needed to.</p> <div class="callout"> <h3>An earlier revision of this page reported 641.9s / 58.3s / 0.31s</h3> <p class="note">Those were <b>Debug</b> numbers — bare <code>zig build</code>, which is what Zig gives you without <code>--release</code>. On this workload Debug is roughly <b>150×</b> slower than the shipped binary, so it inflates every rung and, worse, inflates them unevenly: the record-body rung is dominated by parsing and allocation, which Debug punishes hardest, while the image rung barely moves because it does almost no work either way. That is what turned a real 8.5× into an apparent 188×. The page also carried a footnote about a “separate environment” reporting 0.38s for the same corpus, unreconciled — it was not a separate environment. It was the same machine with an optimized build. See <code>build.zig</code>’s bench notes for the measurement rules this cost us.</p> </div> </section> <section> <h2>The rule every layer obeys</h2> <p>These layers share one contract, and it is unusually strict: <b>a cache may make an answer faster, never different.</b> Every tier below can be deleted at any moment, mid-operation, and the only cost is time.</p> <div class="callout"> <h3>Why that is not the normal arrangement</h3> <p class="note">Most version control systems make derived state load-bearing. Fossil keeps <code>mlink</code>, <code>plink</code>, <code>event</code> and <code>tagxref</code> as persistent SQL tables and reconstructs them only under an explicit <code>fossil rebuild</code>; Git’s index and packfiles are part of the repository you are expected to keep. In zit the append-only log is the whole repository, and everything else is a memo of a pure function over it. That is what lets a corrupt, truncated, stale or half-written cache be treated as a <i>miss</i> rather than as damage.</p> </div> <p>The validation is uniform. Every derived tier is keyed on the log’s own identity — end offset, frame count, last frame offset, last header CRC — plus a self-checksum over its own bytes. Any mismatch, wrong magic, wrong version, or failed checksum is a miss. Nothing is trusted because it was found.</p> </section> <section> <h2>Memos of derived state</h2> <p>These answer “what does history mean” — branches, tags, leaves, ancestry, name resolution. They live under <code>.zit-cache/</code>, the advisory tier beside the store.</p> <div class="stack"> <article class="tier"> <header><h3>The zero-parse image</h3><span class="path">.zit-cache/repo · image section</span><span class="pill warm">warm path</span></header> <p>The hot query answers of a fully-derived state, laid out as dense integer-addressed columns and reloaded as offset-addressed windows into the cache buffer. Nothing is parsed, allocated or relocated on load — the columns <i>are</i> the loaded form.</p> <dl class="spec"> <dt>Holds</dt><dd>Per rid: event mtime, flags (has-event / leaf / closed), active branch id with the main-branch fallback already resolved, and the artifact hash. Plus the leaf list, the resolved symbol table, and the full parent and child adjacency with the primary-link flag.</dd> <dt>Validated by</dt><dd>Log-tail key, self CRC, a layout version, and per-column structural checks: every span inside the image, every CSR non-decreasing and closing at its pool length, every stored rid naming a real slot.</dd> <dt>On mismatch</dt><dd>Falls through to the record body below — still a validated hit, just a slower one.</dd> </dl> <p class="note"><b>Its one allocation</b> is widening the leaf rid list. Everything else is a window.</p> </article> <article class="tier"> <header><h3>The crosslink record body</h3><span class="path">.zit-cache/repo · record section</span><span class="pill cold">middle rung</span></header> <p>The inputs to derivation, stored in persisted order at a named log prefix, so a read verb replays records instead of re-reading and re-CRC’ing the whole log.</p> <dl class="spec"> <dt>Holds</dt><dd>The crosslink input records — what the derive consumed, not what it produced.</dd> <dt>Validated by</dt><dd>The same log-tail key and self-checksum as the image; the two travel in one file.</dd> <dt>On mismatch</dt><dd>Full rebuild from the log, then the cache is rewritten atomically.</dd> </dl> <p class="note">This is the fallback that makes the image safe to change. A layout bump costs one slow open, never a wrong answer — which is why adding adjacency to the image needed no migration path at all.</p> </article> <article class="tier"> <header><h3>The rebuild checkpoint</h3><span class="path">.zit-cache/checkpoint</span><span class="pill warm">warm path</span></header> <p>A prefix-anchored claim: as of check-in C, whose frame closes the log prefix, the fully-verified file set fingerprints to this value — plus the record body for that prefix.</p> <dl class="spec"> <dt>Holds</dt><dd>One anchor and the crosslink records up to it, so a from-scratch derive can resume from the prefix instead of re-parsing a permanently flat segment.</dd> <dt>Validated by</dt><dd>Its own prefix key; <code>zit checkpoint verify</code> re-derives everything it claims and compares byte for byte, trusting nothing.</dd> <dt>On mismatch</dt><dd>Ignored. The first full derivation plants a fresh one at no extra cost.</dd> </dl> <p class="note"><b>Sidecar only, deliberately.</b> An earlier design pinned the anchor’s file set as stored sub-manifest artifacts; it was retired because minting artifacts early created permanent, unreferenced objects — and everything it bought was available for free elsewhere. A cache that writes history is no longer disposable.</p> </article> </div> </section> <section> <h2>Memos of log structure</h2> <div class="stack"> <article class="tier"> <header><h3>The log-scan index</h3><span class="path">.zit-cache/index</span><span class="pill warm">warm path</span></header> <p>Frame locations, so opening the store does not mean walking every frame header.</p> <dl class="spec"> <dt>Holds</dt><dd>The serialized scan of the log’s frame headers, adopted by the lazy content channel on open.</dd> <dt>Validated by</dt><dd>Itself — its own magic, CRC and log key, checked by the adopter. The cache module is a dumb byte carrier for it and never inspects the contents.</dd> <dt>On mismatch</dt><dd>Re-walk the headers. Absence and unreadability are the same case: a miss.</dd> </dl> </article> </div> </section> <section> <h2>Memos of content bytes</h2> <p>These are in-process, not on disk. They exist because reconstructing one blob and reconstructing a thousand share almost all of their work.</p> <div class="stack"> <article class="tier"> <header><h3>The plaintext memo</h3><span class="path">logstore.PlaintextCache</span><span class="pill warm">in process</span></header> <p>A delta family’s hot bases, kept resident for the span of one sweep. Without it, a walk over delta chains re-reconstructs each base once per referencing delta.</p> <dl class="spec"> <dt>Holds</dt><dd>Reconstructed plaintexts keyed by frame offset. A resolved delta’s own plaintext is memoized so a deeper delta built on it hits; raw and zlib top-level frames are not, so a delta-free repository pays nothing.</dd> <dt>Bounded by</dt><dd>64 MiB resident, with an entry cap <i>derived</i> from the byte budget so the two cannot disagree about which is doing the bounding.</dd> <dt>On pressure</dt><dd>Eviction. It changes the work, never the answer — a miss simply reconstructs.</dd> </dl> <p class="note">Turned a projected <b>8.5×</b> read amplification on depth-16 chains into roughly two reconstructions per frame regardless of depth. The verb that most recently needed it was <code>log</code>, which walked history without it and paid for the omission on every repository large enough to notice.</p> </article> <article class="tier"> <header><h3>The sub-manifest memo</h3><span class="path">repostate.SubManifestMemo</span><span class="pill warm">in process</span></header> <p>The same idea one layer up: tree nodes shared between check-ins, held while a partition walk runs.</p> <dl class="spec"> <dt>Bounded by</dt><dd>64 MiB, chosen against a measurement rather than a guess — a pkgsrc-scale tree of ~123k files puts the working set near 30 MB, so the budget holds one comfortably while staying a statable cost.</dd> <dt>Character</dt><dd>A memory-versus-hit-rate dial, not a correctness or capacity bound. Too small costs a little speed and nothing else.</dd> </dl> </article> <article class="tier"> <header><h3>Delta base selection</h3><span class="path">logstore · pack</span><span class="pill cold">write path</span></header> <p>Not a cache — the reason the caches above matter. Stored artifacts are delta-compressed against a chosen base, so reading one blob may mean walking a chain.</p> <dl class="spec"> <dt>Character</dt><dd>A base is a hint, never a requirement. Ineligible or unreadable candidates are silently skipped; there is no error path for a bad candidate.</dd> </dl> <p class="note">One consequence worth knowing: a blob’s <i>logical</i> size never requires walking its chain. A zlib container states its uncompressed length in its first four bytes, and a delta carries its target size in its own header — so <code>zit ls --size</code> costs a header read per file, while the text/binary verdict, which needs every byte, does not.</p> </article> </div> </section> <section> <h2>Memos of the working tree</h2> <div class="stack"> <article class="tier"> <header><h3>The materialization plan</h3><span class="path">.zit-cache/vstat</span><span class="pill warm">warm path</span></header> <p>Which working files are already correct, so <code>update</code> does not read every file to find out.</p> <dl class="spec"> <dt>Trusted only when</dt><dd>The row’s digest is the one the current baseline expects <i>and</i> inode, size and mtime all still match the file on disk. Anything else falls back to a full content compare, which remains the authority.</dd> <dt>Known trade</dt><dd>An edit that restores mtime, size and inode exactly is invisible to the stat check — fossil’s own <code>vfile</code> trade, taken knowingly rather than inherited.</dd> </dl> </article> <article class="tier"> <header><h3>The store cone</h3><span class="path">.zit-cache/store-cone</span><span class="pill cold">policy</span></header> <p>The odd one out: <b>policy, not memo.</b> It records which subtree a sparse store is meant to hold.</p> <dl class="spec"> <dt>Why it lives here</dt><dd>Losing it heals toward dense — elided content reclassifies as transit, so the cost of loss is bandwidth, never data. That loss rule is what makes it safe to co-locate with the disposable tier despite not being a memo.</dd> </dl> </article> <article class="tier"> <header><h3>The worktree registry</h3><span class="path">.zit-cache/worktrees</span><span class="pill cold">advisory</span></header> <p>Which working trees a store serves. Trees stamp themselves on open and update.</p> <dl class="spec"> <dt>Self-healing</dt><dd>The listing lazily prunes entries whose path is gone or whose checkout re-pointed elsewhere. Membership is verified by the store file’s inode, so a same-named impostor never counts.</dd> </dl> </article> </div> </section> <section> <h2>Where the technique comes from</h2> <p class="credits">The image’s column layout is borrowed deliberately, and the borrowing is recorded in the source rather than absorbed silently:</p> <ul class="credits"> <li><b>Roc — Richard Feldman.</b> Reload an on-disk cache image without re-parsing it.</li> <li><b>The Zig compiler — Andrew Kelley.</b> Dense integer-indexed arrays as the load format: relocation-free, offset-addressed sections read in place.</li> <li><b>Casey Muratori and game development practice.</b> Struct-of-arrays column layout, for locality.</li> <li><b>compact_arena.</b> A distinct <code>enum(u32)</code> index family per column, so a slot minted for one column cannot address another — enforced at compile time.</li> </ul> <p class="note">The CRC-32 implementation underneath all the checksums is slicing-by-N, from Kounavis & Berry at Intel, generalized by Kadatch & Jenkins. It is vendored rather than taken from the standard library because the standard one walks a byte at a time, which measured as ~93% of a repository open.</p> </section> <section> <h2>What the image still does not carry</h2> <div class="callout cold"> <h3>The tag tables</h3> <p class="note"><code>tagxref</code> is <code>(rid, tag) → value</code> with mtimes, and it is not in the image. So <code>branch</code> and <code>tag</code> still take the record-body rung — measurably: 0.33–0.36s against <code>ls</code> and <code>info</code> at 0.03–0.04s on the same store.</p> <p class="note"><code>info</code> was on that list too, and for a real reason: <code>amend --comment</code> stores its override as a tag, so on an early fast path it printed the manifest’s original comment instead. An existing test caught it the moment the verb moved. The override is now resolved on the image path and <code>info</code> reads it — which is the argument for having had the test, not an argument that the hazard was imaginary.</p> </div> <p>The sharper lesson from this work is about a failure mode rather than a missing column. Adjacency landed in the image, and the branch walk still answered wrongly — because a branch <i>value</i> was read from the empty tag table beside the correct edges, so every check-in looked like it sat on the main branch. The walk never left the branch and resolved <code>start:feature</code> to a check-in on a different branch, with exit 0 and no diagnostic.</p> <div class="callout"> <h3>Which is why the guarantee is phrased the way it is</h3> <p class="note">“A cache may make an answer faster, never different” is not a slogan — it is the property that has to be <i>tested</i>, tier by tier, because a cache that is merely absent degrades gracefully while a cache that is <b>partial</b> answers confidently and wrongly. Every tier here is compared against the path it replaces, and the comparisons carry a guard that fails when the fixture had nothing to compare.</p> </div> </section> <section> <h2>The layers at a glance</h2> <div class="tablewrap"> <table> <thead><tr><th>Layer</th><th>Lives</th><th>Memoizes</th><th>Loss costs</th></tr></thead> <tbody> <tr><td>Zero-parse image</td><td class="m">.zit-cache/repo</td><td>Derived query answers</td><td>One record-body replay</td></tr> <tr><td>Record body</td><td class="m">.zit-cache/repo</td><td>Derivation inputs</td><td>One full log derive</td></tr> <tr><td>Rebuild checkpoint</td><td class="m">.zit-cache/checkpoint</td><td>A verified log prefix</td><td>Re-parsing the flat prefix</td></tr> <tr><td>Log-scan index</td><td class="m">.zit-cache/index</td><td>Frame locations</td><td>A header walk on open</td></tr> <tr><td>Plaintext memo</td><td class="m">in process</td><td>Reconstructed delta bases</td><td>Re-reconstruction</td></tr> <tr><td>Sub-manifest memo</td><td class="m">in process</td><td>Shared tree nodes</td><td>Re-walking the partition</td></tr> <tr><td>Materialization plan</td><td class="m">.zit-cache/vstat</td><td>Which files are current</td><td>A full content compare</td></tr> <tr><td>Store cone</td><td class="m">.zit-cache/store-cone</td><td>Sparse policy <i>(not a memo)</i></td><td>Bandwidth — heals to dense</td></tr> <tr><td>Worktree registry</td><td class="m">.zit-cache/worktrees</td><td>Trees served by a store</td><td>Self-heals on next open</td></tr> </tbody> </table> </div> </section> <footer> Timings measured on one machine against the tcl corpus through <code>-R</code>, in the shipped <code>--release=fast</code> mode, taking the minimum of repeated runs. The ratios are what the work moved; the absolute seconds are not portable to another machine. </footer> </div> </body> </html>