| Comment: | repocache: extend across appends; import warms at completion
The derived-state cache was all-or-nothing: any log append was a total miss, and the next read re-derived the whole store. On a real 1M-artifact pkgsrc store that was 352s per cold read, re-paid after every commit. Repository.open's miss path now distinguishes growth from rewrite. On growth it replays the cached record body (parse-free), folds only the tail frames through the same crosslink fold a cold derive uses, and rewrites the cache: every append healed by the next read at O(tail). Measured on pkgsrc: 352.31s cold, 2.40s after a commit, output byte-identical to a forced full re-derive. Sound because extension refuses whenever the fold could be wrong rather than stale: validatePrefix proves the old tail frame still verifies at its offset (a rewritten log cannot extend), and a shun fingerprint in the cache header forces the full derive on any shun-table change, since the body was distilled under the old mask. Cache format v2; a v1 cache misses once and rebuilds. import-git now warms the cache at completion, after the atomic rename, against the published store -- clone has done this since warmCacheFromLog existed, so an import-built store no longer defers the whole derive onto its first reader. Tests: extension equals cold derive with a fork and control tag in the tail; a forged-cache canary proving the fold starts from the stored body; shun change takes the full derive with the mask applied. Also backs out 1a4b74f2d5 (parallel staging): measured ~3% on a corpus where staging is ~5s of 149s -- threading complexity the number never justified. Closes #51. |
|---|---|
| Timelines: | family | ancestors | descendants | both | sub-manifest |
| Files: | files | file ages | folders |
| SHA3-256: |
84d3fe8f3201b5b90c6d03d6fc05ac38 |
| User & Date: | dmestas on 2026-08-10 01:23:44.086 |
| Other Links: | branch diff | manifest | tags |
|
2026-08-10
| ||
| 13:41 |
update: cost tracks the diff, not the width
Three fixes, each found by measuring the previous one's residue. markUnchanged: a path byte-identical between baseline and target -- same digest, same mode, with requireClean having already proven the on-disk file matches -- is skipped by every mutation phase: staging both directions, the undo journal, deletion and rewrite. An update used to delete and rewrite the entire tree regardless of how little changed. .zit-vstat: the stat sidecar written at materialization is now the BASELINE PLAN -- version-bound, CRC'd, carrying mode and plan order -- so requireClean rebuilds the baseline with zero tree flattens when it describes the current check-in, and proves most files clean by inode/size/mtime instead of reconstructing their content out of the log. Advisory throughout: any mismatch, staleness or corruption falls back to the flatten and the content compare, which stay authoritative. ManagedSet.fromSortedPaths: building the 66k-entry managed set through insert was quadratic -- an O(n) sorted-position memmove plus the LIVE sortedUnique assert per call -- and measured at 23.9 SECONDS of a pkgsrc update, larger than every real phase combined. The plan's own order is already sorted and unique, so the set is now built in one pass, order enforced per adjacent pair, invariant asserted once. Measured, same store and targets as #52's gate: sqlite (2.2k files): 117.8s flat -> 0.6s adjacent / 1.2s distant pkgsrc (66k files): 42.2s flat -> 6.8s adjacent / 8.9s distant pkgsrc user CPU: ~25s -> 1.3s Residue is stat I/O (two ~66k-file sweeps), not computation. The lockstep flatten prune #52 proposed was never built: the flatten measured 178ms at 66k files once the real costs fell. Tests: dirty AND delete of a skipped-diff file both refuse with the tree untouched; sidecar intact/absent/rebound-to-wrong-check-in all produce identical trees; an unchanged file provably keeps its inode; an updated tree equals a from-scratch materialization. Closes #52. check-in: 8797f53546 user: dmestas tags: sub-manifest | |
| 01:23 |
repocache: extend across appends; import warms at completion
The derived-state cache was all-or-nothing: any log append was a total miss, and the next read re-derived the whole store. On a real 1M-artifact pkgsrc store that was 352s per cold read, re-paid after every commit. Repository.open's miss path now distinguishes growth from rewrite. On growth it replays the cached record body (parse-free), folds only the tail frames through the same crosslink fold a cold derive uses, and rewrites the cache: every append healed by the next read at O(tail). Measured on pkgsrc: 352.31s cold, 2.40s after a commit, output byte-identical to a forced full re-derive. Sound because extension refuses whenever the fold could be wrong rather than stale: validatePrefix proves the old tail frame still verifies at its offset (a rewritten log cannot extend), and a shun fingerprint in the cache header forces the full derive on any shun-table change, since the body was distilled under the old mask. Cache format v2; a v1 cache misses once and rebuilds. import-git now warms the cache at completion, after the atomic rename, against the published store -- clone has done this since warmCacheFromLog existed, so an import-built store no longer defers the whole derive onto its first reader. Tests: extension equals cold derive with a fork and control tag in the tail; a forged-cache canary proving the fold starts from the stored body; shun change takes the full derive with the mask applied. Also backs out 1a4b74f2d5 (parallel staging): measured ~3% on a corpus where staging is ~5s of 149s -- threading complexity the number never justified. Closes #51. check-in: 84d3fe8f32 user: dmestas tags: sub-manifest | |
|
2026-08-09
| ||
| 16:32 |
update: stage plan entries in parallel
Staging reconstructs and writes every file in a plan -- one decompress, delta-chain application and hash verification each -- and on a wide repository that is tens of thousands of files for a single open or update. The work is per-file independent, and the store's read path is offset-addressed pread over an index immutable for the duration, so it parallelises without coordination. The one thing that cannot be written concurrently is last_stage_fault, a module global. Workers keep their own and the winner is chosen by ENTRY INDEX, which is exactly the fault sequential order would have raised, so this is observationally identical to the loop it replaces including which failure gets reported. Below 64 entries it stays sequential, and a thread that fails to spawn is not an error -- its share drains off the same cursor. Measured 2:33.5 to 2:28.8 on a 299 MB sqlite store, about 3%, with CPU never leaving 101%. So staging is NOT the dominant term there: the flatten's per-node reconstruction is, which this does not touch. Committed so it can be measured at pkgsrc scale, where the file count is 27x higher and the size distribution is different. If it does not move that number either, revert it -- it buys complexity for 3%. check-in: 1a4b74f2d5 user: dmestas tags: sub-manifest | |