Timeline

Three fixes, each found by measuring the previous one's residue.

markUnchanged: a path byte-identical between baseline and target -- same digest, same mode, with requireClean having already proven the on-disk file matches -- is skipped by every mutation phase: staging both directions, the undo journal, deletion and rewrite. An update used to delete and rewrite the entire tree regardless of how little changed.

.zit-vstat: the stat sidecar written at materialization is now the BASELINE PLAN -- version-bound, CRC'd, carrying mode and plan order -- so requireClean rebuilds the baseline with zero tree flattens when it describes the current check-in, and proves most files clean by inode/size/mtime instead of reconstructing their content out of the log. Advisory throughout: any mismatch, staleness or corruption falls back to the flatten and the content compare, which stay authoritative.

ManagedSet.fromSortedPaths: building the 66k-entry managed set through insert was quadratic -- an O(n) sorted-position memmove plus the LIVE sortedUnique assert per call -- and measured at 23.9 SECONDS of a pkgsrc update, larger than every real phase combined. The plan's own order is already sorted and unique, so the set is now built in one pass, order enforced per adjacent pair, invariant asserted once.

Measured, same store and targets as #52's gate: sqlite (2.2k files): 117.8s flat -> 0.6s adjacent / 1.2s distant pkgsrc (66k files): 42.2s flat -> 6.8s adjacent / 8.9s distant pkgsrc user CPU: ~25s -> 1.3s Residue is stat I/O (two ~66k-file sweeps), not computation.

The lockstep flatten prune #52 proposed was never built: the flatten measured 178ms at 66k files once the real costs fell.

Tests: dirty AND delete of a skipped-diff file both refuse with the tree untouched; sidecar intact/absent/rebound-to-wrong-check-in all produce identical trees; an unchanged file provably keeps its inode; an updated tree equals a from-scratch materialization. Closes #52. check-in: 8797f53546 user: dmestas tags: sub-manifest

The derived-state cache was all-or-nothing: any log append was a total miss, and the next read re-derived the whole store. On a real 1M-artifact pkgsrc store that was 352s per cold read, re-paid after every commit.

Repository.open's miss path now distinguishes growth from rewrite. On growth it replays the cached record body (parse-free), folds only the tail frames through the same crosslink fold a cold derive uses, and rewrites the cache: every append healed by the next read at O(tail). Measured on pkgsrc: 352.31s cold, 2.40s after a commit, output byte-identical to a forced full re-derive.

Sound because extension refuses whenever the fold could be wrong rather than stale: validatePrefix proves the old tail frame still verifies at its offset (a rewritten log cannot extend), and a shun fingerprint in the cache header forces the full derive on any shun-table change, since the body was distilled under the old mask. Cache format v2; a v1 cache misses once and rebuilds.

import-git now warms the cache at completion, after the atomic rename, against the published store -- clone has done this since warmCacheFromLog existed, so an import-built store no longer defers the whole derive onto its first reader.

Tests: extension equals cold derive with a fork and control tag in the tail; a forged-cache canary proving the fold starts from the stored body; shun change takes the full derive with the mask applied.

Also backs out 1a4b74f2d5 (parallel staging): measured ~3% on a corpus where staging is ~5s of 149s -- threading complexity the number never justified. Closes #51. check-in: 84d3fe8f32 user: dmestas tags: sub-manifest

Staging reconstructs and writes every file in a plan -- one decompress, delta-chain application and hash verification each -- and on a wide repository that is tens of thousands of files for a single open or update. The work is per-file independent, and the store's read path is offset-addressed pread over an index immutable for the duration, so it parallelises without coordination.

The one thing that cannot be written concurrently is last_stage_fault, a module global. Workers keep their own and the winner is chosen by ENTRY INDEX, which is exactly the fault sequential order would have raised, so this is observationally identical to the loop it replaces including which failure gets reported. Below 64 entries it stays sequential, and a thread that fails to spawn is not an error -- its share drains off the same cursor.

Measured 2:33.5 to 2:28.8 on a 299 MB sqlite store, about 3%, with CPU never leaving 101%. So staging is NOT the dominant term there: the flatten's per-node reconstruction is, which this does not touch. Committed so it can be measured at pkgsrc scale, where the file count is 27x higher and the size distribution is different. If it does not move that number either, revert it -- it buys complexity for 3%. check-in: 1a4b74f2d5 user: dmestas tags: sub-manifest

A POLICY CHANGE, not a bug fix. Gates 5 to 7 prove the mirror's manifest is byte-reproducible from the check-in this import builds. Flat mode claims that; sub-manifest mode restructures the file set behind S cards, so its check-in is a different artifact by construction and claims nothing of the kind. Refusing to release the metadata because a mode failed to reproduce something it never promised is a category error.

What makes the drop safe is gate 3 -- the bytes hash to the name a sibling manifest.uuid records, which a project file merely called 'manifest' cannot satisfy. That gate is untouched and still catches the look-alike. Gate 8 still guards the tags member. The B-card and S-card shape checks move with gates 5 to 7: a delta manifest is a real check-in, and refusing it only ever mattered because gate 5 needs a complete F run to compare against.

On a sound source the win is small: bench measures 180 retained on sqlite, and this takes that to about 1. The win is on sources holding shunned or phantom content, where the origin's manifests name bytes the origin no longer has and no export can reproduce them -- permanent stranding today. libfossil goes from 293 of 3819 commits validated to 3817, and its store from 35.0 MB to 30.2 MB.

Costs, stated rather than discovered later: this retires the only signal that a source repository is incomplete, and it leaves mirrorTreeMatches's tree arm unreachable from classification, annotated in place with the condition for deleting it. check-in: 063a66e4c5 user: dmestas tags: sub-manifest

Fossil's git export derives the git commit time by truncating a floating-point Julian Day, so a whole-second origin timestamp lands a second early about half the time. Minting from commit.committer meant zit built those check-ins at a provably different instant than the origin, their manifests were never redundant, and their mirror metadata was kept forever: 4533 of 36383 commits on a real sqlite mirror.

Where gate 3 has proved a mirror manifest authentic and gate 4 parsed it, the D card is now that manifest's own. Per commit, only where the proof succeeded -- an unproven mirror is never trusted. On the sqlite corpus timestamp_mismatch goes 4533 to 0 and the sha1-named era drops 18373 of 18392.

Not under an explicit --attribution author: that flag asks for the author's instants, and adopting the origin's would leave it doing nothing on most commits while still appearing set. Those imports drop nothing, stated rather than silent.

Gate 7 also compared against the committer unconditionally while minting the attributed identity, so under --attribution author it could drop a manifest on a proof about a check-in that was never built. It now compares against the instant actually minted. check-in: 7495b2e141 user: dmestas tags: sub-manifest

A tracked file can be a complete check-in manifest -- a vendored fossil source tarball is the ordinary route, since fossil stopped sterilizing archive exports in 2.18. The import stores it verbatim rather than mutating it the way fossil's own git import does: the file the user committed should be the file we store, and open already refuses to materialize a check-in whose content is absent. But the operator should hear it at import, not at the first open, so such blobs are counted and reported. Two bytes reject ordinary content before anything is parsed.

The --manifest e2e now pins the marker's POSITION, not just its presence: it follows the final card and is not line 1. A refactor moving it to line 1 would still sterilize and would still pass every previous assertion, while silently diverging from fossil and breaking the mirror strip.

Errata for artifact-format 4.1.2 filed separately; the draft states the wrong position, the wrong rejection gate, and a literal no fossil code path emits. check-in: 12ce319d18 user: dmestas tags: sub-manifest

sterilize_manifest() appends its marker after the last card. On an ordinary manifest that is the end of the blob, but on a PGP-clearsigned one the cards are followed by the signature block and the marker lands immediately before it. The strip only handled the trailing case, so a signed check-in's mirror bytes never hashed to the origin's recorded name, the pair proof refused it, and its manifest/manifest.uuid/ manifest.tags were retained permanently.

On a real sqlite mirror that is 1076 check-ins, all 2009-2011. With the splice every one of the corpus's 36383 commits proves: 35307 by the trailing strip, 1076 only by this.

Deliberately narrow -- only the two positions fossil writes. A marker anywhere else is still left alone, so a blob that merely contains the line is not reshaped into one that proves. check-in: a161ee719a user: dmestas tags: sub-manifest

The algorithm was a hardcoded .sha1 that mis-named every import from a SHA3-256 Fossil, in both output modes. Sub-manifest mode reproduces nothing by construction, so it has nothing to inherit; flat mode is meant to inherit its source's algorithm, but that is unimplemented and sqlite shows it cannot be one choice anyway -- its history spans the promotion ratchet, so a single manifest there names both 40- and 64-hex blobs. SHA3-256 is the correct answer for a source whose algorithm cannot be derived, and the only defensible one for new addresses after SHATTERED.

Now one constant, with the store's default policy and the submanifest k-analysis derived from it rather than guessed. Four name-verification sites in the storage layer used the same constant and rejected every SHA3 name; they now use artifact.verify, which reads the algorithm off the name's own width and cannot go stale.

Two test repairs the change exposed: the ISSUE #24 e2e compared parent and child check-in names that both alias one readCheckoutState buffer, passing on the length difference alone, and that fixture no longer produces a mixed-width store -- noted inline, it needs its own fixture.

Hashing is ~2.9x slower than SHA-1 on Apple Silicon, which lengthens hash-bound work. check-in: 51afeef206 user: dmestas tags: sub-manifest

Two CLI-level fixtures for the class behind #42, differing only in their T cards, both dated 2099 so mtime cannot explain the difference:

- self-tagged: carries trunk's tag pair, wins the tip, and open must refuse with exit 1, name the class, and leave the tree untouched. - untagged: a leaf and visible in 'zit leaves', but never resolvable, so open succeeds on the real tip.

Also repairs assertions the earlier trunk-relabel and open-diagnostic work left stale in the two opt-in e2e steps, which 'zig build test' does not run: branch topology now maps the source's main line to trunk, and open's untracked-file diagnostic includes the resolved check-in line. Closes #44. check-in: 33d9286c19 user: dmestas tags: sub-manifest

git fast-export --all emits a 'reset worktrees/NAME/HEAD' pseudo-ref for every detached-HEAD linked worktree. canonicalRef correctly refuses it, but that refusal unwound the whole import, so one worktree in the source aborted a complete history after all the work was done. Skip it, counted as skipped_foreign_refs.

The skip is on the refs/ prefix alone; canonicalRef's traversal refusal still aborts. Fixes #45. check-in: 31346e784b user: dmestas tags: sub-manifest