Timeline

An unlimited timeline re-read and re-parsed every manifest just to print its comment first-line — 163k inflate+parse cycles on pkgsrc. The comment now rides the crosslink record into the derived state (commentOf; image backing falls back to the manifest read), so the render loop is a lookup per row: full timeline 12.3s -> 1.06s. Record-format consumers bumped so old bodies miss instead of misparsing (repocache v3, checkpoint v2 — one re-derive per repo, then warm). zit log's pinned B-log-1 walk untouched. check-in: 57d8de4120 user: dmestas tags: sub-manifest

The lazy content channel re-walked every frame header on each open — O(store) before the first byte of any comment, cat, or annotate. The scan's product (entries, hash index, shun rows, config cursor, tail key) now persists as .zit-cache/index, self-validating and LogKey-keyed: an exact tail adopts whole, a grown log resumes the walk over the tail alone, and anything else falls back to the authoritative scan (--no-cache never consults it). pkgsrc: info 1.7s -> 0.44s, timeline -n 5 4.3s -> 0.96s. check-in: f2bfd36e74 user: dmestas tags: sub-manifest

Replace the pick-the-minimum-stamp linear scan with an intrusive doubly linked recency list; get/put/evict are all constant-time. The scan was invisible at small occupancy and dominant at full: a pkgsrc clone kept the cache pinned at max_entries and spent over half its client CPU sweeping the map once per stored artifact. Measured: clone 12:38 -> 8:39 wall, client CPU 694s -> 463s; cloned store verifies with 0 faults. check-in: 0df83f350a user: dmestas tags: sub-manifest

Seven files relocate with unchanged basenames: git_fast_export(+test), git_import, git_import_impl, git_import_cli, git_import_verify, git_import_benchmark_support. The six standalone module roots (test, refs_test, cli_test, e2e, benchmark, pipe_helper) stay flat because a Zig module cannot import above its root file's directory — the same rule that keeps main/e2e/dst/smoke at src/. Future export-git files follow the same split: roots flat, implementation in src/git/. Path-string edits only; build.zig unchanged. check-in: ce3c87e25c user: dmestas tags: sub-manifest

.zit-cache/checkpoint pairs a validated log-prefix key with the prefix's crosslink record body; Repository.open resumes past the flat segment on a cache miss instead of re-parsing it. Witness shape under a closed emission gate creates zero artifacts; tree shape pins the anchor's file set as the canonical S-tree (batch-oracle byte-identical, one interior root node). verify re-derives every claim from real history; --no-cache bypasses both sidecars. New verb: zit checkpoint [show|build|verify|drop].

pkgsrc (163k commits, 60k files): cold open 8.4s vs 352s full derive; verify passes; 366/367 tree nodes deduplicated against the imported tree. check-in: e4026acd7b user: dmestas tags: sub-manifest

The policy is three names and a prefix: .zit, .zit-checkout, .zit-cache, .zit-tmp-*. Removed: the absorbed sibling reservations (.zit-managed/-undo/ -bisect), the pre-directory cache/vstat spellings and their read tolerance, the retired scratch spellings, and the import-provenance names (refs, .zit-import-map.jsonl).

Behavior shift: in never-re-opened old checkouts these leftovers are now visible to extras and deletable by clean. The container's byte-1 fold stays, so a legacy authoritative sibling still migrates losslessly on first contact (e2e: every-leftover checkout converges to the clean root with membership intact). check-in: c0a116d458 user: dmestas tags: sub-manifest

Sectioned container (ZITCKOT, tagged table, whole-file CRC); payloads are the existing formats unchanged. Byte-1 dispatch migrates legacy checkouts on first write; siblings dead once a container exists. Undo/redo restore and the commit/merge record+membership writes each collapse to one atomic splice. DST gains a legacy-seeded migration sweep (full-legacy or full-container at every crash cut). Registry: the three names move to the legacy tier. Perf on pkgsrc: adjacent update 6.50s (was 6.8), no-op 5.14s (was 7.69); migration of a 66k-file checkout invisible inside one update. check-in: efa29d234f user: dmestas tags: sub-manifest

Tasks 1-5 of the control-file consolidation plan. The checkout root goes from twelve zit-owned names to six live ones; the authoritative-tier container (task 6) and legacy removal (task 7) are deliberately not in this change.

Registry: control_paths now carries every reserved root name with its tier -- authoritative, advisory, scratch, legacy -- and is the single authority the case table, the extras exclusions and the ignore-glob coverage pin all derive from or test against. A name added without its glob line or case coverage is a test failure, proven by mutation. The pin caught two live gaps on arrival: a crashed update's transaction directory had no ignore-glob line, and a crashed import's staging directory was never reserved at all.

import-git --provenance: the per-object map and the source ref manifest are now instrumentation, published only on request. The default destination is the repository and its checkout record -- fossil's own shape. Both files are still written and verified in staging every run; without the flag they leave with the raw generation, and the staging manifest check enforces whichever set was chosen.

Scratch: one prefix. .zit-tmp-txn- (update transactions), .zit-tmp-import- (import staging), .zit-tmp-pack (pack scratch; name only, the truncate-reuse and rewrite-exclusion lock protocol are untouched). One registry prefix row and one glob line cover the tier including names that do not exist yet.

Advisory: .zit-cache is a directory -- repo (derived-state cache) and vstat (materialization plan), with the tree-checkpoint's member reserved. A legacy cache FILE is served through a read fallback and migrated by the first write via rename through a scratch name: the derivation survives with no bytes copied and no re-derive. Verified on a 66k-file pkgsrc checkout: identical SHA-256 across migration of a 35.8MB cache, fallback reads at parity, no-op update 7.69s vs 7.74s.

Legacy spellings of every renamed or relocated name stay reserved and ignore-globbed until the removal release. check-in: f1ac625093 user: dmestas tags: sub-manifest

Three fixes, each found by measuring the previous one's residue.

markUnchanged: a path byte-identical between baseline and target -- same digest, same mode, with requireClean having already proven the on-disk file matches -- is skipped by every mutation phase: staging both directions, the undo journal, deletion and rewrite. An update used to delete and rewrite the entire tree regardless of how little changed.

.zit-vstat: the stat sidecar written at materialization is now the BASELINE PLAN -- version-bound, CRC'd, carrying mode and plan order -- so requireClean rebuilds the baseline with zero tree flattens when it describes the current check-in, and proves most files clean by inode/size/mtime instead of reconstructing their content out of the log. Advisory throughout: any mismatch, staleness or corruption falls back to the flatten and the content compare, which stay authoritative.

ManagedSet.fromSortedPaths: building the 66k-entry managed set through insert was quadratic -- an O(n) sorted-position memmove plus the LIVE sortedUnique assert per call -- and measured at 23.9 SECONDS of a pkgsrc update, larger than every real phase combined. The plan's own order is already sorted and unique, so the set is now built in one pass, order enforced per adjacent pair, invariant asserted once.

Measured, same store and targets as #52's gate: sqlite (2.2k files): 117.8s flat -> 0.6s adjacent / 1.2s distant pkgsrc (66k files): 42.2s flat -> 6.8s adjacent / 8.9s distant pkgsrc user CPU: ~25s -> 1.3s Residue is stat I/O (two ~66k-file sweeps), not computation.

The lockstep flatten prune #52 proposed was never built: the flatten measured 178ms at 66k files once the real costs fell.

Tests: dirty AND delete of a skipped-diff file both refuse with the tree untouched; sidecar intact/absent/rebound-to-wrong-check-in all produce identical trees; an unchanged file provably keeps its inode; an updated tree equals a from-scratch materialization. Closes #52. check-in: 8797f53546 user: dmestas tags: sub-manifest

The derived-state cache was all-or-nothing: any log append was a total miss, and the next read re-derived the whole store. On a real 1M-artifact pkgsrc store that was 352s per cold read, re-paid after every commit.

Repository.open's miss path now distinguishes growth from rewrite. On growth it replays the cached record body (parse-free), folds only the tail frames through the same crosslink fold a cold derive uses, and rewrites the cache: every append healed by the next read at O(tail). Measured on pkgsrc: 352.31s cold, 2.40s after a commit, output byte-identical to a forced full re-derive.

Sound because extension refuses whenever the fold could be wrong rather than stale: validatePrefix proves the old tail frame still verifies at its offset (a rewritten log cannot extend), and a shun fingerprint in the cache header forces the full derive on any shun-table change, since the body was distilled under the old mask. Cache format v2; a v1 cache misses once and rebuilds.

import-git now warms the cache at completion, after the atomic rename, against the published store -- clone has done this since warmCacheFromLog existed, so an import-built store no longer defers the whole derive onto its first reader.

Tests: extension equals cold derive with a fork and control tag in the tail; a forged-cache canary proving the fold starts from the stored body; shun change takes the full derive with the mask applied.

Also backs out 1a4b74f2d5 (parallel staging): measured ~3% on a corpus where staging is ~5s of 149s -- threading complexity the number never justified. Closes #51. check-in: 84d3fe8f32 user: dmestas tags: sub-manifest

Staging reconstructs and writes every file in a plan -- one decompress, delta-chain application and hash verification each -- and on a wide repository that is tens of thousands of files for a single open or update. The work is per-file independent, and the store's read path is offset-addressed pread over an index immutable for the duration, so it parallelises without coordination.

The one thing that cannot be written concurrently is last_stage_fault, a module global. Workers keep their own and the winner is chosen by ENTRY INDEX, which is exactly the fault sequential order would have raised, so this is observationally identical to the loop it replaces including which failure gets reported. Below 64 entries it stays sequential, and a thread that fails to spawn is not an error -- its share drains off the same cursor.

Measured 2:33.5 to 2:28.8 on a 299 MB sqlite store, about 3%, with CPU never leaving 101%. So staging is NOT the dominant term there: the flatten's per-node reconstruction is, which this does not touch. Committed so it can be measured at pkgsrc scale, where the file count is 27x higher and the size distribution is different. If it does not move that number either, revert it -- it buys complexity for 3%. check-in: 1a4b74f2d5 user: dmestas tags: sub-manifest

A POLICY CHANGE, not a bug fix. Gates 5 to 7 prove the mirror's manifest is byte-reproducible from the check-in this import builds. Flat mode claims that; sub-manifest mode restructures the file set behind S cards, so its check-in is a different artifact by construction and claims nothing of the kind. Refusing to release the metadata because a mode failed to reproduce something it never promised is a category error.

What makes the drop safe is gate 3 -- the bytes hash to the name a sibling manifest.uuid records, which a project file merely called 'manifest' cannot satisfy. That gate is untouched and still catches the look-alike. Gate 8 still guards the tags member. The B-card and S-card shape checks move with gates 5 to 7: a delta manifest is a real check-in, and refusing it only ever mattered because gate 5 needs a complete F run to compare against.

On a sound source the win is small: bench measures 180 retained on sqlite, and this takes that to about 1. The win is on sources holding shunned or phantom content, where the origin's manifests name bytes the origin no longer has and no export can reproduce them -- permanent stranding today. libfossil goes from 293 of 3819 commits validated to 3817, and its store from 35.0 MB to 30.2 MB.

Costs, stated rather than discovered later: this retires the only signal that a source repository is incomplete, and it leaves mirrorTreeMatches's tree arm unreachable from classification, annotated in place with the condition for deleting it. check-in: 063a66e4c5 user: dmestas tags: sub-manifest

Fossil's git export derives the git commit time by truncating a floating-point Julian Day, so a whole-second origin timestamp lands a second early about half the time. Minting from commit.committer meant zit built those check-ins at a provably different instant than the origin, their manifests were never redundant, and their mirror metadata was kept forever: 4533 of 36383 commits on a real sqlite mirror.

Where gate 3 has proved a mirror manifest authentic and gate 4 parsed it, the D card is now that manifest's own. Per commit, only where the proof succeeded -- an unproven mirror is never trusted. On the sqlite corpus timestamp_mismatch goes 4533 to 0 and the sha1-named era drops 18373 of 18392.

Not under an explicit --attribution author: that flag asks for the author's instants, and adopting the origin's would leave it doing nothing on most commits while still appearing set. Those imports drop nothing, stated rather than silent.

Gate 7 also compared against the committer unconditionally while minting the attributed identity, so under --attribution author it could drop a manifest on a proof about a check-in that was never built. It now compares against the instant actually minted. check-in: 7495b2e141 user: dmestas tags: sub-manifest

A tracked file can be a complete check-in manifest -- a vendored fossil source tarball is the ordinary route, since fossil stopped sterilizing archive exports in 2.18. The import stores it verbatim rather than mutating it the way fossil's own git import does: the file the user committed should be the file we store, and open already refuses to materialize a check-in whose content is absent. But the operator should hear it at import, not at the first open, so such blobs are counted and reported. Two bytes reject ordinary content before anything is parsed.

The --manifest e2e now pins the marker's POSITION, not just its presence: it follows the final card and is not line 1. A refactor moving it to line 1 would still sterilize and would still pass every previous assertion, while silently diverging from fossil and breaking the mirror strip.

Errata for artifact-format 4.1.2 filed separately; the draft states the wrong position, the wrong rejection gate, and a literal no fossil code path emits. check-in: 12ce319d18 user: dmestas tags: sub-manifest

sterilize_manifest() appends its marker after the last card. On an ordinary manifest that is the end of the blob, but on a PGP-clearsigned one the cards are followed by the signature block and the marker lands immediately before it. The strip only handled the trailing case, so a signed check-in's mirror bytes never hashed to the origin's recorded name, the pair proof refused it, and its manifest/manifest.uuid/ manifest.tags were retained permanently.

On a real sqlite mirror that is 1076 check-ins, all 2009-2011. With the splice every one of the corpus's 36383 commits proves: 35307 by the trailing strip, 1076 only by this.

Deliberately narrow -- only the two positions fossil writes. A marker anywhere else is still left alone, so a blob that merely contains the line is not reshaped into one that proves. check-in: a161ee719a user: dmestas tags: sub-manifest

The algorithm was a hardcoded .sha1 that mis-named every import from a SHA3-256 Fossil, in both output modes. Sub-manifest mode reproduces nothing by construction, so it has nothing to inherit; flat mode is meant to inherit its source's algorithm, but that is unimplemented and sqlite shows it cannot be one choice anyway -- its history spans the promotion ratchet, so a single manifest there names both 40- and 64-hex blobs. SHA3-256 is the correct answer for a source whose algorithm cannot be derived, and the only defensible one for new addresses after SHATTERED.

Now one constant, with the store's default policy and the submanifest k-analysis derived from it rather than guessed. Four name-verification sites in the storage layer used the same constant and rejected every SHA3 name; they now use artifact.verify, which reads the algorithm off the name's own width and cannot go stale.

Two test repairs the change exposed: the ISSUE #24 e2e compared parent and child check-in names that both alias one readCheckoutState buffer, passing on the length difference alone, and that fixture no longer produces a mixed-width store -- noted inline, it needs its own fixture.

Hashing is ~2.9x slower than SHA-1 on Apple Silicon, which lengthens hash-bound work. check-in: 51afeef206 user: dmestas tags: sub-manifest

Two CLI-level fixtures for the class behind #42, differing only in their T cards, both dated 2099 so mtime cannot explain the difference:

- self-tagged: carries trunk's tag pair, wins the tip, and open must refuse with exit 1, name the class, and leave the tree untouched. - untagged: a leaf and visible in 'zit leaves', but never resolvable, so open succeeds on the real tip.

Also repairs assertions the earlier trunk-relabel and open-diagnostic work left stale in the two opt-in e2e steps, which 'zig build test' does not run: branch topology now maps the source's main line to trunk, and open's untracked-file diagnostic includes the resolved check-in line. Closes #44. check-in: 33d9286c19 user: dmestas tags: sub-manifest

git fast-export --all emits a 'reset worktrees/NAME/HEAD' pseudo-ref for every detached-HEAD linked worktree. canonicalRef correctly refuses it, but that refusal unwound the whole import, so one worktree in the source aborted a complete history after all the work was done. Skip it, counted as skipped_foreign_refs.

The skip is on the refs/ prefix alone; canonicalRef's traversal refusal still aborts. Fixes #45. check-in: 31346e784b user: dmestas tags: sub-manifest

The header byte and the config field are both HashPolicy values. The import wrote @intFromEnum(HashAlgo.sha1) into each. The enums overlap numerically and mean different things -- HashAlgo is {sha1=0, sha3_256=1}, HashPolicy is {sha1=0, auto=1, sha3=2, ...} -- so every imported repository advertised policy sha1 by coincidence of ordinal, and the obvious 'switch it to sha3_256' repair would have written 1 = auto, silently arming 7.4's promotion ratchet.

Now derived through artifact.defaultPolicy rather than written as a literal. The import mints SHA-1 names throughout, so 7.2's rule gives auto. When the naming algorithm learns to follow its source the predicate changes and the derivation does not.

Header byte on a fresh import: 00 to 01. check-in: 4efd2e36d2 user: dmestas tags: sub-manifest

mirrorParentProven required the parent's validated_uuid, set only when a commit FULLY dropped. So proving commit N's parent meant commit N-1 had to clear every gate for itself -- tree shape, timestamp, manifest shape -- none of which bear on whether its UUID is correct. One retain anywhere permanently poisoned every descendant.

On a real 36,432-commit sqlite mirror that produced zero drops and 36,252 parent_unproven retentions: its 2000-05-29 genesis commit fails the timestamp gate, and essentially all of history descends from it.

Gate 3 already proves identity per commit and cryptographically. That proof is now recorded as attested_uuid whether or not the commit drops, and the parent proof accepts either. Drop eligibility is unchanged -- attestation governs only what a CHILD may rely on.

The existing test asserted the defect as intent, comment and all; it now asserts the corrected behaviour. (#43) check-in: e7fd62ec4d user: dmestas tags: sub-manifest

Regression tests for issue 42. A blob whose bytes are a valid check-in manifest IS a check-in -- structure types artifacts, and repo-state 4.4 exempts only what a parser declines to classify -- so a repository that tracks such a file holds check-ins of another repository. That derivation is required; what must not regress is the refusal.

Asserts materialization fails with FileContentMissing, names the file, and leaves nothing written -- the fail-closed property stated as a test so it cannot be quietly traded later. Also pins that the diagnostic explains the class and never alleges corruption, since the store is intact and verify says so.

Records the T-card grammar that makes such a manifest resolvable: 'T *branch * trunk' self-targets, while 'T *branch trunk' puts a branch name in the UUID field and the manifest is rejected entirely. (#42) check-in: 0369e62f19 user: dmestas tags: sub-manifest

The format types artifacts by structure, and repo-state 4.4 exempts only what a parser declines to classify -- so a stored blob whose bytes are a valid check-in manifest IS a check-in, joins the DAG, and may carry its own branch tags. A repository tracking such a file (a Fossil-to-Git mirror's 'manifest' is the common source) therefore holds check-ins of another repository, whose blobs it never had. That derivation is conformant; Fossil does the same.

What was not acceptable was spending minutes materializing before finding out, then reporting it as a missing file with no way to see why. open now presence-checks the resolved tip's file set before invoking the engine and refuses naming the class; update prints the same explanation.

Advisory only: a files() fault falls through, and stagePlan remains the sole authority, so this cannot let anything through -- it can only refuse earlier. Verified against a foreign self-tagged manifest that genuinely hijacks trunk: exit 1, working tree untouched. (#42) check-in: a81e9a8745 user: dmestas tags: sub-manifest

parseForImport normalizes a fossil-origin F card into a fresh buffer and the manifest borrows that copy, so every F-card hash the plan reads points into it rather than into the stored bytes. Nothing exercised it: every synthetic corpus here is git-native and never carries the separator, so normalized is null throughout the suite.

The path is correct -- the test passes -- but it was covered only by repositories nobody runs the tests against. (#42) check-in: 15db533ea7 user: dmestas tags: sub-manifest

open picks its own target -- the tip of the configured main branch -- rather than being handed one, so a failure that omits the check-in cannot distinguish materializing the expected one going wrong from resolving a different one than the operator believes is the tip. There was no other way to see which. (#42) check-in: da081a3eca user: dmestas tags: sub-manifest

StageFault held a slice into PlanEntry.path, documented as borrowing a plan that outlives the call. It does not: applyCheckin frees the plan from its own defer on the way out, so the caller rendered freed heap -- ten bytes of garbage where a filename belonged, in the very diagnostic added to make this failure legible.

Path is copied now, as the artifact name already was. Adds the end-to-end test that would have caught it: the earlier test hand-built a fault and so never exercised the lifetime that was wrong. check-in: 41dc7067d8 user: dmestas tags: sub-manifest

Three separate sites printed the identical 'cannot read repository: repository is corrupt' -- deriving the view, reopening the log, and applyCheckin. So a materialization failure was indistinguishable from a failure to read the repository, which is a different claim about a different subsystem.

stagePlan's three refusals now return FileContentMissing carrying the file path and the artifact name: a file the check-in references has no artifact here, or has none stored, or the stored bytes do not match the name. verify cannot surface any of these -- it checks that stored artifacts are valid, never that referenced ones exist.

Being in the closed ApplyError set, the compiler required all four verb mappers to handle it. (#42) check-in: 0b7b4af7fe user: dmestas tags: sub-manifest

'repository is corrupt' says a repository cannot be read, not what in it is unreadable -- and the walk that discovers it visits every artifact in the store, minutes on a large one. Whoever holds the broken repository was left bisecting it or attaching a debugger.

The derive now records which artifact it rejected and why; openRepo reports it, so every verb's open gets the name. Plain data, no allocation, so it survives the errdefer that tears down a half-built view. Repository.open clears it per attempt because callers declare a Repository undefined.

Also splits open's two identical diagnostics: deriving the view and reopening the log for materialization both printed the same line, which made a failure in one indistinguishable from the other. (#42) check-in: 37f746e59f user: dmestas tags: sub-manifest

A real 53,773-commit import evicted 99,097 times, every one forced by the 4096-entry cap and not one by the 64 MiB byte cap, with the resident set at 23,551 KiB. The cache was held to a third of its own stated size by a second, unrelated number, and the 51 percent hit rate that produced looked like a working-set problem rather than an arithmetic accident.

The entry cap now divides the byte budget by a planning entry size, so the two cannot disagree about which one is bounding: bytes bind for normal data, entries only for many tiny items, which is what an entry cap is for. (#40) check-in: ac89bad5fd user: dmestas tags: sub-manifest

A git default branch is master or main; this format's is trunk, and so is every place that must name the main line without being told -- a fresh checkout, zit open resolving a tip, a clone initialising its own state. A repository whose main line was called master satisfied none of them: it cloned and verified clean, then could not be opened, because the branch the destination looked for held no commits.

labelBranches now takes the name to MATCH on and the name to APPLY separately: the head is still found by its source name, the segment is labelled trunk. Side branches keep their source names, and the git-side refs mapping still records sym-heads/master, so nothing about round-trip identity moves.

If a side branch already publishes trunk, the source name is kept rather than colliding two segments under one label. (#42) check-in: 5caf866d86 user: dmestas tags: sub-manifest

Profiling a real pkgsrc-scale import put readv at over half of all CPU samples. A block cache in front of pread is the shape that addresses it: the store is append-only, so a block is retained only when the read that filled it came back FULL -- which proves those bytes already existed, so appends cannot alter them and need no invalidation. Only truncate does.

Deliberately not attached to any caller. Wired into import-git it reached a 99 percent hit rate over 392,421 reads and did NOT make a small corpus faster: at that size the page cache already served those reads, so the syscalls were cheap and the lookup was pure overhead. The prior from the real profile is strong but unproven at scale, and shipping it on 'should help' is how the reverted top-level cache in issue 32 happened.

useReadCache is the one-line opt-in for profiling it. (#39) check-in: f3becbedd1 user: dmestas tags: sub-manifest

A real 53,773-commit run reported 99,097 evictions against 69,485 lookups -- evicting more than it was asked about. But two caps can force that, max_bytes and max_entries, and they bind under different data, so the count alone said only 'raise something'.

Evictions are now split by cause and the resident set is reported, so a run names the budget to raise instead of requiring a blind sweep. (#40) check-in: 88ae8ba619 user: dmestas tags: sub-manifest

PackStats carries the cache's hits, misses, evictions and oversized skips, and import-git prints them. Same reason SubManifestMemo got this in #35: a fixed budget that is wrong for a corpus is invisible from outside the process, and a hit count without its denominator cannot show it. Nothing here gates a decision. (#40) check-in: b6cd09fc11 user: dmestas tags: sub-manifest

The trailer is the four bytes immediately after the payload and was read by its own pread. FileStorage is unbuffered, so that was a second kernel transition for adjacent bytes, paid per frame -- and a deltified artifact walks its whole chain frame by frame.

Also gives PlaintextCache the counters it was missing. It tracked hits with no denominator, so a hit COUNT could not be read as a hit RATE; misses, evicted and oversized make the budgets observable. (#39, #40) check-in: a4f298d3e1 user: dmestas tags: sub-manifest

max_entries was 1 shifted by 20 while its own doc named the u32 count field that carries it, so insert refused a correct set at 1,048,576 paths -- four thousand times below what the format can express. Now the u32 range, with max_file_bytes computed from it rather than a flat 64 MiB, and read sized from stat like every other whole-file read here.

Two dependents moved in the same change, which is the point:

extras borrowed max_entries as a directory-sweep ceiling. Never the same quantity, and a tracked-set format change would have silently widened a filesystem runaway guard. It has its own now.

undo parseSlot allocated count entries BEFORE parsing them, guarded by count over max_files. That guard became vacuous the moment max_files reached the u32 range, leaving a 4-billion-entry claim reaching the allocator. Now bounded by what the remaining payload can back. (#38) check-in: 93adddd49d user: dmestas tags: sub-manifest

The ceiling was a PRODUCT of round cap and batch size, so shrinking the batch to 1 reaches it at a corpus a unit test can hold: 1124 artifacts, 1124+ rounds, stall budget 4.

Records that a converging clone ends exactly one stall deep -- the terminal round carries no cfile -- so the budget can never be 1, and that the count does not grow with the corpus. (#37) check-in: 5aa8e46e47 user: dmestas tags: sub-manifest

The cap was justified as bounding 'a lying on-disk size before any allocation'. The read path takes stat's size and allocates exactly that, so there was no claim to distrust. What it did do was silently refuse a larger cache -- and a cache is advisory, so that refusal is invisible, degrading to a full rebuild with no diagnostic.

Real bounds stay where they bind: the usize cast on read, the u32 image_len field on write. (#38) check-in: c82c04b10d user: dmestas tags: sub-manifest

buildPlan flattened both trees through the non-memoized path, so the baseline walk re-inflated and re-parsed every node the check-in did not touch -- which on a wide repository is nearly all of them.

Removes the redundant parse, not the redundant walk: both still enumerate the whole tree. Pruning that needs pack.zig's lockstep descent (#36). check-in: 9fcc75f669 user: dmestas tags: sub-manifest

max_tracked_file_bytes is not a policy number: it IS logstore.max_payload, and put/putArtifact assert on it, so a file above it would trip an assert during commit rather than be refused. Tied by a comptime assert in main.zig and documented with its three co-constraints (storage frame, sync decode cap, and the delta format's sub-2^32 integers).

Oversize now reports the size and the limit instead of 'input/output error', which named nothing the operator did. Diff keeps the generic path; reaching it means an already-committed file grew past the ceiling. (#34) check-in: 56a44d4010 user: dmestas tags: sub-manifest

The last 42 fell in two passes and a hand-repair.

Round 6 fed each refuted proposal its reviewer's own objection back -- "off by one at the start", "range too short for a biconditional, it must decide both directions", "evidence stops on the dangling No". Those objections were near-solutions, and 38 of 40 second attempts passed review where the first had failed.

Eight were then read and repaired by hand. Seven were ordinary rot. The eighth was not: the reviewer had refused a proposal for the nine-cause fatal-error set because the cited lines "neither state nor deny the claim", and it was right -- no passage states such a set, because it is zit's own enumeration and its members each carry their own citation. It now names the governing section with NO range. Inventing a span to satisfy the checker is the defect this sweep exists to remove.

One of the nine was the checker's fault rather than the code's. sync.zig:4599 cites artifact-format section 3.2 L347-350, byte-exact and correct, but the words "artifact-format" sat on the previous line and the draft-name detection only reads the current one. Reflowed the comment.

964 citations, 964 resolving. The baseline is deleted: this started as a two-way ratchet over a 512-citation backlog, and with the backlog gone an allowance is only somewhere for new rot to hide. A citation whose lines fall outside the section it names now fails the build.

test 3506/3506, fmt. Issue #18. check-in: 8d3c0927be user: dmestas tags: sub-manifest

SubManifestMemo filled to 4 MiB then stopped interning, so the retained set froze on whatever arrived first and later resolves paid a lookup that could not hit. Measured at 30,000 files: working set 7,494 KiB against a 4,096 KiB ceiling, costing 23 points of hit rate (92->69%) and ~27% of wall time.

Now per-entry allocation with an LRU list, trimmed in beginWalk. Eviction cannot happen in resolve: TreeWalk frames borrow the views that resolved them, so freeing mid-walk is a use-after-free -- the arena hid that by never freeing. Budget 64 MiB, from the measured working set scaled to pkgsrc.

Same corpus: 92%, 0 evicted, 26s->19s. Unchanged where budget isn't binding.

repository.zig and git_import_verify.zig halves landed in 76a2905055, swept in by a concurrent commit. (#35) check-in: 9b4f7ae683 user: dmestas tags: sub-manifest

Every earlier pass asked only for a line re-key WITHIN the cited section. A citation whose claim lived in a different section could therefore only come back "unresolved" -- even when the agent had already found the right answer and written it into its reason field. Four rounds of that read as "these need a human", when what it actually showed was that the prompt never permitted the fix.

This pass let each repair choose its own kind. Of 126 confirmed: 63 changed the SECTION, 41 re-keyed a range, and 22 rewrote a claim the drafts no longer make. 104 of those were structurally unproposable before. Unresolved fell from 67 to 5.

The claim rewrites are the ones worth reading. reply_body_cap was documented as an RFC-mandated 512 MiB ceiling; section 4 L830-833 states the opposite, that Fossil imposes no such entity limit, so it is now documented as a zit-local cap -- the same repair the 65535-byte line bound got earlier. A verbatim quotation attributed to the draft, "The card MUST consist of exactly two whitespace-separated tokens", does not appear in it; section 10.2 L2118 says "Exactly two visible tokens are required."

42 review verdicts were refused and are not applied. Two refute agents died mid-run; their proposals were left unapplied rather than taken on trust, and one refute agent lost to a stalled stream cost 14 repairs that the completed run later recovered.

Broken 172 to 42, 96 percent of citations resolving, 0 unresolvable sections. test 3500/3500, fmt. Issue #18. check-in: 76a2905055 user: dmestas tags: sub-manifest

The section sign was not the only anchor in the tree. `App. A.2 L3102-3114` and `ABNF App. B.7 L3566-3575` name an appendix by word, so their ranges bound to whatever section sat further left, or to nothing. Three citations were invisible or misattributed for that reason.

All three are repaired. The decoded version-gate text is App. A.2 L2547, "Decoded error: Fossil version 2.0 or later required." The transcribed gate-fire response body is L2542-2545 -- the error, clone_seqno, push and timestamp lines.

The third keeps no range at all, deliberately. It cites "the seven pragma SP NAME LF productions that carry no argument", and Appendix B is flat now, with those seven SCATTERED through it: uv-push-ok at L2907, send-private at L2925, req-clusters at L2935. No single range covers them, so it names the appendix and stops. A span invented to satisfy the checker would be the same defect this sweep exists to remove, wearing a different costume.

Surfacing three and repairing four takes broken from 173 to 172 while the checker gets strictly stronger. 947 citations, 775 correct, 0 unresolvable sections.

test 3500/3500, fmt. Issue #18. check-in: 471255d369 user: dmestas tags: sub-manifest

Six banner comments named sections that do not exist, in the "Section N" spelling rather than the section-sign one. Earlier renumbering sweeps matched the sign and missed these on the very same page, so they are leftovers of those fixes.

Each banner carries its own section TITLE, so every target is confirmed by matching that title against the draft's own header rather than by inference:

Section 10.3 Configuration Exchange -> 9.3 Section 10.4 Commit-Lock Pragmas -> 9.4 Section 10.5 Private-Content and Discovery Pragmas -> 9.5 Section 3.10 Round Message Layout -> 3.2.1 Section 9.1.1 Unversioned-File Synchronisation -> 8.5.1 Section 9.2.1 Derived Branch, Tag, Leaf, Closed, and Current-Trunk Views -> 8.6.1

Section 3.10 is not a deleted section after all, which both an earlier audit and this author had assumed: it is Round Message Layout, renumbered to 3.2.1. The comments reasoning about its removal, corrected in an earlier commit, remain accurate about the reasoning being void.

No "Section N" reference in the tree now names a section absent from its draft, down from six. Baseline unchanged at 173: this spelling is still invisible to the checker, which is its own item.

test 3500/3500, fmt. Issue #18. check-in: c8cd73f343 user: dmestas tags: sub-manifest

The containment test asked only that lo be at or after the section start and hi at or before its end. A REVERSED range satisfies both halves independently, so it cleared. sync.zig carried a range running from 1878 back to 1443 -- digits that cannot both be right -- and it had passed as correct since this checker was written. The test now also requires lo to be at or before hi.

That site is repaired rather than deferred. The digits were unrecoverable but the claim was not: section 8.2 L1517-1520 states it outright, "An accepting clone reply emits zero or more clone cfile cards in RID order ... then clone_seqno NEXT, push SERVERCODE PROJECTCODE, and final comment", with L1504-1505 for iterating from the supplied RID.

6682409ff0 added a section 8.2 range at L1915-1924 for "the store grew". Section 8.2 ends at L1595; L1915 is config-push payload text. The claim is section 3.7(a) L577-578, "file/cfile, including dangling delta, stored content or igot created a phantom" -- which is what artifact_count_grew measures, and the sibling condition on the line above already cites 3.7(b). Left unfixed this failed the ratchet at 174.

Two checker comments asserted things the code does not do: BARE_DEFAULT claimed ambiguity "never manufactures a failure" -- true, while it manufactures roughly 79 silent CLEARANCES -- and the witnesses comment claimed completeness when the loop breaks on first fit. Five draft aliases added so an unrecognised spelling cannot fall through to the all-drafts path unnoticed.

945 citations, 773 correct, 173 broken, 0 unresolvable sections. Baseline holds at 173 while absorbing a new defect and a newly-detectable class.

test 3500/3500, fmt. Issue #18. check-in: a9b1ee8fb4 user: dmestas tags: sub-manifest

undo's per-file cap read managed.max_file_bytes -- the serialized PATH-TABLE bound (64 MiB) -- as if it were a content ceiling. Any tracked file over 64 MiB (commit allows 1 GiB) failed captureFile, and since a durable pre-image gates any mutation, that aborted checkout/update/merge outright.

Per-file cap is now max_tracked_file_bytes, homed in managed so every layer shares one number. Journal read sizes from stat instead of a derived cap that would have refused journals record can write. (#34) check-in: 1cc2d055b9 user: dmestas tags: sub-manifest

A total-round cap multiplied against the per-round batch size into an accidental ceiling on repository size (~1M artifacts), which a healthy pkgsrc-scale store tripped while converging every round. Guard now counts consecutive rounds that moved no bytes; announcement is not delivery, so a peer announcing phantoms forever still trips it.

Also: import-git reports sub-manifest memo hit rate and whether the retention budget latched (#35). check-in: 6682409ff0 user: dmestas tags: sub-manifest

A citation may carry several ranges: `§9.3 L3278/L3320-3322`, `§7.7 L1465/L1469-1471`, `§4.6.4-m6 (L1014)`. Matching only `§SEC Lnnn` saw the first and was blind to the rest, so half a compound citation could be re-keyed while the other half stayed rotted -- on the same line, looking equally verified. An adversarial reviewer caught that during a re-key pass; the tool could not have.

342 line references sat outside the checker's view. Each now binds to the NEAREST PRECEDING section on its own line, which is the rule that makes this safe: a range written BEFORE any section on the line -- `L545-547 hands clone to §8.1-§8.4, and §8.2 L1504-1506` -- belongs to earlier prose, not to a section further right, and is left alone rather than misattributed. 122 became checkable; 12 leading ranges were correctly skipped. Ranges are never carried across a line break for the same reason.

This also surfaced §10.5, which the old pattern could not see at all because a requirement id sits between section and range (`§10.5 r43 L2477-2479`). §10 is Error Handling with only §10.1; the send-private and client-url claims are §9.5 Private-Content and Discovery Pragmas (L1983-1985: "send-private enables private sync only with x ... Clients repeat it every private round"). Same -1 downshift already verified for §10.3 and §10.4. 21 sites renumbered, and not-a-section is back to 0.

Baseline 80 to 173: 93 defects that were always there and are now visible. The population it measures grew 821 to 943.

test 3495/3495, fmt. Issue #18. check-in: 43c89e841f user: dmestas tags: sub-manifest

d4e0e0d6da fixed six renumbered sections and deliberately left their line ranges stale, on the reasoning that a correct section turns a stale range into an ordinary lookup. This is that lookup: 20 of 100 survived review.

The 20 percent hit rate is lower than the pass before it, and the reason is worth recording. It is NOT that the renumbering targets were wrong -- §11.1 L2604-2607 reads identically before and after that commit, so those citations were never renumbered by it. What the pass found instead is a second, older layer of mis-sectioning underneath the first: claims about server-emitted error cards citing §11.1 Credential Replay when they belong in §10.1 Error Card. Fixing one generation of drift exposed the next.

Two refusals name failure modes the applier could not have caught. One proposal re-keyed half of a COMPOUND citation -- sync.zig:961 carries `§9.3 L3278/L3320-3322` -- which would have welded a freshly verified range to a still-rotted one on the same line. Another submitted a quote that stopped mid-word, with the half of the claim it needed sitting in the sentence it cut off.

The 66 unresolved are now the honest core: three independent adversarial passes have each declined to invent an answer for them.

Baseline 100 to 80; 90 percent of citations now resolve. test 3495/3495, fmt. Issue #18. check-in: dcd6c48df9 user: dmestas tags: sub-manifest

Issue #18: "a reference to a plan must not wear § notation." 80 sites across eight files cited `§E`, meaning item E of storage-track-plan.md -- a document that is not a draft and is not even in this repository. They read exactly like spec citations, including in test names.

`§E` is unambiguous: no draft defines an Appendix E, and every site sits in the storage family (checkout, logstore, managed, pack) whose headers already name storage-track-plan.md. Bare references become `plan E`; where the filename is already present it reads `storage-track-plan.md item E`, which says the same thing without the false notation.

One carried a line number, `§E L119`. A line cite into a file outside the repository cannot be verified by anything, so the range went with it.

That was the last `not a section of any draft` in the tree: 1 to 0. Baseline 101 to 100.

Deliberately NOT swept: §A and §B, which are real draft appendices -- §A alone has 162 subsection citations, and a blanket letter sweep would have destroyed every one. §C (78 sites) is left for a separate ruling: it looks like the same plan, but Appendix C exists in three drafts, so it needs the discriminator applied per site rather than in bulk.

test 3495/3495, fmt. Issue #18. check-in: ee4e0e623a user: dmestas tags: sub-manifest

Two comment families asserted RFC requirements the drafts do not make.

The 65535-byte line bound is ours, not the protocol's. §5.2 L962 says "This protocol imposes no separate maximum line length" and §11.3 L2199 repeats it, yet four tests named it an RFC bound and max_line_len's doc cited §5.2 as its source. The constant stays -- a finite receive window is a resource limit, and §11.3 L2200-2201 leaves payload limits to "declared counts and storage policy" -- but it is now documented as local policy, including that a conforming peer may legally send a longer line and we refuse it. Issue #32 justified this constant as a principled tripwire on the strength of that citation; the doc now records that the justification was void.

The §3.10 comments reasoned about a section that no longer exists -- "§3.10 restates §3.2 and defers to it, so introduces no new order rule". The argument had no subject. Each test asserts real behaviour and is kept; the citations now name the section that actually governs it, and r1 points at §3.2 L376-377, the rule the deleted section deferred TO.

not-a-section 6 to 1. Baseline 106 to 101. test 3495 (1 environmental e2e skip, no failures), fmt. Issue #18. check-in: 8aa2cf2c35 user: dmestas tags: sub-manifest

Six cited sections do not exist in the drafts at all, so no line range could ever have been right. Each was renumbered, and each successor is confirmed by subject match against the section's own text:

§B.4 -> §B Appendix B flattened; cookie-card, url-login-value, pct-user, unreserved and pct-encoded all still exist, every citation exactly 1787 lines above its production §12.1 -> §11.1 Credential Replay §12.2 -> §11.2 Private-Content Exposure and Protected Transport §10.3 -> §9.3 Configuration Exchange -- "the server MUST require Admin; otherwise emit error not authorized to push" §10.4 -> §9.4 Commit-Lock Pragmas -- every site is ci-lock/ci-unlock §9.1.1 -> §8.5.1 Unversioned-File Synchronisation -- "status 3+MTIME-CMP (2 older, 3 identical, 4 newer)"

The §B citations carry corrected line numbers, each asserted to land on the production its comment names. The rest carry the section only: once the section is right the stale range is an ordinary "outside" case, which the re-key pass already handles, and inventing a range here would be the clamping this work exists to prevent.

Deliberately untouched, all needing a decision rather than a lookup: §12.3 (successor §11.3 states the OPPOSITE -- "This protocol defines no separate per-line byte limit before LF" -- while four tests assert a 65535-byte bound as an RFC requirement), §3.10 (removed; the comment reasons about the section's own existence), and §E (a reference to storage-track-plan.md wearing §-notation, which the issue forbids).

not-a-section 36 to 6. Baseline 144 to 135. test 3495/3495, fmt. Issue #18. check-in: d4e0e0d6da user: dmestas tags: sub-manifest

Of 475 candidates, 368 were located by reading the draft text that supports what each comment claims and survived adversarial review; 23 were refuted and 84 returned unresolved. sync.zig and sync_test.zig now resolve at 86 percent, up from 39.

The refusals are the useful part. Several proposals verified byte-for-byte and still pointed at the wrong sentence; one range truncated away the half that carried the claim; one cited text that REFUTES its comment. The unresolved are mostly mis-sectioned rather than mis-lined -- §9.1 cited for push/pull card grammar, §10.1 for the client-version pragma -- and one where an erratum inverted the claim, so the COMMENT is now wrong and no line range would fix it. Those are left for a section decision.

The checker also could not see `§A.1`-style citations at all: 36 of them, unverified while looking checked, 25 of them broken. Appendix headers use `Appendix A. Title` and `A.1. Title`, so both forms are recognised now.

Baseline 477 to 144, over a population that grew 781 to 817.

test 3495/3495, fmt. Issue #18. check-in: 18e5fa7bbb user: dmestas tags: sub-manifest

All four assert that the artifact hash names the uncompressed, fully reconstructed bytes, and cited §4.1 for it. No §4.1 in any of the five drafts says that -- §4.1 defines the container. The statement is sync §7.7 L1465, "Integrity hashes reconstructed artifact bytes, not wire payload."

These were the ones the re-key pass returned unresolved rather than guessing at: wrong section, not drifted lines, so no line range would have been right. Naming the draft because §7.7 also exists in artifact-format.

Baseline 481 to 477. test 3495/3495, fmt. Issue #18. check-in: f1e10128f8 user: dmestas tags: sub-manifest

Each was located by reading the draft text that supports what the code comment actually claims, then adversarially re-checked against a verbatim quote of those lines. Clamping a range into its section would satisfy the checker while pointing at the wrong paragraph, so the checker alone was never the gate.

Of 33 proposals, 2 were refuted on exactly that ground -- the quote verified byte-for-byte and still did not state the claim -- and 4 came back unresolved because no draft's §4.1 states what those comments assert at all. Those are mis-sectioned, not line-rotted, and need a section decision rather than a re-key.

Also restores container.zig's `decode` doc block, which 7d26a31305 left stranded above `decodePrefix` when that function was inserted between the comment and the function it describes.

test 3495/3495, fmt. Issue #18. check-in: e5db38a6df user: dmestas tags: sub-manifest

A bare `§4.1` exists in five drafts, and the report named whichever was witnessed first -- so container.zig's container-format citations were reported against submanifests when they mean sync-protocol. The verdict was right and the message sent a re-keyer to the wrong document.

The report IS the worklist, so that is a defect. Baseline unchanged. check-in: 2fa7460b8f user: dmestas tags: sub-manifest

Most `§X.Y Lnnn` citations no longer point at the text they name. Line numbers rot each time a draft is re-rendered for an erratum, and a citation resolving to the wrong paragraph is worse than none: it looks authoritative.

scripts/check-citations.py parses section spans out of each draft and asserts every cited range falls inside the section it names. It reproduces the survey in #18 -- 65 sections-not-found exactly, 674 vs 667 checked, the delta being citations a wider regex catches.

It splits on "\n" rather than using splitlines(), because drafts carry form-feed page separators that splitlines() also breaks on, drifting every line number further the deeper into the draft it sits. That bug makes correct citations look broken, and it is why this is a script with a reproduced baseline rather than a one-off measurement.

CI pins the count at 512 and fails in BOTH directions, so a new broken citation and a fixed-but-unlowered baseline are each caught. Re-keying the backlog stays separate work; this stops it rotting further.

Also drops parseGimmeCard's assert restating the check above it. parseIgotCard's remaining assert tests a different property and stays.

test 3495/3495, fmt. Issue #18. check-in: 0f80982412 user: dmestas tags: sub-manifest

verifyCommits flattened each check-in's whole S-tree against the published store through a resolver that died with the call, so a node near the root was re-fetched, re-inflated and re-parsed once per descendant check-in -- the cost a89b63ad05 removed from deriveCandidateHints, still present here.

repostate.SubManifestMemo already does this for Repository.flattenIfTree, and documents rebinding `inner` for reuse across walks; the log-only path had no way to hold one. flattenTreeFromLogMemo takes a caller-owned slot and the Verifier keeps one for the whole pass.

Byte-identical output. The win is commit count times inter-commit node sharing: 121 commits over 12,000 files touching 6 per commit, 9.88s to 8.19s. Wide per-commit churn destroys the sharing and is unchanged, never slower.

test 3495/3495, fmt. Issue #33. check-in: c120b93af5 user: dmestas tags: sub-manifest

deriveCandidateHints visited every artifact in the store and reconstructed each one in full, only for parseForImport to discard the blobs. Two bytes decide it instead: §4.4 makes byte 0 of a structural artifact a card letter and byte 1 its delimiter, so a prefix inflate classifies without paying for the whole artifact.

Conservative both ways -- a delta frame or a prefix error falls through to the full read, and only a definite mismatch rejects -- so output is byte-identical, verified on every corpus tried.

Pack of a full-form-dominated store (the shape import-git's automatic pack reads) 10.84s to 8.49s. Neutral on an already-packed store, where deltas dominate and cannot be classified this way.

test 3495/3495, fmt. Issue #32. check-in: 7d26a31305 user: dmestas tags: sub-manifest

Issue #32. `zit stat` gains full-form-bytes and delta-bytes, a partition of content-bytes. The counts alone mislead: a real history is 23% full-form by artifact and 84% by byte, because an artifact is stored whole exactly when delta did not pay.

tree_cache_bytes was 1 MiB, smaller than a wide tree's root fan-out, so nodes were evicted and re-decoded on the next commit that touched them. 64 MiB: byte-identical output, 67.4s to 56.5s importing 45,000 files. Unmeasurable below the old budget, which is why it went unnoticed.

fast_export.Limits.max_events loses its 1,000,000 default rather than gaining a new one. #29 removed that cap at the CLI and left it armed here; six callers silently re-inherited it, the benchmark among them.

test 3495/3495, fmt. check-in: 367539bec4 user: dmestas tags: sub-manifest

The 3,000,000 retained-entry cap and the 1,000,000 event cap both bound by proxy with a round number attached, and each stopped legitimate imports before the real bound was reached. Bytes already bound what costs: max_retained_bytes for retained state, max_total_data for payload. Deleting max_export_events alone was a no-op because the reader carries the same default, so it is now passed maxInt explicitly.

test 3495/3495, gate, e2e, git-import-cli, git-import-e2e. Closes #29. check-in: 255b8f2489 user: dmestas tags: sub-manifest

Issues #27 and #28.

deriveCandidateHints walked each check-in's whole S-card tree and its parent's, for every check-in. Shared subtrees can never pair -- an unchanged node has the same rid on both sides, so pairAgainstParent skips it -- and their blobs were already recorded at introduction, which Pass 2 relies on only as a distinct set (it compacts with dedupSortedRids). So the walk now descends lockstep and prunes any subtree whose S node is the same artifact in both. O(check-ins x tree) becomes O(check-ins x changed paths): sub-manifest loads drop 56 percent on libfossil, to near the distinct-node floor, and the ratio grows with tree size.

The visits that remain were re-inflating. contentOfEntry memoized only .delta frames, so a .zlib sub-manifest re-inflated on every visit -- which is why a pack profile is dominated by flate.Decompress.streamInner rather than by tree-walk arithmetic. PlaintextCache gains memoize_undeltaed, off by default because a linear sweep that reads each artifact once would only duplicate bytes, and opted into by this walk, which revisits.

Output is byte-identical: the wapp store hashes unchanged, and a 189,251 artifact fossil-corpus repack is a fixpoint at 176,143,384 bytes with the same 146,059 re-deltaed.

test 3495/3495, gate, dst 6x1000, e2e 718/718, git-import-e2e, git-import-cli. check-in: a89b63ad05 user: dmestas tags: sub-manifest

pickBase reconstructed each of its five candidates through the delta chain on every call, and the candidate window slides by one artifact per write, so consecutive calls re-read almost the same set. A store-owned memo cuts frame reads 49.7 percent on pack.

The memo is a LogStore field rather than a caller-owned cache threaded through pickBase and putArtifact. A memo keyed by frame offset goes stale when rollbackArtifactBatch truncates and later writes reuse those offsets, and only the store sees that happen -- a caller-owned cache would hand callers an obligation they cannot discharge. It also changes no signature.

assertBaseMemoBelowEnd enforces the invariant at all three post-open truncation sites. Removing the clear from rollbackArtifactBatch makes it fire from a general import-path test, not only its own regression test. check-in: c075a01cfa user: dmestas tags: sub-manifest

wapp import, user CPU: safe 1076s -> 540s, fast 95s -> 53s. Byte-identical output; the 14 pinned vectors are unchanged.

std.AutoHashMap(u64,u32) + Wyhash is gone. The window is now a comptime 16-byte array read through readInt, and the table is open-addressed with capacity from a bounded counter, so it is never more than half full -- that is the probe's termination argument, asserted rather than assumed.

Most of the safe-side win is not bounds checks. Wyhash.init declares a 48-byte buf as undefined, which runtime_safety poison-fills on every call, and the encoder hashed twice per position -- once for the window, once again inside the map on the resulting key. Removing the map removed both.

The probe's explicit bound sits inside `if (std.debug.runtime_safety)`: present in ReleaseSafe where it can trap, compiled out of ReleaseFast. Written unconditionally it cost 12.2% of the fast import; gated it costs nothing in either mode. check-in: bfb63af209 user: dmestas tags: sub-manifest

zit serve on the imported destination, zit clone over a real socket, then the clone compared to Git commit by commit out of its own store. Proves the packed generation is a usable repository, not merely a readable file.

The mirror fixture gains five validated pairs and an orphan look-alike, so gate 3 is exercised in both directions -- the look-alike's bytes must survive. Author and committer instants are now distinct: git's default made them equal, which left the --attribution author comparison inert, and a regression that always wrote the committer instant would have shipped green. storeHolds errors on an undecodable name instead of answering "absent" to a question it cannot ask; twenty drop-set assertions rested on that. check-in: 4a7c197abd user: dmestas tags: sub-manifest

verify_store, verify_map and after_rename_no_replace had no test; five of the eight legacy aliases were inert selectors that PUBLISH instead of failing. The boundaries are now partitioned into named classes and a gate asserts the partition covers the enum exactly, so a new boundary cannot ship an undriven fault arm.

Cancellation and child failure are exercised while generation 1 is open, with the window proven from the export stream rather than asserted in a comment. Every case asserts launched, reaped, no destination, and a fail-closed no-staging-sibling. Post-rename is inverted: the destination must survive and hold exactly the publish manifest. check-in: 67269dd361 user: dmestas tags: sub-manifest

append(gpa, try dupe(...)) evaluates the dupe first, so a growth failure inside append strands it outside the errdefer. Split into dupe, errdefer, append. checkAllAllocationFailures over mirrorDropNames, with a non-empty drop set asserted so the probe cannot pass on a run that allocated nothing. check-in: ab9a72bd10 user: dmestas tags: sub-manifest

verifyFrozen checked headers and existence -- it opened the map without reading a byte. The ordered pipeline now proves the published generation is the raw one minus exactly the candidate-only drop set: both stores reconstructed and rehashed, name sets differenced, every commit resolved and its tree flattened, tree and provenance digests recomputed, parents and attribution compared, refs and tags resolved, and the repository view derived from scratch. Each mismatch returns its own error and blocks the rename.

Eleven corruption fixtures, one per invariant, and each clause is mutation-proven: disabling it reddens its own fixture and nothing else. A clause no fixture catches is a clause that does not exist.

The ledger's strings borrow the importer, which the freeze destroys before verification runs, so the pipeline captures what it needs before the freeze rather than reading dangling memory.

Real corpus unchanged by the added checks: 7,010,160 bytes published, 0 faults, fixpoint byte-identical.

test 3484/3484, gate, e2e 716/716, git-import-e2e 11/11, git-import-cli 247/247, dst 6x1000. check-in: 60d9edab68 user: dmestas tags: sub-manifest

Translation writes a raw generation, packing writes a second through the classifier's candidate-only drop set, verification reopens both, and only the packed one is published. Stages are one-way and publish rejects every earlier state.

Measured end to end on the wapp mirror at k=5: a fresh import publishes 7.0 MB where the unpacked baseline was 63 MB. zit verify on the result reports 944 artifacts and 0 faults, a second pack is a no-op, and a second import into the same destination refuses with DEST byte-identical.

The catastrophic mode this task introduces -- the 66 MB raw generation riding the whole-directory rename into DEST -- was guarded only by statement order, since .verified is both the pre- and post-discard stage. Two guards now: publish refuses while a raw generation is present, and a staging manifest requires exactly {.zit, .zit-checkout, .zit-import-map.jsonl, refs} immediately before the sync.

Also fixes a use-after-free introduced with the S-tree hints: the partition-key map borrowed an S card's filename, which every frame below the root frees on return. The key is owned now, pinned by a pack-level test that segfaults without it.

test 3471/3471, gate, e2e 716/716, git-import-e2e 11/11, git-import-cli 234/234, dst 6x1000. check-in: 8530aa1440 user: dmestas tags: sub-manifest

deriveCandidateHints read only a check-in's F run, and an S-bearing check-in keeps its files in sub-manifests -- so a tree-shaped repository got no same-path hints at all and every blob fell back to the chronological window. Pack was almost inert on exactly the shape sub-manifests produce.

Measured on the wapp mirror imported at k=5: 66.0 MB packs to 63.9 MB before (508 re-deltaed), 7.2 MB after (1028). 89% instead of 3%. zit verify on the result: 1518 artifacts, 0 faults. Fixpoint holds -- a second pack is byte-identical.

Two lanes, per design §6: blobs take earlier same-path versions nearest-first from the shared map, unchanged in shape and now fed by the full recursive enumeration; S nodes take the PRIMARY PARENT's node under an exactly equal canonical partition key, which needs both trees walked and cannot be an offset-nearest pool -- that is the heuristic the design forbids. Check-in lineage is untouched.

Recursion is bounded by max_tree_depth; a seen-set collapses diamonds and rejects cycles. A referenced S node that will not resolve is corruption and aborts, as the design requires.

test 3461/3461, gate, e2e 716/716, git-import-e2e 11/11, dst 6x1000. check-in: 6e2fc456f8 user: dmestas tags: sub-manifest

A Fossil->Git mirror carries root manifest/manifest.uuid/manifest.tags that are not project content: wapp has 252 distinct manifest blobs across 252 commits while its native history tracks no such file. The classifier proves per commit that those bytes are reconstructible from what the import already holds, so packing can drop them.

Fail-closed and cryptographic: stripped-manifest hash equals the uuid, F cards equal the tree, P cards equal validated parents, D card equals the committer instant. A real top-level file named manifest cannot satisfy that by chance, and any inconclusive signal retains every present candidate -- never a strict subset.

Three corrections to the design, all established against the corpus: presence is per-repository (fossil's manifest setting emits each of the three independently), gate 6 must exclude every present candidate since F cards list none of them, and gate 3 must strip the sterilizer line the export appends or the uuid never matches. update.zig's manifest_paths and sterilizer are now public rather than duplicated -- it already had all three names and the exact trailer.

Measured on the pinned wapp corpus: 252/252 commits validate, drop set 512 = 252 manifest + 252 manifest.uuid + 8 distinct manifest.tags.

test 3461/3461, gate, e2e 716/716, git-import-e2e 11/11, dst 6x1000. check-in: d189b0da1d user: dmestas tags: sub-manifest

applyDirect's >= skip means a tag on a separate control artifact, rather than on the check-in manifest, depends on crosslink order; names emitBranchControls as the dependent. Also corrects branchValueOf's doc, left stale by c8c7c2696e. check-in: 941ddcb981 user: dmestas tags: sub-manifest

fast-export emits each commit once, under whichever ref reached it first, so that header was provenance and not branch. A single-branch repository imported with ZERO branches; multi-head repositories were scrambled in proportion to merge activity, which at sqlite scale read as 7,421 leaves on master under repo-state §7.2's same-branch leaf test. Issue #26.

labelBranches claims the primary's first-parent chain, then each head shallowest tip first, then inherits; emitBranchControls writes one control per segment root. Two things the plan did not anticipate: objects are folded to check-ins by hash first, because two git commits can translate to one byte-identical check-in and §5.2 allows one branch tag per rid; and controls are emitted deepest-first, because §5.3's timestamp skip would otherwise let an ancestor's propagation win over a root whose commit shares its second.

source_ref is deleted rather than demoted -- after the change it had no readers. makeBranchControl now composes the §6.2 pair through artifact.composeCommitTags instead of a lone *branch card.

Content was never affected: artifacts, parent links and tip content were always correct. Only derived branch state was wrong, which is why a full benchmark ran clean through it. Repairing an existing imported store means re-importing; check-in hashes survive, since branch state lives in separate controls.

Two pre-existing expectations were rewritten, deliberately: both demanded two active branch tags on one rid, which §5.2 forbids, and were unsatisfiable rather than merely stale.

Verified: fmt, test 3281/3281, gate, e2e 716/716, git-import-e2e 11/11, dst 6x1000. Shapes A/B/C match git branch exactly, 11/11 tip parity. check-in: 4d7a22b93c user: dmestas tags: sub-manifest

repo-state §6.1 requires reading the configured value and falling back to trunk only when main-branch is unset -- "MUST NOT assume a fixed string in the presence of an explicit setting". branchValueOf, which backs the §7.2 same-branch leaf test, used the literal.

It hid because all three uses are comparisons: when both sides fall back identically the result is unchanged. It bites when one check-in is tagged and another is not, which is the imported-repository shape -- displayed as one branch by branchOf, leaf-tested as another by branchValueOf.

The name is now a field on RepoState, set through one RepoView setter so the two readers cannot drift, with branchOf asserting its argument agrees. That assert immediately found a second source: update.zig re-read the config itself and passed a value the state never saw. It also treats an empty stored name as unset, as the view already did. check-in: c8c7c2696e user: dmestas tags: sub-manifest

An S-bearing manifest carries its files in sub-manifests, not in its own F run, so reading bare .f found none of them and every tracked path was classified `added` -- identical output whether or not anything had changed. Issue #25.

CheckinBaseline and DiffBaselineResolver now flatten first, the same step commit already performs. §5.1's "downstream MUST NOT distinguish" is exactly this: a raw .f read is how a reader opts out of the derivation seam.

Submanifest-specific, confirmed both ways: a --no-submanifests import was already clean. Not import-specific though -- any tree-shaped checkout hits it, so a clone from a tree-emitting peer would too.

Regression test asserts clean status and empty diff after materializing a tree check-in, and that a real edit still reports `edited`. check-in: 1e8c8a198d user: dmestas tags: sub-manifest

hexToDigest required exactly digest_len*2 hex characters. zit init writes the sha3-256 policy, so that looked like the only case -- but import-git opens its destination under git's sha1, naming every check-in in 40 hex. So commit rejected the parent of every imported repository and reported it corrupt on a store zit verify called clean. Issue #24.

Length now comes from the name, as artifact §7.2 allows both widths. pack.zig carried the same copy and would have dropped the primary-parent hint the same way; cmd/sync.zig's callers move to the slice form.

commitParentCorrupt now names which step failed. Collapsing an undecodable name, an absent parent and real damage into one sentence is why verify and commit could disagree with no way to tell which was right.

Reproduced on 12 commits and 40 files -- not scale-dependent, and not submanifest-specific: it fires before the S-card branch, so flat imports failed identically. Regression test commits over an imported sha1 history; mutation-proved by restoring the width check. check-in: ffea6238e6 user: dmestas tags: sub-manifest

Same source, same path, two imports: check-in manifests byte-identical once F/S/Z/P are stripped, blob provenance identical, P topology re-derived through each import's own map. check-in: 81b01a1157 user: dmestas tags: sub-manifest

Imports with sub-manifests off: flat manifests throughout and the spec-default config frame (k=0, enable off). Flat is §8.1's default shape; it is this verb's analyze-and-pin that is the deviation.

TreeFormat.Shape becomes flat | tree{k, height_max}, so flat-with-a-k and tree-without-one stop being representable and every tree-only site traps on the wrong tag. buildFlat drains the same MergedCursor, so ordering, dedup and rename/copy/delete stay single-sourced. SubmanifestChoice replaces submanifest_k from Request down to UserOptions.

Publication refuses if the store holds a SUBMANIFEST anyway, derived from work done rather than from the flag. Preflight analysis still runs under flat -- skipping it would put a second divergence inside the region the benchmark times.

Closes #23. check-in: 9b96469ae6 user: dmestas tags: sub-manifest

The latch is one-way and means 'an S reference is stored here', not 'trees are enabled'. The import set enable_submanifests and left it false, which stayed hidden only because the §7.2 gate is the disjunction of the two -- a later 'zit submanifest disable' would have left the repository advertising avoid-submanifests while holding S-bearing artifacts.

Derived from emitted_nodes, not from the k pin: an import whose sources all stay flat stores no S reference and must leave the latch false. check-in: 0b6e41e54f user: dmestas tags: sub-manifest

Additive; covers the tagged trailing block and the legacy widths. check-in: 3392ff5a3e user: dmestas tags: sub-manifest

Deletes the duplicate resolver in update.zig. The surviving path decodes S-card hashes case-insensitively per §3.5; the deleted one ran them through lowercase-only validateName and rejected a legal uppercase hash. TreeError codes mapped exhaustively rather than via else. check-in: 4a3061280e user: dmestas tags: sub-manifest

Four blocks of prose attributed to §6.4 a sentence that appears nowhere in the draft; replaced with verbatim draft text.

§6.4 requires the relative margin m to be documented, which a source comment does not discharge. `zit commit --help` now states the 15% threshold, with a test that renders drift_margin_pct and requires the help to contain it -- the prose and the constant cannot drift apart.

The config frame's trailing block carries a shape tag, so an extension a reader cannot name costs one re-scan instead of an unreadable config (login, remote and project fields included). Legacy widths still parse.

Names the §7.2 avoid-submanifests predicate: the pragma keys on the setting alone, not the armed disjunction, and that divergence is now visible at the call site.

These four land together because D's deletion and E's addition are interleaved in one region of repository.zig with no hunk boundary between them. check-in: 999efb0fd9 user: dmestas tags: sub-manifest

The auto-retune erratum was accepted and is in the draft body; these comments still said it was submitted and unapplied. Comments and one test name only. check-in: a2eb78b7be user: dmestas tags: sub-manifest

Bench now builds ReleaseSafe, which is the mode the O(store) asserts fired in, so without this the clone arm hits them at width. check-in: c1f44ba0ed user: dmestas tags: sub-manifest

ReleaseSafe asserts made the clone answer O(store), and serve's 5s deadline raced read + answer + write as one unit -- so a slow answer was cancelled mid-flight and the dead pread surfaced as error.Storage.

Asserts gone (zit verify covers them); the deadline now races read and write separately. Not scale-dependent -- machine speed vs a wall clock.

Stalled peer still cut at 5.0s. test/gate/e2e/dst green. check-in: 94bebf57f1 user: dmestas tags: trunk

The emitted `optimize` field was the string "ReleaseFast", hardcoded into the JSON format string rather than derived from anything. It reported ReleaseFast for every record the benchmark has ever written, whatever the executable was actually built as - so the one field whose job is to say which binary produced a measurement could not be wrong in a detectable way, only in a silent one.

Now `@tagName(builtin.mode)`, alongside the os/arch tags beside it that were already derived that way.

Found by switching bench to ReleaseSafe and watching `executable_sha256` change while `optimize` did not. check-in: be18ff3902 user: dmestas tags: sub-manifest

`--release` resolves to ReleaseSafe here as a correctness decision: src/ carries over a thousand asserts that turn a detected inconsistency into a crash instead of a durable write, and ReleaseFast compiles every one of them away. The bench executables opted out of that on the grounds that "a benchmark must measure optimized code".

Optimized, yes - but it has to be the code users run, or the number describes a binary nobody executes. This matters more than a uniform constant would suggest, because the cost is not uniform: the same rationale block records `zit pack` at 599 ms fast against 15.2 s safe on one store. A measurement taken in the mode that strips the asserts cannot be compared against, or used to make a claim about, the shipped artifact.

`bench_zit` is the sharpest case - the serve benchmark spawns it as the real `zit` binary, so it stood in for production while being built unlike it. check-in: 30b7d5e766 user: dmestas tags: sub-manifest

The 5s per-connection deadline raced read + answer + write as one unit, so a slow answer was indistinguishable from a stalled peer -- and cancelling mid-answer surfaced as error.Storage, telling the peer the repository was damaged. Issue #22.

Read and write are raced separately, each with the full deadline; the answer runs unraced since it is bounded by the store, not the peer.

Verified: a peer that sends one body byte and stalls is still cut at 5.0s; pikchr reclone passes. test 2327/2327, gate, e2e 671/671, dst 6x1000, fuzz serve_request_garbage. closed check-in: 73bdc2fef1 user: dmestas tags: issue22-diagnostics

Under runtime_safety (ReleaseSafe -- the shipped build) emitCloneReply reconstructed every served artifact and its delta basis just to assert a declared size. That pushed a clone round past serve's 5s per-connection deadline, which cancels mid-answer; the in-flight pread then failed and the peer got HTTP 500 on a healthy store. Issue #22.

zit verify checks the same properties on demand, which is where O(store) verification belongs.

Also names the failure server-side: serve collapses every AnswerError into a bare 500, so serveAnswer now logs which one.

Reclone verified green: wapp 1038, althttpd 1695, pikchr 2842, libfossil 17218 (18 rounds). test 2327/2327, gate, e2e 671/671, dst 6x1000. check-in: 879b867bb1 user: dmestas tags: issue22-diagnostics

crashCheckOnce repairs by truncating below the earliest injected bitflip and reopening. When the truncation empties the log, that reopen writes a fresh header - and the injected short-write fault could land on it, so open returned Storage, the sweep mapped it to Io, and the run failed. Nothing there was a statement about the store: the simulator faulted its own recovery.

The repair and its reopen now run against the healthy disk phase 2 heals to, the all-zero FaultCfg. Faults come back before the invariant check, which is workload again.

Pre-existing: trunk fails seed 201 with the same Io. check-in: 9e3107e570 user: dmestas tags: sub-manifest

The generator's own note says a real check-in DAG is acyclic by construction - a manifest's hash covers its P-card - while the phantom-parent flavor lets a hash be named before its record exists, so a later pick can close a loop and must be checked. The check only ever walked primary_parent_of, and the merge parent was drawn by pickInsertedIndex with the comment "no cycle risk".

That holds for the primary chain alone. Section 3.3 ancestry, which the B1 tie-break walks, follows merge parents too: a merge edge into an already inserted check-in, paired with a phantom primary edge out of it, closes a loop the guard never looked at. ancestryMaximal then finds every candidate dominated and no maximal element, and panics - seed 192 built exactly that, 9 to 14 to 13 and back.

parent_indices records every parent index a record names, wouldCloseCycle walks that graph visited-once, and the merge draw is checked like the primary one. A losing draw degrades to the single-parent record rather than drawing again, so the seeded stream shifts as little as it can.

Pre-existing: trunk fails seed 192 identically, and gen.zig is byte-identical there. Only the default six seeds hid it. check-in: 992ff19a23 user: dmestas tags: sub-manifest

A name can reach the same check-in through two index entries, so one selection offers the same rid twice. MtimeMaxPick.offer keeps the first as `best` and appends the second to `tied`; winner then appends `best` back, so `tied` holds that rid twice.

Ancestor sets are self-inclusive, so each copy found itself in the other's set. Dominance skipped only the same INDEX, so both copies were dominated, every candidate was eliminated, and the maximal element a finite DAG must have came up null - a panic on `best.?`, reached through resolveName.

Dominance is about being a different check-in, not sitting at a different offset. The new test drives offer/winner directly, and also pins that a duplicate cannot shadow a real descendant. check-in: d3f58d2875 user: dmestas tags: sub-manifest

Trunk's Slice 1/2 made both open paths header-only, so a rotted payload now opens clean and surfaces at content() instead of at open. The storage sweep still assumed open was the tripwire, so two of its own claims were wrong once the merge combined them with this branch's store:

- crashCheckOnce repaired only "if the open itself reports corruption", a precondition Slice 2 removed. Injected rot reached the invariant check and read as data loss. NeedsRepair, private to the harness, now carries "refused AND genuinely damaged" to the repair that already existed.

- rotWithinFrame spelled the per-frame overhead as 22, the header alone, so a frame's own trailing frame_crc32 counted as inert pad. A bitflip in those four bytes makes a reader recompute a correct checksum, compare it to a corrupted stored one, and refuse the frame; the sweep called that data loss. logstore now exports frame_fixed_overhead and the sweep derives it instead of respelling it.

Both are trunk defects, not merge damage: trunk alone fails dst at seed 8. The merge only shifted the draws far enough to reach the first one at seed 3, inside the default gate's range.

cli.zig's verb enum was the single textual conflict: a pure union of trunk's verify and this branch's submanifest and import-git.

fmt, test, gate, dst green. check-in: 584314dcb7 user: dmestas tags: sub-manifest

"the peer answered with an unexpected HTTP status" sends an investigation nowhere. 500 (the server could not build a reply), 413 (the request exceeded the body ceiling), 400 (a malformed container) and 417 (a rejected Expect: 100-continue) are four different bugs, and the message distinguished none of them. Issue #22 has been open on exactly that sentence.

transport.zig discarded the code at the status check because Zig errors carry no payload and the closed error set is deliberate. Rather than widen TransportError with a variant per status, the status is left on the Transport as last_status for the caller to read after the error, and syncRunFail names it: "the peer answered HTTP 500 (Internal Server Error)".

Immediately useful: re-running the issue #22 gate against the fossil corpus with this build turns its failure_reason from the old sentence into HTTP 500, which -- together with the reply-size arithmetic -- narrows the cause to OutOfMemory or ServerStorage and eliminates every 4xx path. The issue's own hypothesis, a per-round shun-snapshot readiness problem, is not among the candidates: that arm is documented unreachable and the code agrees.

Also corrects three comments Slice 2 falsified, all on the same read path:

logstore's open no longer CRC-verifies payloads, so the serve snapshot's contentImpl/wireImpl read faults are no longer "post-scan EIO or bitrot after a verified scan" -- ordinary stored corruption now reaches serve there, on the read. That is a better shape (one bad frame fails the round that touches it rather than making the server refuse to serve at all) but it makes that arm live traffic, and anything reasoning about how often it fires must start from that.

runPack and runShun both claimed their read-only open was an "integrity pre-flight" that aborted a corrupt source before a byte was written. It no longer verifies payloads. The guarantee still holds -- packLog pulls every artifact through srcContent, which propagates error.Corrupt rather than skipping the frame, and readFrameAt CRC-checks every read, so a rotted payload still aborts with the temp deleted and .zit untouched -- but it is now collected during the copy loop instead of at open. Both verbs RENAME OVER the original, so runPack's comment now states the consequence plainly: if srcContent is ever changed to answer null on a corrupt read, the rewrite would silently omit the damaged artifact and then delete the only copy of it.

Verified: fmt clean, test 2327/2327, gate, e2e 671/671. The new message checked against a stub server answering 500. check-in: b4637e26df user: dmestas tags: issue22-diagnostics

openReadOnly is what every CLI verb opens through, and until now it still checksummed every payload byte on every invocation -- so nothing users ran had gotten faster. It scans header-only now:

10k frames 42 MB 42.2ms -> 10.3ms 10k frames 410 MB 298.0ms -> 12.3ms 100k frames 416 MB 511.9ms -> 115.4ms 100 frames 419 MB 321.4ms -> 0.10ms

Open cost is a function of frame count rather than repository size. On a store of a few large artifacts it is effectively free.

This completes the sequence: vendor a faster crc, measure the path production actually takes, scan header-only on the append path, ship the fsck verb, then delete the eager check. Each step was gated on the one before it, and the deletion was deliberately last because it is the only one that gives something up.

What is given up: opening a repository no longer detects payload corruption. It surfaces on the read instead -- readFrameAt crc-checks every frame unconditionally now -- or on demand through "zit verify". That is the split git draws between ordinary operation and fsck, and it was signed off knowing the tradeoff. A damaged store also stays usable for everything that does not touch the damaged bytes, which the old eager check made impossible.

Slices 0 and 4 are closed too: the 4 KiB chunk widening landed inside verify, the only place that loop still runs, and the sidecar index is dead on its own gate -- the 100k-frame shape was to justify it by exceeding 150ms and came in at 119ms. mmap is dead alongside it, separately, because a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats reclamation rather than merely slowing it. Neither rejection is recorded at the logstore extension comment yet; that is the one piece of follow-up this leaves behind.

Verified on the merged tree: fmt clean, test 2327/2327, gate 19/19. On the branch tip, additionally e2e 671/671 and dst 6 seeds x 1000 steps. check-in: 6d9172d488 user: dmestas tags: trunk

The speedup reaches production. openReadOnly is the path every CLI verb takes -- Repository.open falls back to it, and Repository.logStore, the content channel behind cat, artifact, checkout and sync, calls it with no cache in front. It scanned every payload byte on every invocation:

10k frames 42 MB 42.2ms -> 10.3ms 10k frames 410 MB 298.0ms -> 12.3ms 100k frames 416 MB 511.9ms -> 115.4ms 100 frames 419 MB 321.4ms -> 0.10ms

Open cost is now a function of frame count, not repository size.

verified_on_open was the flag openReadOnly set to tell readFrameAt that the scan had already checked every frame, so reads could skip their own crc. With no scan doing that, the exemption had nothing left to rest on and the flag is deleted. readFrameAt now crc-checks unconditionally: it is the only thing standing between a rotted payload and a caller that believes it, and restoring any skip there means restoring the scan.

Detection is relocated, not removed. A payload bitflip no longer voids the open -- the rest of the repository stays usable -- and surfaces the first time anyone reads those bytes: "zit cat" and "zit artifact" both refuse with 'repository is corrupt', and "zit verify" names the artifact and the offset. That verb shipped first for exactly this reason.

openForVerify is deleted. It existed because openReadOnly both aborted on a damaged store and granted read-time trust, neither of which verify could accept; both properties are gone, it became byte-identical to openReadOnly, and its own doc comment said to remove it in this commit.

The DST prove-it-can-fail arm is inverted rather than deleted. It asserted that openReadOnly WOULD serve latent rot silently, across a seed range where at least one corpus lands a payload rot on a raw frame whose header survived. That hazard is gone, so the same range now asserts the opposite: the read path never serves rot as success. trust_on_read goes with it -- there are no longer two paths to select between. Losing the eager whole-log scan cost nothing here, because per-read verification is what was catching this all along.

Net -40 lines.

Verified: fmt clean, test 2327/2327, gate, e2e 671/671, dst 6 seeds x 1000 steps. The Slice 1 tripwire -- the assertion that openReadOnly still rejected eagerly, written to fail if the flag was deleted without an fsck verb -- fired on this commit, which is the one it was aimed at. closed check-in: b224ffeb40 user: dmestas tags: logstore-slice2

zit now imports sqlite at defaults: 36,416 commits, 2,109 refs, 44m35s, no flags. Two independent walls stood in front of that.

Refs (issue 21). One file per ref made a ref's name a path, so the filesystem got a vote on which repositories are representable. sqlite has refs/remotes/origin/EQP-improvements and .../eqp-improvements at different commits -- legal, distinct git refs -- and on APFS the second landed on the first as a bare PathAlreadyExists, raised during publication after the whole import had run. Case was only the reachable instance; path length, reserved device names and each platform's own Unicode fold table are the same bug in other clothes.

They are one sorted file now, "refs", whose lines pair a canonical name with its target. Nothing reads it back -- "refs" is a reserved root the working-tree scan skips, and branches resolve through repostate's sym- tags -- so it is interop output and its job is to be legible. Sorting makes it byte-identical for a given ref set, so two imports of one repository can be compared.

This supersedes the ASCII-fold refusal from the previous commit. That was the right interim while a ref was a filename; with one file the collision is representable, so refusing it would be gratuitous. What survives is the genuine collision -- distinct sources normalizing to a byte-identical canonical name, refs/heads/./release against refs/heads/release -- which no layout can hold, and which now names both sources instead of only failing.

Budget (issue 20). The 4 GiB payload default refused sqlite's 12.2 GiB stream, and could not simply be raised: fileCommandCeiling divided the same number, so a larger allowance for a whole history was also a larger allowance for one commit's retained command list. Those bound different things and only one is memory -- payload streams and is released per event (sqlite's import held 21 MiB), while a commit's command list is held whole. The arena has max_command_arena_bytes of its own now and keeps the old 4 GiB; the cumulative total goes to 1 TiB.

The file-command ceiling is therefore structural: about 104 million commands in one commit, and no flag raises it. Its diagnostic says so rather than pointing at a knob that no longer moves it. The help text claimed the payload flag fixed both; it only ever fixed one. check-in: 8ae7041c2c user: dmestas tags: sub-manifest

Opening a repository checksummed every payload byte, so open cost scaled with the whole store rather than with what the caller asked for. The append path now scans header-only:

10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms

openReadOnly -- the path every CLI verb reaches -- deliberately still verifies payloads. Nothing has moved on the production path yet. That is Slice 2, and it is gated on this merge's other half: zit verify, the on-demand fsck that has to exist before eager detection is removed.

The 100k-frame run settles a question the plan left open. Slice 4's gate was 'build the sidecar index only if that shape exceeds 150ms after the scan change'; it lands at 119ms. The sidecar is dead, and mmap with it -- separately, a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats the reclamation guarantee rather than merely slowing it. Both should be recorded as evaluated-and-rejected at the logstore extension comment; that is not done here.

Also adds --mode append|read-only to the open benchmark. Every open number this effort had produced measured LogStore.open, which no CLI verb enters; the two modes agreed within noise before the scan change, which is what makes the after numbers above a comparison rather than two guesses.

Verified on the merged tree: fmt clean, test 2327/2327. gate 19/19 and e2e 671/671 were watched green at the branch tip, which is content-identical -- trunk did not move between the branch point and this merge. check-in: 90bd1b8015 user: dmestas tags: trunk

Slice 2 will extend header-only scanning to openReadOnly and delete verified_on_open, which removes eager corruption detection entirely. This is the on-demand replacement that has to exist first -- the same split git makes between normal operation and git fsck.

Two passes over the store. FrameVerifier walks every frame under .verify_payloads, which is the mode the open paths are shedding; it is built on StorageFile rather than an opened LogStore, so it still works on a store that will not open -- the case an fsck exists for. Then openForVerify opens .headers_only and reconstructs each artifact, which catches a frame whose checksum is honest about bytes that are not what their name claims.

Both passes report rather than abort. A payload CRC fault carries a header-proven frame_len, so the walk steps over it and inventories the rest; only a header-window fault, whose length cannot be trusted, stops the walk. A torn tail is not a fault -- open recovers it silently, and it means an interrupted write, not rot. An unreadable payload is a finding too, not the end of the inventory.

Exit codes stay 0/1/2 as every other verb documents them; 2 remains usage error and never corruption. Findings stream to stderr as they are found, the clean summary goes to stdout, and on any fault stdout is empty.

Scope stops at content artifacts. A manifest whose bytes hash correctly is byte-identical to what commit wrote and what sync validated on ingest, so re-parsing it would catch a zit bug rather than storage rot. The help text says so.

openForVerify is deliberately not openReadOnly: openReadOnly sets verified_on_open, which makes readFrameAt skip the per-read whole-frame CRC, and it scans .verify_payloads, so on exactly the damaged store an operator is fsck-ing it aborts before pass 2 runs. Slice 2 makes the two identical and openForVerify is deleted in that commit; its doc comment says so.

The chunk buffer in scanOneFrame's payload loop widens from 4 KiB to 64 KiB, matching cloneInto's existing precedent. Post-Slice-2 that loop runs only under verify, where an operator's wall-clock is what it costs.

zit verify is not an RFC-mandated verb. repo-state 7.3 blesses full rebuild as a fallback, but the verb's shape is a local design decision, not a protocol requirement.

Verified: fmt clean, test 2327/2327, gate 19/19, e2e 671/671. Four mutants, all compiling, each red for its own semantic reason: pointing the frame pass at .headers_only, stubbing artifact.verify to true, dropping the unreadable-store tally, and reverting the payload read to a hard error. Exit arms smoke-checked against the built binary. closed check-in: bcb9f8eb6e user: dmestas tags: logstore-open-cost

sqlite carries refs/remotes/origin/EQP-improvements and .../eqp-improvements pointing at different commits. Both are legal, distinct git refs. Zit stores a ref by creating a file at its canonical path, so on APFS the second one landed on the first: bare PathAlreadyExists, raised by writeRefs during publication, after 49 minutes of completed import. No ref name, no cause.

The refusal already existed. finalizeRefs rejects two source refs that resolve to one canonical leaf -- 'refs/heads/./release' against 'refs/heads/release' is a tested case. It just compared bytes, and a filesystem does not. Keying the canonical index by an ASCII fold makes the existing rule see the collision it was written to catch, uniformly, instead of leaving it to whichever disk the import ran against.

ASCII rather than Unicode is deliberate and documented at the fold: APFS, HFS+ and NTFS each fold a different frozen table, so following them would make the importable set depend on the volume. A-Z is what they agree on.

The error now names both source refs, both canonical spellings, and whether case alone separates them -- a normalization clash and a case clash want different fixes from the operator. Retained on the importer and formatted by the CLI, the same split FileCommandOverrun uses.

This refuses on Linux what Linux tolerated, which is the point: the repository means the same thing everywhere or it is not portable. Making sqlite importable rather than merely diagnosable is a layout question -- one file per ref cannot represent it -- and stays open. check-in: f0331d2f5a user: dmestas tags: sub-manifest

The open-time scan streamed every payload byte through the crc, making open cost O(repository bytes) rather than O(what the caller asked for). ScanMode.headers_only validates each frame's header window and then steps over the payload by frameSpan arithmetic, leaving open cost O(frame count).

Measured, same binary, both modes in one run:

10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms

Every fault that lives in a header window is still caught: readFrameHeader runs unchanged, so a bad prefix crc, a bad header crc, an out-of-range hash_len and an unknown flag kind all still void the open. Torn-tail detection survives intact -- a torn frame is a clean prefix, never damaged bytes, so offset + frame_len > total catches every truncation the payload and trailer short-read checks used to, using a length the header window already proved trustworthy. What is no longer caught at open is a bitflip in already-landed payload bytes, which becomes content()'s job; it crc-checks the frame it reads regardless.

openReadOnly -- the path every CLI verb reaches -- deliberately keeps .verify_payloads. No guarantee has moved on the production path yet, and none should until the explicit fsck verb exists to replace it. This half is what makes the cost measurable without taking on that decision.

The one test that changed is the one that had to: a corrupted payload on a complete frame no longer voids open. It now asserts the fault surfaces on the first content() read AND that openReadOnly still rejects the file eagerly. That second assertion is the tripwire for deleting verified_on_open without shipping the verb first. check-in: bae910f00e user: dmestas tags: logstore-open-cost