Timeline

Opening a repository checksummed every payload byte, so open cost scaled with the whole store rather than with what the caller asked for. The append path now scans header-only:

10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms

openReadOnly -- the path every CLI verb reaches -- deliberately still verifies payloads. Nothing has moved on the production path yet. That is Slice 2, and it is gated on this merge's other half: zit verify, the on-demand fsck that has to exist before eager detection is removed.

The 100k-frame run settles a question the plan left open. Slice 4's gate was 'build the sidecar index only if that shape exceeds 150ms after the scan change'; it lands at 119ms. The sidecar is dead, and mmap with it -- separately, a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats the reclamation guarantee rather than merely slowing it. Both should be recorded as evaluated-and-rejected at the logstore extension comment; that is not done here.

Also adds --mode append|read-only to the open benchmark. Every open number this effort had produced measured LogStore.open, which no CLI verb enters; the two modes agreed within noise before the scan change, which is what makes the after numbers above a comparison rather than two guesses.

Verified on the merged tree: fmt clean, test 2327/2327. gate 19/19 and e2e 671/671 were watched green at the branch tip, which is content-identical -- trunk did not move between the branch point and this merge. check-in: 90bd1b8015 user: dmestas tags: trunk

Slice 2 will extend header-only scanning to openReadOnly and delete verified_on_open, which removes eager corruption detection entirely. This is the on-demand replacement that has to exist first -- the same split git makes between normal operation and git fsck.

Two passes over the store. FrameVerifier walks every frame under .verify_payloads, which is the mode the open paths are shedding; it is built on StorageFile rather than an opened LogStore, so it still works on a store that will not open -- the case an fsck exists for. Then openForVerify opens .headers_only and reconstructs each artifact, which catches a frame whose checksum is honest about bytes that are not what their name claims.

Both passes report rather than abort. A payload CRC fault carries a header-proven frame_len, so the walk steps over it and inventories the rest; only a header-window fault, whose length cannot be trusted, stops the walk. A torn tail is not a fault -- open recovers it silently, and it means an interrupted write, not rot. An unreadable payload is a finding too, not the end of the inventory.

Exit codes stay 0/1/2 as every other verb documents them; 2 remains usage error and never corruption. Findings stream to stderr as they are found, the clean summary goes to stdout, and on any fault stdout is empty.

Scope stops at content artifacts. A manifest whose bytes hash correctly is byte-identical to what commit wrote and what sync validated on ingest, so re-parsing it would catch a zit bug rather than storage rot. The help text says so.

openForVerify is deliberately not openReadOnly: openReadOnly sets verified_on_open, which makes readFrameAt skip the per-read whole-frame CRC, and it scans .verify_payloads, so on exactly the damaged store an operator is fsck-ing it aborts before pass 2 runs. Slice 2 makes the two identical and openForVerify is deleted in that commit; its doc comment says so.

The chunk buffer in scanOneFrame's payload loop widens from 4 KiB to 64 KiB, matching cloneInto's existing precedent. Post-Slice-2 that loop runs only under verify, where an operator's wall-clock is what it costs.

zit verify is not an RFC-mandated verb. repo-state 7.3 blesses full rebuild as a fallback, but the verb's shape is a local design decision, not a protocol requirement.

Verified: fmt clean, test 2327/2327, gate 19/19, e2e 671/671. Four mutants, all compiling, each red for its own semantic reason: pointing the frame pass at .headers_only, stubbing artifact.verify to true, dropping the unreadable-store tally, and reverting the payload read to a hard error. Exit arms smoke-checked against the built binary. closed check-in: bcb9f8eb6e user: dmestas tags: logstore-open-cost

sqlite carries refs/remotes/origin/EQP-improvements and .../eqp-improvements pointing at different commits. Both are legal, distinct git refs. Zit stores a ref by creating a file at its canonical path, so on APFS the second one landed on the first: bare PathAlreadyExists, raised by writeRefs during publication, after 49 minutes of completed import. No ref name, no cause.

The refusal already existed. finalizeRefs rejects two source refs that resolve to one canonical leaf -- 'refs/heads/./release' against 'refs/heads/release' is a tested case. It just compared bytes, and a filesystem does not. Keying the canonical index by an ASCII fold makes the existing rule see the collision it was written to catch, uniformly, instead of leaving it to whichever disk the import ran against.

ASCII rather than Unicode is deliberate and documented at the fold: APFS, HFS+ and NTFS each fold a different frozen table, so following them would make the importable set depend on the volume. A-Z is what they agree on.

The error now names both source refs, both canonical spellings, and whether case alone separates them -- a normalization clash and a case clash want different fixes from the operator. Retained on the importer and formatted by the CLI, the same split FileCommandOverrun uses.

This refuses on Linux what Linux tolerated, which is the point: the repository means the same thing everywhere or it is not portable. Making sqlite importable rather than merely diagnosable is a layout question -- one file per ref cannot represent it -- and stays open. check-in: f0331d2f5a user: dmestas tags: sub-manifest

The open-time scan streamed every payload byte through the crc, making open cost O(repository bytes) rather than O(what the caller asked for). ScanMode.headers_only validates each frame's header window and then steps over the payload by frameSpan arithmetic, leaving open cost O(frame count).

Measured, same binary, both modes in one run:

10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms

Every fault that lives in a header window is still caught: readFrameHeader runs unchanged, so a bad prefix crc, a bad header crc, an out-of-range hash_len and an unknown flag kind all still void the open. Torn-tail detection survives intact -- a torn frame is a clean prefix, never damaged bytes, so offset + frame_len > total catches every truncation the payload and trailer short-read checks used to, using a length the header window already proved trustworthy. What is no longer caught at open is a bitflip in already-landed payload bytes, which becomes content()'s job; it crc-checks the frame it reads regardless.

openReadOnly -- the path every CLI verb reaches -- deliberately keeps .verify_payloads. No guarantee has moved on the production path yet, and none should until the explicit fsck verb exists to replace it. This half is what makes the cost measurable without taking on that decision.

The one test that changed is the one that had to: a corrupted payload on a complete frame no longer voids open. It now asserts the fault surfaces on the first content() read AND that openReadOnly still rejects the file eagerly. That second assertion is the tripwire for deleting verified_on_open without shipping the verb first. check-in: bae910f00e user: dmestas tags: logstore-open-cost

benchOpen timed LogStore.open -- the append path. Production never takes it: Repository.open falls back to openReadOnly, and Repository.logStore, the lazy content channel behind blob, cat, checkout and sync, calls openReadOnly with no cache in front. Every open number this effort has produced described a path no CLI verb enters.

That was survivable while the two paths ran the same scan. It stops being survivable the moment either one scans header-only, at which point the bench would report the optimized path while production still paid the full cost. --mode append|read-only defaults to append so existing invocations report unchanged numbers, and the mode is in the output line because the two are only comparable when a captured log can tell them apart.

Calibration on this tree, before any scan change: the modes agree within noise on all four shapes -- run B is 1% apart. That agreement is the check that the flag measures what it claims. check-in: e56a663b7a user: dmestas tags: logstore-open-cost

ExportPayloadLimitExceeded was a bare error name. Importing sqlite cost 17 minutes to learn a single integer, and the operator still had to guess it: the stream is 12.2 GiB against a 4 GiB default.

One error carried two unrelated bounds. max_total_data is the operator's --max-export-payload-bytes; max_data is a fixed 1 GiB per-payload ceiling no flag reaches. Reporting the wrong one sends an operator to a knob that cannot help, so PayloadOverrun records which bound refused, and the single -payload arm says plainly that no option raises it.

The cumulative arm refuses to name a remedy value. Parsing stops at the overrun, so the observed total is a lower bound; 'above 9' would name a number that fails again. It reports the bound and says the full payload is larger, which is the honest shape of what the parser knows.

Also corrects the budget's own doc comment, which claimed four GiB was 'the working set an import may claim on a commodity machine'. Measured resident size importing sqlite's 12.2 GiB stream: 21 MiB. Payload is streamed and released per event. What the cumulative half actually bounds is total history, so the default over-provisions memory by three orders of magnitude while refusing a mainstream repository. check-in: ee703cde16 user: dmestas tags: sub-manifest

Trunk since d186a44e brought the repository shun table and its whole- branch verification fixes, the release-assert guard, the `test`/`gate` lane split, the structurally bounded logstore priority window, and the vendored crc32. Eight files conflicted. The decisions worth naming:

Lanes. `git-import-e2e` was a compile-only proof hanging off `test`; that is exactly the cost trunk moved to `gate`, so it moves with `e2e` and `smoke`. `import-git-bench` stays on `test` because it registers a real unit-test run rather than a compile proof. `declared_steps` is the union plus `gate`.

Sync ingest. Both sides rewrote it: trunk added the §7.7 receive-side shun admission, this branch added §7.1 phantom minting for a pushed tree's missing sub-manifests. They compose in one direction — admission is asked first, so a card §7.7 refuses is never stored and mints no phantoms either. The S references of an artifact this repository declines to hold are not names it should then go begging for.

Client construction. Trunk factored the three verbs' `Client` literal into `syncClient` and threaded `RepoAdmission`; this branch had widened `project_code` into `RepoSyncIdentity`. The helper now carries both, which is what trunk's own comment asked for: three copies of that literal is how a repository filter comes to be wired into two verbs and not the third.

Per-round server refresh. Trunk made the snapshot mandatory — `error.ServerStorage` rather than serving with no shun table — where this branch read its four sub-manifest fields through fail-open defaults. Trunk's reading wins for all six fields: serving a stale default is the failure, not the fallback.

Two guards trunk owns caught real defects in this branch's code, which is the whole point of them:

The total switch in `shunConsequence` refused to compile against the `submanifest` artifact type, exactly as its comment promised. It has an arm now.

The release-assert guard found nine asserts on untrusted input across six functions. Two were reachable faults, not style: `lookupTree` asserted §4.4 on a sub-manifest a peer supplies, so a malformed node panicked a ReleaseSafe reader and was walked anyway under `--release=fast`. It returns `SubUnresolvable` now, which §5.2 already defines as "absent, or is not a SUBMANIFEST". The empty-S-run assert became an early `null`. The two sub-manifest writers refuse an empty group through a new `SubmanifestWriteError` rather than asserting; the builder's internal path keeps its allocation-only surface through `serializeLeafCore`, since the partitioner is what establishes non- emptiness there. The rest were tautologies or restatements of invariants the card parser already enforces, and now say so in a comment instead of carrying a panic.

Verified on the merged tree: `gate`, `e2e`, `dst` (6 seeds x 1000 steps), `smoke` (S1-S10, including trunk's new shun interop), and `git- import-e2e` all green, `zig fmt --check` clean. Beyond the suite, the importer was run for real: the empty-email repro imports and its check- in carries `U mohye77`, and marktext imports end to end with the analyzed pin at k=5. check-in: 562cf5be74 user: dmestas tags: sub-manifest

parseIdentity refused an identity carrying an empty address twice over, once as `open + 1 == close` and again as `email.len == 0`. That is not malformed input. Setting `user.email` to the empty string is ordinary git configuration; `git fast-export` writes the identity with an empty angle-bracket pair, and `git fast-import` reads that same stream back to the same commit. The construct is valid by git's own definition on both sides of its own tooling, and we rejected it.

The cost was not cosmetic. Two such commits, reachable only through a merge's second parent, aborted the import of an entire 437 commit repository with `import-git: import failed: Malformed`, naming neither the line nor the rule.

Only the email clauses are relaxed. The name clauses stay: git's own porcelain refuses an empty ident name, and the importer maps the identity's name onto the U card login, so a nameless identity would produce a nameless check-in. The two questions were conflated in a single condition, which is a large part of why this went unnoticed, so the guard now separates them and says why.

This is the third defect on this branch of the same shape: a property of our own fixtures promoted into a rule about the world. The first two were bounds tuned to the width of our benchmark corpus. This one is sharper, a refusal of input the tool we integrate with emits by default. Every identity fixture in the parser tests carried a non-empty email, so a uniform corpus could not falsify a rule that keys on emptiness.

Both tests are witnesses rather than decoration: each fails with Malformed against the previous parser and passes against this one. The unit test pins the construct and the empty-name refusal beside it; the differential test has git itself produce the commit and then asserts the imported check-in carries the login mohye77 rather than an empty user. check-in: 1f21f96558 user: dmestas tags: sub-manifest

`zit import-git SOURCE DEST` reads a Git repository through `git fast- export` and builds a zit repository from it. The destination appears only after the export, artifact creation and publication have all completed, so a failure anywhere leaves no half-published repository behind; this is the first caller of the logstore artifact batch added earlier on this branch, which is what makes a check-in's blobs arrive as a unit.

Mode 160000 now parses as a gitlink instead of error.UnsupportedMode. That completes the split begun in the previous check-in: reporting what the stream said is the parser's job, and deciding that zit has no submodule concept is the importer's. The refusal still happens, but it now explains itself instead of failing as a parse error on input git legitimately emits.

Signed commits are refused by default rather than imported with the signature silently dropped. Fossil has no signature card, so importing one is a loss the operator should choose; --drop-signatures accepts it explicitly.

The sub-manifest parameter is analyzed, not guessed. Preflight measures the source's own name set and pins its measured optimum, and --submanifest-k overrides that only to match another repository's existing pin, which is what lets two repositories share sub-manifests by content address. On a 100,000 file corpus the analyzed value cuts the objective by roughly 3.4x against the fixed default it replaces.

Correctness is checked differentially against real Git rather than against fixtures we wrote: every materialized tree is compared to `git ls-tree` for the corresponding commit, refs and annotated tags are compared to the source, and both comparators now assert that a gitlink is absent rather than merely tolerating it, so the omission is a contract instead of an accident.

The rest is wiring. extras learns the import provenance names, root.zig exports the importer modules across the executable and library boundary, tidy and the CI step guard learn the new test roots and build steps, and the e2e help check learns that a verb spelled import_git in Zig is import-git on the command line. check-in: dc0aa2f858 user: dmestas tags: sub-manifest

zig build test was 9m13s, and the tests it actually ran were a small fraction of that: it also compiled the e2e and smoke harnesses in the pinned ReleaseSafe and then never executed them. test is now unit tests only -- 2m54s cold, 1m42s warm -- and the new gate step carries those two compile proofs.

The dst, bench and smoke UNIT tests deliberately stay on test. They are unit tests despite their siblings' names, zig build dst/smoke run the simulation and the interop check rather than them, and CI invokes the lanes one by one and never invokes gate -- so moving them would have dropped them from CI while every step still looked green. ci_test's declared_steps records gate as intentionally not-in-CI, and names that failure mode for the next lane someone adds. check-in: 7363238ad5 user: dmestas tags: trunk