Timeline

Opening a repository checksummed every payload byte, so open cost scaled with the whole store rather than with what the caller asked for. The append path now scans header-only:

10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms

openReadOnly -- the path every CLI verb reaches -- deliberately still verifies payloads. Nothing has moved on the production path yet. That is Slice 2, and it is gated on this merge's other half: zit verify, the on-demand fsck that has to exist before eager detection is removed.

The 100k-frame run settles a question the plan left open. Slice 4's gate was 'build the sidecar index only if that shape exceeds 150ms after the scan change'; it lands at 119ms. The sidecar is dead, and mmap with it -- separately, a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats the reclamation guarantee rather than merely slowing it. Both should be recorded as evaluated-and-rejected at the logstore extension comment; that is not done here.

Also adds --mode append|read-only to the open benchmark. Every open number this effort had produced measured LogStore.open, which no CLI verb enters; the two modes agreed within noise before the scan change, which is what makes the after numbers above a comparison rather than two guesses.

Verified on the merged tree: fmt clean, test 2327/2327. gate 19/19 and e2e 671/671 were watched green at the branch tip, which is content-identical -- trunk did not move between the branch point and this merge. check-in: 90bd1b8015 user: dmestas tags: trunk

Slice 2 will extend header-only scanning to openReadOnly and delete verified_on_open, which removes eager corruption detection entirely. This is the on-demand replacement that has to exist first -- the same split git makes between normal operation and git fsck.

Two passes over the store. FrameVerifier walks every frame under .verify_payloads, which is the mode the open paths are shedding; it is built on StorageFile rather than an opened LogStore, so it still works on a store that will not open -- the case an fsck exists for. Then openForVerify opens .headers_only and reconstructs each artifact, which catches a frame whose checksum is honest about bytes that are not what their name claims.

Both passes report rather than abort. A payload CRC fault carries a header-proven frame_len, so the walk steps over it and inventories the rest; only a header-window fault, whose length cannot be trusted, stops the walk. A torn tail is not a fault -- open recovers it silently, and it means an interrupted write, not rot. An unreadable payload is a finding too, not the end of the inventory.

Exit codes stay 0/1/2 as every other verb documents them; 2 remains usage error and never corruption. Findings stream to stderr as they are found, the clean summary goes to stdout, and on any fault stdout is empty.

Scope stops at content artifacts. A manifest whose bytes hash correctly is byte-identical to what commit wrote and what sync validated on ingest, so re-parsing it would catch a zit bug rather than storage rot. The help text says so.

openForVerify is deliberately not openReadOnly: openReadOnly sets verified_on_open, which makes readFrameAt skip the per-read whole-frame CRC, and it scans .verify_payloads, so on exactly the damaged store an operator is fsck-ing it aborts before pass 2 runs. Slice 2 makes the two identical and openForVerify is deleted in that commit; its doc comment says so.

The chunk buffer in scanOneFrame's payload loop widens from 4 KiB to 64 KiB, matching cloneInto's existing precedent. Post-Slice-2 that loop runs only under verify, where an operator's wall-clock is what it costs.

zit verify is not an RFC-mandated verb. repo-state 7.3 blesses full rebuild as a fallback, but the verb's shape is a local design decision, not a protocol requirement.

Verified: fmt clean, test 2327/2327, gate 19/19, e2e 671/671. Four mutants, all compiling, each red for its own semantic reason: pointing the frame pass at .headers_only, stubbing artifact.verify to true, dropping the unreadable-store tally, and reverting the payload read to a hard error. Exit arms smoke-checked against the built binary. closed check-in: bcb9f8eb6e user: dmestas tags: logstore-open-cost

The open-time scan streamed every payload byte through the crc, making open cost O(repository bytes) rather than O(what the caller asked for). ScanMode.headers_only validates each frame's header window and then steps over the payload by frameSpan arithmetic, leaving open cost O(frame count).

Measured, same binary, both modes in one run:

10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms

Every fault that lives in a header window is still caught: readFrameHeader runs unchanged, so a bad prefix crc, a bad header crc, an out-of-range hash_len and an unknown flag kind all still void the open. Torn-tail detection survives intact -- a torn frame is a clean prefix, never damaged bytes, so offset + frame_len > total catches every truncation the payload and trailer short-read checks used to, using a length the header window already proved trustworthy. What is no longer caught at open is a bitflip in already-landed payload bytes, which becomes content()'s job; it crc-checks the frame it reads regardless.

openReadOnly -- the path every CLI verb reaches -- deliberately keeps .verify_payloads. No guarantee has moved on the production path yet, and none should until the explicit fsck verb exists to replace it. This half is what makes the cost measurable without taking on that decision.

The one test that changed is the one that had to: a corrupted payload on a complete frame no longer voids open. It now asserts the fault surfaces on the first content() read AND that openReadOnly still rejects the file eagerly. That second assertion is the tripwire for deleting verified_on_open without shipping the verb first. check-in: bae910f00e user: dmestas tags: logstore-open-cost

benchOpen timed LogStore.open -- the append path. Production never takes it: Repository.open falls back to openReadOnly, and Repository.logStore, the lazy content channel behind blob, cat, checkout and sync, calls openReadOnly with no cache in front. Every open number this effort has produced described a path no CLI verb enters.

That was survivable while the two paths ran the same scan. It stops being survivable the moment either one scans header-only, at which point the bench would report the optimized path while production still paid the full cost. --mode append|read-only defaults to append so existing invocations report unchanged numbers, and the mode is in the output line because the two are only comparable when a captured log can tell them apart.

Calibration on this tree, before any scan change: the modes agree within noise on all four shapes -- run B is 1% apart. That agreement is the check that the flag measures what it claims. check-in: e56a663b7a user: dmestas tags: logstore-open-cost

zig build test was 9m13s, and the tests it actually ran were a small fraction of that: it also compiled the e2e and smoke harnesses in the pinned ReleaseSafe and then never executed them. test is now unit tests only -- 2m54s cold, 1m42s warm -- and the new gate step carries those two compile proofs.

The dst, bench and smoke UNIT tests deliberately stay on test. They are unit tests despite their siblings' names, zig build dst/smoke run the simulation and the interop check rather than them, and CI invokes the lanes one by one and never invokes gate -- so moving them would have dropped them from CI while every step still looked green. ci_test's declared_steps records gate as intentionally not-in-CI, and names that failure mode for the next lane someone adds. check-in: 7363238ad5 user: dmestas tags: trunk