| Comment: | merge logstore-slice2: the open-cost work reaches production
openReadOnly is what every CLI verb opens through, and until now it still checksummed every payload byte on every invocation -- so nothing users ran had gotten faster. It scans header-only now: 10k frames 42 MB 42.2ms -> 10.3ms 10k frames 410 MB 298.0ms -> 12.3ms 100k frames 416 MB 511.9ms -> 115.4ms 100 frames 419 MB 321.4ms -> 0.10ms Open cost is a function of frame count rather than repository size. On a store of a few large artifacts it is effectively free. This completes the sequence: vendor a faster crc, measure the path production actually takes, scan header-only on the append path, ship the fsck verb, then delete the eager check. Each step was gated on the one before it, and the deletion was deliberately last because it is the only one that gives something up. What is given up: opening a repository no longer detects payload corruption. It surfaces on the read instead -- readFrameAt crc-checks every frame unconditionally now -- or on demand through "zit verify". That is the split git draws between ordinary operation and fsck, and it was signed off knowing the tradeoff. A damaged store also stays usable for everything that does not touch the damaged bytes, which the old eager check made impossible. Slices 0 and 4 are closed too: the 4 KiB chunk widening landed inside verify, the only place that loop still runs, and the sidecar index is dead on its own gate -- the 100k-frame shape was to justify it by exceeding 150ms and came in at 119ms. mmap is dead alongside it, separately, because a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats reclamation rather than merely slowing it. Neither rejection is recorded at the logstore extension comment yet; that is the one piece of follow-up this leaves behind. Verified on the merged tree: fmt clean, test 2327/2327, gate 19/19. On the branch tip, additionally e2e 671/671 and dst 6 seeds x 1000 steps. |
|---|---|
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA3-256: |
6d9172d4886ea6ebf4c2d97a457d0452 |
| User & Date: | dmestas on 2026-07-31 21:15:04.320 |
| Other Links: | manifest | tags |
|
2026-08-01
| ||
| 14:46 |
merge issue22-diagnostics: reclone got HTTP 500 on a healthy store
ReleaseSafe asserts made the clone answer O(store), and serve's 5s deadline raced read + answer + write as one unit -- so a slow answer was cancelled mid-flight and the dead pread surfaced as error.Storage. Asserts gone (zit verify covers them); the deadline now races read and write separately. Not scale-dependent -- machine speed vs a wall clock. Stalled peer still cut at 5.0s. test/gate/e2e/dst green. check-in: 94bebf57f1 user: dmestas tags: trunk | |
| 07:22 |
merge trunk: land the logstore slices, repair the sweep's rot oracle
Trunk's Slice 1/2 made both open paths header-only, so a rotted payload now opens clean and surfaces at content() instead of at open. The storage sweep still assumed open was the tripwire, so two of its own claims were wrong once the merge combined them with this branch's store: - crashCheckOnce repaired only "if the open itself reports corruption", a precondition Slice 2 removed. Injected rot reached the invariant check and read as data loss. NeedsRepair, private to the harness, now carries "refused AND genuinely damaged" to the repair that already existed. - rotWithinFrame spelled the per-frame overhead as 22, the header alone, so a frame's own trailing frame_crc32 counted as inert pad. A bitflip in those four bytes makes a reader recompute a correct checksum, compare it to a corrupted stored one, and refuse the frame; the sweep called that data loss. logstore now exports frame_fixed_overhead and the sweep derives it instead of respelling it. Both are trunk defects, not merge damage: trunk alone fails dst at seed 8. The merge only shifted the draws far enough to reach the first one at seed 3, inside the default gate's range. cli.zig's verb enum was the single textual conflict: a pure union of trunk's verify and this branch's submanifest and import-git. fmt, test, gate, dst green. check-in: 584314dcb7 user: dmestas tags: sub-manifest | |
|
2026-07-31
| ||
| 21:51 |
sync: name the HTTP status the peer answered with
"the peer answered with an unexpected HTTP status" sends an investigation nowhere. 500 (the server could not build a reply), 413 (the request exceeded the body ceiling), 400 (a malformed container) and 417 (a rejected Expect: 100-continue) are four different bugs, and the message distinguished none of them. Issue #22 has been open on exactly that sentence. transport.zig discarded the code at the status check because Zig errors carry no payload and the closed error set is deliberate. Rather than widen TransportError with a variant per status, the status is left on the Transport as last_status for the caller to read after the error, and syncRunFail names it: "the peer answered HTTP 500 (Internal Server Error)". Immediately useful: re-running the issue #22 gate against the fossil corpus with this build turns its failure_reason from the old sentence into HTTP 500, which -- together with the reply-size arithmetic -- narrows the cause to OutOfMemory or ServerStorage and eliminates every 4xx path. The issue's own hypothesis, a per-round shun-snapshot readiness problem, is not among the candidates: that arm is documented unreachable and the code agrees. Also corrects three comments Slice 2 falsified, all on the same read path: logstore's open no longer CRC-verifies payloads, so the serve snapshot's contentImpl/wireImpl read faults are no longer "post-scan EIO or bitrot after a verified scan" -- ordinary stored corruption now reaches serve there, on the read. That is a better shape (one bad frame fails the round that touches it rather than making the server refuse to serve at all) but it makes that arm live traffic, and anything reasoning about how often it fires must start from that. runPack and runShun both claimed their read-only open was an "integrity pre-flight" that aborted a corrupt source before a byte was written. It no longer verifies payloads. The guarantee still holds -- packLog pulls every artifact through srcContent, which propagates error.Corrupt rather than skipping the frame, and readFrameAt CRC-checks every read, so a rotted payload still aborts with the temp deleted and .zit untouched -- but it is now collected during the copy loop instead of at open. Both verbs RENAME OVER the original, so runPack's comment now states the consequence plainly: if srcContent is ever changed to answer null on a corrupt read, the rewrite would silently omit the damaged artifact and then delete the only copy of it. Verified: fmt clean, test 2327/2327, gate, e2e 671/671. The new message checked against a stub server answering 500. check-in: b4637e26df user: dmestas tags: issue22-diagnostics | |
| 21:15 |
merge logstore-slice2: the open-cost work reaches production
openReadOnly is what every CLI verb opens through, and until now it still checksummed every payload byte on every invocation -- so nothing users ran had gotten faster. It scans header-only now: 10k frames 42 MB 42.2ms -> 10.3ms 10k frames 410 MB 298.0ms -> 12.3ms 100k frames 416 MB 511.9ms -> 115.4ms 100 frames 419 MB 321.4ms -> 0.10ms Open cost is a function of frame count rather than repository size. On a store of a few large artifacts it is effectively free. This completes the sequence: vendor a faster crc, measure the path production actually takes, scan header-only on the append path, ship the fsck verb, then delete the eager check. Each step was gated on the one before it, and the deletion was deliberately last because it is the only one that gives something up. What is given up: opening a repository no longer detects payload corruption. It surfaces on the read instead -- readFrameAt crc-checks every frame unconditionally now -- or on demand through "zit verify". That is the split git draws between ordinary operation and fsck, and it was signed off knowing the tradeoff. A damaged store also stays usable for everything that does not touch the damaged bytes, which the old eager check made impossible. Slices 0 and 4 are closed too: the 4 KiB chunk widening landed inside verify, the only place that loop still runs, and the sidecar index is dead on its own gate -- the 100k-frame shape was to justify it by exceeding 150ms and came in at 119ms. mmap is dead alongside it, separately, because a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats reclamation rather than merely slowing it. Neither rejection is recorded at the logstore extension comment yet; that is the one piece of follow-up this leaves behind. Verified on the merged tree: fmt clean, test 2327/2327, gate 19/19. On the branch tip, additionally e2e 671/671 and dst 6 seeds x 1000 steps. check-in: 6d9172d488 user: dmestas tags: trunk | |
| 21:12 |
logstore: openReadOnly scans header-only; verified_on_open is gone
The speedup reaches production. openReadOnly is the path every CLI verb takes -- Repository.open falls back to it, and Repository.logStore, the content channel behind cat, artifact, checkout and sync, calls it with no cache in front. It scanned every payload byte on every invocation: 10k frames 42 MB 42.2ms -> 10.3ms 10k frames 410 MB 298.0ms -> 12.3ms 100k frames 416 MB 511.9ms -> 115.4ms 100 frames 419 MB 321.4ms -> 0.10ms Open cost is now a function of frame count, not repository size. verified_on_open was the flag openReadOnly set to tell readFrameAt that the scan had already checked every frame, so reads could skip their own crc. With no scan doing that, the exemption had nothing left to rest on and the flag is deleted. readFrameAt now crc-checks unconditionally: it is the only thing standing between a rotted payload and a caller that believes it, and restoring any skip there means restoring the scan. Detection is relocated, not removed. A payload bitflip no longer voids the open -- the rest of the repository stays usable -- and surfaces the first time anyone reads those bytes: "zit cat" and "zit artifact" both refuse with 'repository is corrupt', and "zit verify" names the artifact and the offset. That verb shipped first for exactly this reason. openForVerify is deleted. It existed because openReadOnly both aborted on a damaged store and granted read-time trust, neither of which verify could accept; both properties are gone, it became byte-identical to openReadOnly, and its own doc comment said to remove it in this commit. The DST prove-it-can-fail arm is inverted rather than deleted. It asserted that openReadOnly WOULD serve latent rot silently, across a seed range where at least one corpus lands a payload rot on a raw frame whose header survived. That hazard is gone, so the same range now asserts the opposite: the read path never serves rot as success. trust_on_read goes with it -- there are no longer two paths to select between. Losing the eager whole-log scan cost nothing here, because per-read verification is what was catching this all along. Net -40 lines. Verified: fmt clean, test 2327/2327, gate, e2e 671/671, dst 6 seeds x 1000 steps. The Slice 1 tripwire -- the assertion that openReadOnly still rejected eagerly, written to fail if the flag was deleted without an fsck verb -- fired on this commit, which is the one it was aimed at. closed check-in: b224ffeb40 user: dmestas tags: logstore-slice2 | |
| 20:54 |
merge logstore-open-cost: header-only open scan and the verify verb
Opening a repository checksummed every payload byte, so open cost scaled with the whole store rather than with what the caller asked for. The append path now scans header-only: 10k frames 42 MB 42.2ms -> 11.2ms 10k frames 410 MB 298.0ms -> 12.7ms 100k frames 416 MB 511.9ms -> 119.2ms 100 frames 419 MB 321.4ms -> 0.10ms openReadOnly -- the path every CLI verb reaches -- deliberately still verifies payloads. Nothing has moved on the production path yet. That is Slice 2, and it is gated on this merge's other half: zit verify, the on-demand fsck that has to exist before eager detection is removed. The 100k-frame run settles a question the plan left open. Slice 4's gate was 'build the sidecar index only if that shape exceeds 150ms after the scan change'; it lands at 119ms. The sidecar is dead, and mmap with it -- separately, a mapping would keep shunned bytes resident and readable until every mapping dropped, which defeats the reclamation guarantee rather than merely slowing it. Both should be recorded as evaluated-and-rejected at the logstore extension comment; that is not done here. Also adds --mode append|read-only to the open benchmark. Every open number this effort had produced measured LogStore.open, which no CLI verb enters; the two modes agreed within noise before the scan change, which is what makes the after numbers above a comparison rather than two guesses. Verified on the merged tree: fmt clean, test 2327/2327. gate 19/19 and e2e 671/671 were watched green at the branch tip, which is content-identical -- trunk did not move between the branch point and this merge. check-in: 90bd1b8015 user: dmestas tags: trunk | |