50 descendants of ea53cd39834b17cf
|
2026-07-16
| ||
| 16:40 | Make pack monotonic: preserve stored wire deltas verbatim instead of reconstructing every frame to full form, which inflated a delta-bearing store many times over closed check-in: 50a4b5d049 user: dmestas tags: packfix | |
| 16:15 | Add a manifest-transform projection to the bench harness (zig build bench -- storage --probe-manifest): read-only, it estimates the compressed win from replacing resolvable 40/64-hex manifest hashes with rid-ordinal varints, so survey 2.5 is decided on a real zlib delta rather than a guess. closed check-in: 64fd19938f user: dmestas tags: probeman | |
| 15:32 |
Add a read-only --probe-dict projection to the storage bench to gate dictionary-frame work.
Before building dictionary-compressed full-form frames, measure the win. The new 'zig build bench -- storage --probe-dict' mode (optional .zit path) samples a deterministic content-addressed dictionary from the corpus: up to 64 evenly spaced other-class artifacts plus 16 manifests, capped at 256 KiB. For every full-form zlib frame it then projects storing the artifact as a fossil delta against that dictionary: it deltas the content, verifies delta.apply reconstructs the bytes exactly, zlib-wraps the delta as a section 4.1 container, and keeps the projection only when it strictly beats the current frame. The one-time dictionary cost is included and every non-reconstructing or unprofitable frame is counted at its current size, so the number is honest, not a best case. It stays read-only: the store is opened through openReadOnly and never written. A clear PROCEED/DEFER line reports the projected reduction against a 15 percent gate. Technique after Yann Collet's zstd dictionary training (https://github.com/facebook/zstd) and the RFC 1950 section 2.2 FDICT preset- dictionary concept (https://www.rfc-editor.org/rfc/rfc1950), with results validated by Chrome's shared-dictionary transport deployment (https://developer.chrome.com/blog/shared-dictionary-compression). The matcher is Fossil's own delta format, so the projected bytes at rest stay a section 4.1 zlib container and no new codec is introduced. closed check-in: c9bea85ebc user: dmestas tags: probedict | |
| 14:54 | Add a storage measurement mode to the bench harness (zig build bench -- storage): report bytes at rest by frame kind and artifact type, per-frame overhead, full-form compression headroom, and serve CPU, so storage-efficiency work is chosen on measured data rather than guesses. check-in: a48a0010ef user: dmestas tags: trunk | |
| 14:51 | Add a storage bench mode reporting bytes at rest by frame-kind and artifact-type plus serve CPU, so the storage-efficiency ideas can be weighed against measured numbers before any claim. closed check-in: e782228b2d user: dmestas tags: sbench | |
| 14:25 | Add a zit pack command that repacks the store into a compact new-generation log: whole-frame-verify the source read-only, re-ingest every artifact in offset order re-deltaing full-form frames against lineage bases, then atomically swap via a fsynced temp — a crash before the rename leaves the original byte-identical, and it doubles as an integrity scrub. check-in: de49e67c9f user: dmestas tags: trunk | |
| 14:16 | Add a zit pack verb that repacks the store into a compact new generation: re-ingest every artifact in offset order through a parent-manifest and same-path-blob lineage hint so full-form frames re-delta against their basis, recompute each reconstruction's hash against its name before writing it (repo-state §7.7), and swap the rewritten log in with an atomic rename so a crash before the swap leaves the original store byte-identical. closed check-in: 43dccab01c user: dmestas tags: pack | |
| 13:38 | Store cloned artifacts as compact delta frames: retain fossil's reverse-delta containers through the dangling queue and persist each basis before the delta that points at it, so a clone from a fossil origin keeps the server's deltas instead of re-inflating every artifact to full form — cutting on-disk size to roughly 0.6x of the full-form store, matching fossil's own delta count. check-in: 9853f8b93e user: dmestas tags: trunk | |
| 13:28 | Keep fossil's reverse-delta containers through the dangling queue and persist each basis frame before the delta that points at it, so a clone from a fossil origin stores compact delta frames instead of re-inflating every artifact to full form. closed check-in: 0ebc7991f0 user: dmestas tags: storage | |
| 13:26 | Merge the sync-protocol re-sync onto trunk: ratify the container-capacity and clone-igot clarifications that match the reference implementation and zit's own behavior. check-in: 61fde8dec1 user: dmestas tags: trunk | |
| 13:23 | Re-sync the sync-protocol draft: the container length field is a receiver capacity, so a decoded stream shorter than the advertised value must not be rejected (4.1), and igot is never applicable during clone rounds (8.2 / Table 2) — both aligning the text with the reference implementation. closed check-in: d708b4a2f2 user: dmestas tags: rfc-resync | |
| 11:26 | Stop the clone client re-announcing every held artifact with igot each round: a clone enables no push direction (sync 8.2/8.5) so the server ignores those cards, and dropping them cuts a 50k-artifact clone request from ~88 MB to under 2 KB while convergence still rides the server's RID cursor. check-in: 289e073d9c user: dmestas tags: trunk | |
| 11:16 | Stop the clone client re-announcing every held artifact with igot each round: a clone enables no push direction (RFC 8.2/8.5) so the server ignores those cards, and dropping them cuts a 50k-artifact clone request from ~88 MB to under 2 KB. closed check-in: ebc8401895 user: dmestas tags: igot | |
| 10:33 | Wire zit_storage and zit_logstore into the fuzz executable so the seeded fuzz suite compiles again: its sim helpers and logstore targets import those modules, which every other build artifact already provides but the fuzz exe was missing. check-in: 490977ea08 user: dmestas tags: trunk | |
| 10:25 | Wire zit_storage and zit_logstore into the fuzz executable so the seeded fuzz suite compiles again: its sim helpers and logstore targets import those modules, which every other build artifact already provides but the fuzz exe was missing. closed check-in: 1c7715c645 user: dmestas tags: fuzzbuild | |
| 09:58 | Add --no-cache to the read-only repo-state verbs (leaves, branch, info, tag): bypass the .zit-cache memo and force a full-log re-derive, giving a first-class integrity/debug escape hatch that re-verifies the whole log instead of trusting the O(1) tail check. check-in: 5af2298bc9 user: dmestas tags: trunk | |
| 09:55 | Add --no-cache to the read-only repo-state verbs to bypass the persisted cache and force a full-log re-derive that re-verifies whole-log integrity closed check-in: 6e3529ad92 user: dmestas tags: nocache | |
| 09:40 | Add a persisted derivation cache so repo-state verbs skip re-deriving all of history: cache the crosslink inputs keyed on the append-only log's tail and replay them on a hit, cutting a warm 'zit leaves' on a 130k-artifact repo from ~103s to 0.21s with byte-identical output and a full rebuild on any log change or corruption. check-in: e1f401fa0d user: dmestas tags: trunk | |
| 09:38 | Document that a cache hit skips whole-log integrity, so interior bitrot preserving size and tail is served until a rebuild — deleting .zit-cache restores full detection. closed check-in: 31e140ddbc user: dmestas tags: cache | |
| 09:28 | Memoize the derived repo view in a disposable .zit-cache sidecar so read-only verbs answer sub-second at fossil scale instead of rescanning the whole log on every invocation. check-in: 08504d5f5d user: dmestas tags: cache | |
| 07:28 | Extract the sync 4.1 zlib container codec into one leaf module: the three copies had drifted — logstore and repostate rejected valid containers whose zlib stream decodes to fewer bytes than the declared capacity, stricter than 4.1's output-capacity semantics — so unify on the spec-correct decode and retire the duplication. check-in: 0b8ff7a1f8 user: dmestas tags: trunk | |
| 07:23 | Extract the §4.1 zlib container codec into one leaf module (src/container.zig) so logstore, repostate, and sync share a single §4.1-capacity-semantics implementation instead of three drifting copies. closed check-in: 0e5cc09150 user: dmestas tags: container | |
| 07:05 | Retain received cfiles verbatim on the clone client: store each artifact's received container or delta as-is after verifying its hash over the reconstructed content, so delta compactness survives clone-of-clone and client disk roughly halves — and reject a lying-named push the server previously accepted. check-in: a5ef5d18c0 user: dmestas tags: trunk | |
| 06:53 | Retain received cfile containers and deltas verbatim on the clone client so delta compactness survives a clone-of-clone, verifying each reconstructed artifact's hash at the persist commit hook before storing the wire form rather than re-expanding every received delta to full form. closed check-in: 5526cf3536 user: dmestas tags: d4 | |
| 06:13 | Speed up repo-state derivation on open: replace the O(N-squared) crosslink scans with parent/child/tag adjacency indexes, verify each frame's whole-frame CRC once at scan instead of twice per verb, and cache delta bases during the derivation sweep — cutting the algorithmic crosslink term over 100x with byte-identical derived state. check-in: baa7053ee1 user: dmestas tags: trunk | |
| 06:11 | Assert a read-only-verified store is never appended to: the append frame path skips the redundant whole-frame CRC only because openReadOnly proved it at scan, so appending to such a store would leave a frame unchecked. closed check-in: 7c3d36865e user: dmestas tags: scale | |
| 06:00 | Accelerate repo-state derivation on open: replace the quadratic crosslink scans with per-parent/child/rid adjacency indexes, verify each frame's CRC once at open instead of again on every read, and memoize delta bases across the derivation sweep, so the query verbs stay fast on large repositories without changing any derived answer. check-in: 76b0f8f1d0 user: dmestas tags: scale | |
| 05:04 | Delta-encode cfiles on clone: store an artifact as a delta against its previous version when materially smaller, and serve that stored delta container verbatim with the reconstructed-size USIZE per sync 7.2, cutting clone reply size 58-87% on version-chained repos while full-form artifacts still serve unchanged. check-in: d7e16fadb1 user: dmestas tags: trunk | |
| 05:01 | Drop the unused putContainer store method and its two tests: its only intended caller is the deferred client-verbatim persist hook, so it stays dead surface until that lands. closed check-in: 673d5070bb user: dmestas tags: delta | |
| 04:50 | Use the known reconstructed size for a delta corpus artifact's USIZE instead of re-parsing the delta header: the bench just created the delta from cur, so its reconstructed target is cur and its length is the section 7.2 USIZE directly, dropping a redundant header parse. check-in: 3bdb4f96c0 user: dmestas tags: delta | |
| 04:47 | Delta-encode blobs and manifests on ingest and serve them verbatim as section 7.2 delta cfiles on clone: store a commit's blob (against the same path's parent blob) and manifest (against the primary parent) as a delta when it comes in materially smaller and within the write-depth cap, carry the reconstructed-target size from the delta header as the cfile USIZE rather than the container's raw-delta-length prefix, and retransmit the stored delta container byte-for-byte so a chained-corpus clone reply drops 58-87 percent while a full-form corpus is byte-identical to before. check-in: 71235bf3be user: dmestas tags: delta | |
| 03:57 | Serve stored cfile containers verbatim on clone: reuse each artifact's already-compressed 4.1 container instead of recompressing it on every clone reply, moving the per-clone compression cost off the serve path to ingest and cutting clone-serve time ~76% with byte-identical wire output. check-in: f4fe62b160 user: dmestas tags: trunk | |
| 03:51 | Restrict verbatim cfile serving to the full form: the delta serve path had no production writer and wrote the delta's length as USIZE instead of the reconstructed artifact size (sync §7.2), so drop it and let delta-stored artifacts fall back to a conformant reconstructed full-form cfile. closed check-in: 8e430e7bd7 user: dmestas tags: perf | |
| 03:24 | Serve stored cfile containers verbatim on clone, reusing each artifact's already-compressed container instead of recompressing every clone reply so the per-clone compression cost moves off the serve path to ingest. check-in: 84d47c3a21 user: dmestas tags: perf | |
| 02:41 | Land the out-of-band sync clone benchmark on trunk: it splits clone time into build/frame/decode/respond/reframe/consume/persist/reopen phases behind a convergence-verified in-process clone, and attributes per-artifact cfile compression as a faithful share of the server respond phase, so the first clone-speed optimization targets measured work rather than a guess. check-in: 5f6f1286d1 user: dmestas tags: trunk | |
| 02:35 | Report the sync clone benchmark's server phase honestly: drop the unmeasurable walk+compress vs decode+parse split and show only the standalone compression micro as a share of respond, since scanRequest and emitCloneReply are private so parse and walk can't be told apart at the public boundary. closed check-in: 53962101c0 user: dmestas tags: bench | |
| 02:10 | Split the sync benchmark's server phase into timed decode, respond and reframe steps plus a standalone cfile-compression measurement, so clone-speed work can tell inbound igot decode+parse apart from the actual per-artifact compression instead of trusting one opaque server label. check-in: 997edb13b5 user: dmestas tags: bench | |
| 01:44 | Add an out-of-band sync clone benchmark that splits round time by phase and tallies card/byte counters behind a convergence-verified clone, so clone-speed work starts from measured attribution rather than a guess. check-in: a5d904a3ef user: dmestas tags: bench | |
| 00:53 | Merge trunk back onto cli after the growable delta queue, aligning the branch tip with trunk. check-in: a5260cc2eb user: dmestas tags: cli | |
| 00:44 | Merge the growable pending-delta queue onto trunk: heap-backed and bounded by the message-body cap instead of a fixed 256 slots, so a real-world clone's thousands of dangling deltas per sync 7.4 are all retained and reconstructed — verified pulling multi-GB of fossil's own and sqlite's source past the former wall. check-in: a070f7fcb5 user: dmestas tags: trunk | |
| 00:36 | Grow the pending-delta queue on the heap bounded by the message-body cap instead of a fixed 256 slots, so a real clone's many dangling deltas per sync 7.4 are all retained and reconstructed rather than aborting the exchange once past 256. check-in: 8af6244a64 user: dmestas tags: cli | |
|
2026-07-15
| ||
| 22:37 | Merge the login-card polish onto trunk: pin the section A.5 auth vectors directly against the RFC bytes, scrub the plaintext password buffer after deriving the secret, and cover the legacy pre-credential config decode. check-in: 22d0c3de75 user: dmestas tags: trunk | |
| 22:31 | Pin the section A.5 login vectors directly, zero the plaintext password buffer after deriving the secret, and cover the legacy config decode path check-in: 91988dd110 user: dmestas tags: cli | |
| 22:23 | Merge trunk back onto cli after the login card. check-in: 56fbf01f2d user: dmestas tags: cli | |
| 22:23 | Merge the section 6 client login card onto trunk: zit push signs a login card with the derived shared secret so a fossil server authorizes the write, persisting only the 40-hex derived secret at 0600 and never the password — completing authenticated push, the last interop direction. check-in: 52cb4d83bf user: dmestas tags: trunk | |
| 22:11 | Sign push login cards with the derived shared secret so a fossil server authorizes the write, persisting only the 40-hex secret at 0600 and never the password check-in: f21e962a00 user: dmestas tags: cli | |
| 20:32 | Merge trunk back onto cli after server delta ingest. check-in: b3167f749e user: dmestas tags: cli | |
| 20:31 | Merge server-side delta ingest onto trunk: the server push path reconstructs a delta-form file card against its held basis per section 7.4 with section 7.7 integrity, then persists it, so a fossil push of a delta-encoded manifest lands on zit serve. check-in: b49f1fa330 user: dmestas tags: trunk | |
| 20:28 | Accept a delta-form file card on the server push-ingest path: reconstruct against the held basis per §7.4, verify §7.7 integrity, then persist, so a fossil push of a delta-encoded manifest lands on zit serve check-in: 6a4dd5b819 user: dmestas tags: cli | |
| 20:02 | Merge trunk back onto cli after delta reception, aligning for the push interop probe. check-in: 44be941a4f user: dmestas tags: cli | |
| 20:02 | Merge section 7.4 delta reception onto trunk: reconstruct delta-form file and cfile cards against their basis with pre-store hash verification, defer a dangling delta in the pending queue and complete the chain to a fixpoint when the basis arrives, so an incremental pull of fossil's delta-encoded manifests converges. check-in: ea53cd3983 user: dmestas tags: trunk | |