Artifact 3b2c6fca9de70acd5c9f670f0066fff13b196da23a9bdb2ae6e9ad2c36010a83:
- File
docs/plans/bench/step3-pack-redeltify-unstamped-run.json
— part of check-in
[efaf2d3ae1]
at
2026-07-18 08:21:09
on branch storage
— feat(§6): pack re-deltification with chronological+wire-delta rebasing and monotonic-shrink guard
fossil-scm-clone: zit=110083984B fossil=72245248B ratio=1.5238 (packed store, exact accounting header64+zlib+delta+config==110083984: zlib 8377 frames / 84,763,872 B 77.0% avg 10118 B; delta 59122 frames / 25,319,960 B 23.0% avg 428 B; config 1 frame / 88 B; raw 0. Artifact-type: checkin/manifest 20848 / 23,816,728 B 21.6%, other 6929 / 4,706,432 B 4.3%, file-blob 39722 / 81,560,672 B 74.1%. As-cloned baseline was 117,528,088 B zlib 9647/92,073,728; delta 57852/25,454,120; ratio 1.6268.)
sqlite: zit=175246544B fossil=147709952B ratio=1.1864 (packed store, exact accounting header64+zlib+delta+config==175246544: zlib 9260 frames / 113,048,584 B 64.5% avg 12208 B; delta 121734 frames / 62,197,808 B 35.5% avg 510 B; config 1 frame / 88 B; raw 0. Artifact-type: checkin/manifest 36515 / 65,427,440 B 37.3%, other 4775 / 5,462,856 B 3.1%, file-blob 89704 / 104,356,096 B 59.5%. As-cloned baseline was 207,344,544 B zlib 10847/140,422,768; delta 120147/66,921,536; ratio 1.4037.)
vs baseline:
fossil-scm-clone: zit=117528088B fossil=72245248B ratio=1.627 (raw=0B/0f; zlib=92073728B/9647f 78.3%; delta=25454120B/57852f 21.7%; config=176B/2f. Header+padding overhead 3711845B 3.2%. Artifact-type: checkin/manifest 26721968B/20848, other 4835264B/6929, file-blob 85970616B/39722. Accounting exact.)
sqlite: zit=207344544B fossil=147709952B ratio=1.404 (raw=0B/0f; zlib=140422768B/10847f 67.7%; delta=66921536B/120147f 32.3%; config=176B/2f. Header+padding overhead 7212808B 3.5%. Artifact-type: checkin/manifest 81006048B/36515, other 5531960B/4775, file-blob 120806296B/89704. Accounting exact.) (user: dmestas, size: 18686) [annotate] [blame] [check-ins using] [more...]
{ "schema": "zit-storage-pack-redeltify/1", "plan": "docs/plans/storage-delta-base-selection.md", "plan_section": "§4 (bench harness) re-run at the §8 step-3 checkpoint (pack re-deltification pass — the extra_deltification analog per §6.1), measured against the §8 step-1 baseline and the §8 step-2 (write-path pickBase) run.", "step": "8/step-3 — pack re-deltification present in the working tree (src/pack.zig packLog re-deltas full-form frames and re-bases verbatim wire deltas via pickBase, with the §6.1 monotonic-shrink guard keeping the smaller of {re-encoded, existing} frame). CLONE/SYNC ingest, then `zit pack`, then at-rest re-measure.", "stamped": false, "stamped_note": "File name is 'unstamped': the fossil-bench corpora are NOT yet pinned (corpora/corpora.json trunk_hash == null for both). Observed trunk-leaf hashes are recorded below for reproducibility, but a future run could see upstream drift until these are written back into corpora/corpora.json as pins per corpora/snapshot.sh.", "key_finding": { "summary": "MEASURED NET WIN on both clone-heavy corpora. With the §6.1 monotonic-shrink guard now present in src/pack.zig, the step-3 `zit pack` re-deltification pass made the store SMALLER than the as-cloned store on BOTH corpora: fossil-scm-clone 117,528,088 B -> 110,083,984 B (-6.3%, ratio 1.6268 -> 1.5238 of fossil); sqlite 207,344,544 B -> 175,246,544 B (-15.5%, ratio 1.4037 -> 1.1864 of fossil). A fresh CLONE stores each artifact as fossil's own wire delta VERBATIM (putDeltaContainer); pack decodes every wire delta to full bytes and re-deltifies through zit's pickBase, and — critically — keeps the re-encoded frame ONLY when it is smaller than the existing stored frame (monotonic-shrink guard). That guard makes pack non-worsening per frame, so the whole-store size can only stay flat or shrink; where pickBase's in-window candidate set (lineage parent + chronological window + same-path predecessors) finds a base that beats fossil's original choice, the frame shrinks; where it does not, fossil's verbatim wire delta is retained. Net effect: both frame classes shrank on both corpora (fscm delta avg 439 B -> 428 B, zlib avg 9544 B -> 10118 B but fewer full-form frames 9647 -> 8377; sqlite delta avg 556 B -> 510 B, zlib frames 10847 -> 9260). This DIRECTLY CONTRADICTS the earlier draft of this file (a prior working-tree state without the guard, which regressed fscm to 156,635,392 B); the guard is the difference and this run supersedes that draft. The §3 correctness invariant held throughout (see correctness_gate): every frame was hash-verified during pack and the pre-pack and post-pack leaf sets are byte-identical.", "pack_redeltify_present_in_tree": true, "monotonic_shrink_guard_present": true, "packed_store_smaller_than_ascloned": true, "regression_direction": "packed < as-cloned on BOTH corpora (improvement)", "accept_threshold": "<75% — delta_len*4 < target_len*3 integer math (src/logstore.zig), fossil content.c:916 parity", "root_cause_of_win": "pickBase re-deltification plus the §6.1 monotonic-shrink guard: re-encode wins only when strictly smaller than the retained fossil wire delta, so the store is monotonically non-worsening and shrinks wherever the in-window base search improves on fossil's original base choice. sqlite (delta-heavy, deep same-path histories) benefits most (-15.5%); fossil-scm-clone less (-6.3%).", "step6_decision_gate": "§8 step-6 gate is 'packed zit > ~1.3x fossil on EITHER corpus => evaluate Odess/Finesse'. fossil-scm-clone packed ratio 1.5238 STILL exceeds 1.3 (gate tripped by that corpus); sqlite packed ratio 1.1864 is under 1.3. §6.2 same-path candidates (step 4) are the plan's next lever for the fossil-scm-clone remainder." }, "provenance": { "generated_utc": "2026-07-18", "host": { "os": "macOS (Darwin 25.5.0)", "kernel": "Darwin", "arch": "arm64" }, "tool_versions": { "zit": "0.0.0 (targets Fossil RFC rev 00), built -Doptimize=ReleaseFast from the storage worktree working tree (uncommitted step-3 pack changes atop commit d859c6a9)", "fossil": "2.28 [52445a27f1] 2026-03-11", "zig": "0.16.0" }, "zit_checkout": "d859c6a9a5a5e83f067445193e62d6cc10bdc5d0 (branch storage) HEAD, PLUS uncommitted working-tree edits: src/pack.zig (new, untracked), src/logstore.zig, src/main.zig, src/dst.zig, src/fuzz_tests.zig, src/logstore_fuzz.zig, src/root.zig, src/sim/storage_sweep.zig, src/smoke.zig, build.zig", "working_tree_dirty": true, "harnesses": [ "fossil-bench @ /Users/dmestas/fossil-bench (drivers/DRIVER_SPEC.md); loopback fossil server (fossil server --localhost) serving each snapshot from corpora/_snapshots/; zit clone / fossil clone off the same served corpus; store-size = raw file bytes over the destination store. NOTE: drivers/zit/driver.sh store_file() resolves '.zit.fossil' but the zit clone store file is named '.zit' — this run measured the actual '.zit' file (wc -c) directly.", "zig build bench -- storage <path> (src/storage_benchmark.zig): frame-class bytes-at-rest + artifact-type split + full-form compression headroom + serve CPU, read-only over the physical .zit log; openReadOnly whole-frame-CRC-verifies every frame first." ], "comparison_class": "approximate", "comparison_note": "Total store size is not like-for-like across store shapes (fossil-bench clone_store_total_bytes caveat / DRIVER_SPEC store-content-size unsupported by both drivers). fossil's number is its whole .fossil (content blobs + SQLite b-tree indices + derived tables in one file); zit's headline is the whole .zit log (content frames + 64-byte header). zit keeps its derived index in a SEPARATE .zit-cache file (reported below), which fossil folds into its single file — so an index-inclusive total is also given. Honest comparison class: 'approximate', not 'like-for-like'." }, "ingest_path": { "method": "CLONE/SYNC — fossil server serves the snapshot over loopback; both zit and fossil clone it anonymously. This exercises zit putDeltaContainer verbatim-frame storage (the production ingest path per plan §1/§5.2.3): the peer's wire deltas land at rest verbatim, so the sender's base choice is inherited. THEN `zit pack` runs the step-3 re-deltification pass over that store.", "both_impls_cloned_same_served_corpus": true, "pack_run": "zit pack (default behavior, no flag) in the cloned working directory; atomic temp+rename; every frame hash-verified during pack; monotonic-shrink guard keeps min(re-encoded, existing) per frame." }, "harness_gaps": { "note": "Rule 7 — reported only what the harness measured. The following requested metrics are NOT emitted by the current storage harness and were NOT synthesized:", "chain_depth_p50_p99": "NOT MEASURED. src/storage_benchmark.zig emits no delta-chain-depth histogram; plan §4.3 lists this as a metric to ADD ('chain-depth histogram') and it has not been added at this checkout. No p50/p99 fabricated.", "put_us_per_artifact": "NOT MEASURED by the storage harness. 'zig build bench -- storage' times SERVE CPU only (produce §4.1/§7.2 wire form), not the put path. A pure put microbenchmark exists (zig build bench -- put) but runs a SYNTHETIC seeded corpus, not the CLONE-ingested real corpus, so it is not substituted here (Rule 7). A clone-ingest proxy (network + decode + put, NOT a pure put) and a pack-throughput proxy (pack wall / artifacts) are recorded per corpus, each clearly labeled a proxy.", "per_frame_delta_vs_base_ratio": "NOT MEASURED (plan §4.3 second add). Not synthesized." }, "corpora": [ { "corpus": "fossil-scm-clone", "fossil_bench_name": "fossil", "tier": "medium", "source": "https://fossil-scm.org/home", "observed_trunk_leaf_hash": "61998ec8aa4f567cbb21f9ad3eeac11e8320184c1d3c822c5a4f3d82dcdaa3cd", "leaves_match_fossil": true, "pinned": false, "artifacts_cloned": 67499, "correctness_gate": "PASSED — `zit leaves` BEFORE pack and AFTER pack are byte-identical (leaf-set sha256 8abd1afb4f64a6b4527e5bab5c4c4190fdc3cc4a93448a847c2e38e2f3f4f9c1 both times; trunk leaf 61998ec8...); invariant §3 byte-exact content round-trip + trunk hash preserved across the at-rest transform; pack additionally hash-verified every reconstructed frame while writing the new generation.", "zit_ascloned": { "at_rest_bytes_total": 117528088, "zit_cache_bytes": 4950808, "at_rest_bytes_index_inclusive": 122478896, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 9647, "on_disk_bytes": 92073728, "pct_store": 78.3, "avg_bytes_per_frame": 9544 }, "delta": { "frames": 57852, "on_disk_bytes": 25454120, "pct_store": 21.7, "avg_bytes_per_frame": 439 }, "config": { "frames": 2, "on_disk_bytes": 176, "pct_store": 0.0 } }, "frame_header_overhead_bytes": 3711845, "accounting": "exact (header 64 + zlib + delta + config == 117528088)", "artifact_type_bytes": { "checkin_manifest": { "count": 20848, "on_disk_bytes": 26721968, "pct_store": 22.7 }, "other_artifact": { "count": 6929, "on_disk_bytes": 4835264, "pct_store": 4.1 }, "file_blob": { "count": 39722, "on_disk_bytes": 85970616, "pct_store": 73.1 } }, "serve_cpu": { "artifacts": 67499, "total_ms": 550.35, "us_per_artifact": 8.154, "mb_per_s": 197.2 }, "ratio_zit_over_fossil": 1.6268 }, "zit_packed": { "at_rest_bytes_total": 110083984, "zit_cache_bytes": 4950808, "at_rest_bytes_index_inclusive": 115034792, "pack_stats_line": "packed: 67499 artifacts, 59122 re-deltaed, 117528088 -> 110083984 bytes", "artifacts_redeltaed": 59122, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 8377, "on_disk_bytes": 84763872, "pct_store": 77.0, "avg_bytes_per_frame": 10118 }, "delta": { "frames": 59122, "on_disk_bytes": 25319960, "pct_store": 23.0, "avg_bytes_per_frame": 428 }, "config": { "frames": 1, "on_disk_bytes": 88, "pct_store": 0.0 } }, "frame_header_overhead_bytes": 3710120, "accounting": "exact (header 64 + zlib + delta + config == 110083984)", "artifact_type_bytes": { "checkin_manifest": { "count": 20848, "on_disk_bytes": 23816728, "pct_store": 21.6 }, "other_artifact": { "count": 6929, "on_disk_bytes": 4706432, "pct_store": 4.3 }, "file_blob": { "count": 39722, "on_disk_bytes": 81560672, "pct_store": 74.1 } }, "full_form_compression_headroom": { "sampled_frames": 1024, "uncompressed": 2245879, "compressed": 987041, "ratio_pct": 43.9 }, "serve_cpu": { "artifacts": 67499, "total_ms": 528.23, "us_per_artifact": 7.826, "mb_per_s": 192.0 }, "ratio_zit_over_fossil": 1.5238, "ratio_zit_index_inclusive_over_fossil": 1.5923, "delta_vs_ascloned": { "total_bytes_change": -7444104, "total_pct_change": -6.3, "zlib_bytes_change": -7309856, "zlib_pct_change": -7.9, "delta_bytes_change": -134160, "delta_pct_change": -0.5, "note": "Store shrank. zlib full-form frames 9647 -> 8377 (some full-form artifacts now delta-encode against an in-window base); delta avg 439 B -> 428 B. Monotonic-shrink guard retained fossil's verbatim wire delta wherever the re-encode was not smaller." } }, "fossil": { "at_rest_bytes_total": 72245248, "clone_wall_s": 20.25, "measure": "wc -c of a fresh `fossil clone` destination (.fossil) off the same served corpus, no post-clone rebuild/vacuum" }, "pack_throughput_proxy": { "note": "NOT the harness put metric. pack wall / artifacts over the whole re-deltification pass (decode-all + up-to-5 delta.create/artifact + guard-compare + write + fsync). Labeled proxy only.", "pack_wall_s": 338.22, "us_per_artifact": 5010.7 }, "clone_ingest_proxy": { "note": "NOT the harness put metric — clone wall / artifacts (network + decode + put). Labeled proxy only.", "zit_clone_wall_s": 53.52, "zit_us_per_artifact": 792.9, "fossil_clone_wall_s": 20.25 } }, { "corpus": "sqlite", "fossil_bench_name": "sqlite", "tier": "large", "source": "https://sqlite.org/src", "observed_trunk_leaf_hash": "6924895d2cbc50971d8da1810423ad15c638dabc31b4f1083dd6076e20492ef4", "leaves_match_fossil": true, "pinned": false, "artifacts_cloned": 130994, "correctness_gate": "PASSED — `zit leaves` BEFORE pack and AFTER pack are byte-identical (leaf-set sha256 c2ae94f17937efb7a392a5c09ba637cffc847ce362af7f621d114d20113f3542 both times; trunk leaf 6924895d...); invariant §3 byte-exact content round-trip + trunk hash preserved across the at-rest transform; pack additionally hash-verified every reconstructed frame while writing the new generation.", "zit_ascloned": { "at_rest_bytes_total": 207344544, "zit_cache_bytes": 8135708, "at_rest_bytes_index_inclusive": 215480252, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 10847, "on_disk_bytes": 140422768, "pct_store": 67.7, "avg_bytes_per_frame": 12945 }, "delta": { "frames": 120147, "on_disk_bytes": 66921536, "pct_store": 32.3, "avg_bytes_per_frame": 556 }, "config": { "frames": 2, "on_disk_bytes": 176, "pct_store": 0.0 } }, "frame_header_overhead_bytes": 7212808, "accounting": "exact (header 64 + zlib + delta + config == 207344544)", "artifact_type_bytes": { "checkin_manifest": { "count": 36515, "on_disk_bytes": 81006048, "pct_store": 39.1 }, "other_artifact": { "count": 4775, "on_disk_bytes": 5531960, "pct_store": 2.7 }, "file_blob": { "count": 89704, "on_disk_bytes": 120806296, "pct_store": 58.3 } }, "serve_cpu": { "artifacts": 130994, "total_ms": 1270.02, "us_per_artifact": 9.695, "mb_per_s": 150.3 }, "ratio_zit_over_fossil": 1.4037 }, "zit_packed": { "at_rest_bytes_total": 175246544, "zit_cache_bytes": 8135708, "at_rest_bytes_index_inclusive": 183382252, "pack_stats_line": "packed: 130994 artifacts, 121734 re-deltaed, 207344544 -> 175246544 bytes", "artifacts_redeltaed": 121734, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 9260, "on_disk_bytes": 113048584, "pct_store": 64.5, "avg_bytes_per_frame": 12208 }, "delta": { "frames": 121734, "on_disk_bytes": 62197808, "pct_store": 35.5, "avg_bytes_per_frame": 510 }, "config": { "frames": 1, "on_disk_bytes": 88, "pct_store": 0.0 } }, "frame_header_overhead_bytes": 7211329, "accounting": "exact (header 64 + zlib + delta + config == 175246544)", "artifact_type_bytes": { "checkin_manifest": { "count": 36515, "on_disk_bytes": 65427440, "pct_store": 37.3 }, "other_artifact": { "count": 4775, "on_disk_bytes": 5462856, "pct_store": 3.1 }, "file_blob": { "count": 89704, "on_disk_bytes": 104356096, "pct_store": 59.5 } }, "full_form_compression_headroom": { "sampled_frames": 1024, "uncompressed": 11636752, "compressed": 3230184, "ratio_pct": 27.8 }, "serve_cpu": { "artifacts": 130994, "total_ms": 1188.40, "us_per_artifact": 9.072, "mb_per_s": 134.8 }, "ratio_zit_over_fossil": 1.1864, "ratio_zit_index_inclusive_over_fossil": 1.2415, "delta_vs_ascloned": { "total_bytes_change": -32098000, "total_pct_change": -15.5, "zlib_bytes_change": -27374184, "zlib_pct_change": -19.5, "delta_bytes_change": -4723728, "delta_pct_change": -7.1, "note": "Store shrank on both classes. zlib full-form frames 10847 -> 9260, delta avg 556 B -> 510 B. sqlite's deeper same-path histories give pickBase more in-window bases that beat fossil's original wire-delta choice." } }, "fossil": { "at_rest_bytes_total": 147709952, "clone_wall_s": 103.48, "measure": "wc -c of a fresh `fossil clone` destination (.fossil) off the same served corpus, no post-clone rebuild/vacuum" }, "pack_throughput_proxy": { "note": "NOT the harness put metric. pack wall / artifacts over the whole re-deltification pass. Labeled proxy only.", "pack_wall_s": 585.63, "us_per_artifact": 4470.7 }, "clone_ingest_proxy": { "note": "NOT the harness put metric — clone wall / artifacts (network + decode + put). Labeled proxy only.", "zit_clone_wall_s": 128.38, "zit_us_per_artifact": 980.1, "fossil_clone_wall_s": 103.48 } } ], "reproducibility": { "note": "Plan §4 acceptance: same corpus => same bytes. The zit AS-CLONED at-rest totals reproduce the §8 step-1 baseline and §8 step-2 run BYTE-FOR-BYTE (fossil-scm-clone 117,528,088; sqlite 207,344,544) and the fossil clone sizes reproduce exactly (72,245,248; 147,709,952), confirming deterministic clone-ingest storage. The PACKED totals are produced by the step-3 pass with the monotonic-shrink guard; pack is documented as idempotent (pack∘pack byte-identical) in-tree.", "ascloned_byte_totals_match_step1_baseline": true }, "headline": { "note": "Step-3 `zit pack` re-deltification is a MEASURED NET WIN on clone-heavy stores at this checkout: the §6.1 monotonic-shrink guard keeps min(re-encoded, existing) per frame, so the store cannot regress and shrinks wherever pickBase's in-window base search improves on fossil's verbatim wire delta. fossil-scm-clone went 1.6268x -> 1.5238x of fossil (-6.3% bytes); sqlite went 1.4037x -> 1.1864x of fossil (-15.5% bytes). This supersedes the earlier draft of this file (guard-less working tree) which had regressed. The §8 step-6 >1.3x gate is still tripped by fossil-scm-clone (1.5238x); §6.2 same-path candidates (step 4) are the plan's next lever there.", "fossil_scm_clone_ascloned_ratio": 1.6268, "fossil_scm_clone_packed_ratio": 1.5238, "sqlite_ascloned_ratio": 1.4037, "sqlite_packed_ratio": 1.1864 } }