Artifact 121bdee18b7e5b1b23cd58a05c333b621d3622eb9c2552e9104447e034630cfc:
- File
docs/plans/bench/step4-samepath-unstamped-run.json
— part of check-in
[0375766879]
at
2026-07-18 15:09:17
on branch storage
— feat(§6.2): same-path candidates in pack via manifest walk
Same-path predecessors sourced from check-in manifest walk, offered ahead of chronological window candidates (fossil's mlink analog per §6.2).
fossil-scm-clone: zit=108143984B fossil=72245248B ratio=1.4969 [Step-4 packed store (measured this run), accounting EXACT (header64 + zlib + delta + config == 108,143,984): zlib 8369 frames / 82,268,624 B (76.1%, avg 9830 B); delta 59130 frames / 25,875,208 B (23.9%, avg 437 B); config 1 frame / 88 B; raw 0. Artifact-type split: checkin/manifest 20848 / 23,816,728 B (22.0%), other-artifact 6929 / 4,706,432 B (4.4%), file-blob 39722 / 79,620,672 B (73.6%). Header+padding overhead 3,710,123 B (3.4%).] | sqlite: zit=174352688B fossil=147709952B ratio=1.1804 [Step-4 packed store (measured this run), accounting EXACT (header64 + zlib + delta + config == 174,352,688): zlib 9248 frames / 113,074,896 B (64.9%, avg 12226 B); delta 121746 frames / 61,277,640 B (35.1%, avg 503 B); config 1 frame / 88 B; raw 0. Artifact-type split: checkin/manifest 36515 / 65,427,440 B (37.5%), other-artifact 4775 / 5,462,856 B (3.1%), file-blob 89704 / 103,462,240 B (59.3%). Header+padding overhead 7,211,093 B (4.1%).] vs baseline: fossil-scm-clone: zit=117528088B fossil=72245248B ratio=1.627 [raw=0B/0f; zlib=92073728B/9647f (78.3%); delta=25454120B/57852f (21.7%); config=176B/2f. Header+padding overhead 3711845B (3.2%). Artifact-type: checkin/manifest 26721968B/20848, other 4835264B/6929, file-blob 85970616B/39722. Accounting exact.] | sqlite: zit=207344544B fossil=147709952B ratio=1.404 [raw=0B/0f; zlib=140422768B/10847f (67.7%); delta=66921536B/120147f (32.3%); config=176B/2f. Header+padding overhead 7212808B (3.5%). Artifact-type: checkin/manifest 81006048B/36515, other 5531960B/4775, file-blob 120806296B/89704. Accounting exact.] (user: dmestas, size: 16978) [annotate] [blame] [check-ins using] [more...]
{ "schema": "zit-storage-pack-samepath/1", "plan": "docs/plans/storage-delta-base-selection.md", "plan_section": "\u00a74 (bench harness) re-run at the \u00a78 step-4 checkpoint (same-path candidates in pack, \u00a76.2 \u2014 fossil's mlink analog), measured against the \u00a78 step-3 (chronological-window pack) run and the \u00a78 step-1 baseline.", "step": "8/step-4 \u2014 same-path candidates present in the working tree (src/pack.zig deriveCandidateHints walks check-in manifests to build path\u2192[blob rids] and feeds each blob's same-path predecessors as priority hints ahead of the chronological window; src/logstore.zig gatherCandidates/putArtifactPacked take an ordered `priority: []const Rid` slice; the \u00a76.1 monotonic-shrink guard is retained). CLONE/SYNC ingest, then `zit pack`, then at-rest re-measure.", "stamped": false, "stamped_note": "File name is 'unstamped': the fossil-bench corpora are NOT yet pinned (corpora/corpora.json trunk_hash == null for both). A future run could see upstream drift until these are written back as pins.", "provenance": { "generated_utc": "2026-07-18", "host": { "os": "macOS (Darwin 25.5.0)", "arch": "arm64" }, "tool_versions": { "zit": "0.0.0 (targets Fossil RFC rev 00), built -Doptimize=ReleaseFast from the storage worktree working tree (uncommitted step-4 same-path edits atop commit efaf2d3ae1)", "fossil": "2.28 [52445a27f1] 2026-03-11", "zig": "0.16.0" }, "zit_checkout": "efaf2d3ae18b0b76e1c23502ba2bed506694c30c (branch storage) HEAD = \u00a76 step-3 pack commit, PLUS uncommitted working-tree edits: src/pack.zig (untracked; deriveCandidateHints/recordPathBlob/dedupSortedRids same-path walk), src/logstore.zig (gatherCandidates priority slice), src/main.zig (\u00a76.2 tests), src/dst.zig, src/fuzz_tests.zig, src/logstore_fuzz.zig, src/sim/storage_sweep.zig", "working_tree_dirty": true, "harnesses": [ "fossil-bench @ /Users/dmestas/fossil-bench (drivers/DRIVER_SPEC.md); loopback `fossil server --localhost` serving each snapshot from corpora/_snapshots/; zit clone / fossil clone off the same served corpus; store-size = raw file bytes over the destination store (zit clone store file is `.zit`, measured with wc -c directly).", "zig build bench -- storage <path> (src/storage_benchmark.zig): frame-class bytes-at-rest + artifact-type split + full-form compression headroom + serve CPU, read-only over the physical .zit log; openReadOnly whole-frame-CRC-verifies every frame first." ], "comparison_class": "approximate", "comparison_note": "Total store size is not like-for-like across store shapes. fossil's number is its whole .fossil (content blobs + SQLite b-tree indices + derived tables in one file); zit's headline is the whole .zit log (content frames + 64-byte header). zit keeps its derived index in a SEPARATE .zit-cache file (reported); an index-inclusive total is also given. Honest comparison class: 'approximate', not 'like-for-like'." }, "ingest_path": { "method": "CLONE/SYNC \u2014 fossil server serves the snapshot over loopback; both zit and fossil clone it anonymously. This exercises zit putDeltaContainer verbatim-frame storage (production ingest per plan \u00a71/\u00a75.2.3): the peer's wire deltas land at rest verbatim, so the sender's base choice is inherited. THEN `zit pack` runs the step-4 same-path re-deltification pass over that store.", "both_impls_cloned_same_served_corpus": true, "pack_run": "zit pack (default behavior, no flag) in the cloned working directory; atomic temp+rename; every frame hash-verified during pack; \u00a76.1 monotonic-shrink guard keeps min(re-encoded, existing) per frame; \u00a76.2 same-path predecessors offered ahead of the chronological window." }, "harness_gaps": { "note": "Rule 7 \u2014 reported only what the harness measured. storage_benchmark.zig is UNCHANGED at this checkout (not in the step-4 diff), so the following requested metrics are NOT emitted and were NOT synthesized:", "chain_depth_p50_p99": "NOT MEASURED. src/storage_benchmark.zig emits no delta-chain-depth histogram; plan \u00a74.3 lists it as a metric to ADD and it has not been added at this checkout. No p50/p99 fabricated.", "put_us_per_artifact": "NOT MEASURED by the storage harness. `zig build bench -- storage` times SERVE CPU only, not the put path. The pure-put microbench (`zig build bench -- put`) runs a SYNTHETIC seeded corpus, not the CLONE-ingested real corpus, so it is not substituted (Rule 7). A clone-ingest proxy (network+decode+put) and a pack-throughput proxy (pack wall / artifacts) are recorded per corpus, each labeled a proxy.", "per_frame_delta_vs_base_ratio": "NOT MEASURED (plan \u00a74.3 second add). Not synthesized." }, "corpora": [ { "corpus": "fossil-scm-clone", "fossil_bench_name": "fossil", "tier": "medium", "source": "https://fossil-scm.org/home", "leaves_match_fossil": true, "pinned": false, "artifacts_cloned": 67499, "correctness_gate": "PASSED \u2014 `zit leaves` BEFORE and AFTER pack are byte-identical (leaf-set sha256 36fac0795bb163c2f3a362cfa4a415910d234e8cc9bb457727b41d88849a54b5 == 36fac0795bb163c2f3a362cfa4a415910d234e8cc9bb457727b41d88849a54b5); invariant \u00a73 byte-exact content round-trip preserved across the at-rest transform; pack hash-verified every reconstructed frame while writing the new generation (pack streams and re-hashes every artifact to its name).", "leaves_sha_pre": "36fac0795bb163c2f3a362cfa4a415910d234e8cc9bb457727b41d88849a54b5", "leaves_sha_post": "36fac0795bb163c2f3a362cfa4a415910d234e8cc9bb457727b41d88849a54b5", "zit_ascloned": { "at_rest_bytes_total": 117528088, "zit_cache_bytes": 4950808, "at_rest_bytes_index_inclusive": 122478896, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 9647, "on_disk_bytes": 92073728, "pct_store": 78.3, "avg_bytes_per_frame": 9544 }, "delta": { "frames": 57852, "on_disk_bytes": 25454120, "pct_store": 21.7, "avg_bytes_per_frame": 439 }, "config": { "frames": 2, "on_disk_bytes": 176, "pct_store": 0.0, "avg_bytes_per_frame": 88 } }, "frame_header_overhead_bytes": 3711845, "accounting": "exact (header 64 + raw + zlib + delta + config == 117528088)", "artifact_type_bytes": { "checkin_manifest": { "count": 20848, "on_disk_bytes": 26721968, "pct_store": 22.7 }, "other_artifact": { "count": 6929, "on_disk_bytes": 4835264, "pct_store": 4.1 }, "file_blob": { "count": 39722, "on_disk_bytes": 85970616, "pct_store": 73.1 } }, "serve_cpu": { "artifacts": 67499, "total_ms": 547.58, "us_per_artifact": 8.112, "mb_per_s": 198.2 }, "ratio_zit_over_fossil": 1.6268 }, "zit_packed": { "at_rest_bytes_total": 108143984, "zit_cache_bytes": 4950808, "at_rest_bytes_index_inclusive": 113094792, "pack_stats_line": "packed: 67499 artifacts, 59130 re-deltaed, 117528088 -> 108143984 bytes", "artifacts_redeltaed": 59130, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 8369, "on_disk_bytes": 82268624, "pct_store": 76.1, "avg_bytes_per_frame": 9830 }, "delta": { "frames": 59130, "on_disk_bytes": 25875208, "pct_store": 23.9, "avg_bytes_per_frame": 437 }, "config": { "frames": 1, "on_disk_bytes": 88, "pct_store": 0.0, "avg_bytes_per_frame": 88 } }, "frame_header_overhead_bytes": 3710123, "accounting": "exact (header 64 + raw + zlib + delta + config == 108143984)", "artifact_type_bytes": { "checkin_manifest": { "count": 20848, "on_disk_bytes": 23816728, "pct_store": 22.0 }, "other_artifact": { "count": 6929, "on_disk_bytes": 4706432, "pct_store": 4.4 }, "file_blob": { "count": 39722, "on_disk_bytes": 79620672, "pct_store": 73.6 } }, "full_form_compression_headroom": { "sampled_frames": 1024, "uncompressed": 2245879, "compressed": 987041, "ratio_pct": 43.9 }, "serve_cpu": { "artifacts": 67499, "total_ms": 543.83, "us_per_artifact": 8.057, "mb_per_s": 183.1 }, "ratio_zit_over_fossil": 1.4969, "ratio_zit_index_inclusive_over_fossil": 1.5654, "delta_vs_ascloned": { "total_bytes_change": -9384104, "total_pct_change": -8.0 } }, "fossil": { "at_rest_bytes_total": 72245248, "clone_wall_s": 21.39, "measure": "wc -c of a fresh `fossil clone` destination (.fossil) off the same served corpus, no post-clone rebuild/vacuum" }, "pack_throughput_proxy": { "note": "NOT the harness put metric. pack wall / artifacts over the whole re-deltification pass (decode-all + up-to-5 delta.create/artifact + same-path candidates + guard-compare + write + fsync). Labeled proxy only.", "pack_wall_s": 371.68, "us_per_artifact": 5506.5 }, "clone_ingest_proxy": { "note": "NOT the harness put metric \u2014 clone wall / artifacts (network + decode + put). Labeled proxy only.", "zit_clone_wall_s": 56.21, "zit_us_per_artifact": 832.8, "fossil_clone_wall_s": 21.39 } }, { "corpus": "sqlite", "fossil_bench_name": "sqlite", "tier": "large", "source": "https://sqlite.org/src", "leaves_match_fossil": true, "pinned": false, "artifacts_cloned": 130994, "correctness_gate": "PASSED \u2014 `zit leaves` BEFORE and AFTER pack are byte-identical (leaf-set sha256 848070e84d034dcd0cfd50dd702ad83c6a46691cc2064d96b51199c38ed4fd6a == 848070e84d034dcd0cfd50dd702ad83c6a46691cc2064d96b51199c38ed4fd6a); invariant \u00a73 byte-exact content round-trip preserved across the at-rest transform; pack hash-verified every reconstructed frame while writing the new generation (pack streams and re-hashes every artifact to its name).", "leaves_sha_pre": "848070e84d034dcd0cfd50dd702ad83c6a46691cc2064d96b51199c38ed4fd6a", "leaves_sha_post": "848070e84d034dcd0cfd50dd702ad83c6a46691cc2064d96b51199c38ed4fd6a", "zit_ascloned": { "at_rest_bytes_total": 207344544, "zit_cache_bytes": 8135708, "at_rest_bytes_index_inclusive": 215480252, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 10847, "on_disk_bytes": 140422768, "pct_store": 67.7, "avg_bytes_per_frame": 12945 }, "delta": { "frames": 120147, "on_disk_bytes": 66921536, "pct_store": 32.3, "avg_bytes_per_frame": 556 }, "config": { "frames": 2, "on_disk_bytes": 176, "pct_store": 0.0, "avg_bytes_per_frame": 88 } }, "frame_header_overhead_bytes": 7212808, "accounting": "exact (header 64 + raw + zlib + delta + config == 207344544)", "artifact_type_bytes": { "checkin_manifest": { "count": 36515, "on_disk_bytes": 81006048, "pct_store": 39.1 }, "other_artifact": { "count": 4775, "on_disk_bytes": 5531960, "pct_store": 2.7 }, "file_blob": { "count": 89704, "on_disk_bytes": 120806296, "pct_store": 58.3 } }, "serve_cpu": { "artifacts": 130994, "total_ms": 1272.21, "us_per_artifact": 9.712, "mb_per_s": 150.0 }, "ratio_zit_over_fossil": 1.4037 }, "zit_packed": { "at_rest_bytes_total": 174352688, "zit_cache_bytes": 8135708, "at_rest_bytes_index_inclusive": 182488396, "pack_stats_line": "packed: 130994 artifacts, 121746 re-deltaed, 207344544 -> 174352688 bytes", "artifacts_redeltaed": 121746, "frame_class_bytes": { "raw": { "frames": 0, "on_disk_bytes": 0, "pct_store": 0.0 }, "zlib": { "frames": 9248, "on_disk_bytes": 113074896, "pct_store": 64.9, "avg_bytes_per_frame": 12226 }, "delta": { "frames": 121746, "on_disk_bytes": 61277640, "pct_store": 35.1, "avg_bytes_per_frame": 503 }, "config": { "frames": 1, "on_disk_bytes": 88, "pct_store": 0.0, "avg_bytes_per_frame": 88 } }, "frame_header_overhead_bytes": 7211093, "accounting": "exact (header 64 + raw + zlib + delta + config == 174352688)", "artifact_type_bytes": { "checkin_manifest": { "count": 36515, "on_disk_bytes": 65427440, "pct_store": 37.5 }, "other_artifact": { "count": 4775, "on_disk_bytes": 5462856, "pct_store": 3.1 }, "file_blob": { "count": 89704, "on_disk_bytes": 103462240, "pct_store": 59.3 } }, "full_form_compression_headroom": { "sampled_frames": 1024, "uncompressed": 11636752, "compressed": 3230184, "ratio_pct": 27.8 }, "serve_cpu": { "artifacts": 130994, "total_ms": 1196.6, "us_per_artifact": 9.135, "mb_per_s": 133.2 }, "ratio_zit_over_fossil": 1.1804, "ratio_zit_index_inclusive_over_fossil": 1.2355, "delta_vs_ascloned": { "total_bytes_change": -32991856, "total_pct_change": -15.9 } }, "fossil": { "at_rest_bytes_total": 147709952, "clone_wall_s": 110.98, "measure": "wc -c of a fresh `fossil clone` destination (.fossil) off the same served corpus, no post-clone rebuild/vacuum" }, "pack_throughput_proxy": { "note": "NOT the harness put metric. pack wall / artifacts over the whole re-deltification pass (decode-all + up-to-5 delta.create/artifact + same-path candidates + guard-compare + write + fsync). Labeled proxy only.", "pack_wall_s": 587.35, "us_per_artifact": 4483.8 }, "clone_ingest_proxy": { "note": "NOT the harness put metric \u2014 clone wall / artifacts (network + decode + put). Labeled proxy only.", "zit_clone_wall_s": 134.1, "zit_us_per_artifact": 1023.7, "fossil_clone_wall_s": 110.98 } } ], "reproducibility": { "note": "Plan \u00a74 acceptance: same corpus => same bytes. The zit AS-CLONED at-rest totals reproduce the \u00a78 step-1/step-2/step-3 baseline BYTE-FOR-BYTE (fossil-scm-clone 117,528,088; sqlite 207,344,544) and the fossil clone sizes reproduce exactly, confirming deterministic clone-ingest storage. The PACKED totals are produced by the step-4 pass (same-path candidates + \u00a76.1 guard).", "ascloned_byte_totals_match_step1_baseline": true }, "headline": { "note": "Step-4 \u00a76.2 same-path candidates in pack, measured incremental gain over step-3 (chronological-window pack).", "step3_packed_bytes": { "fossil-scm-clone": 110083984, "sqlite": 175246544 }, "fossil-scm-clone": { "ascloned_ratio": 1.6268, "packed_ratio": 1.4969, "packed_bytes": 108143984, "step3_packed_bytes": 110083984, "vs_step3_bytes_change": -1940000, "vs_step3_pct_change": -1.76 }, "sqlite": { "ascloned_ratio": 1.4037, "packed_ratio": 1.1804, "packed_bytes": 174352688, "step3_packed_bytes": 175246544, "vs_step3_bytes_change": -893856, "vs_step3_pct_change": -0.51 } } }