Timeline

Extend the full-form (.zlib) compression-headroom probe with a dictionary sub-arm: train a candidate preset dictionary on the sampled full-form population and recompress the SAME frames with it preloaded, reporting the net byte gain over the empty-window baseline. This is the GO/NO-GO signal for adding a preset-dictionary frame class. At-rest MEASUREMENT only: it never touches the store or the production §4.1 codec, and it preloads the dictionary through a throwaway reference-difference method because Zig 0.16 std flate hardcodes FDICT=0 and exposes no setDictionary.

The dictionary-compression lineage this arm measures: RFC 1950 reserves the FDICT bit and the 32 KiB preset-dictionary window, and its §9 says the most- referenced strings belong at the dictionary tail (shortest back-references); Yann Collet's zstd --train/COVER is the reference for training a shared dictionary from a population of small, mutually-similar records; Chromium productionised shared-dictionary transport at scale (SDCH, later the Compression-Dictionary-Transport work); Fabrice Bellard's LZEXE established the practical art of squeezing highly-repetitive input with a tiny back- reference machine. closed check-in: 1027cba83c user: dmestas tags: storage

Lands amend, artifact, bisect, log PATH, annotate, merge --cherrypick/--backout, and addremove — the VCS-verb-survey recommendations — onto trunk, already integrated with the storage branch's pack re-deltification.

- amend: mint display-metadata T cards, never rewrite history (also corrects the git-reflex table's wrong immutability lesson). - artifact: the plumbing window (whatis + raw dump) over the one parser. - bisect: DAG binary search (bad/good/next/status/reset), state in .zit-bisect. - log PATH: per-file history folded into log (fossil finfo). - annotate: per-line provenance; the version chain is ordered by TOPOLOGY (matching fossil's generation walk), not mtime, so a parent is never misattributed to its child on a timestamp collision. - merge --cherrypick/--backout: alternate-baseline special merges recording a 4.7.14 Q card, rendered by info; the plain merge surface is byte-identical. - addremove: one sweep, add every extra and drop every missing.

Built decomposed (focused implementers + adversarial review) after the monolithic workflow thrashed on API instability. 1678 unit + 376 e2e + dst 6x1000 green. check-in: e5fffafdb8 user: dmestas tags: trunk

fossil-style best-of-N delta base selection (pickBase, plan §5) plus a `zit pack` re-deltification pass with a monotonic-shrink guard (§6.1) and same-path candidates (§6.2). At-rest only: all four RFCs conformant, wire interop preserved, every repacked frame hash-verified against the log.

Measured (whole-file, index-inclusive): sqlite: 207.3 MB -> 174.4 MB (1.40x -> 1.18x vs fossil 147.7 MB) fossil-scm: 117.5 MB -> 108.1 MB (1.63x -> 1.50x vs fossil 72.2 MB)

The monotonic-shrink guard (keep the smaller of {existing frame, re-encoding}) is what turned re-deltifying a clone of fossil-quality wire deltas into a net win instead of a +33% regression. check-in: bab9303056 user: dmestas tags: trunk

Same-path predecessors sourced from check-in manifest walk, offered ahead of chronological window candidates (fossil's mlink analog per §6.2).

fossil-scm-clone: zit=108143984B fossil=72245248B ratio=1.4969 [Step-4 packed store (measured this run), accounting EXACT (header64 + zlib + delta + config == 108,143,984): zlib 8369 frames / 82,268,624 B (76.1%, avg 9830 B); delta 59130 frames / 25,875,208 B (23.9%, avg 437 B); config 1 frame / 88 B; raw 0. Artifact-type split: checkin/manifest 20848 / 23,816,728 B (22.0%), other-artifact 6929 / 4,706,432 B (4.4%), file-blob 39722 / 79,620,672 B (73.6%). Header+padding overhead 3,710,123 B (3.4%).] | sqlite: zit=174352688B fossil=147709952B ratio=1.1804 [Step-4 packed store (measured this run), accounting EXACT (header64 + zlib + delta + config == 174,352,688): zlib 9248 frames / 113,074,896 B (64.9%, avg 12226 B); delta 121746 frames / 61,277,640 B (35.1%, avg 503 B); config 1 frame / 88 B; raw 0. Artifact-type split: checkin/manifest 36515 / 65,427,440 B (37.5%), other-artifact 4775 / 5,462,856 B (3.1%), file-blob 89704 / 103,462,240 B (59.3%). Header+padding overhead 7,211,093 B (4.1%).] vs baseline: fossil-scm-clone: zit=117528088B fossil=72245248B ratio=1.627 [raw=0B/0f; zlib=92073728B/9647f (78.3%); delta=25454120B/57852f (21.7%); config=176B/2f. Header+padding overhead 3711845B (3.2%). Artifact-type: checkin/manifest 26721968B/20848, other 4835264B/6929, file-blob 85970616B/39722. Accounting exact.] | sqlite: zit=207344544B fossil=147709952B ratio=1.404 [raw=0B/0f; zlib=140422768B/10847f (67.7%); delta=66921536B/120147f (32.3%); config=176B/2f. Header+padding overhead 7212808B (3.5%). Artifact-type: checkin/manifest 81006048B/36515, other 5531960B/4775, file-blob 120806296B/89704. Accounting exact.] check-in: 0375766879 user: dmestas tags: storage

fossil-scm-clone: zit=110083984B fossil=72245248B ratio=1.5238 (packed store, exact accounting header64+zlib+delta+config==110083984: zlib 8377 frames / 84,763,872 B 77.0% avg 10118 B; delta 59122 frames / 25,319,960 B 23.0% avg 428 B; config 1 frame / 88 B; raw 0. Artifact-type: checkin/manifest 20848 / 23,816,728 B 21.6%, other 6929 / 4,706,432 B 4.3%, file-blob 39722 / 81,560,672 B 74.1%. As-cloned baseline was 117,528,088 B zlib 9647/92,073,728; delta 57852/25,454,120; ratio 1.6268.)

sqlite: zit=175246544B fossil=147709952B ratio=1.1864 (packed store, exact accounting header64+zlib+delta+config==175246544: zlib 9260 frames / 113,048,584 B 64.5% avg 12208 B; delta 121734 frames / 62,197,808 B 35.5% avg 510 B; config 1 frame / 88 B; raw 0. Artifact-type: checkin/manifest 36515 / 65,427,440 B 37.3%, other 4775 / 5,462,856 B 3.1%, file-blob 89704 / 104,356,096 B 59.5%. As-cloned baseline was 207,344,544 B zlib 10847/140,422,768; delta 120147/66,921,536; ratio 1.4037.)

vs baseline:

fossil-scm-clone: zit=117528088B fossil=72245248B ratio=1.627 (raw=0B/0f; zlib=92073728B/9647f 78.3%; delta=25454120B/57852f 21.7%; config=176B/2f. Header+padding overhead 3711845B 3.2%. Artifact-type: checkin/manifest 26721968B/20848, other 4835264B/6929, file-blob 85970616B/39722. Accounting exact.)

sqlite: zit=207344544B fossil=147709952B ratio=1.404 (raw=0B/0f; zlib=140422768B/10847f 67.7%; delta=66921536B/120147f 32.3%; config=176B/2f. Header+padding overhead 7212808B 3.5%. Artifact-type: checkin/manifest 81006048B/36515, other 5531960B/4775, file-blob 120806296B/89704. Accounting exact.) check-in: efaf2d3ae1 user: dmestas tags: storage

fossil-scm-clone: zit=117528088B fossil=72245248B ratio=1.6268 [zlib 9647 frames / 92,073,728 B (78.3%, avg 9544 B); delta 57852 frames / 25,454,120 B (21.7%, avg 439 B); config 2 frames / 176 B; raw 0 frames. Header+padding overhead 3,711,845 B (3.2%). Exact accounting: 64 + zlib + delta + config == 117,528,088. Artifact types: checkin/manifest 20848 / 26,721,968 B (22.7%), other 6929 / 4,835,264 B (4.1%), file-blob 39722 / 85,970,616 B (73.1%).] | sqlite: zit=207344544B fossil=147709952B ratio=1.4037 [zlib 10847 frames / 140,422,768 B (67.7%, avg 12945 B); delta 120147 frames / 66,921,536 B (32.3%, avg 556 B); config 2 frames / 176 B; raw 0 frames. Header+padding overhead 7,212,808 B (3.5%). Exact accounting: 64 + zlib + delta + config == 207,344,544. Artifact types: checkin/manifest 36515 / 81,006,048 B (39.1%), other 4775 / 5,531,960 B (2.7%), file-blob 89704 / 120,806,296 B (58.3%).]

vs baseline: fossil-scm-clone: zit=117528088B fossil=72245248B ratio=1.627 [raw=0B/0f; zlib=92073728B/9647f (78.3%); delta=25454120B/57852f (21.7%); config=176B/2f. Header+padding overhead 3711845B (3.2%). Artifact-type: checkin/manifest 26721968B/20848, other 4835264B/6929, file-blob 85970616B/39722. Accounting exact.] | sqlite: zit=207344544B fossil=147709952B ratio=1.404 [raw=0B/0f; zlib=140422768B/10847f (67.7%); delta=66921536B/120147f (32.3%); config=176B/2f. Header+padding overhead 7212808B (3.5%). Artifact-type: checkin/manifest 81006048B/36515, other 5531960B/4775, file-blob 120806296B/89704. Accounting exact.] check-in: d859c6a9a5 user: dmestas tags: storage

zit vs fossil at-rest bytes over the fossil-bench fossil-scm + sqlite corpora, imported via the clone/sync path (putDeltaContainer verbatim frames exercised):

fossil-scm-clone: zit 117,528,088 B vs fossil 72,245,248 B -> 1.63x (67,499 artifacts) sqlite: zit 207,344,544 B vs fossil 147,709,952 B -> 1.40x (130,994 artifacts)

Frame classes, artifact-type split, compression headroom, and serve CPU (203.9 / 138.4 MB/s) from 'zig build bench -- storage'. Chain-depth p50/p99 and a pure put us/artifact are NOT emitted by the current harness (plan §4.3 metrics-to-add); not synthesized (Rule 7). Comparison class: approximate (whole-file sizes, differing store shapes). Corpora unpinned; observed trunk hashes recorded in the JSON. check-in: 9aaf485847 user: dmestas tags: storage

Two independent slices recovered from an interrupted session, each gated green.

- Sync now dispatches on method + Content-Type at any URL path (§4 L759-764), so a stock 'fossil clone http://host:port/' at the bare root interoperates; the /xfer-only requirement and its 404 are gone. - 'zit log' is reduced to the signed one-verb surface: one check-in ID, help only, no -n windowing; usage errors exit 2, a bad ID exits 1.

Unit + e2e (209/209) green; fmt clean; dst 6x1000 ok. check-in: dbb2735bff user: dmestas tags: trunk