| Comment: | storage-bench: add a dict-trained headroom arm (§2.2 preset-dictionary GO/NO-GO)
Extend the full-form (.zlib) compression-headroom probe with a dictionary sub-arm: train a candidate preset dictionary on the sampled full-form population and recompress the SAME frames with it preloaded, reporting the net byte gain over the empty-window baseline. This is the GO/NO-GO signal for adding a preset-dictionary frame class. At-rest MEASUREMENT only: it never touches the store or the production §4.1 codec, and it preloads the dictionary through a throwaway reference-difference method because Zig 0.16 std flate hardcodes FDICT=0 and exposes no setDictionary. The dictionary-compression lineage this arm measures: RFC 1950 reserves the FDICT bit and the 32 KiB preset-dictionary window, and its §9 says the most- referenced strings belong at the dictionary tail (shortest back-references); Yann Collet's zstd --train/COVER is the reference for training a shared dictionary from a population of small, mutually-similar records; Chromium productionised shared-dictionary transport at scale (SDCH, later the Compression-Dictionary-Transport work); Fabrice Bellard's LZEXE established the practical art of squeezing highly-repetitive input with a tiny back- reference machine. |
|---|---|
| Timelines: | family | ancestors | storage |
| Files: | files | file ages | folders |
| SHA3-256: |
1027cba83cc7a791a19ac90a2d30721e |
| User & Date: | dmestas on 2026-07-18 17:47:47.706 |
| Other Links: | branch diff | manifest | tags |
|
2026-07-18
| ||
| 17:47 |
storage-bench: add a dict-trained headroom arm (§2.2 preset-dictionary GO/NO-GO)
Extend the full-form (.zlib) compression-headroom probe with a dictionary sub-arm: train a candidate preset dictionary on the sampled full-form population and recompress the SAME frames with it preloaded, reporting the net byte gain over the empty-window baseline. This is the GO/NO-GO signal for adding a preset-dictionary frame class. At-rest MEASUREMENT only: it never touches the store or the production §4.1 codec, and it preloads the dictionary through a throwaway reference-difference method because Zig 0.16 std flate hardcodes FDICT=0 and exposes no setDictionary. The dictionary-compression lineage this arm measures: RFC 1950 reserves the FDICT bit and the 32 KiB preset-dictionary window, and its §9 says the most- referenced strings belong at the dictionary tail (shortest back-references); Yann Collet's zstd --train/COVER is the reference for training a shared dictionary from a population of small, mutually-similar records; Chromium productionised shared-dictionary transport at scale (SDCH, later the Compression-Dictionary-Transport work); Fabrice Bellard's LZEXE established the practical art of squeezing highly-repetitive input with a tiny back- reference machine. closed check-in: 1027cba83c user: dmestas tags: storage | |
| 17:12 | Merge trunk into storage: bring in the converged trunk (storage pack work + the cli verb slice — annotate, bisect, extras, merge3, undo, cherrypick/backout) so the dict lever (§2.2 preset-dictionary compression) builds on current state. check-in: 09748596dc user: dmestas tags: storage | |