Artifact 3a2f028e4f87a395b2b779ac63260fcb68a7a9c7ffc9fea5b065d5bc065c4719:
- File
scripts/soak.README
— part of check-in
[9f00c44a8f]
at
2026-07-20 03:36:42
on branch h1-soak
— Add scripts/soak.sh: out-of-band deep DST + fuzz soak runner
The pre-commit gate runs the DST and fuzz harnesses shallow. Every comment deferring depth to an out-of-band runner pointed at a script that did not exist; scripts/ held only mirror-export.sh. This is that runner.
On a ReleaseSafe build (asserts on) it runs the repostate+sync DST at a deep seed/step sweep, then every fuzz target enumerated from the runner's own usage output, so the list cannot drift, under an equal per-target wall-clock budget. --rounds grows from measured speed to fill each budget, so a cheap target sweeps millions of rounds while a pack round-trip sweeps thousands.
On the first failure it appends a runnable reproduce line to a dated log, drops a flag file, and exits non-zero; it never files and never fixes. It owns an explicit fresh seed per fuzz invocation, because the fuzz runner's buffered stderr is dropped on the error-return path: a crashing target prints only its error name, not the seed it flushed before running.
Logs and the flag live outside the repo. Installing the schedule and the first full nightly are operator actions, documented in the header and soak.README. (user: dmestas, size: 2924) [annotate] [blame] [check-ins using] [more...]
soak.sh — out-of-band deep DST + fuzz soak runner ================================================= What it is ---------- The pre-commit gate and CI run the DST and fuzz harnesses SHALLOW, for regression detection only: `zig build dst` sweeps a few seeds x a few thousand steps, and CI fuzzes one seed x 100 rounds ("regression detection, not a bug hunt", in CI's own words). soak.sh is the bug hunt — the SAME harnesses at nightly depth on a ReleaseSafe build (asserts ON). It adds machine-hours, not new invariants: every "depth belongs to an out-of-band runner" comment in the tree refers to this script. It never auto-files and never auto-fixes. On the first failure it appends the harness's own `reproduce with seed=...` line (reshaped into a runnable command) to a dated log, drops a flag file, and exits non-zero. Running it ---------- Full nightly (multi-hour): ./scripts/soak.sh Bounded smoke (minutes): DST_SEEDS=3 DST_STEPS=500 \ FUZZ_TARGET_BUDGET=6 FUZZ_SEEDS=3 ./scripts/soak.sh Config env vars and their nightly defaults are documented in the script header (DST_SEEDS=200, DST_STEPS=10000, FUZZ_TARGET_BUDGET=300s/target, FUZZ_SEEDS=20, ZIT_SOAK_DIR=$HOME/.zit-soak). Logs and the flag file live under ZIT_SOAK_DIR, outside the repo, so a soak run never shows up in `fossil status`. Depth by time, not by flat rounds: a flat --rounds would run the cheapest fuzz target in seconds and the heaviest (~276x per-round cost, src/fuzz_tests.zig: 63-78) for hours. Each target instead gets an equal wall-clock budget and its --rounds is grown from measured speed to fill each slice, so the per-target cost ratios honor themselves with no second table to drift. The target list is enumerated from the runner's own usage output (the ci.yml:146-159 pattern), so that cannot drift either. Operator responsibilities (NOT done by the script) -------------------------------------------------- 1. Install the schedule. The script only RUNS the soak; installing a launchd/cron entry is an operator action. Copy-paste-ready launchd plist and cron line are in the script header. The self-hosted runner being down does not block a manual `./scripts/soak.sh`. 2. Run the first full nightly and retain its log. 3. Triage findings. A raised flag ($ZIT_SOAK_DIR/SOAK-FAILED) is a P6 GO/NO-GO input, not a footnote. Reproduce with the logged line, disposition, then delete the flag. A clean run never clears a flag the operator has not yet seen. Reproduce lines --------------- dst: zig build dst -Doptimize=ReleaseSafe -- --seed N --steps <steps> fuzz: zig build fuzz -Doptimize=ReleaseSafe -- <target> --seed 0xSEED --rounds R Both replay the exact failure: one PRNG stream per run, the seed is the whole universe. The dst line carries --steps (the harness's own line omits it); the fuzz line carries the target and the --rounds the soak had grown to.