Artifact 40912ded7272fccb1c547b299a7d72521506cedbb5831518011d0c08fe1038c7:
- File
scripts/soak.README
— part of check-in
[4df2fad0e6]
at
2026-09-18 15:49:53
on branch trunk
— soak: sweep the DST on the production build too
ReleaseSafe keeps every assert as a detector; the build that ships is ReleaseFast, where only the simulation's own state-replay oracle can catch a bug that appears when the checks are gone. The nightly now runs the same DST sweep on both builds. DST_FAST=0 skips the second. (user: dmestas, size: 3194) [annotate] [blame] [check-ins using]
soak.sh — out-of-band deep DST + fuzz soak runner ================================================= What it is ---------- The pre-commit gate and CI run the DST and fuzz harnesses SHALLOW, for regression detection only: `zig build dst` sweeps a few seeds x a few thousand steps, and CI fuzzes one seed x 100 rounds ("regression detection, not a bug hunt", in CI's own words). soak.sh is the bug hunt — the SAME harnesses at nightly depth on a ReleaseSafe build (asserts ON). It adds machine-hours, not new invariants: every "depth belongs to an out-of-band runner" comment in the tree refers to this script. It never auto-files and never auto-fixes. On the first failure it appends the harness's own `reproduce with seed=...` line (reshaped into a runnable command) to a dated log, drops a flag file, and exits non-zero. Running it ---------- Full nightly (multi-hour): ./scripts/soak.sh Bounded smoke (minutes): DST_SEEDS=3 DST_STEPS=500 \ FUZZ_TARGET_BUDGET=6 FUZZ_SEEDS=3 ./scripts/soak.sh Config env vars and their nightly defaults are documented in the script header (DST_SEEDS=200, DST_STEPS=10000, FUZZ_TARGET_BUDGET=300s/target, FUZZ_SEEDS=20, ZIT_SOAK_DIR=$HOME/.zit-soak). Logs and the flag file live under ZIT_SOAK_DIR, outside the repo, so a soak run never shows up in `fossil status`. Depth by time, not by flat rounds: a flat --rounds would run the cheapest fuzz target in seconds and the heaviest (~276x per-round cost, src/fuzz_tests.zig: 63-78) for hours. Each target instead gets an equal wall-clock budget and its --rounds is grown from measured speed to fill each slice, so the per-target cost ratios honor themselves with no second table to drift. The target list is enumerated from the runner's own usage output (the ci.yml:146-159 pattern), so that cannot drift either. Operator responsibilities (NOT done by the script) -------------------------------------------------- 1. Install the schedule. The script only RUNS the soak; installing a launchd/cron entry is an operator action. Copy-paste-ready launchd plist and cron line are in the script header. The self-hosted runner being down does not block a manual `./scripts/soak.sh`. 2. Run the first full nightly and retain its log. 3. Triage findings. A raised flag ($ZIT_SOAK_DIR/SOAK-FAILED) is a P6 GO/NO-GO input, not a footnote. Reproduce with the logged line, disposition, then delete the flag. A clean run never clears a flag the operator has not yet seen. The DST sweep runs twice: on ReleaseSafe, where every assert is a detector, and again on ReleaseFast, the build that ships, where only the simulation's own state-replay oracle can catch a bug. DST_FAST=0 skips the second. Reproduce lines --------------- dst: zig build dst --release=safe -- --seed N --steps <steps> (or --release=fast when the failing phase is dst-fast) fuzz: zig build fuzz --release=safe -- <target> --seed 0xSEED --rounds R Both replay the exact failure: one PRNG stream per run, the seed is the whole universe. The dst line carries --steps (the harness's own line omits it); the fuzz line carries the target and the --rounds the soak had grown to.