Artifact d3ba53d7dcdc0660f2e3f0223b8633111f4ac1639a803626740a2e91b6ec5661:
- File
scripts/soak.README
— part of check-in
[8ebc3e4dd8]
at
2026-09-24 01:02:57
on branch trunk
— soak: record a finding and keep sweeping; CI: cap the runners' zig cache
A failing fuzz target is abandoned and the sweep moves on, so one recurring crash no longer hides the targets after it; the exit lists every finding. Each workflow clears its zig-local cache whole at job start once it passes 8 GB, since Zig never prunes it. (user: dmestas, size: 3377) [annotate] [blame] [check-ins using]
soak.sh — out-of-band deep DST + fuzz soak runner ================================================= What it is ---------- The pre-commit gate and CI run the DST and fuzz harnesses SHALLOW, for regression detection only: `zig build dst` sweeps a few seeds x a few thousand steps, and CI fuzzes one seed x 100 rounds ("regression detection, not a bug hunt", in CI's own words). soak.sh is the bug hunt — the SAME harnesses at nightly depth on a ReleaseSafe build (asserts ON). It adds machine-hours, not new invariants: every "depth belongs to an out-of-band runner" comment in the tree refers to this script. It never auto-files and never auto-fixes. On a failure it appends the harness's reproduce line to a dated log and drops a flag file, then keeps going: a failing fuzz target is abandoned and the sweep moves to the next, and a failing DST phase moves to the next phase. At the end it lists every finding and exits non-zero if there was one, so one recurring crash cannot hide the targets after it. Running it ---------- Full nightly (multi-hour): ./scripts/soak.sh Bounded smoke (minutes): DST_SEEDS=3 DST_STEPS=500 \ FUZZ_TARGET_BUDGET=6 FUZZ_SEEDS=3 ./scripts/soak.sh Config env vars and their nightly defaults are documented in the script header (DST_SEEDS=200, DST_STEPS=10000, FUZZ_TARGET_BUDGET=300s/target, FUZZ_SEEDS=20, ZIT_SOAK_DIR=$HOME/.zit-soak). Logs and the flag file live under ZIT_SOAK_DIR, outside the repo, so a soak run never shows up in `fossil status`. Depth by time, not by flat rounds: a flat --rounds would run the cheapest fuzz target in seconds and the heaviest (~276x per-round cost, src/fuzz_tests.zig: 63-78) for hours. Each target instead gets an equal wall-clock budget and its --rounds is grown from measured speed to fill each slice, so the per-target cost ratios honor themselves with no second table to drift. The target list is enumerated from the runner's own usage output (the ci.yml:146-159 pattern), so that cannot drift either. Operator responsibilities (NOT done by the script) -------------------------------------------------- 1. Install the schedule. The script only RUNS the soak; installing a launchd/cron entry is an operator action. Copy-paste-ready launchd plist and cron line are in the script header. The self-hosted runner being down does not block a manual `./scripts/soak.sh`. 2. Run the first full nightly and retain its log. 3. Triage findings. A raised flag ($ZIT_SOAK_DIR/SOAK-FAILED) is a P6 GO/NO-GO input, not a footnote. Reproduce with the logged line, disposition, then delete the flag. A clean run never clears a flag the operator has not yet seen. The DST sweep runs twice: on ReleaseSafe, where every assert is a detector, and again on ReleaseFast, the build that ships, where only the simulation's own state-replay oracle can catch a bug. DST_FAST=0 skips the second. Reproduce lines --------------- dst: zig build dst --release=safe -- --seed N --steps <steps> (or --release=fast when the failing phase is dst-fast) fuzz: zig build fuzz --release=safe -- <target> --seed 0xSEED --rounds R Both replay the exact failure: one PRNG stream per run, the seed is the whole universe. The dst line carries --steps (the harness's own line omits it); the fuzz line carries the target and the --rounds the soak had grown to.