Artifact [a0830062f2]

Artifact a0830062f25e2b8871e84ba6a94f6d61c36fb2fe91cec04bc71a96fda2a7df22:

• File .github/workflows/soak.yml — part of check-in [8ebc3e4dd8] at 2026-09-24 01:02:57 on branch trunk — soak: record a finding and keep sweeping; CI: cap the runners' zig cache

  A failing fuzz target is abandoned and the sweep moves on, so one recurring crash no longer hides the targets after it; the exit lists every finding. Each workflow clears its zig-local cache whole at job start once it passes 8 GB, since Zig never prunes it. (user: dmestas, size: 4610) [annotate] [blame] [check-ins using]

name: soak

# The nightly bug hunt. CI runs the deterministic simulation and the fuzz
# corpus shallow, for regression detection; this runs the same harnesses at
# depth on a ReleaseSafe build (asserts on), every night, on one runner, and
# fails loudly with a reproduce line when either finds something. It is
# scripts/soak.sh with a schedule, which the script's own README leaves to
# the operator. Budgets are the script's nightly defaults and can be
# overridden from the dispatch form for a bounded manual run.
on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:
    inputs:
      dst_seeds:
        description: "seeds swept by the deterministic simulation"
        default: "200"
      dst_steps:
        description: "steps per seed"
        default: "10000"
      fuzz_target_budget:
        description: "wall-clock seconds granted to each fuzz target"
        default: "300"
      fuzz_seeds:
        description: "fresh-seed invocations per fuzz target"
        default: "20"

permissions:
  contents: read

# One soak at a time; a second trigger waits rather than doubling the load
# on a single-slot runner.
concurrency:
  group: soak
  cancel-in-progress: false

jobs:
  soak:
    runs-on: [self-hosted, Linux, X64, linux-x64-zit]
    # 200 seeds x 10000 steps plus 27 targets x 300 s is a few hours; the
    # ceiling keeps a hang from holding the runner into the working day.
    timeout-minutes: 420
    env:
      # Kept in lockstep with src/ci_test.zig's `pinned_zig_version` and
      # `pinned_zig_shasums`, which fail `zig build test` if this file drifts.
      ZIG_VERSION: 0.16.0
      ZIG_TARGET: x86_64-linux
      ZIG_SHASUM: 70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00
    steps:
      - uses: actions/checkout@v5

      - name: Install Zig ${{ env.ZIG_VERSION }}
        run: |
          set -euo pipefail
          cache="${HOME}/.cache/zit-ci"
          # Zig never prunes its compilation cache: every source change leaves
          # a generation behind, and these runners grew 9-14 GB of it in a week.
          # Past the cap it is cleared WHOLE, at job start, when nothing else is
          # building on this single-slot runner. Not pruned by age: Zig does not
          # touch an entry it reuses, so an old entry may still be live, and a
          # manifest naming a deleted output costs more than one cold compile.
          cap_kb=$((8 * 1024 * 1024))
          if [ -d "${cache}/zig-local" ]; then
            used_kb=$(du -sk "${cache}/zig-local" | awk '{print $1}')
            if [ "${used_kb}" -gt "${cap_kb}" ]; then
              echo "zig cache is ${used_kb} KB, over the ${cap_kb} KB cap; clearing it"
              rm -rf "${cache}/zig-local"
            fi
          fi
          mkdir -p "${cache}/zig-local" "${cache}/zig-global" "${cache}/toolchains"
          echo "ZIG_LOCAL_CACHE_DIR=${cache}/zig-local" >> "$GITHUB_ENV"
          echo "ZIG_GLOBAL_CACHE_DIR=${cache}/zig-global" >> "$GITHUB_ENV"
          tarball="zig-${ZIG_TARGET}-${ZIG_VERSION}.tar.xz"
          kept="${cache}/toolchains/${tarball}"
          digest() { sha256sum "$1" | awk '{print $1}'; }
          if [ ! -f "$kept" ] || [ "$(digest "$kept")" != "${ZIG_SHASUM}" ]; then
            rm -f "$kept"
            curl -fsSL -o "${kept}.part" "https://ziglang.org/download/${ZIG_VERSION}/${tarball}"
            mv "${kept}.part" "$kept"
          fi
          actual="$(digest "$kept")"
          if [ "$actual" != "${ZIG_SHASUM}" ]; then
            echo "zig tarball digest mismatch: expected ${ZIG_SHASUM}, got ${actual}" >&2
            rm -f "$kept"
            exit 1
          fi
          tar -xf "$kept"
          echo "${GITHUB_WORKSPACE}/zig-${ZIG_TARGET}-${ZIG_VERSION}" >> "$GITHUB_PATH"

      - name: Deep DST and fuzz soak
        env:
          ZIT_SOAK_DIR: ${{ runner.temp }}/soak
          DST_SEEDS: ${{ inputs.dst_seeds || '200' }}
          DST_STEPS: ${{ inputs.dst_steps || '10000' }}
          FUZZ_TARGET_BUDGET: ${{ inputs.fuzz_target_budget || '300' }}
          FUZZ_SEEDS: ${{ inputs.fuzz_seeds || '20' }}
        run: |
          set -euo pipefail
          rm -rf "${ZIT_SOAK_DIR}"
          mkdir -p "${ZIT_SOAK_DIR}"
          bash scripts/soak.sh

      # The log and, on a failure, the flag with its reproduce line, kept
      # whether the soak passed or not.
      - name: Keep the soak log
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: soak-${{ github.run_id }}
          path: ${{ runner.temp }}/soak
          if-no-files-found: warn
          retention-days: 30