Artifact d34f7b08b15977d63957e14c6f93260161db5f4374393169b4d7f44be77dba94:
- File
src/ci_test.zig
— part of check-in
[7363238ad5]
at
2026-07-31 02:43:47
on branch trunk
— build: split the unit lane from the full gate
zig build test was 9m13s, and the tests it actually ran were a small fraction of that: it also compiled the e2e and smoke harnesses in the pinned ReleaseSafe and then never executed them. test is now unit tests only -- 2m54s cold, 1m42s warm -- and the new gate step carries those two compile proofs.
The dst, bench and smoke UNIT tests deliberately stay on test. They are unit tests despite their siblings' names, zig build dst/smoke run the simulation and the interop check rather than them, and CI invokes the lanes one by one and never invokes gate -- so moving them would have dropped them from CI while every step still looked green. ci_test's declared_steps records gate as intentionally not-in-CI, and names that failure mode for the next lane someone adds. (user: dmestas, size: 27530) [annotate] [blame] [check-ins using]
//! Structural pins over the GitHub Actions workflow that gates this repository. //! //! The pipeline itself is the real test — a green run against a real runner is //! the only thing that proves fossil installs, Zig builds, and smoke can spawn //! a server. What a green run cannot prove is that the workflow still invokes //! everything it is supposed to invoke: a build step added to build.zig months //! from now is silently *not* run by CI, and CI stays green while covering //! less. That is the failure this module exists to catch, and it is a //! repository-file fact, so it is pinned the same way src/packaging_test.zig //! pins the shipped distribution: embed the file, assert on its text, let //! `zig build test` be the thing that notices. //! //! The workflow arrives through build.zig's `build_unit_test_module` as the //! anonymous import `ci_workflow`, deliberately on the test-only module: the //! exported `zit` module must never depend on a repo-root file outside //! `build.zig.zon`'s `.paths`, or a consumer fetching the package fails before //! compiling a line. `.github` is allowlisted in packaging_test.zig's //! `test_only_roots` for exactly that reason. //! //! That allowlist is a *packaging* exemption only. The workflow file still has //! to be version-controlled, or a clean checkout fails to build on the embed — //! and because it lives under a dot-directory, bare `fossil extras` never //! reports it as untracked (`fossil extras --dotfiles` does). That near-miss is //! why src/packaging_test.zig now pins tracking separately from packaging. //! //! These are text assertions, not a YAML parse. A pinned command sitting inside //! a YAML comment would satisfy them. That trade is taken knowingly: a parser //! for one file is more machinery than the risk warrants, and the workflow is //! short enough that a human reads it whole. What the pins buy is the //! *negative space* — the reviewed-exception list below — which no amount of //! reading catches reliably. const std = @import("std"); const ci_workflow = @embedFile("ci_workflow"); const build_source = @embedFile("build_source"); fn expect_contains(haystack: []const u8, needle: []const u8, where: []const u8) !void { if (std.mem.indexOf(u8, haystack, needle) != null) return; std.debug.print("\nci pin: {s} does not contain \"{s}\"\n", .{ where, needle }); return error.CiPinMissing; } /// Counts non-overlapping occurrences of `needle`. Used for the single-job pin: /// "one gating job" is a claim about how many `runs-on:` keys exist, which a /// containment check cannot make. fn count_occurrences(haystack: []const u8, needle: []const u8) usize { if (needle.len == 0) return 0; var seen: usize = 0; var scan: usize = 0; while (std.mem.indexOfPos(u8, haystack, scan, needle)) |hit| { seen += 1; scan = hit + needle.len; } return seen; } /// The exact Zig toolchain the release is built and tested with. Pinned as a /// constant here and asserted against the workflow so the two cannot drift: /// a runner silently upgrading Zig turns "0.1.0 builds on 0.16.0" into an /// unverified claim. const pinned_zig_version = "0.16.0"; /// Published SHA-256 digests of the three pinned Zig toolchain tarballs /// (ziglang.org/download/index.json, 2026-07-20). The workflow matrix verifies /// each native target against its own digest; pinning all three here prevents a /// platform leg from silently accepting a swapped or mismatched tarball. const pinned_zig_shasums = [_][]const u8{ "b23d70deaa879b5c2d486ed3316f7eaa53e84acf6fc9cc747de152450d401489", "70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00", "ea4b09bfb22ec6f6c6ceac57ab63efb6b46e17ab08d21f69f3a48b38e1534f17", }; /// The command that turns the digests above from recorded values into /// enforced ones. Pinned separately because the two are independently /// deletable: dropping this line while leaving `ZIG_SHASUM:` in the `env:` /// block would leave a digest sitting in the file that nothing checks — the /// same "a value appearing in the file is not the value being used" trap the /// battery's whole-line anchoring guards against. /// Anchored to the whole checking command, not the bare tool name: `sha256sum /// --check --strict` appearing anywhere satisfies a loose substring while the /// value actually fed to it is something other than the pinned `ZIG_SHASUM` — /// a digest check that verifies the wrong digest. Naming the env var and the /// tarball on the same line is the reverse assertion: the digest this file /// pins is provably the digest the workflow checks. const pinned_digest_check = "if [ \"$actual\" != \"${ZIG_SHASUM}\" ]; then"; /// The release battery, in the order the workflow runs it — asserted, not just /// described: the pin below requires each entry to appear later in the file than /// the one before it. /// /// Each entry is a whole `run:` line, `run: ` prefix and trailing newline /// included. Anchoring matters more than it looks: as a bare substring /// `zig build` is contained in `zig build test`, `zig build e2e`, `zig build /// dst` and `zig build smoke`, so deleting the bare-build step would have left /// the pin passing while claiming coverage it no longer had. /// /// `zig fmt --check build.zig src` rather than `--check .`: the bare dot walks /// `.zig-cache/` and `zig-out/`, whose generated sources are not ours to /// format and whose presence differs between a cold runner and a warm local /// tree. Naming the two real source roots keeps the check meaning the same /// thing in both places. const battery = [_][]const u8{ "run: zig fmt --check build.zig src\n", "run: zig build\n", "run: zig build test\n", "run: zig build e2e\n", "run: zig build dst\n", "run: zig build smoke\n", }; /// Every step build.zig declares. Asserted as an exact set, in neither /// direction forgiving: a step added to build.zig and a step deleted from it /// both fail here, which is the point — the scan below is only meaningful if it /// is looking at the step list this file was written against. /// /// An earlier revision guarded the scan with `checked < 8`, a floor sitting /// exactly at the current count. It fired on any legitimate step removal, said /// nothing about *which* step went missing, and would have been satisfied by a /// step being renamed. Naming the set costs the same edit and reports the /// actual difference. const declared_steps = [_][]const u8{ "test", "gate", "run", "e2e", "fuzz", "dst", "bench", "smoke", "cov" }; /// Steps declared in build.zig that CI deliberately does not run, each with the /// reason it is out of scope. This list is the point of the whole module: the /// pin below fails when build.zig grows a step that is neither invoked by CI /// nor named here, which forces the "should CI run this?" decision at the /// moment the step is written instead of never. /// Steps CI runs, but not as a bare `run: zig build <step>` line — the sweep /// below drives `fuzz` one target per invocation inside a loop, so the plain /// whole-line invocation the forward pin looks for never appears. Each entry /// carries the exact anchor that proves the step really is invoked, so this is /// an alternate proof of coverage, NOT an exemption from it: an entry whose /// anchor is absent fails exactly like a missing invocation. Keeping it /// separate from `not_in_ci` matters — conflating "runs, in a loop" with /// "deliberately skipped" is how a step silently stops being covered while a /// stale excuse keeps the pin green. const ci_parameterized = [_]struct { step: []const u8, anchor: []const u8, how: []const u8 }{ .{ .step = "fuzz", .anchor = "zig build fuzz -- \"${target}\" --seed \"${FUZZ_SEED}\" --rounds \"${FUZZ_ROUNDS}\"\n", .how = "swept over every target enumerated from the step's own usage output, at a fixed seed and round budget", }, }; const not_in_ci = [_]struct { step: []const u8, why: []const u8 }{ .{ .step = "run", .why = "developer convenience wrapper around the installed exe" }, .{ .step = "gate", .why = "aggregate of lanes CI already invokes one by one; running it too would rebuild and rerun the same work while reporting one failure where CI currently reports which lane broke. Note this means `gate` itself is never exercised in CI -- if a lane is ever added to `gate` WITHOUT a matching `run:` line below, it will be gated locally and nowhere else" }, .{ .step = "bench", .why = "measures throughput; a shared runner's timings are noise, and rule 7 forbids claims not backed by a real measurement" }, .{ .step = "cov", .why = "out-of-band by design (needs kcov installed); never gates a commit" }, }; test "the workflow triggers on every push and on pull requests" { // Anchored as the whole trigger block rather than as three loose // substrings: bare `on:` is contained in `runs-on:`, so it asserted nothing // about the triggers at all. try expect_contains(ci_workflow, "on:\n push:\n pull_request:\n", ".github/workflows/ci.yml"); // No `branches:` filter under the triggers: the release runs on // `release-0.1`, and a workflow that only fires for the default branch // would leave the entire release unverified until the merge. if (std.mem.indexOf(u8, ci_workflow, "branches:") != null) { std.debug.print("\nci pin: .github/workflows/ci.yml filters triggers by branch; " ++ "the release branch (release-0.1) must be gated too, so the triggers stay unfiltered\n", .{}); return error.CiPinMissing; } } test "the workflow pins every native Zig toolchain by version and digest" { // Target-specific digests live in the matrix; the environment assignment // must select the current leg's value rather than restating one platform. try expect_contains(ci_workflow, "ZIG_VERSION: " ++ pinned_zig_version ++ "\n", ".github/workflows/ci.yml"); try expect_contains(ci_workflow, "ZIG_SHASUM: ${{ matrix.zig_shasum }}\n", ".github/workflows/ci.yml"); try expect_contains(ci_workflow, "zig_shasum: " ++ pinned_zig_shasums[0] ++ "\n", ".github/workflows/ci.yml"); try expect_contains(ci_workflow, "zig_shasum: " ++ pinned_zig_shasums[1] ++ "\n", ".github/workflows/ci.yml"); try expect_contains(ci_workflow, "zig_shasum: " ++ pinned_zig_shasums[2] ++ "\n", ".github/workflows/ci.yml"); // The digest is only a supply-chain guarantee if something checks it. See // `pinned_digest_check`. try expect_contains(ci_workflow, pinned_digest_check, ".github/workflows/ci.yml"); } test "the workflow bounds how long a run may hang" { // The bounded-waits rule applied to CI itself. GitHub's default is 360 // minutes, which is indistinguishable from "no timeout" in practice: a // hung smoke server would hold the gate open for six hours without ever // reporting failure. // // So the values are pinned, not merely the key. Asserting that // `timeout-minutes:` appears twice is satisfied by `timeout-minutes: 360` // in both places — the default restated, certifying neither an interval nor // a bound. Each pin is a whole line with its YAML indentation, the way the // `permissions:\n contents: read\n` pin is: four spaces is the job // ceiling, eight is a step bound, so the indentation is what makes "the // job has a ceiling *and* smoke has a tighter one" a claim rather than a // count of keys. try expect_contains(ci_workflow, "\n timeout-minutes: 20\n", ".github/workflows/ci.yml"); // smoke spawns a server: it is the one step that can hang without producing // output, so it carries the tightest bound in the battery. try expect_contains(ci_workflow, "\n timeout-minutes: 5\n", ".github/workflows/ci.yml"); } test "the workflow takes the least privilege it can and cancels superseded runs" { // The battery only reads the checkout. Without an explicit block the job // inherits the repo/org default, which is read/write on many // configurations — a wider token than any step here has a use for. try expect_contains(ci_workflow, "permissions:\n contents: read\n", ".github/workflows/ci.yml"); // Unfiltered `push:` plus `pull_request:` doubles up on PR branches; a // superseded run should not keep burning a runner. Anchored to the whole // line with its `concurrency:` indentation, like the `permissions:` pin // above: the bare substring is satisfied by the phrase in a comment, or by // `cancel-in-progress: true` nested under something that does not govern // this workflow's runs. try expect_contains(ci_workflow, "\n cancel-in-progress: true\n", ".github/workflows/ci.yml"); } test "the workflow installs fossil and records the version it got" { // smoke drives the real `fossil` binary; without it the conformance claim // the release rests on is not exercised at all. Anchored to the install // command rather than to the bare word "fossil": that word appears in this // file's own explanatory comments, so the loose pin stayed green even with // the entire install step deleted. // The self-hosted runner is a developer machine with fossil already // installed (2.28), so the workflow verifies rather than installs — but it // must still HARD-FAIL when fossil is absent, or smoke would be skipped and // the conformance claim would quietly stop being exercised. Pinned to the // guard, not to the bare word "fossil": that word appears in this file's // own comments, so a loose pin stayed green with the whole step deleted. try expect_contains(ci_workflow, "command -v fossil >/dev/null 2>&1 || {", ".github/workflows/ci.yml"); // The runner's fossil predates the local 2.28, and the release notes have // to name the exact version smoke passed against (readiness item S7). That // is only knowable if the run prints it — so pin the whole command line, // which a prose mention of "fossil version" cannot satisfy. try expect_contains(ci_workflow, "\n fossil version\n", ".github/workflows/ci.yml"); } test "the workflow runs the whole release battery, in order" { // `previous` starts unset rather than at 0 so the first entry is not // required to sit at a strictly positive offset — the comparison below is // strict, and a battery entry legitimately could begin at byte 0. var previous: ?usize = null; for (battery) |command| { const at = std.mem.indexOf(u8, ci_workflow, command) orelse { std.debug.print("\nci pin: .github/workflows/ci.yml has no line \"{s}\"", .{command}); return error.CiPinMissing; }; // Each battery command runs exactly once. Ordering alone cannot make // this claim: duplicating a step leaves every entry's *first* offset // unchanged, so the comparison below stays satisfied while the battery // silently burns a slow step twice. const runs = count_occurrences(ci_workflow, command); if (runs != 1) { std.debug.print("\nci pin: .github/workflows/ci.yml runs \"{s}\" {d} times; " ++ "each battery command is run exactly once\n", .{ command, runs }); return error.CiPinMissing; } // Ordering is part of the claim: format check before build before // tests, so a run fails at the cheapest step that can catch the fault // rather than after the slow ones have burned runner minutes. // Strict, not `<`: with the exactly-once pin above, equal offsets are // unreachable, so the strictness is a statement of intent rather than a // live guard. Note the remaining softness — `indexOf` finds the *first* // occurrence, so a command named in a comment above its real step is // what gets ordered. // That is the text-assertion trade this module's header declares. if (previous) |before| if (at <= before) { std.debug.print("\nci pin: .github/workflows/ci.yml runs \"{s}\" out of order; " ++ "the battery in src/ci_test.zig is ordered cheapest-failure-first\n", .{command}); return error.CiPinMissing; }; previous = at; } } test "the workflow enumerates the complete fuzz registry through list mode" { // A failed bare fuzz invocation is summarized by `zig build`; on Linux its // leading target names can be truncated before the workflow parses them. // Successful list mode makes complete enumeration part of the CLI contract. try expect_contains( ci_workflow, "targets=\"$(zig build fuzz -- --list)\"\n", ".github/workflows/ci.yml", ); } test "the workflow is one matrix job gated on explicit native runners" { // One matrix job runs the identical battery on every native release target. // Its conjunctive labels prevent a leg from falling onto an arbitrary // self-hosted runner, while each matrix entry names one dedicated machine. try expect_contains( ci_workflow, "\n runs-on: [self-hosted, \"${{ matrix.os }}\", \"${{ matrix.arch }}\", \"${{ matrix.runner }}\"]\n", ".github/workflows/ci.yml", ); try expect_contains(ci_workflow, "runner: mac-mini-zit\n", ".github/workflows/ci.yml"); try expect_contains(ci_workflow, "runner: linux-x64-zit\n", ".github/workflows/ci.yml"); try expect_contains(ci_workflow, "runner: linux-arm64-zit\n", ".github/workflows/ci.yml"); const jobs = count_occurrences(ci_workflow, "runs-on:"); if (jobs != 1) { std.debug.print("\nci pin: .github/workflows/ci.yml declares {d} jobs (`runs-on:` keys); " ++ "0.1.1 gates all native targets through exactly one matrix job\n", .{jobs}); return error.CiPinMissing; } } test "every build.zig step is either run by CI or a reviewed exception" { // The negative-space pin. Scans build.zig for `b.step("<name>"` / // `env.b.step("<name>"` declarations and requires each name to be // accounted for. Text matching, as elsewhere in this file: a step whose // name is computed rather than a literal is invisible here, which is // stated rather than assumed. build.zig declares them all literally today. const marker = ".step(\""; var seen = [_]bool{false} ** declared_steps.len; var scan: usize = 0; while (std.mem.indexOfPos(u8, build_source, scan, marker)) |hit| { const start = hit + marker.len; const end = std.mem.indexOfScalarPos(u8, build_source, start, '"') orelse { // Unreachable while build.zig compiles, but failing loudly beats a // `break` that would silently truncate the scan. std.debug.print("\nci pin: build.zig has an unterminated .step(\"...) literal " ++ "at byte offset {d}; the scan cannot continue\n", .{start}); return error.CiPinMissing; }; scan = end + 1; const name = build_source[start..end]; for (declared_steps, &seen) |expected, *hit_before| { if (std.mem.eql(u8, expected, name)) { hit_before.* = true; break; } } else { std.debug.print("\nci pin: build.zig declares step \"{s}\", which src/ci_test.zig's " ++ "`declared_steps` does not list. Add it there (and decide whether CI runs it) " ++ "so the scan keeps describing the real step set.\n", .{name}); return error.CiPinMissing; } // The invocation is matched as a whole `run:` line, not as a substring: // `zig build e2e` contains `zig build e`, so an unanchored match would // let a step named "e" read as covered by the e2e leg. var invocation_buf: [128]u8 = undefined; const invocation = std.fmt.bufPrint(&invocation_buf, "run: zig build {s}\n", .{name}) catch { std.debug.print("\nci pin: build.zig declares a step name longer than {d} bytes " ++ "(\"{s}\"); widen invocation_buf\n", .{ invocation_buf.len, name }); return error.CiPinMissing; }; if (std.mem.indexOf(u8, ci_workflow, invocation) != null) continue; // Covered, but invoked in a loop rather than as a bare `run:` line. // The anchor is verified, not merely trusted: an entry naming a step // whose anchor has drifted out of the workflow is a step CI stopped // running, and it must fail like any other gap. var parameterized_covered = false; for (ci_parameterized) |parameterized| { if (!std.mem.eql(u8, parameterized.step, name)) continue; if (std.mem.indexOf(u8, ci_workflow, parameterized.anchor) == null) { std.debug.print("\nci pin: src/ci_test.zig records step \"{s}\" as run by CI " ++ "({s}), but .github/workflows/ci.yml no longer contains its invocation. " ++ "Either restore the sweep or move the step to `not_in_ci` with a reason.\n", .{ name, parameterized.how }); return error.CiPinMissing; } parameterized_covered = true; break; } if (parameterized_covered) continue; for (not_in_ci) |exception| { if (std.mem.eql(u8, exception.step, name)) break; } else { std.debug.print("\nci pin: build.zig declares step \"{s}\", which " ++ ".github/workflows/ci.yml never runs (`zig build {s}`) and src/ci_test.zig's " ++ "`not_in_ci` list does not excuse. Either add it to the workflow or " ++ "record why CI skips it.\n", .{ name, name }); return error.CiPinMissing; } } // The other half of the exact-set assertion, and the vacuity guard: a scan // that matched nothing would otherwise pass having checked nothing. for (declared_steps, seen) |expected, found| { if (found) continue; std.debug.print("\nci pin: src/ci_test.zig expects build.zig to declare step \"{s}\", " ++ "but the scan for `.step(\"...)` did not find it — the step was removed or " ++ "renamed, or the scan pattern has drifted from the source\n", .{expected}); return error.CiPinMissing; } } test "the reviewed not-in-CI exceptions are real build.zig steps that CI does not run" { // The other direction: an exception for a step that no longer exists is // dead permission. Left unchecked, `not_in_ci` accumulates entries that // quietly excuse nothing while looking like they excuse something. for (not_in_ci) |exception| { var declaration_buf: [128]u8 = undefined; const declaration = std.fmt.bufPrint(&declaration_buf, ".step(\"{s}\"", .{exception.step}) catch { std.debug.print("\nci pin: not_in_ci names a step longer than {d} bytes " ++ "(\"{s}\"); widen declaration_buf\n", .{ declaration_buf.len, exception.step }); return error.CiPinMissing; }; if (std.mem.indexOf(u8, build_source, declaration) == null) { std.debug.print("\nci pin: src/ci_test.zig excuses step \"{s}\" from CI (\"{s}\"), " ++ "but build.zig no longer declares it; drop the stale exception\n", .{ exception.step, exception.why }); return error.CiPinMissing; } // And the negative space itself, which is this module's declared // purpose and was the one direction nothing asserted: an excused step // must actually be absent from the workflow. Without this, adding // `zig build bench` to CI would silently falsify that entry's stated // reason — "a shared runner's timings are noise, and rule 7 forbids // claims not backed by a real measurement" — with no test failing and // the excuse left in the file reading as though it still held. // // Matched as a whole `run:` line for the same reason the forward // direction is: `zig build e2e` contains `zig build e`. var invocation_buf: [128]u8 = undefined; const invocation = std.fmt.bufPrint(&invocation_buf, "run: zig build {s}\n", .{exception.step}) catch { std.debug.print("\nci pin: not_in_ci names a step longer than {d} bytes " ++ "(\"{s}\"); widen invocation_buf\n", .{ invocation_buf.len, exception.step }); return error.CiPinMissing; }; if (std.mem.indexOf(u8, ci_workflow, invocation) == null) continue; std.debug.print("\nci pin: .github/workflows/ci.yml runs \"{s}\", but src/ci_test.zig's " ++ "`not_in_ci` excuses it from CI (\"{s}\"). Either drop the exception or drop the " ++ "step from the workflow — the recorded reason no longer describes what CI does.\n", .{ exception.step, exception.why }); return error.CiPinMissing; } } test "the shipped release mode keeps assertions alive" { // src/ carries over a thousand `std.debug.assert` calls. They are the // mechanism by which a detected inconsistency becomes a crash instead of a // durable write — TigerStyle's "downgrade catastrophic correctness bugs // into liveness bugs" — and that trade exists only while the assertion // does. ReleaseFast and ReleaseSmall compile every one to nothing, which in // an append-only store converts each guarded corruption into an unguarded // one: the bad frame is permanent, the crash would not have been. // // So `--release` must resolve to ReleaseSafe. Without the preference, // `standardOptimizeOption(.{})` leaves the choice to whoever packages, and // the failure is silent in both directions: nothing warns at build time and // nothing differs at run time until a corruption that would have been // caught is written instead. // // This pins the DEFAULT, not the ceiling. An explicit // `-Doptimize=ReleaseFast` still works for someone who has weighed it; // what it prevents is reaching that mode by omission. const preference = ".preferred_optimize_mode = .ReleaseSafe"; if (std.mem.indexOf(u8, build_source, preference) == null) { std.debug.print("\nci pin: build.zig no longer states `{s}` in its " ++ "standardOptimizeOption call. A release built without it strips every " ++ "std.debug.assert in src/, turning guarded corruption into silent " ++ "corruption in an append-only store. Restore the preference, or if the " ++ "project has genuinely decided to ship unchecked, delete this test in " ++ "the same commit so the decision is visible in history.\n", .{preference}); return error.CiPinMissing; } // Negative space: the preference is worthless if the call it sits in is not // the one that produces the top-level `optimize`. Pin the pairing, not just // the substring — a stray mention in a comment must not satisfy this. const call = "standardOptimizeOption(.{ .preferred_optimize_mode = .ReleaseSafe })"; if (std.mem.indexOf(u8, build_source, call) == null) { std.debug.print("\nci pin: build.zig mentions the ReleaseSafe preference but not as " ++ "`{s}`. The preference only binds when it is passed to the call that yields the " ++ "top-level optimize mode.\n", .{call}); return error.CiPinMissing; } }