Artifact [29f0efc333]

Artifact 29f0efc3338c14eb87085cd87591d01d78ae1feac8b3f6ab7f5a2533ac01a7db:

• File src/sim/faulty_dir.zig — part of check-in [0cea021cf9] at 2026-07-20 04:21:55 on branch h2-checkout-dst — sim: working-tree crash-recovery DST sweep for update

  Add sim/faulty_dir.zig, a decorator std.Io that cuts a run at a seeded mutating-op boundary the way a power loss does: the crashing write persists at most a torn prefix and every later mutation is a frozen no-op, so the checkout layer's own recovery moves no byte. Crash-point enumeration follows TigerBeetle's storage-fault-injection lineage, ported to the directory surface update.zig and checkout.zig write through.

  Add sim/checkout_sweep.zig: it materializes a check-in, replays an A-to-B update.applyCheckin under a crash at every reachable cut, reopens, and asserts unmanaged files are preserved, the checkout record names only a complete target, and the tree recovers to the target via the undo journal. Joins the dst gate per seed at bounded cuts; the soak walks every boundary.

  Finding: update.writePayload writes working files in place (createFile truncate, write, sync), not through the atomic write-temp+fsync+rename that checkout.write, the managed set, and the undo journal use. A crash mid-write can leave a file matching neither its pre- nor its post-image. The sweep pins this; a working-tree write-journal is the fix. (user: dmestas, size: 13665) [annotate] [blame] [check-ins using] [more...]

//! Deterministic working-tree crash simulator (H2, native harness design — no
//! RFC clause touches the working tree). A decorator `std.Io` that sits in front
//! of a healthy backend io and, at a single seeded operation index, "crashes":
//! it kills the run's forward progress the way a power loss does — the crashing
//! mutation applies at most a torn prefix, and every mutating op after it is a
//! no-op error, so whatever the checkout layer's own rollback tries to do next
//! moves no byte. The seed is the whole universe: the same seed reproduces the
//! same cut and the same on-disk aftermath exactly.
//!
//! The crash-point enumeration technique — walk the operation sequence and cut
//! at every reachable boundary, then assert recovery from each cut — is
//! TigerBeetle's storage-fault-injection lineage (the VOPR's disk model),
//! adapted here from block storage to the POSIX directory surface `update.zig`
//! and `checkout.zig` write through. It mirrors sim/faulty_file.zig's
//! discipline: every fault is a draw from the run PRNG, corruption is a pure
//! function of the payload so a retry never self-heals, and the healthy default
//! (no `crash_at_op`) is a perfect disk that DST phase 2 recovers over.
//!
//! Mechanism: the decorator keeps the backend's `userdata` and copies the
//! backend `VTable`, overriding only the mutating entries. The non-mutating
//! entries stay the backend's own function pointers, so reads/opens/stats never
//! lie — a crash tears WRITES, never the ability to observe what survived.
//! The active decorator is a single-run install (one sweep at a time, single
//! threaded, exactly like update.zig's `test_fault`); the overrides reach it
//! through `active`.

const std = @import("std");

const Io = std.Io;
const Dir = std.Io.Dir;
const File = std.Io.File;

/// The one decorator wired for the current run. Set by `install`, cleared by
/// `uninstall`; the vtable overrides read it. Single-threaded by construction
/// (the DST sweep drives one operation at a time), the same shape update.zig's
/// `test_fault` global uses to reach a file-private fault seam.
var active: ?*FaultyDir = null;

/// Crash schedule for one run. `crash_at_op == null` is a perfect disk (the
/// healthy default DST phase 2 recovers over); a set value crashes as the
/// `crash_at_op`-th mutating operation begins.
pub const CrashCfg = struct {
    /// One-indexed mutating-operation number to crash at, or null for never.
    /// Counting covers every write/sync/create/delete/rename the run performs,
    /// so a value in `[1, total_ops]` selects a reachable cut.
    crash_at_op: ?u64 = null,
    /// When true, the crashing WRITE first persists a random strict prefix of
    /// its bytes (a torn write) before failing — the byte-granular power-loss
    /// tear that leaves a file matching neither its pre- nor post-image. When
    /// false the crash is operation-granular (the write does not apply at all).
    torn_write: bool = false,
    /// When true, the torn prefix is additionally bit-flipped with a
    /// payload-derived mask (pure function of the offset, never self-heals),
    /// mirroring faulty_file.zig's `corruptionMask`.
    corrupt_torn: bool = false,
};

pub const FaultyDir = struct {
    /// The healthy io every non-crashing op delegates to.
    backend: Io,
    /// The backend's own vtable, kept so the overrides can call through it.
    backend_vtable: *const Io.VTable,
    /// A copy of `backend_vtable` with the mutating entries pointed at the
    /// overrides below. `io()` hands out `&self.vtable`, so `self` must outlive
    /// every use (the caller pins it as a `var`).
    vtable: Io.VTable,
    random: std.Random,
    cfg: CrashCfg,
    /// Mutating operations begun so far (monotonic; the crash index is compared
    /// against it).
    op_index: u64 = 0,
    /// Latched once the cut fires; every later mutating op is a frozen no-op.
    crashed: bool = false,
    /// The operation index the crash landed at (0 until it fires) — the value a
    /// failure report prints so the cut is auditable.
    crash_op_index: u64 = 0,

    /// The injected failure. `AccessDenied` is a member of every mutating error
    /// set on this path (write/sync/create/delete/rename), so one constant maps
    /// cleanly into all of them; update.zig collapses any of them to its own
    /// `error.Storage`, exactly as a real device fault would.
    const injected = error.AccessDenied;

    pub fn init(backend: Io, random: std.Random, cfg: CrashCfg) FaultyDir {
        var v = backend.vtable.*;
        v.fileWritePositional = wrapWrite;
        v.fileSync = wrapSync;
        v.dirCreateFile = wrapCreateFile;
        v.dirCreateFileAtomic = wrapCreateFileAtomic;
        v.dirCreateDir = wrapCreateDir;
        v.dirDeleteFile = wrapDeleteFile;
        v.dirDeleteDir = wrapDeleteDir;
        v.dirRename = wrapRename;
        return .{
            .backend = backend,
            .backend_vtable = backend.vtable,
            .vtable = v,
            .random = random,
            .cfg = cfg,
        };
    }

    /// The decorated io. Shares the backend's `userdata`, so the un-overridden
    /// backend entries receive exactly the context they expect.
    pub fn io(self: *FaultyDir) Io {
        return .{ .userdata = self.backend.userdata, .vtable = &self.vtable };
    }

    pub fn install(self: *FaultyDir) void {
        std.debug.assert(active == null); // one decorator at a time
        active = self;
    }

    pub fn uninstall() void {
        active = null;
    }

    const Decision = enum { proceed, crash_now, frozen };

    fn gate(self: *FaultyDir) Decision {
        if (self.crashed) return .frozen;
        self.op_index += 1;
        if (self.cfg.crash_at_op) |k| {
            if (self.op_index >= k) {
                self.crashed = true;
                self.crash_op_index = self.op_index;
                return .crash_now;
            }
        }
        return .proceed;
    }

    fn wrapWrite(
        ud: ?*anyopaque,
        file: File,
        header: []const u8,
        data: []const []const u8,
        splat: usize,
        offset: u64,
    ) File.WritePositionalError!usize {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.fileWritePositional(ud, file, header, data, splat, offset),
            .frozen => return injected,
            .crash_now => {
                if (self.cfg.torn_write) self.applyTorn(ud, file, data, offset);
                return injected;
            },
        }
    }

    /// Persist a strict prefix of the crashing write (the torn tail a power loss
    /// leaves in the page cache), optionally payload-corrupted. Best-effort: a
    /// backend hiccup here only means a slightly shorter tear, never a false
    /// invariant verdict.
    fn applyTorn(self: *FaultyDir, ud: ?*anyopaque, file: File, data: []const []const u8, offset: u64) void {
        for (data) |buf| {
            if (buf.len < 2) continue;
            const k = 1 + self.random.uintLessThan(usize, buf.len - 1); // 1..=len-1
            if (self.cfg.corrupt_torn) {
                var scratch: [512]u8 = undefined;
                const n = @min(k, scratch.len);
                @memcpy(scratch[0..n], buf[0..n]);
                var i: usize = 0;
                while (i < n) : (i += 1) scratch[i] ^= corruptionMask(offset + i);
                _ = self.backend_vtable.fileWritePositional(ud, file, &.{}, &.{scratch[0..n]}, 1, offset) catch {};
            } else {
                _ = self.backend_vtable.fileWritePositional(ud, file, &.{}, &.{buf[0..k]}, 1, offset) catch {};
            }
            return;
        }
    }

    fn wrapSync(ud: ?*anyopaque, file: File) File.SyncError!void {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.fileSync(ud, file),
            .frozen, .crash_now => return injected,
        }
    }

    fn wrapCreateFile(ud: ?*anyopaque, dir: Dir, sub_path: []const u8, opts: Dir.CreateFileOptions) File.OpenError!File {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.dirCreateFile(ud, dir, sub_path, opts),
            .frozen, .crash_now => return injected,
        }
    }

    fn wrapCreateFileAtomic(ud: ?*anyopaque, dir: Dir, sub_path: []const u8, opts: Dir.CreateFileAtomicOptions) Dir.CreateFileAtomicError!File.Atomic {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.dirCreateFileAtomic(ud, dir, sub_path, opts),
            .frozen, .crash_now => return injected,
        }
    }

    fn wrapCreateDir(ud: ?*anyopaque, dir: Dir, sub_path: []const u8, perm: Dir.Permissions) Dir.CreateDirError!void {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.dirCreateDir(ud, dir, sub_path, perm),
            .frozen, .crash_now => return injected,
        }
    }

    fn wrapDeleteFile(ud: ?*anyopaque, dir: Dir, sub_path: []const u8) Dir.DeleteFileError!void {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.dirDeleteFile(ud, dir, sub_path),
            .frozen, .crash_now => return injected,
        }
    }

    fn wrapDeleteDir(ud: ?*anyopaque, dir: Dir, sub_path: []const u8) Dir.DeleteDirError!void {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.dirDeleteDir(ud, dir, sub_path),
            .frozen, .crash_now => return injected,
        }
    }

    fn wrapRename(ud: ?*anyopaque, old_dir: Dir, old_sub_path: []const u8, new_dir: Dir, new_sub_path: []const u8) Dir.RenameError!void {
        const self = active.?;
        switch (self.gate()) {
            .proceed => return self.backend_vtable.dirRename(ud, old_dir, old_sub_path, new_dir, new_sub_path),
            .frozen, .crash_now => return injected,
        }
    }
};

/// Payload-derived, pure function of the byte offset (faulty_file.zig §B): every
/// retry corrupts identically and the mask is always nonzero, so it always flips.
fn corruptionMask(off: u64) u8 {
    const h = std.hash.Wyhash.hash(0, std.mem.asBytes(&off));
    return @truncate(1 | (h & 7));
}

test "faulty_dir: a healthy schedule delegates every op unchanged" {
    const t = std.testing;
    var tmp = t.tmpDir(.{});
    defer tmp.cleanup();

    var prng = std.Random.DefaultPrng.init(1);
    var faulty = FaultyDir.init(t.io, prng.random(), .{}); // no crash
    faulty.install();
    defer FaultyDir.uninstall();
    const io = faulty.io();

    var file = try tmp.dir.createFile(io, "f", .{ .read = false, .truncate = true, .resolve_beneath = true });
    try file.writePositionalAll(io, "hello world", 0);
    try file.sync(io);
    file.close(io);
    try t.expect(!faulty.crashed);

    const back = try tmp.dir.readFileAlloc(t.io, "f", t.allocator, .limited(64));
    defer t.allocator.free(back);
    try t.expectEqualStrings("hello world", back);
}

test "faulty_dir: the crash op errors and freezes every later mutation" {
    const t = std.testing;
    var tmp = t.tmpDir(.{});
    defer tmp.cleanup();

    var prng = std.Random.DefaultPrng.init(2);
    // Crash at the first mutating op: the very first createFile fails, and the
    // decorator latches so the follow-up delete is a no-op error too.
    var faulty = FaultyDir.init(t.io, prng.random(), .{ .crash_at_op = 1 });
    faulty.install();
    defer FaultyDir.uninstall();
    const io = faulty.io();

    try t.expectError(error.AccessDenied, tmp.dir.createFile(io, "f", .{ .truncate = true, .resolve_beneath = true }));
    try t.expect(faulty.crashed);
    try t.expectEqual(@as(u64, 1), faulty.crash_op_index);
    // Frozen: a subsequent delete cannot move a byte.
    try t.expectError(error.AccessDenied, tmp.dir.deleteFile(io, "anything"));
}

test "faulty_dir: a torn write leaves a strict prefix, matching neither image" {
    const t = std.testing;
    var tmp = t.tmpDir(.{});
    defer tmp.cleanup();

    // First lay down the pre-image with a healthy decorator.
    {
        var prng0 = std.Random.DefaultPrng.init(0);
        var healthy = FaultyDir.init(t.io, prng0.random(), .{});
        healthy.install();
        defer FaultyDir.uninstall();
        const hio = healthy.io();
        var f = try tmp.dir.createFile(hio, "f", .{ .truncate = true, .resolve_beneath = true });
        try f.writePositionalAll(hio, "AAAAAAAAAAAAAAAA", 0);
        try f.sync(hio);
        f.close(hio);
    }

    // Now crash mid-write of the post-image: the file already exists, so open +
    // positional overwrite tears a prefix of the new bytes over the old.
    var prng = std.Random.DefaultPrng.init(7);
    var faulty = FaultyDir.init(t.io, prng.random(), .{ .crash_at_op = 1, .torn_write = true });
    faulty.install();
    defer FaultyDir.uninstall();
    const io = faulty.io();

    var f = try tmp.dir.openFile(io, "f", .{ .mode = .write_only, .resolve_beneath = true });
    try t.expectError(error.AccessDenied, f.writePositionalAll(io, "BBBBBBBBBBBBBBBB", 0));
    f.close(io);

    const back = try tmp.dir.readFileAlloc(t.io, "f", t.allocator, .limited(64));
    defer t.allocator.free(back);
    // A strict, non-empty prefix of B's overwrote A's: neither all-A nor all-B.
    try t.expect(back.len == 16);
    try t.expect(std.mem.indexOfScalar(u8, back, 'B') != null); // some new bytes landed
    try t.expect(std.mem.indexOfScalar(u8, back, 'A') != null); // but not all of them
}