Artifact 5bb73d38f08b926f14b81f6d22c67be3a116f8fce0baf61c5121cc71dae70173:
- File
src/logstore.zig
— part of check-in
[230c2bbebe]
at
2026-07-30 10:34:15
on branch shun-implementation
— shun S4: stop advertising what the shun table names
LogIndexStore keeps a shunned holding in its enumeration and MASKS it rather than removing it, so a shun row appended between two rounds of one clone cannot renumber the survivors under the pager's cursor. held() answers false, the name leaves by_name, and SS9.5's 'igot for every eligible holding after the SS3.3 filters' is where that mask lives.
The plan overstated what was missing. S1's log-level mask already nulled content and wire, so a clone silently skipped a shunned artifact before this slice. What was actually broken is narrower and worse: because by_name still resolved the name, emitCloneReply's basis proof fired in a safety build and the SERVE PROCESS PANICKED -- a fossil clone got 'server did not reply', not an absent artifact. Reproduced against fossil 2.28 and pinned as smoke S9.
That also means the planned e2e ('a clone does not receive it') was never mutation-sensitive: it passes with this slice fully removed. The observable that pins the slice is the ABSENCE of an igot card naming a shunned artifact in a non-clone round's inventory walk, which now has its own test and a second round in the raw-HTTP e2e case.
Smoke uses a defer-shunned fixture, not the planned packed one: pack rebuilds the delta chains and so removes the exact hazard the check exists to catch.
logstore.zig carries a comment-only change: LogStore.shunned's doc claimed two questions were open that S4 and S5 have since closed. (user: dmestas, size: 181934) [annotate] [blame] [check-ins using]
//! The framed, append-only, content-addressed log over the `StorageFile` //! capability. One file: a 64-byte header followed by 8-byte-aligned //! frames, each self-verifying via a header checksum (over the fixed //! fields) plus a whole-frame checksum (over header+payload). The header //! checksum is what makes torn-vs-corrupt discrimination exact and //! size-independent: without it, a torn write (a clean prefix, safe to //! truncate) and a corrupted length field (unsafe to trust) both read as //! "not enough of a frame," and nothing short of decoding payload bytes //! that may not exist could tell them apart. //! //! No RFC section behind this file: native storage-engine design. The //! oracle is this file's own framing invariants, checked by //! sim/storage_sweep.zig's DST sweep: every acknowledged put stays //! readable and hash-verifies; a torn tail truncates to exactly the last //! complete frame; the index rebuilt on open matches the write-time //! model; and corruption always surfaces as an error, never as silently //! wrong bytes. //! //! A second, narrower checksum guards the frame's fixed 6-byte prefix //! (`len|flags|hash_len`) on its own, in a 10-byte window whose size //! never depends on `hash_len`. Without it, scanning a frame whose //! `hash_len` byte is itself the bitflipped one has no way to know how //! many bytes the rest of the header needs before it can check the //! header checksum -- a corrupted `hash_len` that happens to demand more //! bytes than the file has left reads exactly like a torn tail. The //! prefix checksum removes that ambiguity: `hash_len` is proven correct //! (or rejected outright) from a fixed-size read, before it is ever used //! to size anything else. const std = @import("std"); const storage = @import("storage.zig"); const crc32 = @import("crc32.zig"); const delta = @import("delta.zig"); /// The §4.1 zlib container codec — the leaf module shared with `repostate` and /// `sync`. The store compresses artifacts through it (the hash still names the /// UNCOMPRESSED bytes, §4.1 L1206-1208) and inflates them back via /// `decodeContainer` below. const codec = @import("container.zig"); /// Local artifact id: dense, 1-based, assigned in intern/put order. 0 is /// never issued (mirrors repostate.zig's MemStore). pub const Rid = u64; /// FILE header size: magic(4) | version u16(2) | hash_kind u8(1) | /// reserved(53) | crc32(4). pub const header_len: u64 = 64; /// Explicit payload bound: keeps a single frame's read/decode work /// bounded no matter how large a check-in gets. pub const max_payload: u64 = 1 << 30; pub const max_hash_len: u8 = 32; pub const min_hash_len: u8 = 1; /// The two COMPLETE artifact-name widths, as raw digests. artifact-format §3.2 /// L347-348: "Exactly two artifact-name forms exist: a 40-hexadecimal-digit /// name, which denotes a SHA-1 hash [RFC3174], and a 64-hexadecimal-digit" /// name — 20 and 32 raw bytes, algorithm inferred by length. The 64-hex form is /// SHA3-256, not SHA-256 (§3.2 L349: "name, which denotes a SHA3-256 hash /// [FIPS202]"); the two algorithms share a digest width and nothing else, so /// the constant is named for the one the draft names. `isCompleteDigest` is /// deliberately NOT applied to `put`/`putArtifact`, whose `min_hash_len .. /// max_hash_len` bound stays opaque; it exists for the one caller whose key /// must be a complete name and never a prefix (`putShun` — see `ShunRecord`). pub const sha1_digest_len: usize = 20; pub const sha3_256_digest_len: usize = 32; /// Is `n` the length of a complete (non-abbreviated) artifact digest? A whole /// digest is bijective with the complete hex name; any other width is a prefix /// or a non-name, and cannot key a set the drafts describe as keyed by complete /// artifact name. Rejecting every other width is not local taste: §3.2 L349-350 /// makes the two-forms rule normative — "A candidate name of any other length /// MUST be rejected as invalid" — and the bijection carries that disposition /// from the hex name to the digest form. pub fn isCompleteDigest(n: usize) bool { return n == sha1_digest_len or n == sha3_256_digest_len; } /// Every frame's on-disk span is a multiple of this. pub const frame_align: u64 = 8; /// Frozen on-disk format version — the `.zit` ship-format identity. A reader /// accepts only this exact stored version and rejects any other, older /// included (`validateHeader`): the writer emits exactly this value, so a /// header carrying anything else is not a zit-written file. An unknown frame /// flag-kind is likewise a clean reject, never UB (`scanOneFrame`). Bumped /// only on a layout change — a copied `.zit` file is a complete, /// self-verifying repository, so this field is what absorbs any future layout /// evolution; widening to a compatibility range belongs to that change, not /// to a version this reader has never produced. /// /// Adding `FrameKind.shun` did NOT bump this, deliberately: an older binary /// meeting a shun frame fails the whole open with `error.Corrupt` (fail-closed /// — see `FrameKind`), which is exactly what a tombstone needs, whereas a bump /// would reject every shun-free log that binary itself wrote. pub const version: u16 = 1; /// Smallest possible frame span: hash_len=1, payload=0 -> /// align8(26+1+0) = 32. const min_frame: u64 = 32; /// Explicit upper bound on delta-chain resolution: a `content()` on a /// delta frame walks at most this many `delta_of` hops before rejecting /// the chain as corrupt. Caps read recursion and read-amplification; /// generous versus the short chains a real store produces. pub const max_delta_chain: u8 = 32; /// The deepest delta chain `putArtifact` will CREATE (write policy), kept /// strictly below the READ bound `max_delta_chain` so a retained foreign /// chain (a peer with no write cap) still has headroom to resolve. A base /// already at this depth is stored full, so a locally-produced chain never /// exceeds this and `content()`'s read walk stays shallow. pub const delta_chain_write_max: u8 = 16; /// fossil `N_NEIGHBOR` (content.c): the most delta bases `pickBase` will ever /// try for one artifact. Bounds the write-path fan-out to at most this many /// `delta.create` runs per put — the whole reason the candidate window is a /// fixed slice, not the unbounded log. pub const max_base_candidates: usize = 5; /// The most frames `putArtifact`'s backward window walk will ever step over to /// gather its chronological candidates, independent of phantom density. The /// candidate cap (`max_base_candidates`) bounds deltas EXAMINED, not frames /// SCANNED: a run of trailing phantom frames (a normal sync/clone artifact of /// interned forward-references) is skipped by `continue` without filling a /// candidate slot, so without a scan bound one put could step over the entire /// store to gather < 5 real bases — hidden O(n^2) over N puts. This caps the /// per-put scan to a fixed window: at most this many entries are inspected, /// so the write-path cost stays O(1) in store size no matter how dense the /// phantoms. Generous versus `max_base_candidates` so an ordinary phantom run /// never starves the window of real bases; a base past the window only costs /// compactness, never correctness (candidates are hints). pub const max_base_scan: usize = 64; /// fossil's per-source floor (content.c:908): a candidate whose reconstructed /// content is shorter than this is skipped BEFORE `delta.create` — the cheap /// length early-out. A delta's fixed framing overhead swamps any saving a /// sub-50-byte source could offer, so running the encoder is wasted work. const delta_min_source: u64 = 50; /// The §5 accept ratio (fossil content.c:914): a delta is worth storing only /// when it is STRICTLY under 75% of the uncompressed artifact — `delta.len * 4 /// < bytes.len * 3`. Integer math, no float (`delta_accept_den`/`num` = 4/3), /// replacing the former looser 9/10 write-path threshold. const delta_accept_num: u64 = 3; const delta_accept_den: u64 = 4; /// The §5 accept test in one pure predicate (fossil content.c:914): is a delta /// of `delta_len` bytes worth storing for a `target_len`-byte artifact? True /// iff strictly under 75% — `delta_len * 4 < target_len * 3`. Integer math on /// u64, overflow-free by a wide margin: `target_len <= max_payload` (< 2^30, the /// caller's asserted bound) so `target_len * 3 < 2^32`; `delta_len` is a delta /// of a `<= max_payload` target, exceeding it only by bounded framing overhead, /// so it stays far below 2^31 and `delta_len * 4` far below 2^33 — neither /// product approaches u64's range. Exact at the integer boundary — 75/100 /// rejects, 74/100 accepts. fn deltaProfitable(delta_len: usize, target_len: usize) bool { return @as(u64, delta_len) * delta_accept_den < @as(u64, target_len) * delta_accept_num; } /// The winning base and the raw (uncompressed) delta reconstructing the /// artifact from it — `pickBase`'s result. The caller owns `delta` and stores /// it (e.g. via `putDelta`), or frees it if it chooses not to. pub const Pick = struct { base: Rid, delta: []u8 }; const magic = "zitL"; /// Payload encoding. `raw` = uncompressed bytes; `zlib` = a fossil zlib /// container (4-byte BE uncompressed length + RFC1950 stream, sync §4.1); /// `delta` = a zlib-contained delta whose `delta_of` is the byte offset of /// the base frame, so resolving a chain link costs one seek and never a /// second index lookup; `config` = a distinguished singleton repo-config /// record, NOT content-addressed and never interned; `shun` = one repository /// shun-table row, likewise never interned (see `ShunRecord`). Any other flag /// byte is an unknown kind and rejected on open. The /// content hash always names the UNCOMPRESSED, fully-reconstructed bytes, so /// encoding never affects naming (sync §4.1 L1206-1208). /// /// FORMAT-VERSION NOTE for `shun = 4`: `version` deliberately stays 1. A `.zit` /// carrying a shun frame, read by a binary that predates this kind, hits /// `readFrameHeader`'s unknown-kind reject and fails the ENTIRE open with /// `error.Corrupt` — fail-closed by design, never a skipped frame and never /// UB. That is the behaviour a tombstone requires: an older binary MUST NOT /// silently ignore one. Bumping `version` would instead reject every shun-free /// log the older binary wrote, which is strictly worse. pub const FrameKind = enum(u8) { raw = 0, zlib = 1, delta = 2, config = 3, shun = 4, _ }; /// One past the highest `FrameKind` value: the bound every NUMERIC flag-byte /// check tests against, and the width of every per-kind table (`storestat`). /// DERIVED by folding the enum's own fields, never hand-written — `FrameKind` /// is non-exhaustive, so `switch` arms over it ARE compile-checked when a kind /// is added but a numeric comparison against a named kind is NOT, and such a /// comparison left behind turns every repository holding the new kind into /// `error.Corrupt` for every consumer. Tied to the enum, a future kind widens /// these checks by itself. /// /// The density check is what makes `flags < frame_kind_count` exactly equal to /// "a kind `FrameKind` names": with every value in `0 .. count-1` named there /// is no hole for an unnamed kind to slip through, and a sparse future enum is /// a compile error here rather than a silently-admitted flag byte. pub const frame_kind_count: usize = blk: { const fields = @typeInfo(FrameKind).@"enum".fields; var highest: usize = 0; for (fields) |f| highest = @max(highest, @as(usize, f.value)); if (fields.len != highest + 1) @compileError("FrameKind values must be dense from 0: a hole would let a numeric flag check admit an unnamed kind"); break :blk highest + 1; }; /// Repo-local config: default-branch name, project id, the sync project /// code (§9.1 `projectcode`, the repository's shared identity — distinct /// from `project_id`), and an opaque hash-policy byte the storage layer /// round-trips but never interprets (exactly like the header's /// `hash_kind`). It has a home IN the file as one distinguished `config` /// frame; "last config frame wins" is derived by scan order, reusing the /// log's own append-prefix property. pub const max_config_str: usize = 255; /// Fixed config record bound: `[hash_policy u8][branch_len u16][branch] /// [pid_len u16][pid][pcode_len u16][pcode][login_len u16][login] /// [secret_len u16][secret][url_len u16][url]` — six length-prefixed strings /// after the policy byte (the login credential, sync §6.3, is the fourth and /// fifth; the saved remote URL, sync §4/§6, is the sixth). pub const max_config_payload: usize = 1 + 6 * (2 + max_config_str); // = 1543 /// A config frame is never keyed by hash — it is found only via /// `config_offset`, never through `index`/`lookup` — so its 1-byte embedded /// hash is inert bookkeeping (`appendFrame` requires some hash). It is never /// an index key and never compared. const config_sentinel: []const u8 = "\x00"; /// One repo config. Both slices borrow a backing buffer (`OwnedConfig`); /// `hash_policy` is opaque to storage. pub const Config = struct { hash_policy: u8, default_branch: []const u8, project_id: []const u8, /// sync §9.1 — the repository's shared project code (`projectcode`, /// an artifact-hash-shaped identifier). Empty on a repo that predates /// it; `init` generates one and a clone adopts the origin's. Defaulted /// so existing construction sites need not restate it. project_code: []const u8 = "", /// sync §6.3 — the login name a `zit push` authenticates as, or empty /// when no credential is stored. Paired with `login_secret_hex`. login_user: []const u8 = "", /// sync §6.3/DECISION-3 — the 40-hex derived shared secret /// (`SHA1(project-code "/" login "/" password)`), NOT the plaintext /// password: it is all a signer needs (§6.4) and is scoped to this /// repo+login. Empty when no credential is stored; exactly 40 bytes /// otherwise. Persisted only under 0600 file permissions by the caller. login_secret_hex: []const u8 = "", /// sync §4/§6 — the default peer URL a bare `zit sync`/`pull`/`push` /// targets, or empty when no remote is saved. A LOCAL, non-synced value: /// it never crosses the wire, it just spares the operator from retyping /// the endpoint. Anonymous only — a credential never lives in the URL. /// Defaulted so existing construction sites need not restate it. remote_url: []const u8 = "", }; /// A `readConfig` result: the owned frame payload plus a `Config` whose /// slices point into it. Mirrors `OwnedConfig`'s owned-backing shape. pub const OwnedConfig = struct { backing: []u8, config: Config, pub fn deinit(self: *OwnedConfig, gpa: std.mem.Allocator) void { gpa.free(self.backing); self.* = undefined; } }; /// The repository shun table's per-row record. /// /// SPEC PROVENANCE, stated exactly. sync §3.3 L518-525 defines the table: /// "The repository shun table is a persistent, repository-local set keyed by /// complete artifact name, distinct from peer-has, peer-lacks, and other /// round-local state. Local administration MAY add or remove names. If /shun /// configuration exchange is implemented, accepted rows update the same set; /// Section 9.3 leaves their row serialization out of scope. Omitting that /// exchange does not omit the local set, which is empty when no local or /// exchanged entries supply names." Every property this type fixes is that /// sentence and not local taste: DURABLE from "persistent", per-repository /// from "repository-local", and keyed by the whole digest from "keyed by /// complete artifact name" — which §7.7 L1435-1437 restates on the admission /// side ("A target is shunned for admission when its complete name is in the /// repository shun table"). §3.3 L527-530 is where the set is READ: "The /// ordinary phantom scan excludes peer-lacks hashes, names in the repository /// shun table, and private hashes unless private sync is enabled; shun-sha1 /// does not itself suppress an otherwise eligible gimme." §8.8 L1777-1779 and /// §9.5 L1988-1992 re-reference that same filter. What remains out of scope is /// only a row's WIRE serialization (§1 L185-188, §9.3 L1915-1917), and §1 /// L191-194 draws that line explicitly: "Omitting the optional /shun table /// group omits its configuration exchange, not the repository-local shun set /// defined in Section 3.3." The `epoch`/`reason` payload below is on the local /// side of it — a row's serialization, which no draft fixes. /// /// HISTORY: durability, repository-locality, and whole-digest keying were /// derived here from the read-side filter alone, before §3.3 carried the /// definition; the applied text then stated all three verbatim, and nothing in /// this type had to change. /// /// Shape: the FRAME's own `hash` field carries the shunned artifact's binary /// digest, so a row is self-describing on a raw byte scan with no side table; /// the payload carries only the operator metadata below. Like `config`, a shun /// frame is never interned into `entries`/`index` — `lookup(hash)` keeps /// meaning "the CONTENT frame for hash", never a tombstone. pub const ShunRecord = struct { /// The shunned artifact's digest. Owned by the store (freed in `deinit`); /// borrowed by `shunList` callers until the next `putShun` or `deinit`. hash: []const u8, /// Unix seconds the shun was recorded, as supplied by the operator. Signed /// and round-tripped verbatim; storage never interprets or bounds it. epoch: i64, /// Free operator text, at most `max_shun_reason` bytes. Empty when none was /// given — a real value, not a missing field. Owned like `hash`. reason: []const u8, }; /// Bound on a shun row's free-text reason. Fixed so a row's payload has a /// stack-sized upper bound the open-time scan can read without allocating. pub const max_shun_reason: usize = 200; /// Fixed shun record bound: `[epoch i64 LE][reason_len u16 LE][reason]` — one /// scalar plus one length-prefixed string, mirroring `max_config_payload`'s /// shape. The scan rejects any `.shun` frame declaring more than this before /// reading a byte of it. pub const max_shun_payload: usize = 8 + 2 + max_shun_reason; // = 210 pub const OpenError = error{ OutOfMemory, Storage, Corrupt }; /// No read happens on the write path, so `Corrupt` can't occur here — /// advertising it would be dead mechanism. pub const PutError = error{ OutOfMemory, Storage }; /// `putShun`'s error set: `PutError` widened by the two input rules a shun row /// must satisfy. They are RETURNED, not asserted, because an assert compiles to /// nothing in `ReleaseFast`/`ReleaseSmall` and each violation is unrecoverable /// once it lands: a wrong-width key writes a durable frame the scan then /// rejects (`isCompleteDigest`), permanently un-openable and — being mid-file, /// not at the tail — not truncatable by torn-tail recovery either; an over-long /// reason overruns `encodeShun`'s fixed stack buffer. Both inputs reach here /// from an operator (the shun CLI verb) and later from a peer, so neither is a /// caller-bug-only class the way `putConfig`'s internally-generated fields are. pub const ShunError = PutError || error{ BadKey, ReasonTooLong }; /// A phantom is purely in-memory bookkeeping: no I/O, so no `Storage`. pub const InternError = error{OutOfMemory}; pub const ContentError = error{ OutOfMemory, Storage, Corrupt }; pub const CommitError = error{Storage}; /// The framed, content-addressed log. `open` rebuilds `entries`/`index` /// from the file each time — there is no persistent side index yet. /// O(n) on open is correct at this scale; a persistent, regenerable /// sidecar is a noted future extension, built only if a benchmark shows /// open time missing its target at large artifact counts. pub const LogStore = struct { gpa: std.mem.Allocator, file: storage.StorageFile, hash_kind: u8, /// Structural end of the log: header + every complete frame written so /// far. Always a multiple of `frame_align`. end: u64, /// Total complete frames (content + config) seen by the scan, and the byte /// offset of the last one. Populated by `scan`; the derivation-cache key /// (`logKey`) reads them so a cached verb can validate the log's tail in /// O(1) instead of re-walking every frame. Zero/`header_len` on an empty log. frame_count: u64 = 0, last_frame_offset: u64 = header_len, entries: std.ArrayList(Entry) = .empty, index: std.StringHashMapUnmanaged(Rid) = .empty, /// Byte offset of the live (latest in scan order) config frame, or null /// when the log carries none. Config is found ONLY through this — never /// via `index`/`entries` (§E). config_offset: ?u64 = null, /// The repository shun table (see `ShunRecord` for its normative /// definition), rebuilt by `scan` from every `.shun` frame. Deliberately /// the same two-part shape as `entries`/`index`: the list owns each row's /// `hash` and `reason` and holds them in APPEND order — which, since the /// log only grows, IS persisted order — so `shunList` is a borrow, not a /// sorted copy; the map keys on the same owned digest for an O(1) /// `isShunned`. Unlike `index` (whose keys `entries` owns), a duplicate /// digest here would leak, so `internShun` goes through `getOrPut` and /// frees the loser. shun_entries: std.ArrayList(ShunRecord) = .empty, shun_index: std.StringHashMapUnmanaged(usize) = .empty, /// Set when a rollback truncate fails and `end` can no longer be proven /// equal to the file's physical end. Every append refuses to run while it /// is set, so a later frame never lands at a diverged offset (the storage /// invariant: offset bookkeeping equals the file end). One-way: a store /// only ever goes healthy -> poisoned, never back. poisoned: bool = false, /// Set once `openReadOnly` has scanned and whole-frame-CRC-verified every /// complete frame in the file (and proven `end == file_size`, so no writer /// can extend it). While set, `readFrameAt` trusts that verification and /// skips the redundant per-read whole-frame CRC — a read-only verb must not /// CRC the entire log a second time after open already did. Every frame's /// CRC is thus verified exactly once (at scan), never zero times: the header /// prefix/header CRCs still run on every read, and a short payload read is /// still corruption. The append path (`open`) leaves this false, so writes /// and post-write reads keep full at-use verification. verified_on_open: bool = false, const Entry = struct { /// Owned duplicate; also the index's key (stable across `entries` /// reallocation since it's a separate heap allocation). hash: []u8, /// Frame start offset. Meaningless while `phantom`. offset: u64, /// On-disk (container/wire) payload length: raw bytes for `raw`, /// the zlib-container length for `zlib`/`delta`. payload_len: u32, /// Payload encoding of this frame. `.raw` while `phantom` (no frame). kind: FrameKind, /// A phantom was `intern`ed but never `put`: no frame on disk. phantom: bool, /// Delta-chain depth: 0 for a phantom or a non-delta frame; one more /// than the base's depth for a `.delta` frame. Set at `insert`/`scan` /// and read by `putArtifact` to enforce `delta_chain_write_max`, so a /// locally-created chain never grows past the write cap. depth: u8 = 0, }; /// Opens for append, creating or repairing the file when necessary, and /// rebuilds the in-memory index by scanning every complete frame. pub fn open(gpa: std.mem.Allocator, file: storage.StorageFile, hash_kind: u8) OpenError!LogStore { const file_size = file.size() catch return error.Storage; var self: LogStore = .{ .gpa = gpa, .file = file, .hash_kind = hash_kind, .end = header_len }; errdefer self.deinit(gpa); if (file_size == 0) { try writeHeader(file, hash_kind); std.debug.assert(self.end % frame_align == 0); return self; } if (file_size < header_len) { // Torn header: nothing before byte `header_len` can be trusted // as a frame boundary either, so start clean. file.truncate(0) catch return error.Storage; try writeHeader(file, hash_kind); std.debug.assert(self.end % frame_align == 0); return self; } self.hash_kind = try validateHeader(file); try scan(&self, true); std.debug.assert(self.end % frame_align == 0); return self; } /// Opens an existing log strictly for reads. Unlike `open`, this path never /// creates, rewrites, or truncates storage: an empty file, torn header, or /// torn frame tail is corruption. Complete-frame checksum and encoding /// faults are rejected by the same scanner used by the append path. pub fn openReadOnly(gpa: std.mem.Allocator, file: storage.StorageFile) OpenError!LogStore { const file_size = file.size() catch return error.Storage; if (file_size < header_len) return error.Corrupt; var self: LogStore = .{ .gpa = gpa, .file = file, .hash_kind = 0, .end = header_len }; errdefer self.deinit(gpa); self.hash_kind = try validateHeader(file); try scan(&self, false); std.debug.assert(self.end == file_size); std.debug.assert(self.end % frame_align == 0); // scan() whole-frame-CRC-verified every frame and proved end == file // size; reads may now skip the redundant per-frame CRC (see the field). self.verified_on_open = true; return self; } pub fn deinit(self: *LogStore, gpa: std.mem.Allocator) void { for (self.entries.items) |e| gpa.free(e.hash); self.entries.deinit(gpa); self.index.deinit(gpa); // The shun list owns both strings of every row; `shun_index` borrows // the same `hash` allocation as its key, so it frees nothing itself. for (self.shun_entries.items) |s| { gpa.free(s.hash); gpa.free(s.reason); } self.shun_entries.deinit(gpa); self.shun_index.deinit(gpa); self.* = undefined; } /// The append-only-log identity a derivation cache is keyed on: the /// structural end offset, the total complete-frame count, the byte offset of /// the last complete frame, and that frame's fixed-header CRC. The log only /// ever grows by whole frames, so a cache built at key K stays valid exactly /// while the live log still matches K: an equal `end` (file size) proves no /// append and no truncation changed the length, and the last frame still /// sitting at `last_frame_offset` with header CRC `last_header_crc` and /// ending at `end` pins the truncate-then-reappend-to-the-same-length corner /// that `end` alone would miss. `frame_count` is recorded for the identity /// but, under the append-only invariant, is subsumed by `end` — see /// `validateTail`, which stays O(1) by checking only the tail. Whole-log /// payload integrity remains `openReadOnly`'s job on the miss/rebuild path, /// so a cache hit will not detect interior bitrot that preserves the file /// size and tail frame; deleting the `.zit-cache` (or any append, which /// forces a rebuild) restores full-log detection. pub const LogKey = struct { end: u64, frame_count: u64, last_frame_offset: u64, last_header_crc: u32, }; pub const KeyError = error{ Storage, Corrupt }; /// The `LogKey` for this already-scanned store — the value a cache write /// records. `end`, `frame_count`, and `last_frame_offset` come from the scan /// (`open`/`openReadOnly` already walked every frame); only the last frame's /// header CRC needs one extra small read, so this is O(1), never a re-walk. /// An empty log (no frames) keys on `end == header_len` alone. Never mutates /// storage. pub fn logKey(self: *const LogStore) KeyError!LogKey { if (self.frame_count == 0) { return .{ .end = self.end, .frame_count = 0, .last_frame_offset = self.end, .last_header_crc = 0 }; } const info = try tailHeaderInfo(self.file, self.last_frame_offset); // The recorded last frame must end exactly at the structural end. std.debug.assert(info.frame_end == self.end); return .{ .end = self.end, .frame_count = self.frame_count, .last_frame_offset = self.last_frame_offset, .last_header_crc = info.header_crc, }; } /// O(1) cache-freshness check: is `file` still the append-only log `key` was /// computed over? True iff the file size equals `key.end` and — for a /// non-empty log — the last frame still occupies `key.last_frame_offset` /// with the same header CRC and end. Reads at most one frame header (plus a /// stat); no full scan, so a cached verb stays sub-second even on a /// multi-gigabyte log. Any I/O failure, a shrunk/grown file, or a changed /// tail is a miss (false), never an error — a cache is advisory. pub fn validateTail(file: storage.StorageFile, key: LogKey) bool { const file_size = file.size() catch return false; if (file_size != key.end) return false; // any append or truncation changes the length if (key.frame_count == 0) return key.end == header_len; // empty log: length is the whole identity // The recorded tail offset must be a real interior frame boundary. if (key.last_frame_offset < header_len or key.last_frame_offset >= file_size) return false; const info = tailHeaderInfo(file, key.last_frame_offset) catch return false; return info.frame_end == key.end and info.header_crc == key.last_header_crc; } const TailInfo = struct { frame_end: u64, header_crc: u32 }; /// Read and CRC-verify the fixed header of the frame at `offset`, returning /// its end offset and header CRC. A torn or corrupt header is `error.Corrupt` /// (`readFrameHeader` already validates the prefix and header CRCs). fn tailHeaderInfo(file: storage.StorageFile, offset: u64) KeyError!TailInfo { const fh = switch (readFrameHeader(file, offset) catch |e| switch (e) { error.Storage => return error.Storage, error.Corrupt => return error.Corrupt, }) { .torn => return error.Corrupt, // a recorded frame offset is never torn .header => |h| h, }; var hcrc = crc32.Crc32.init(); hcrc.update(fh.header_bytes[0 .. 18 + @as(usize, fh.hash_len)]); const frame_len = frameSpan(fh.hash_len, fh.payload_len); return .{ .frame_end = offset + frame_len, .header_crc = hcrc.final() }; } /// Clone = copy + derive: byte-copies `source` into `dest`, then opens /// `dest` — rebuilding the index and recovering /// config with no network (no network capability is even in scope here). /// The result is an independent, equal repository: same artifact set, same /// config, byte-identical log. A damaged copy needs no clone-specific /// error — `open`'s torn-tail truncation drops exactly the incomplete /// tail, so the copy degrades gracefully through the shared derive path. /// Precondition (documented, not asserted — assertions carry no I/O): /// `dest` is empty. `hash_kind` is a don't-care: `dest` is non-empty after /// the copy, so `open` reads the copied header's stored kind. pub fn cloneInto( gpa: std.mem.Allocator, source: storage.StorageFile, dest: storage.StorageFile, ) OpenError!LogStore { const src_size = source.size() catch return error.Storage; var chunk: [64 * 1024]u8 = undefined; var copied: u64 = 0; // Bounded: `copied` rises by `n > 0` every pass, so the loop runs at // most ceil(src_size / chunk.len) times before `copied == src_size`. while (copied < src_size) { const n = source.pread(copied, &chunk) catch return error.Storage; // A 0-count before EOF on a stable file is a device failure, not a // boundary — return rather than spin (no unbounded loop). if (n == 0) return error.Storage; dest.append(chunk[0..n]) catch return error.Storage; copied += n; } std.debug.assert(copied == src_size); return open(gpa, dest, 0); } /// Resolves `hash` to its rid, creating a phantom (no I/O) when /// unknown. Phantoms never reach disk. pub fn intern(self: *LogStore, hash: []const u8) InternError!Rid { std.debug.assert(hash.len >= min_hash_len and hash.len <= max_hash_len); if (self.index.get(hash)) |rid| return rid; try self.entries.ensureUnusedCapacity(self.gpa, 1); const owned = try self.gpa.dupe(u8, hash); errdefer self.gpa.free(owned); const rid: Rid = self.entries.items.len + 1; try self.index.put(self.gpa, owned, rid); self.entries.appendAssumeCapacity(.{ .hash = owned, .offset = 0, .payload_len = 0, .kind = .raw, .phantom = true }); return rid; } /// Stores `bytes` verbatim (a `raw` frame) under `hash`. Idempotent for /// an already-content-bearing hash (returns the existing rid, appends /// nothing); upgrades a phantom in place. pub fn put(self: *LogStore, hash: []const u8, bytes: []const u8) PutError!Rid { std.debug.assert(bytes.len <= max_payload); return self.insert(hash, bytes, .raw, 0, 0); } /// Store `bytes` under `hash`, delta-encoded against the size-minimizing /// base `pickBase` (§5) chooses from a small candidate window, else as a /// §4.1 zlib container. The window is the caller's lineage `base` hint /// (priority) plus the most recent content frames in log order — a bounded /// best-of-N search (fossil `extra_deltification`), accepting a delta only /// when it is under 75% of the artifact. Every candidate is a HINT, never a /// requirement: a null/phantom/config/unreadable/over-deep/unprofitable /// base is silently skipped — there is no error path for an ineligible /// base, only a compactness difference. Idempotent + phantom-upgrade like /// `put`/`putZlib`: an already content-bearing `hash` keeps its stored /// form and no encode work runs. This one method hides the whole storage /// policy so callers pass only the base hint they naturally hold. pub fn putArtifact(self: *LogStore, hash: []const u8, bytes: []const u8, base: ?Rid) PutError!Rid { std.debug.assert(hash.len >= min_hash_len and hash.len <= max_hash_len); std.debug.assert(bytes.len <= max_payload); // Idempotent: return before any encode work when the artifact is held. const held: ?Rid = self.index.get(hash); if (held) |rid| { if (!self.entries.items[@intCast(rid - 1)].phantom) return rid; } var pri_buf: [1]Rid = undefined; const priority: []const Rid = if (base) |b| blk: { pri_buf[0] = b; break :blk pri_buf[0..1]; } else pri_buf[0..0]; var cand_buf: [max_base_candidates]Rid = undefined; const n = self.gatherCandidates(priority, cand_buf[0..]); if (try self.pickBase(self.gpa, hash, bytes, cand_buf[0..n])) |pick| { defer self.gpa.free(pick.delta); return try self.putDelta(hash, pick.base, pick.delta); } return self.putZlib(hash, bytes); } /// Fill `cand_buf` with the best-of-N base candidate window (§5.2) for a /// write: the caller's `priority` hints first (in order — a lineage parent, /// or the §6.2 same-path predecessors pack derives), then the most recent /// content frames in log order. Insertion order IS log order, so a backward /// walk over `entries` yields the chronological window for free — no manifest /// parsing on the write path. Priority candidates are deduped against each /// other and the chronological walk skips any rid already listed. Returns the /// count written (≤ `max_base_candidates`); `pickBase` re-checks full /// eligibility, so this only gathers plausible rids. Shared by `putArtifact` /// and `putArtifactPacked` so the two write paths pick from an identical /// window shape. fn gatherCandidates(self: *LogStore, priority: []const Rid, cand_buf: []Rid) usize { std.debug.assert(cand_buf.len >= max_base_candidates); // The priority window is bounded to the candidate cap: both callers // (putArtifact's 1-elem hint, pack's fixed-size hint buffer) structurally // hold it, and the `for (priority)` loop below only caps entries ADDED, // not iterations — state the upper bound locally so a future caller with a // duplicate-heavy slice cannot silently scan past the window. std.debug.assert(priority.len <= max_base_candidates); var n: usize = 0; // Priority hints fill first, in caller order, bounded to the window and // deduped so a hint that repeats (or coincides across lanes) costs no slot. for (priority) |b| { if (n >= max_base_candidates) break; std.debug.assert(b >= 1 and b <= self.entries.items.len); if (ridListed(cand_buf[0..n], b)) continue; cand_buf[n] = b; n += 1; } // Newest-first; `idx` strictly decreases to 0. Two independent bounds // stop the walk: `n < max_base_candidates` (window full) and // `scanned < max_base_scan` (frames inspected). The scan cap is the one // that bites under load — a run of trailing phantom frames `continue`s // without filling a slot, so without it a single put could step over the // whole store to gather < 5 candidates (hidden O(n^2) over N puts). The // cap holds the per-put scan to a fixed window regardless of phantom // density; a real base past the window only costs compactness. A held // artifact that reaches this walk is necessarily a phantom (a // content-bearing hit returns early in the caller), so the phantom skip // already excludes "self"; the priority-dedup skips any hint already // listed. var idx: usize = self.entries.items.len; var scanned: usize = 0; while (idx > 0 and n < max_base_candidates and scanned < max_base_scan) { idx -= 1; scanned += 1; const rid: Rid = @intCast(idx + 1); if (self.entries.items[idx].phantom) continue; if (ridListed(cand_buf[0..n], rid)) continue; // priority hint already listed cand_buf[n] = rid; n += 1; } return n; } /// Whether `rid` already appears in `listed` — the bounded (≤ /// `max_base_candidates`) dedup guard `gatherCandidates` runs so a priority /// hint and the chronological walk never offer the same base twice. fn ridListed(listed: []const Rid, rid: Rid) bool { for (listed) |r| if (r == rid) return true; return false; } /// The outcome of `putArtifactPacked`: either a fresh re-encoding won the /// §6.1 monotonic-shrink comparison and was written (`rewrote`), or the /// existing source frame is smaller-or-equal and NOTHING was written /// (`kept`) — the caller then copies that existing frame verbatim. pub const PackPut = union(enum) { rewrote: Rid, kept }; /// `putArtifact` with the §6.1 monotonic-shrink guard for `zit pack`: gather /// the same best-of-N window, run `pickBase`, and encode the winning form — /// but COMMIT the new frame only when its on-disk span is STRICTLY smaller /// than `floor_span` (the artifact's existing stored-frame span in the source /// store, from `frameSpanOf`). A tie or a larger re-encoding returns `.kept` /// and writes nothing, so the caller re-stores the existing frame — often a /// compact wire delta whose base fossil already chose well — verbatim. /// `pickBase`'s <75%-of-FULL accept is a FIRST-encoding rule; in pack the /// reference size is the EXISTING frame, not the full artifact, so this guard /// keeps every REWRITTEN frame non-worsening RELATIVE TO THE FLOOR THE CALLER /// DECLARES: this frame is size-neutral or smaller than `floor_span`, never /// larger. That is a per-frame guarantee only while the caller's floor is a /// frame it could actually re-store instead; a caller with no keepable frame /// passes `null` and gets the smallest encoding available, which may still /// exceed the source frame it replaces. `null` is a distinct STATE, not an /// in-band span: "I have no frame to keep" and "the frame I would keep spans /// N bytes" are different claims, and an in-band sentinel in the span's own /// type could not tell a caller with no keepable frame from one that computed /// a garbage span — the second silently forces a rewrite of a frame that /// should have been kept. `zit pack` passes `null` for exactly one shape: a /// delta whose base the reclaim filter dropped, whose source frame is /// unre-storable. (Outside this function pack's /// `copySrcFrame` §3.3 deep-chain fallback can likewise enlarge a KEPT frame; /// see pack.zig.) `priority` is the caller's ordered base hints /// (a lineage parent, or the §6.2 same-path predecessors), offered ahead of /// the chronological window. Idempotent + phantom-upgrade like `putArtifact` /// (a re-put of a held artifact returns `.rewrote` with its existing rid). pub fn putArtifactPacked(self: *LogStore, hash: []const u8, bytes: []const u8, priority: []const Rid, floor_span: ?u64) PutError!PackPut { std.debug.assert(hash.len >= min_hash_len and hash.len <= max_hash_len); std.debug.assert(bytes.len <= max_payload); if (floor_span) |fs| std.debug.assert(fs >= min_frame); // Idempotent: return before any encode work when the artifact is held. if (self.index.get(hash)) |rid| { if (!self.entries.items[@intCast(rid - 1)].phantom) return .{ .rewrote = rid }; } var cand_buf: [max_base_candidates]Rid = undefined; const n = self.gatherCandidates(priority, cand_buf[0..]); // The best new encoding, compressed to its on-disk container form so its // span is comparable to the existing frame's. A profitable delta compares // its zlib-contained delta; otherwise the fallback is a full zlib // container (exactly what `putZlib` would store). if (try self.pickBase(self.gpa, hash, bytes, cand_buf[0..n])) |pick| { defer self.gpa.free(pick.delta); const container = codec.encode(self.gpa, pick.delta, .default) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, error.InputTooLarge => unreachable, // pick.delta <= max_payload + bounded overhead }; defer self.gpa.free(container); // Keep the smaller; the existing frame wins ties (strict `<`). No // floor ⇒ nothing to keep, so the new encoding always wins. if (beatsFloor(frameSpan(hash.len, container.len), floor_span)) { return .{ .rewrote = try self.putDeltaContainer(hash, pick.base, container) }; } return .kept; } const container = codec.encode(self.gpa, bytes, .default) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, error.InputTooLarge => unreachable, // asserted bytes.len <= max_payload (< 2^32) }; defer self.gpa.free(container); if (beatsFloor(frameSpan(hash.len, container.len), floor_span)) { return .{ .rewrote = try self.putContainer(hash, container) }; } return .kept; } /// The §6.1 monotonic-shrink comparison, in one place for both of /// `putArtifactPacked`'s encodings: a new frame of `span` bytes is written /// iff it is STRICTLY smaller than the frame the caller would otherwise keep. /// A caller with no keepable frame (`null`) has nothing to lose by writing, /// so it always writes — which is why `.kept` is impossible under a null /// floor. fn beatsFloor(span: u64, floor_span: ?u64) bool { const floor = floor_span orelse return true; return span < floor; } /// The §5 delta-base policy as ONE deep module: choose the size-minimizing /// base for storing `bytes` (named `hash`) from `candidates`, or null ⇒ /// store the full form. `candidates` are RIDs the caller believes are /// plausible bases (lineage parent, chronological neighbors, same-path /// predecessors), in priority order; at most `max_base_candidates` are /// examined. This owns the WHOLE decision — the caller owns only WHICH /// candidates to offer: /// * best-of-N: run `delta.create` (delta.zig:294) against every ELIGIBLE /// candidate and keep the SMALLEST delta (ties resolve to the earlier, /// higher-priority candidate via the strict `<`); /// * eligibility: content-bearing (non-phantom), non-config, chain-depth /// headroom (`depth < delta_chain_write_max`), and never the artifact /// itself (a self-delta is degenerate) — all free checks, run before /// any I/O so an ineligible candidate costs nothing; /// * cheap length early-out: a candidate whose content is under /// `delta_min_source` bytes is skipped BEFORE `delta.create` (fossil /// content.c:908); /// * accept iff the smallest delta clears the 75% bar (`deltaProfitable`, /// fossil content.c:914). /// On accept, returns the winning base rid and the raw delta bytes (caller /// owns and stores them, e.g. via `putDelta`); otherwise null. Ineligible / /// unreadable candidates are silently skipped — a base is a hint, never a /// requirement, so there is no error path for a bad candidate. Only /// `OutOfMemory`/`Storage` escape. pub fn pickBase( self: *LogStore, gpa: std.mem.Allocator, hash: []const u8, bytes: []const u8, candidates: []const Rid, ) PutError!?Pick { std.debug.assert(hash.len >= min_hash_len and hash.len <= max_hash_len); std.debug.assert(bytes.len <= max_payload); var best: ?[]u8 = null; var best_base: Rid = 0; // Free the retained best delta if an allocation/storage fault escapes // mid-search; the success and reject paths hand it off / free it // explicitly, so this only fires on an error return. errdefer if (best) |b| gpa.free(b); // Bounded fan-out: at most `max_base_candidates` `delta.create` runs, // even if the caller over-offers (fossil's N_NEIGHBOR = 5). const limit = @min(candidates.len, max_base_candidates); var i: usize = 0; while (i < limit) : (i += 1) { const cand = candidates[i]; std.debug.assert(cand >= 1 and cand <= self.entries.items.len); const entry = &self.entries.items[@intCast(cand - 1)]; // Eligibility — all free, no I/O: skip a phantom, the config // singleton, a base already at the write-chain cap, and the // artifact itself. if (entry.phantom) continue; if (entry.kind == .config) continue; if (entry.depth >= delta_chain_write_max) continue; if (std.mem.eql(u8, entry.hash, hash)) continue; // Routed through `content`, so it inherits the shun mask: a shunned // candidate reads null here and is skipped like a phantom. That is // a WRITE-path consequence of a READ guard, and the right one — a // new delta rooted on a frame reclamation is about to drop would be // unresolvable the moment those bytes went away. It does not // conflict with §7.7, which governs a receiver honouring a basis a // peer already chose, not this store choosing a fresh one. const base_bytes = self.content(gpa, cand) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, error.Storage => return error.Storage, error.Corrupt => continue, // an unreadable base is not a basis } orelse continue; // a phantom or shunned candidate carries no usable content defer gpa.free(base_bytes); // Cheap length early-out (fossil content.c:908): a source too small // to ever pay for a delta's framing is skipped before the encoder. if (base_bytes.len < delta_min_source) continue; const raw = delta.create(gpa, base_bytes, bytes) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, error.TargetTooLarge => continue, // unencodable target; skip }; // Keep the strictly smaller delta; a tie keeps the earlier (higher // priority) candidate. The loser is freed immediately so at most // one delta buffer is ever live. if (best == null or raw.len < best.?.len) { if (best) |b| gpa.free(b); best = raw; best_base = cand; } else { gpa.free(raw); } } const smallest = best orelse return null; // Accept iff strictly under 75% of the uncompressed artifact. The // globally smallest delta is <= every other, so if any candidate clears // 75% the smallest does too — testing only the smallest is exact. if (!deltaProfitable(smallest.len, bytes.len)) { gpa.free(smallest); return null; } return .{ .base = best_base, .delta = smallest }; } /// Stores `bytes` compressed into a fossil zlib container (§4.1, default /// level) under `hash`. The hash still names the UNCOMPRESSED bytes, so /// storage compression never changes an artifact's address (§4.1 /// L1206-1208). Idempotent + phantom-upgrade like `put`. pub fn putZlib(self: *LogStore, hash: []const u8, bytes: []const u8) PutError!Rid { std.debug.assert(bytes.len <= max_payload); const container = codec.encode(self.gpa, bytes, .default) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, error.InputTooLarge => unreachable, // asserted bytes.len <= max_payload (< 2^32) }; defer self.gpa.free(container); return self.insert(hash, container, .zlib, 0, 0); } /// Store an already-encoded full-form §4.1 zlib `container` verbatim under /// `hash` as a `.zlib` frame, so a clone that received this exact container /// re-serves it byte-for-byte (sync §7.2/§4.1 L858-860: "retransmitted /// verbatim"). Unlike `putZlib` — which compresses raw plaintext — the /// wire bytes are preserved, not re-created. PRECONDITION (§7.7 L1332): /// the caller MUST have verified `hash` names the bytes `container` /// inflates to; this store performs no hash recompute (it lacks the /// plaintext to). Idempotent + phantom-upgrade like `put`. pub fn putContainer(self: *LogStore, hash: []const u8, container: []const u8) PutError!Rid { // §4.1 — a container carries at least its 4-byte length prefix. std.debug.assert(container.len >= 4 and container.len <= max_payload); return self.insert(hash, container, .zlib, 0, 0); } /// Stores `raw_delta` (a delta reconstructing this artifact from the frame /// at `base_rid`) as a zlib-contained `delta` frame by encoding it into a /// §4.1 container, then storing that container verbatim via /// `putDeltaContainer`. The ingest path (`putArtifact`) uses this to store /// a locally created delta. Idempotent + phantom-upgrade like `put`. pub fn putDelta(self: *LogStore, hash: []const u8, base_rid: Rid, raw_delta: []const u8) PutError!Rid { std.debug.assert(raw_delta.len >= 1 and raw_delta.len <= max_payload); const container = codec.encode(self.gpa, raw_delta, .default) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, error.InputTooLarge => unreachable, // asserted raw_delta.len <= max_payload (< 2^32) }; defer self.gpa.free(container); return self.putDeltaContainer(hash, base_rid, container); } /// Store an already-§4.1-encoded delta `container` (a compressed /// [FossilDelta] reconstructing this artifact from the frame at `base_rid`) /// verbatim as a `.delta` frame; `delta_of` records the base frame's byte /// offset. The stored container IS the wire payload a `cfile` serves /// verbatim (§7.2), so a clone that received this exact delta container /// re-serves it byte-for-byte — unlike `putDelta`, the container is NOT /// re-encoded. PRECONDITION (§7.7 L1332): the caller MUST have verified /// `hash` names the bytes this container reconstructs to (the delta applied /// to `base_rid`'s content); this store performs no hash recompute. /// Idempotent + phantom-upgrade like `put`. pub fn putDeltaContainer(self: *LogStore, hash: []const u8, base_rid: Rid, container: []const u8) PutError!Rid { std.debug.assert(base_rid >= 1 and base_rid <= self.entries.items.len); const base = &self.entries.items[@intCast(base_rid - 1)]; std.debug.assert(!base.phantom); std.debug.assert(base.offset >= header_len and base.offset < self.end); // §4.1 — a delta container carries at least its 4-byte length prefix. std.debug.assert(container.len >= 4 and container.len <= max_payload); const base_off = base.offset; // Bookkeeping only: the WRITE cap is `putArtifact`'s job (it refuses a // base at `delta_chain_write_max`); the verbatim caller (`persistStore`) // enforces the READ bound `max_delta_chain` for a retained foreign // chain. `putDeltaContainer` itself is the low-level primitive and // stays permissive. Saturating so depth never wraps on a pathologically // deep foreign chain. const depth: u8 = base.depth +| 1; return self.insert(hash, container, .delta, base_off, depth); } /// Shared insert core: append one frame carrying `payload` under `hash` /// with encoding `kind` (and `delta_of` for `.delta`), then update the /// index. Idempotent for an already-content-bearing hash; upgrades a /// phantom in place. On a bookkeeping failure after the frame landed, the /// file is truncated back so disk and memory stay in lockstep; a rollback /// truncate that itself fails poisons the store. fn insert(self: *LogStore, hash: []const u8, payload: []const u8, kind: FrameKind, delta_of: u64, depth: u8) PutError!Rid { std.debug.assert(hash.len >= min_hash_len and hash.len <= max_hash_len); std.debug.assert(payload.len <= max_payload); std.debug.assert((depth == 0) == (kind != .delta)); // depth is delta-only if (self.index.get(hash)) |rid| { const entry = &self.entries.items[@intCast(rid - 1)]; if (!entry.phantom) return rid; const res = try self.appendFrame(hash, payload, kind, delta_of); entry.offset = res.offset; entry.payload_len = res.len; entry.kind = kind; entry.phantom = false; entry.depth = depth; return rid; } try self.entries.ensureUnusedCapacity(self.gpa, 1); const owned = try self.gpa.dupe(u8, hash); errdefer self.gpa.free(owned); const res = try self.appendFrame(hash, payload, kind, delta_of); const rid: Rid = self.entries.items.len + 1; self.index.put(self.gpa, owned, rid) catch |e| { // Frame already landed on disk but bookkeeping failed: undo the // write so the file matches in-memory state exactly. If the // rollback truncate itself fails, `end` and the file end diverge // and no later append can be trusted — poison the store. self.file.truncate(res.offset) catch { self.poisoned = true; }; self.end = res.offset; return e; }; self.entries.appendAssumeCapacity(.{ .hash = owned, .offset = res.offset, .payload_len = res.len, .kind = kind, .phantom = false, .depth = depth }); return rid; } /// The payload encoding of `rid` (`.raw` for a phantom). Lets a serve /// loop pick a full-content versus delta cfile without a frame read. pub fn kindOf(self: *const LogStore, rid: Rid) FrameKind { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); return self.entries.items[@intCast(rid - 1)].kind; } /// The on-disk (container/wire) payload length of `rid`; 0 for a phantom. /// Symmetric with `hashOf`; the DST sweep uses it to bound the crc-covered /// span of a frame whose uncompressed content differs in length. pub fn payloadLenOf(self: *const LogStore, rid: Rid) u32 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); return self.entries.items[@intCast(rid - 1)].payload_len; } /// The total on-disk span (aligned prefix + hash + payload) of `rid`'s /// frame — the byte cost `zit pack`'s §6.1 monotonic-shrink guard uses as /// the floor when deciding whether a re-encoding is worth writing. Asserts a /// real (non-phantom) frame; a phantom carries no frame to measure. pub fn frameSpanOf(self: *const LogStore, rid: Rid) u64 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); const entry = &self.entries.items[@intCast(rid - 1)]; std.debug.assert(!entry.phantom); return frameSpan(entry.hash.len, entry.payload_len); } /// The delta-chain depth of `rid`: 0 for a non-delta frame (or a phantom), /// one more than its base for a `.delta` frame. Lets the verbatim persist /// path (`persistStore`) refuse a retained delta whose base already sits at /// the `max_delta_chain` READ bound, so a stored chain never exceeds what /// `content()` can walk. pub fn depthOf(self: *const LogStore, rid: Rid) u8 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); return self.entries.items[@intCast(rid - 1)].depth; } /// The byte offset of `rid`'s frame in the log file — the stable key a /// `PlaintextCache` uses to memoise a frame's reconstructed plaintext across /// a derivation sweep. Stable because the log is append-only: a written /// frame never moves, so its offset names it for the sweep's lifetime. pub fn offsetOf(self: *const LogStore, rid: Rid) u64 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); std.debug.assert(!self.entries.items[@intCast(rid - 1)].phantom); // a phantom has no frame return self.entries.items[@intCast(rid - 1)].offset; } /// Whether `rid` is a phantom (interned but carrying no frame). Lets a /// caller distinguish "held on disk" from "announced but absent" without /// reconstructing content. pub fn isPhantom(self: *const LogStore, rid: Rid) bool { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); return self.entries.items[@intCast(rid - 1)].phantom; } /// The delta-chain depth of the entry whose frame starts at `offset`, for /// recovering a delta's depth during the open-time scan. Entries are /// appended in ascending offset order there (real frames only, no /// phantoms), so a binary search resolves the base with no auxiliary map. /// Returns 0 when no entry starts at `offset` (a dangling `delta_of` that /// `content()` rejects). Precondition: `entries` is offset-sorted — true /// only mid-`scan`, which is this method's sole caller. fn depthForBaseOffset(self: *const LogStore, offset: u64) u8 { const items = self.entries.items; var lo: usize = 0; var hi: usize = items.len; while (lo < hi) { const mid = lo + (hi - lo) / 2; const o = items[mid].offset; if (o == offset) return items[mid].depth; if (o < offset) lo = mid + 1 else hi = mid; } return 0; } /// THE READ MASK: is `entry`'s artifact in the repository shun table? /// Shared verbatim by `content`, `contentCached` and `wire` so the domain /// argument below is stated once instead of re-derived at three call sites. /// /// A shunned artifact reads as ABSENT even while its content frame is still /// on disk. That is what makes reclamation INVISIBLE — dropping the bytes /// later changes no verb's output — and so what makes deferring the reclaim /// safe. /// /// Deliberately skips the public `isShunned`'s `isCompleteDigest` assert /// (20 or 32 bytes) — but shares its one probe body (`shunHasName`) so the /// two can never drift. An ENTRY's hash is not in that domain: `put`/ /// `intern` accept `min_hash_len .. max_hash_len` (1..32) and keep a name /// opaque, and the fuzz fixtures in this file legitimately drive /// `content()` on a 9-byte name. Asserting here would therefore trap on a /// perfectly valid store. /// /// Answering false for an out-of-domain width is EXACT here, not fail-open, /// and the argument is different from `isShunned`'s caller contract (which /// is about wire names an assert must not let through silently): every key /// in this map was PROVEN complete before it was inserted — `putShun` /// returns `error.BadKey` otherwise, and the open scan fails the whole open /// with `error.Corrupt` on a `.shun` frame whose `hash_len` is not complete. /// A non-complete entry hash therefore provably cannot be a key, so "not /// found" is the true answer and not merely a tolerated miss. /// /// THE SYNC-SIDE RECONCILIATION, and what is still open. `content`/`wire` /// are RID-KEYED, so they answer null for a shunned artifact whose frame is /// on disk. Three gates were written when read-ness and held-ness agreed; /// reconciling them was always a sync decision, not a storage one, and two /// of the three have since been decided in the sync layer: /// /// * `sync.zig` L1018 — `consumeDelta`'s basis-held test. STILL OPEN. A /// locally shunned-but-held DELTASRC reads null, so an admitted target /// falls into the dangling-delta branch: the target is queued, its /// phantom is removed, and a phantom record is minted for a rid that is /// not a phantom. Nothing ever completes it (the basis is already /// stored, so no later store of that NAME fires `completeFor`, and /// §7.4's `held` guard discards a re-sent basis), and /// `pending.hasTarget` suppresses the gimme — the artifact ends up /// neither stored, phantom, nor requestable. This is the receive side /// (sync §7.7) and belongs to the slice that implements it. /// * The SEND-side enumeration — DECIDED. `cmd/sync.zig`'s /// `LogIndexStore` now applies sync §9.5 L1988-1992's "every eligible /// holding after the Section 3.3 shun … filters" to the enumeration /// itself: a shunned frame keeps its adapter rid (§8.2 L1504-1506 /// licenses a GAP, never a renumbering) and is MASKED — `held` false, /// `content`/`wire` null, name absent from the wire-name index. So the /// disagreement this comment used to record at `heldImpl` is gone: the /// igot walk no longer advertises what no round can send, and a `gimme` /// for the name lands on §3.3 L532-535's "If absent, silently skip". /// `loadContentStore` (the pull CLIENT's own holdings snapshot) still /// interns-but-does-not-hold a shunned artifact — both reads answer null /// under this mask, so the entry becomes a PHANTOM. That is not an /// announce-side hole: the client's igot walk gates on `held`, so a /// phantom is never advertised; it is gimme-eligible, and `Client.shun` /// (§3.3 L527-530) is what excludes it. What remains is only the oddity /// of a phantom record standing for content this repository physically /// holds — no wire consequence today. /// * `sync.zig`'s §3.3 phantom scan — DECIDED. Both `Client.shun` and /// `Server.shun` now carry the repository table, so §3.3 L527-530's /// exclusion is wired and absorbs the phantom half of this. /// /// The one repair storage still must not pick unilaterally is the first /// bullet's: declaring a held-but-shunned basis available (gate on `held`) /// versus reading it through an explicitly unmasked door. Note that the /// send side deliberately does NOT depend on this mask — `LogIndexStore` /// states its own — so choosing the unmasked door cannot silently reopen /// the send-side hole. fn shunned(self: *const LogStore, entry: *const Entry) bool { return self.shunHasName(entry.hash); } /// The one place the shun table is probed. `shunned` (the read mask) and /// `isShunned` (the public predicate) are two DOORS onto this body, not two /// implementations of it: they differ only in the caller contract each /// enforces on the way in. Widening the set's notion of membership — name /// normalization, an exemption concept, a second index — lands here once and /// reaches both, which is the point. Two inlined `contains` calls under /// divergent contracts could silently answer from different rules, and the /// read path is the half where that fails OPEN. fn shunHasName(self: *const LogStore, hash: []const u8) bool { return self.shun_index.contains(hash); } /// WHICH RULE a shared read body applies before it touches the frame. Every /// rid-keyed reader names its choice at the call site, so "the masked set is /// exactly {`content`, `contentCached`, `wire`}" is a fact the compiler /// enforces rather than a claim five doc comments make: the mask is written /// ONCE per shared body instead of hand-copied per reader, and a new reader /// cannot end up unmasked by forgetting a line — there is no default to /// fall through to. const ReadMask = enum { /// An ARTIFACT read: a name in the repository shun table reads as /// ABSENT even while its frame is on disk. masked, /// In-repository frame MAINTENANCE that shows the bytes to nobody: the /// shun table is not consulted. See `contentUnmasked` for the full /// argument and for why `pack.zig` is the only legitimate caller. maintenance, }; /// Owned uncompressed bytes for a rid, or null while it's a phantom or its /// name is in the repository shun table (see `shunned`). Re-reads and /// re-verifies the frame (both checksums, embedded hash) on every call, /// then decodes by kind: `raw` copied out, `zlib` inflated, `delta` /// resolved by walking `delta_of` and applying via `delta.apply`. /// Never trusts a stale scan result. pub fn content(self: *LogStore, gpa: std.mem.Allocator, rid: Rid) ContentError!?[]u8 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); const entry = &self.entries.items[@intCast(rid - 1)]; if (entry.phantom) return null; return try self.contentOfEntry(gpa, entry, .masked, null); } /// The frame read every rid-keyed content reader shares, INCLUDING the rule /// it reads under: apply `mask`, then resolve `entry`'s offset and, for a /// `.delta` frame, memoize the result when the caller brought a cache. Both /// the decode and the policy live here so the masked pair /// (`content`/`contentCached`) and the maintenance reader /// (`contentUnmasked`) cannot drift in either. fn contentOfEntry(self: *LogStore, gpa: std.mem.Allocator, entry: *const Entry, mask: ReadMask, cache: ?*PlaintextCache) ContentError!?[]u8 { // The precondition `offsetOf` states for the same dereference: a phantom // has no frame, and its `offset` is not a frame start — `readFrameAt` // below would decode arbitrary log bytes under this entry's name. Every // caller filters phantoms first; this is what keeps the next one honest. std.debug.assert(!entry.phantom); // BEFORE the frame read, not after. A masked reader must not reconstruct // a shunned artifact's plaintext at all, let alone memoise it where a // deeper delta could then be served from it. `contentCached`'s "identical // bytes to `content(rid)`" includes identical masking. if (mask == .masked and self.shunned(entry)) return null; const bytes = try self.contentAtOffset(gpa, entry.offset, max_delta_chain, entry.hash, cache); // Keep a resolved delta's plaintext hot for any deeper delta that names // this frame as its base; raw/zlib top-level frames are left uncached so // a delta-free sweep never dups a byte. Best-effort: a failed insert // just means no memo, never a wrong or missing answer. if (cache) |c| { if (entry.kind == .delta) c.put(gpa, entry.offset, bytes) catch {}; } return bytes; } /// Like `content`, but resolves delta bases through `cache` — a bounded /// plaintext memo the caller owns for the span of one derivation sweep. A /// full sweep over delta chains would otherwise re-reconstruct each base /// once per referencing delta (the k(k+1)/2 read-amplification the plan /// projects at ~8.5x on depth-16 chains); memoizing a base's plaintext on /// first use makes every later reference a lookup, so each frame is fully /// reconstructed at most about twice regardless of chain depth. A `.delta` /// frame's own plaintext is also memoized so a deeper delta built on it /// hits; `.raw`/`.zlib` top-level results are never inserted, so an /// all-raw repository pays zero cache overhead. Identical bytes to /// `content(rid)`; the cache changes only the work, never the answer. pub fn contentCached(self: *LogStore, gpa: std.mem.Allocator, rid: Rid, cache: *PlaintextCache) ContentError!?[]u8 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); const entry = &self.entries.items[@intCast(rid - 1)]; if (entry.phantom) return null; return try self.contentOfEntry(gpa, entry, .masked, cache); } /// THE MAINTENANCE READER: `content`/`contentCached` with the shun mask /// deliberately absent (phantoms still read null — a phantom has no frame to /// read). `cache` may be null for the uncached shape. /// /// THIS IS NOT AN ARTIFACT READ. The masked readers answer "what does this /// repository surface under this name", and a shunned name surfaces nothing. /// This answers a different question — "what is in this frame" — and exists /// for exactly one kind of caller: in-repository frame MAINTENANCE that /// relocates bytes without showing them to anybody. `pack.zig` is that /// caller, and the only one. /// /// Why it has to exist. A repack reads every source frame and rewrites it /// into a fresh generation. Read through the mask, `packLog`'s `orelse /// continue` would silently DROP every shunned-but-held frame — deleting the /// artifact AND its interned name, so `lookup`/`resolveHexPrefix` start /// answering null where they answered a rid. That is `zit pack` changing /// what an unrelated verb prints, which is exactly the invariant the mask /// exists to protect ("reclamation is invisible"), and it is a /// data-destroying reclamation policy no draft states. /// /// Any path that hands bytes to an operator, a peer, a working tree or a /// derived view MUST use `content`/`contentCached`/`wire`. Reaching for this /// one to "fix a null" means you want the masked reader. /// /// `rid`'s range is CHECKED here, not asserted as on the masked readers. /// This door and `wireUnmasked` exist precisely to bypass a safety rule, and /// in `ReleaseFast`/`ReleaseSmall` both the assert AND the slice bounds check /// are elided — leaving `rid == 0` to underflow before the `@intCast` and /// read off the front of `entries`. One compare, on a path that is about to /// do file I/O, holds the bound in every optimize mode. pub fn contentUnmasked(self: *LogStore, gpa: std.mem.Allocator, rid: Rid, cache: ?*PlaintextCache) ContentError!?[]u8 { if (rid == 0 or rid > self.entries.items.len) return error.Corrupt; const entry = &self.entries.items[@intCast(rid - 1)]; if (entry.phantom) return null; return try self.contentOfEntry(gpa, entry, .maintenance, cache); } /// Reconstructs the uncompressed content of the frame at `offset`. /// `depth_remaining` bounds delta recursion: each `delta` hop consumes /// one, and a delta reached at zero is `error.Corrupt` (never unbounded /// recursion). `expect_hash`, when non-null, requires the frame's embedded /// hash to match (the top-level integrity check); recursion into a base /// passes null since a base carries its own name. `cache`, when non-null, /// memoizes delta-base plaintext (see `contentCached`). /// /// CARRIES NO SHUN MASK, AND MUST NOT — this is normative, not stylistic. /// sync §7.7 L1444-1450: /// /// For an admitted target whose DELTASRC name is in the repository shun /// table, the receiver does not apply admission independently to that /// source: content still held locally MAY serve as the delta basis; if /// the content is unavailable, ordinary dangling-delta handling creates /// or retains the source phantom, but the Section 3.3 scan excludes it, /// so it is not requested. /// /// zit satisfies that STRUCTURALLY and for free. This function is keyed by /// OFFSET, and the delta recursion below reaches a base through `delta_of` /// — a byte offset — passing `expect_hash = null` because a base carries /// its own name. Base resolution therefore never re-enters the rid-keyed /// entry points (`content`/`contentCached`/`wire`) where the hash-keyed /// mask lives: the mask and the basis path are physically distinct code /// paths, so a held-but-shunned base keeps serving every delta rooted on it /// while the base itself reads as absent by name. /// /// Pushing the mask down here would break §7.7 by making that base /// unusable and every dependent delta unresolvable. It would also be a /// silent breakage — the deltas simply stop reconstructing — so the §7.7 /// test in `logstore_test.zig` pins the property rather than leaving this /// comment as its only defence. fn contentAtOffset(self: *LogStore, gpa: std.mem.Allocator, offset: u64, depth_remaining: u8, expect_hash: ?[]const u8, cache: ?*PlaintextCache) ContentError![]u8 { std.debug.assert(depth_remaining <= max_delta_chain); const fv = try self.readFrameAt(gpa, offset); if (expect_hash) |eh| { if (!std.mem.eql(u8, fv.hash[0..fv.hash_len], eh)) { gpa.free(fv.payload); return error.Corrupt; } } switch (fv.kind) { .raw => return fv.payload, // ownership transfers to the caller .zlib => { defer gpa.free(fv.payload); return decodeContainer(gpa, fv.payload, max_payload); }, .delta => { defer gpa.free(fv.payload); if (depth_remaining == 0) return error.Corrupt; // A base must be an earlier, in-range frame: a forward or // self reference is corruption, and a mid-frame/garbage // offset fails `readFrameAt`'s checksums below. if (fv.delta_of < header_len or fv.delta_of >= offset) return error.Corrupt; const raw_delta = try decodeContainer(gpa, fv.payload, max_payload); defer gpa.free(raw_delta); // A memoized base: apply against the borrowed plaintext, never // re-reading or re-reconstructing the base frame this sweep. if (cache) |c| { if (c.get(fv.delta_of)) |hit| { return delta.apply(gpa, hit, raw_delta) catch |e| switch (e) { error.OutOfMemory => error.OutOfMemory, error.MalformedDelta, error.ChecksumMismatch => error.Corrupt, }; } } const base = try self.contentAtOffset(gpa, fv.delta_of, depth_remaining - 1, null, cache); defer gpa.free(base); // A base is referenced by at least this delta, so memoize it. if (cache) |c| c.put(gpa, fv.delta_of, base) catch {}; return delta.apply(gpa, base, raw_delta) catch |e| switch (e) { error.OutOfMemory => error.OutOfMemory, error.MalformedDelta, error.ChecksumMismatch => error.Corrupt, }; }, .config, .shun => { // A delta base (or content lookup) resolving to a config or // shun frame is corruption: neither is content-addressed and // neither is ever a base (§E; `ShunRecord` for shun). gpa.free(fv.payload); return error.Corrupt; }, _ => unreachable, // readFrameAt rejects any flag byte outside {raw,zlib,delta,config,shun} } } /// One fully validated frame read from `offset`: kind, `delta_of`, the /// embedded hash, and the owned on-disk payload (raw bytes, or the zlib /// container for `zlib`/`delta`). Caller owns `payload`. const FrameView = struct { kind: FrameKind, delta_of: u64, hash: [max_hash_len]u8, hash_len: u8, payload: []u8, }; /// Reads and checksum-verifies the frame at `offset`, returning its /// decoded fields and owned payload. Shares the fixed-header validation /// (`readFrameHeader`) with the open-time scanner; unlike the scanner, a /// short read here is `error.Corrupt` — the frame was known to exist, so /// missing bytes mean the file lied or lost data, not a torn tail to /// recover. Any flag byte outside {raw,zlib,delta,config,shun} is rejected. fn readFrameAt(self: *LogStore, gpa: std.mem.Allocator, offset: u64) ContentError!FrameView { const fh = switch (try readFrameHeader(self.file, offset)) { .torn => return error.Corrupt, .header => |h| h, }; const pl: usize = fh.payload_len; if (pl > max_payload) return error.Corrupt; // wire-derived bound before allocation const payload = try gpa.alloc(u8, pl); errdefer gpa.free(payload); const payload_offset: u64 = offset + fh.fixed_and_crc_len; const npay = self.file.pread(payload_offset, payload) catch return error.Storage; if (npay < pl) return error.Corrupt; // Whole-frame CRC: skipped only when `openReadOnly` already verified // every frame at scan time (the file is proven immutable-length there). // The header prefix/header CRCs in `readFrameHeader` always run, so a // frame is never read wholly unchecked; a short payload read above is // still corruption regardless. if (!self.verified_on_open) { var trailer: [4]u8 = undefined; const nt = self.file.pread(payload_offset + pl, &trailer) catch return error.Storage; if (nt < 4) return error.Corrupt; var fcrc = crc32.Crc32.init(); fcrc.update(fh.header_bytes[0..fh.fixed_and_crc_len]); fcrc.update(payload); if (fcrc.final() != std.mem.readInt(u32, &trailer, .little)) return error.Corrupt; } return .{ .kind = fh.kind, .delta_of = fh.delta_of, .hash = fh.hash, .hash_len = fh.hash_len, .payload = payload, }; } /// A stored artifact's cfile wire payload (sync §7.2), served verbatim. /// Two forms, matching the two stored container kinds: /// * `full` — a §4.1 zlib container of the whole artifact; USIZE is its /// own 4-byte big-endian prefix (`writeCfileVerbatim`). /// * `delta` — a §4.1 zlib container of a [FossilDelta], plus the base /// artifact's hash (DELTASRC) and the reconstructed-target size read /// from the delta header (delta §3.2). USIZE is that `target_size` — /// NEVER the container's 4-byte prefix, which is the raw delta's length /// (sync §7.2 L1142-1144: "in delta form it is not delta size"). /// `deinit` frees the owned `container` in either arm. pub const Wire = union(enum) { full: []u8, delta: Delta, pub const Delta = struct { container: []u8, base_hash_buf: [max_hash_len]u8, base_hash_len: u8, /// delta §3.2 — the reconstructed target length; the cfile USIZE. target_size: u64, pub fn baseHash(self: *const Delta) []const u8 { return self.base_hash_buf[0..self.base_hash_len]; } }; pub fn container(self: *const Wire) []const u8 { return switch (self.*) { .full => |c| c, .delta => |d| d.container, }; } pub fn deinit(self: *Wire, gpa: std.mem.Allocator) void { switch (self.*) { .full => |c| gpa.free(c), .delta => |d| gpa.free(d.container), } self.* = undefined; } }; /// The stored cfile payload for `rid` served verbatim (sync §4.1/§7.2), or /// null when the frame holds no stored container — a `.raw` frame or a /// phantom (the serve loop then falls back to compressing `content()`) — or /// when `rid`'s name is in the repository shun table (see `shunned`; the /// `content()` fallback masks too, so a shunned `.raw` artifact has no path /// out either). A `.zlib` frame yields the `full` arm; a `.delta` frame /// yields the `delta` arm, carrying the base hash (read from the base frame) /// and the target size (read from the delta header — one small inflate). The /// stored container IS the wire payload (normative §4.1/§7.2), so this /// exposes no representation the RFC does not already fix. Caller owns the /// `Wire`. pub fn wire(self: *LogStore, gpa: std.mem.Allocator, rid: Rid) ContentError!?Wire { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); const entry = &self.entries.items[@intCast(rid - 1)]; if (entry.phantom) return null; return self.wireOfEntry(gpa, entry, .masked); } /// `wire` with the shun mask deliberately absent — the maintenance door, on /// exactly the terms `contentUnmasked` documents at length: `pack.zig`'s /// verbatim frame copy needs the stored container of a shunned-but-held /// frame to re-emit it into the packed generation. NOT a serve path; a `Wire` /// obtained here MUST NOT reach a peer. `rid`'s range is CHECKED for the /// reason `contentUnmasked` gives. pub fn wireUnmasked(self: *LogStore, gpa: std.mem.Allocator, rid: Rid) ContentError!?Wire { if (rid == 0 or rid > self.entries.items.len) return error.Corrupt; const entry = &self.entries.items[@intCast(rid - 1)]; if (entry.phantom) return null; return self.wireOfEntry(gpa, entry, .maintenance); } /// The kind-directed container read both `wire` doors share, INCLUDING the /// rule it reads under, so they cannot drift in either. fn wireOfEntry(self: *LogStore, gpa: std.mem.Allocator, entry: *const Entry, mask: ReadMask) ContentError!?Wire { // `offsetOf`'s precondition, for the same dereference: a phantom's // `offset` is not a frame start, so `readFrameAt` would decode arbitrary // log bytes and could hand back a checksum-passing neighbour frame under // this entry's name. Callers filter phantoms; this keeps the next honest. std.debug.assert(!entry.phantom); // The stored container IS the wire payload. Masking `content` but not // this would leave a shunned artifact's exact bytes reachable by the one // path whose whole job is handing them to a peer. Before the kind switch, // not inside an arm: every kind that can yield a container must mask, and // a guard that only covers the arm the fixtures happen to exercise is a // leak wearing a test's clothes. if (mask == .masked and self.shunned(entry)) return null; switch (entry.kind) { .raw => return null, .zlib => { const fv = try self.readFrameAt(gpa, entry.offset); std.debug.assert(fv.kind == .zlib); return .{ .full = fv.payload }; }, .delta => { const fv = try self.readFrameAt(gpa, entry.offset); errdefer gpa.free(fv.payload); std.debug.assert(fv.kind == .delta); // Structural phantom safety (D3): a stored delta's base ALWAYS // precedes it in the log; a forward/self reference is corruption. if (fv.delta_of < header_len or fv.delta_of >= entry.offset) return error.Corrupt; // The reconstructed-target size (the §7.2 USIZE) comes from the // delta's own header, not the container prefix (that is the raw // delta length). One small inflate reads it without applying a // single segment. const raw_delta = try decodeContainer(gpa, fv.payload, max_payload); defer gpa.free(raw_delta); const target_size = delta.targetSize(raw_delta) catch return error.Corrupt; // The base frame's own header carries its hash; reading it also // checksum-validates that `delta_of` still points at a real frame. const base = try self.readFrameAt(gpa, fv.delta_of); gpa.free(base.payload); // A non-content frame is never a basis. `content()` already // rejects this exact log, so without the check `wire()` would // hand a peer a delta cfile naming a base that is a config // record or a tombstone — a chain no receiver can resolve, and // disagreement between the two read paths over one file. if (base.kind == .config or base.kind == .shun) return error.Corrupt; var out: Wire = .{ .delta = .{ .container = fv.payload, .base_hash_buf = undefined, .base_hash_len = base.hash_len, .target_size = target_size, } }; @memcpy(out.delta.base_hash_buf[0..base.hash_len], base.hash[0..base.hash_len]); return out; }, // Neither a config nor a shun frame is ever an artifact rid (both // are never interned), so neither reaches here; any other kind is // rejected on open. .config, .shun, _ => unreachable, } } /// Borrows the hash naming `rid`. Every rid, phantom or not, has one. pub fn hashOf(self: *const LogStore, rid: Rid) []const u8 { std.debug.assert(rid >= 1 and rid <= self.entries.items.len); return self.entries.items[@intCast(rid - 1)].hash; } /// Returns caller-owned non-phantom artifact RIDs in the order their /// frames were persisted. Dense RID order is deliberately not used: /// `intern` may create a phantom before a later `put` writes its frame. pub fn ridsInPersistedOrder(self: *const LogStore, gpa: std.mem.Allocator) error{OutOfMemory}![]Rid { var count: usize = 0; for (self.entries.items) |entry| { if (!entry.phantom) count += 1; } const rids = try gpa.alloc(Rid, count); count = 0; for (self.entries.items, 0..) |entry, i| { if (entry.phantom) continue; rids[count] = @intCast(i + 1); count += 1; } std.mem.sort(Rid, rids, self, ridOffsetLessThan); return rids; } fn ridOffsetLessThan(self: *const LogStore, a: Rid, b: Rid) bool { return self.entries.items[@intCast(a - 1)].offset < self.entries.items[@intCast(b - 1)].offset; } pub fn lookup(self: *const LogStore, hash: []const u8) ?Rid { return self.index.get(hash); } /// Resolve a lowercase-hex artifact-name prefix to the single non-phantom /// rid whose full name starts with it, or null when nothing matches /// (artifact §3.6 prefix lookup). `error.Ambiguous` when more than one /// stored artifact shares the prefix — a prefix matching several artifacts /// is REFUSED, never silently narrowed to one (repo-state §8.3 item 8). /// General-purpose: this queries EVERY held artifact, file blobs included, /// since a blob is a non-phantom frame here just like a manifest. No /// allocation — each entry's raw digest is hex-encoded into a fixed stack /// buffer. Bound: `self.entries.items.len`. Preconditions (asserted): the /// prefix is 4..2*max_hash_len bytes and every byte is a lowercase hex /// digit — the caller canonicalizes; a non-hex or out-of-band prefix is the /// caller's not-found family, never reaches here. pub fn resolveHexPrefix(self: *const LogStore, prefix: []const u8) error{Ambiguous}!?Rid { std.debug.assert(prefix.len >= 4 and prefix.len <= 2 * max_hash_len); for (prefix) |c| std.debug.assert((c >= '0' and c <= '9') or (c >= 'a' and c <= 'f')); const lower = "0123456789abcdef"; var found: ?Rid = null; for (self.entries.items, 0..) |entry, i| { if (entry.phantom) continue; // announced but no frame — carries no content var hexbuf: [2 * max_hash_len]u8 = undefined; const hex_len = entry.hash.len * 2; std.debug.assert(hex_len <= hexbuf.len); for (entry.hash, 0..) |b, j| { hexbuf[j * 2] = lower[b >> 4]; hexbuf[j * 2 + 1] = lower[b & 0x0f]; } const hex = hexbuf[0..hex_len]; if (hex.len < prefix.len) continue; // a longer prefix cannot name a shorter hash if (std.mem.startsWith(u8, hex, prefix)) { if (found != null) return error.Ambiguous; // §8.3 item 8: >1 match refuses found = @intCast(i + 1); // rids are 1-based (entries index + 1) } } if (found) |rid| std.debug.assert(!self.entries.items[@intCast(rid - 1)].phantom); return found; } pub fn commit(self: *LogStore) CommitError!void { self.file.sync() catch return error.Storage; } /// Appends a new config frame and makes it the live config (§E). Latest /// wins by scan order, so an edit is one append — no rewrite, no mutable /// pointer. On a torn append `appendFrame` rolls the file back and /// `config_offset` is left untouched, so failure leaves the prior config /// live. pub fn putConfig(self: *LogStore, cfg: Config) PutError!void { std.debug.assert(cfg.default_branch.len <= max_config_str); std.debug.assert(cfg.project_id.len <= max_config_str); std.debug.assert(cfg.project_code.len <= max_config_str); std.debug.assert(cfg.login_user.len <= max_config_str); std.debug.assert(cfg.login_secret_hex.len <= max_config_str); // sync §6.3 — a stored secret is either absent or the full 40-hex // derived value; no other width is ever persisted. std.debug.assert(cfg.login_secret_hex.len == 0 or cfg.login_secret_hex.len == 40); std.debug.assert(cfg.remote_url.len <= max_config_str); var buf: [max_config_payload]u8 = undefined; const payload = encodeConfig(&buf, cfg); const res = try self.appendFrame(config_sentinel, payload, .config, 0); std.debug.assert(res.offset + payload.len < self.end); // frame landed before the new end self.config_offset = res.offset; } /// The live config, or null when the log carries none (§E: absent config /// means the caller — CLI/repostate — owns the default; storage bakes /// none). Re-reads and re-verifies the frame on every call. Caller owns /// the returned `OwnedConfig`. pub fn readConfig(self: *LogStore, gpa: std.mem.Allocator) ContentError!?OwnedConfig { const off = self.config_offset orelse return null; const fv = try self.readFrameAt(gpa, off); errdefer gpa.free(fv.payload); if (fv.kind != .config) return error.Corrupt; const parsed = try parseConfig(fv.payload); return .{ .backing = fv.payload, .config = parsed }; } /// Record `hash` in this repository's shun table, durably: one appended /// `.shun` frame carrying the digest in its own `hash` field and /// `epoch`/`reason` in its payload. Never interned, so no rid is issued and /// `lookup(hash)` is unaffected. Never removes anything — reclaiming the /// shunned artifact's bytes is a separate, later operation. /// /// IDEMPOTENT AS SET MEMBERSHIP: a digest already in the set appends NO /// frame and leaves the existing row's `epoch`/`reason` alone. The durable /// record is the one that was actually written; this is a set, not a /// last-write-wins cell. /// /// INPUT RULES, both CHECKED and returned as `ShunError` rather than /// asserted — see `ShunError` for why an assert is the wrong instrument for /// either (both survive into `ReleaseFast`, both are unrecoverable once the /// frame lands, and both take operator text): /// /// * `hash` is a COMPLETE digest, 20 or 32 bytes (`isCompleteDigest`), /// else `error.BadKey`. This is the one put path that does NOT take the /// opaque `min_hash_len .. max_hash_len` bound. The shun table is keyed /// by complete artifact name, so a whole-digest key makes an abbreviated /// prefix structurally unrepresentable rather than merely discouraged — /// a prefix admitted here would sit in the set never matching the full /// digest anyone queries, i.e. fail-OPEN, the one failure a tombstone /// must not have. It would also be durable: `scan` rejects such a frame /// outright, so admitting one here bricks the repository for every later /// open. The write end therefore states the rule exactly as strongly as /// the read end does. `hash_kind` is still never read: the algorithm is /// inferred from the width, exactly as artifact-format §3.2 L347-348 /// defines the two name forms, so this checks a length and still /// interprets no digest. /// * `reason` is at most `max_shun_reason` bytes, else /// `error.ReasonTooLong`. Unlike `putConfig`'s internally-generated /// fields this is free operator text — the first payload field in this /// file taking any — and `encodeShun` copies it into a fixed /// `max_shun_payload` stack buffer, so the bound is a memory-safety /// bound, not a policy one. An operator-facing caller (the CLI verb) may /// still truncate for ergonomics; it is not required to, because this /// rejects rather than trusts. /// /// Every fallible step runs BEFORE the append, so nothing that can fail /// follows the frame landing: a partially-recorded tombstone (durable on /// disk, absent in memory) would read as fail-OPEN until the next reopen, /// which is exactly the failure a tombstone must not have. pub fn putShun(self: *LogStore, hash: []const u8, reason: []const u8, epoch: i64) ShunError!void { if (!isCompleteDigest(hash.len)) return error.BadKey; if (reason.len > max_shun_reason) return error.ReasonTooLong; if (self.shunHasName(hash)) return; const owned_hash = try self.gpa.dupe(u8, hash); errdefer self.gpa.free(owned_hash); const owned_reason = try self.gpa.dupe(u8, reason); errdefer self.gpa.free(owned_reason); try self.shun_entries.ensureUnusedCapacity(self.gpa, 1); try self.shun_index.ensureUnusedCapacity(self.gpa, 1); // The slice's central invariant — a shun frame is NEVER interned — // stated here, where the append that could violate it happens, and not // only in the tests. Frame-level, deliberately: asserting the digest is // absent from `index` would be WRONG (shunning an artifact this repo // holds is the ordinary case); what must not change is the intern // count. A refactor that routed this append through the content path // would make `lookup(hash)` resolve a tombstone as content. const entries_before = self.entries.items.len; var buf: [max_shun_payload]u8 = undefined; const payload = encodeShun(&buf, epoch, reason); const res = try self.appendFrame(hash, payload, .shun, 0); std.debug.assert(res.offset >= header_len and res.offset + payload.len < self.end); // frame landed before the new end std.debug.assert(self.entries.items.len == entries_before); self.shunAssumeCapacity(owned_hash, epoch, owned_reason); } /// THE ONE DOOR onto the in-memory set — the only place either half is /// written. Takes ownership of two already-duplicated strings, and both /// containers' capacity is already reserved by the caller, so the two writes /// below are infallible and nothing can run between them. That is what lets /// `putShun` place its only irreversible step (the append) last, and it is /// also what makes a HALF-WRITTEN row — listed but not queryable, or /// queryable but not listed — unrepresentable rather than merely asserted /// against afterwards. /// /// Which matters more since the read mask than it did before it. `shunned` /// answers from `shun_index` while `shunList` reads `shun_entries`, so a /// desync no longer just misreports a row to an operator: it leaves an /// artifact displayed as suppressed and still SERVED. A `std.debug.assert` /// pairing check is the wrong instrument for a rule with that consequence — /// it compiles to nothing in `ReleaseFast`/`ReleaseSmall`, i.e. it is absent /// from every build that ships. Funnelling both writers through here removes /// the failure mode instead of watching for it. /// /// The two halves cannot disagree about the BYTES either: the map borrows /// the row's own allocation as its key, and its value is that row's position. /// PRECONDITION: `owned_hash` is not already a key. fn shunAssumeCapacity(self: *LogStore, owned_hash: []u8, epoch: i64, owned_reason: []u8) void { std.debug.assert(!self.shunHasName(owned_hash)); const row = self.shun_entries.items.len; self.shun_entries.appendAssumeCapacity(.{ .hash = owned_hash, .epoch = epoch, .reason = owned_reason }); self.shun_index.putAssumeCapacity(owned_hash, row); } /// Add one scanned shun row to the set, first-writer-wins. A log CAN carry /// two frames for one digest (a concurrent writer, a hand-built fixture); /// the repeat is dropped BEFORE anything is allocated, so — unlike `index`, /// whose keys `entries` owns — this map never takes a duplicate copy of the /// row's own allocation that it would then have to free again. Same shape as /// `putShun`'s idempotence check, and lands in `shunAssumeCapacity` for the /// same reason: one door onto the set. fn internShun(self: *LogStore, hash: []const u8, epoch: i64, reason: []const u8) error{OutOfMemory}!void { if (self.shunHasName(hash)) return; const owned_hash = try self.gpa.dupe(u8, hash); errdefer self.gpa.free(owned_hash); const owned_reason = try self.gpa.dupe(u8, reason); errdefer self.gpa.free(owned_reason); try self.shun_entries.ensureUnusedCapacity(self.gpa, 1); try self.shun_index.ensureUnusedCapacity(self.gpa, 1); self.shunAssumeCapacity(owned_hash, epoch, owned_reason); } /// Is `hash` in this repository's shun table? O(1), no I/O — the check a /// per-round sync filter (sync §3.3 L527-530) can afford per hash. /// /// Asserts the SAME complete-digest domain `putShun` checks, and for a /// sharper reason than symmetry: the two ends of one predicate must accept /// one domain or the read end fails OPEN. Sync and CLI callers handle /// 40/64-char lowercase-hex WIRE names throughout; passing one here would /// otherwise compile, miss silently, and report a shunned artifact as /// servable. `lookup` is unasserted because its bad-key miss is fail-SAFE /// (you lose content); a false from here suppresses a suppression. /// /// CALLER CONTRACT — binding on the sync layer that will wire this in, and /// stated here because this slice is where the domain is fixed: an assert /// is the right instrument only while every argument is locally derived. A /// name off the wire is not. sync §3.3 L527-530 puts this predicate in the /// per-round phantom scan, so once a peer's announced names reach it, a /// name of any other width would trap the process in a safe build — a /// one-packet remote denial of service. The sync boundary MUST therefore /// reject a name that is not one of artifact-format §3.2's two forms (a /// protocol error on that peer's round) BEFORE decoding it to a digest and /// calling this. Widening this predicate to answer false instead is the /// wrong repair: an out-of-domain query silently answered "not shunned" is /// the fail-open case the assert exists to make impossible. pub fn isShunned(self: *const LogStore, hash: []const u8) bool { std.debug.assert(isCompleteDigest(hash.len)); return self.shunHasName(hash); } /// The whole shun table in persisted order, BORROWED — no allocator, no /// error, O(1). Append order is persisted order (the log only grows), so /// there is nothing to sort and nothing to copy. Every row's `hash` and /// `reason` belong to the store and stay valid until the next `putShun` or /// `deinit`. pub fn shunList(self: *const LogStore) []const ShunRecord { return self.shun_entries.items; } /// Structural end of the log (header + every complete frame written so /// far), for callers that track write progress without duplicating /// frame-layout arithmetic (e.g. the DST sweep's crash-recovery model). pub fn size(self: *const LogStore) u64 { return self.end; } /// The number of interned frames (phantom or real) held so far — lets /// `pack.packLog` assert its destination is a just-opened empty log before /// it re-bases anything into it (§6.1 base-availability precondition: a /// hint translated via `dst.lookup` must only ever resolve a frame THIS /// pack wrote, never a pre-existing unrelated one). pub fn entryCount(self: *const LogStore) usize { return self.entries.items.len; } const AppendResult = struct { offset: u64, len: u32 }; /// Encodes one frame and appends it in a single `StorageFile.append` /// call — one call is one atomic torn-write draw point, matching the /// crash simulator's fault granularity. On failure, truncates back to /// the prior structural end so a partial write never lingers; if that /// rollback truncate itself fails the store is poisoned and refuses every /// later append. fn appendFrame(self: *LogStore, hash: []const u8, bytes: []const u8, kind: FrameKind, delta_of: u64) PutError!AppendResult { if (self.poisoned) return error.Storage; // A read-only store proved every frame's whole-frame CRC at scan time and // reads skip the redundant re-CRC; appending here would add a frame that // never gets that check. The append path always opens via `open`. std.debug.assert(!self.verified_on_open); const hl: u64 = hash.len; const pl: u64 = bytes.len; const frame_len = frameSpan(hl, pl); std.debug.assert(frame_len <= max_payload + 64); const frame = try self.gpa.alloc(u8, @intCast(frame_len)); defer self.gpa.free(frame); _ = encodeFrame(frame, hash, bytes, @intFromEnum(kind), delta_of); const offset = self.end; if (self.file.append(frame)) |_| { self.end += frame_len; std.debug.assert(self.end % frame_align == 0); return .{ .offset = offset, .len = @intCast(pl) }; } else |_| { // A torn append may have left a prefix; if the rollback truncate // can't remove it, `end` no longer matches the file end — poison. self.file.truncate(self.end) catch { self.poisoned = true; }; return error.Storage; } } }; /// A bounded plaintext memo for a single derivation sweep: base frame offset to /// its reconstructed plaintext. Owned by the sweep caller (repoview.load), not /// by the store — see `LogStore.contentCached`. Explicitly bounded on BOTH a /// byte budget and an entry count; an insert that would exceed either evicts /// least-recently-used entries first, and a single item larger than the whole /// byte budget is simply never cached (the budget never grows). No unbounded /// allocation and no unbounded memoization — deleting the cache can only change /// speed, never any answer. pub const PlaintextCache = struct { /// Default resident-byte budget: comfortably holds a delta family's hot /// bases without letting the memo grow without limit. pub const default_max_bytes: usize = 64 << 20; /// Default hard cap on resident entries, bounding per-entry metadata even /// under many tiny items. pub const default_max_entries: usize = 4096; const Node = struct { bytes: []u8, last_used: u64 }; map: std.AutoHashMapUnmanaged(u64, Node) = .empty, bytes_used: usize = 0, /// Resident plaintext never exceeds this many bytes (fixed for a cache's /// lifetime; the field form lets a test drive eviction with a small budget). max_bytes: usize = default_max_bytes, /// Resident entry count never exceeds this. max_entries: usize = default_max_entries, /// Monotonic recency clock; the entry with the smallest stamp is the LRU /// victim. u64 never wraps within a single sweep's insert/lookup count. tick: u64 = 0, /// Observability only (never gates logic): the number of `get` calls that /// returned a memoised plaintext. A persist/derivation sweep over delta /// chains proves basis reuse by this rising above zero — a re-reconstruction /// that the memo replaced with a lookup. hits: u64 = 0, pub fn init() PlaintextCache { return .{}; } pub fn deinit(self: *PlaintextCache, gpa: std.mem.Allocator) void { var it = self.map.valueIterator(); while (it.next()) |n| gpa.free(n.bytes); self.map.deinit(gpa); self.* = undefined; } /// The memoized plaintext for `offset`, or null; bumps its LRU recency. /// The returned slice is borrowed and valid until the next `put`/`deinit`. pub fn get(self: *PlaintextCache, offset: u64) ?[]const u8 { const n = self.map.getPtr(offset) orelse return null; self.tick += 1; self.hits += 1; n.last_used = self.tick; return n.bytes; } /// Memoize a private copy of `src` under `offset`, evicting LRU entries to /// stay within both bounds. A no-op when `offset` is already resident (base /// offsets are unique keys) or when `src` alone exceeds the byte budget. /// Only `error.OutOfMemory` escapes. pub fn put(self: *PlaintextCache, gpa: std.mem.Allocator, offset: u64, src: []const u8) error{OutOfMemory}!void { if (self.map.contains(offset)) return; if (src.len > self.max_bytes) return; // never grow the budget for one item // Evict until the newcomer fits under both caps (bounded by the // resident count — each pass removes exactly one entry). while (self.bytes_used + src.len > self.max_bytes or @as(usize, self.map.count()) >= self.max_entries) { if (!self.evictOne(gpa)) break; // nothing left to evict } if (self.bytes_used + src.len > self.max_bytes) return; // still over: skip const owned = try gpa.dupe(u8, src); errdefer gpa.free(owned); self.tick += 1; try self.map.put(gpa, offset, .{ .bytes = owned, .last_used = self.tick }); self.bytes_used += owned.len; } /// Evict the least-recently-used entry; false when the map is empty. fn evictOne(self: *PlaintextCache, gpa: std.mem.Allocator) bool { var victim_key: ?u64 = null; var victim_tick: u64 = std.math.maxInt(u64); var victim_len: usize = 0; var it = self.map.iterator(); while (it.next()) |e| { if (e.value_ptr.last_used < victim_tick) { victim_tick = e.value_ptr.last_used; victim_key = e.key_ptr.*; victim_len = e.value_ptr.bytes.len; } } const k = victim_key orelse return false; const n = self.map.getPtr(k).?; gpa.free(n.bytes); _ = self.map.remove(k); std.debug.assert(self.bytes_used >= victim_len); self.bytes_used -= victim_len; return true; } }; fn writeHeader(file: storage.StorageFile, hash_kind: u8) OpenError!void { var buf: [header_len]u8 = [_]u8{0} ** header_len; @memcpy(buf[0..4], magic); std.mem.writeInt(u16, buf[4..6], version, .little); buf[6] = hash_kind; var crc = crc32.Crc32.init(); crc.update(buf[0..60]); std.mem.writeInt(u32, buf[60..64], crc.final(), .little); file.append(&buf) catch return error.Storage; } /// Returns the stored `hash_kind` on success. Rejects: bad magic, a stored /// version other than this reader's, or a header crc mismatch. fn validateHeader(file: storage.StorageFile) OpenError!u8 { var buf: [header_len]u8 = undefined; const n = file.pread(0, &buf) catch return error.Storage; if (n < header_len) return error.Corrupt; if (!std.mem.eql(u8, buf[0..4], magic)) return error.Corrupt; const stored_version = std.mem.readInt(u16, buf[4..6], .little); if (stored_version != version) return error.Corrupt; const stored_crc = std.mem.readInt(u32, buf[60..64], .little); var crc = crc32.Crc32.init(); crc.update(buf[0..60]); if (crc.final() != stored_crc) return error.Corrupt; return buf[6]; } /// Outcome of reading one frame at a scan offset: either a torn tail (the /// file ends mid-frame — truncate and stop) or a validated frame. const ScanOutcome = union(enum) { torn, frame: struct { hash: [max_hash_len]u8, hash_len: u8, frame_len: u64, payload_len: u32, kind: FrameKind, delta_of: u64 }, }; /// Fixed prefix window: len(4)|flags(1)|hash_len(1)|prefix_crc32(4). Its /// size does not depend on `hash_len` — that is what lets `hash_len` be /// proven correct (or rejected) before it is used to size anything else. const prefix_len: usize = 10; /// A frame's fully validated fixed header — everything before the payload. /// The payload and its whole-frame crc are the caller's concern: the /// content reader allocates and returns the payload, the open-time scanner /// streams it for the crc without allocating. const FrameHeader = struct { payload_len: u32, kind: FrameKind, delta_of: u64, hash: [max_hash_len]u8, hash_len: u8, /// Span of prefix+hash+delta_of+header_crc, i.e. the payload's offset /// relative to the frame start; the whole-frame crc is seeded over /// exactly this many bytes of `header_bytes`. fixed_and_crc_len: usize, /// The verified header bytes, so the caller seeds the whole-frame crc /// without re-reading them. header_bytes: [22 + max_hash_len]u8, }; /// Either a torn tail (a short read on the prefix or header window) or a /// validated fixed header. const HeaderScan = union(enum) { torn, header: FrameHeader, }; /// Reads and checksum-verifies a frame's fixed header at `offset`, shared by /// the content reader (`readFrameAt`) and the open-time scanner /// (`scanOneFrame`). Two-stage: first the fixed-size prefix window /// (hash_len-independent, so a short read there is unambiguously a torn write /// — nothing about its size can be thrown off by corruption), which proves /// `hash_len` trustworthy via its own crc; only then is `hash_len` used to /// size the rest of the header read, at which point the same "short read = /// torn, checksum fails = corrupt" reasoning applies validly (append() only /// ever writes one frame per call and a torn write is a clean prefix — never /// damaged bytes; a full window whose checksum fails can only be a bitflip on /// already-landed bytes). A short read on either window is `.torn`; the /// callers decide its disposition (the scanner recovers, the content reader /// treats it as `error.Corrupt`). Any flag byte outside /// {raw,zlib,delta,config,shun} is rejected. fn readFrameHeader(file: storage.StorageFile, offset: u64) error{ Corrupt, Storage }!HeaderScan { var prefix: [prefix_len]u8 = undefined; const n0 = file.pread(offset, &prefix) catch return error.Storage; if (n0 < prefix_len) return .torn; const payload_len = std.mem.readInt(u32, prefix[0..4], .little); const flags = prefix[4]; const hash_len = prefix[5]; if (hash_len < min_hash_len or hash_len > max_hash_len) return error.Corrupt; var pcrc = crc32.Crc32.init(); pcrc.update(prefix[0..6]); if (pcrc.final() != std.mem.readInt(u32, prefix[6..][0..4], .little)) return error.Corrupt; // A prefix-verified flag byte is trustworthy (a torn write is a clean // truncation, never damaged bytes), so any kind `FrameKind` does not name is // a genuine unknown-kind frame, rejected — not a torn tail. The bound is // `frame_kind_count`, folded from the enum, so adding a kind admits it here // without a second edit and without a stale numeric comparison rejecting // every repository that holds one. if (flags >= frame_kind_count) return error.Corrupt; const hl: usize = hash_len; const fixed_and_crc_len: usize = frameHeaderSpan(hash_len); var hbuf: [22 + max_hash_len]u8 = undefined; const n1 = file.pread(offset, hbuf[0..fixed_and_crc_len]) catch return error.Storage; if (n1 < fixed_and_crc_len) return .torn; var hcrc = crc32.Crc32.init(); hcrc.update(hbuf[0 .. 18 + hl]); if (hcrc.final() != std.mem.readInt(u32, hbuf[18 + hl ..][0..4], .little)) return error.Corrupt; var fh: FrameHeader = .{ .payload_len = payload_len, .kind = @enumFromInt(flags), .delta_of = std.mem.readInt(u64, hbuf[10 + hl ..][0..8], .little), .hash = undefined, .hash_len = hash_len, .fixed_and_crc_len = fixed_and_crc_len, .header_bytes = undefined, }; @memcpy(fh.hash[0..hl], hbuf[10..][0..hl]); @memcpy(fh.header_bytes[0..fixed_and_crc_len], hbuf[0..fixed_and_crc_len]); return .{ .header = fh }; } /// Reads and validates exactly one frame at `offset`, for the open-time scan. /// Delegates the fixed-header validation to `readFrameHeader`; a torn header, /// a frame that would run past `total`, or a torn payload/trailer all report /// `.torn` (recoverable tail), while a failed whole-frame crc is /// `error.Corrupt`. The payload is streamed through the crc in bounded chunks /// so a large frame never forces a full-payload allocation. fn scanOneFrame(file: storage.StorageFile, offset: u64, total: u64) OpenError!ScanOutcome { const fh = switch (try readFrameHeader(file, offset)) { .torn => return .torn, .header => |h| h, }; const frame_len = frameSpan(fh.hash_len, fh.payload_len); if (offset + frame_len > total) return .torn; var crc = crc32.Crc32.init(); crc.update(fh.header_bytes[0..fh.fixed_and_crc_len]); var remaining: u64 = fh.payload_len; var pos = offset + @as(u64, fh.fixed_and_crc_len); var chunk: [4096]u8 = undefined; while (remaining > 0) { const take: usize = @intCast(@min(remaining, @as(u64, chunk.len))); const got = file.pread(pos, chunk[0..take]) catch return error.Storage; if (got < take) return .torn; crc.update(chunk[0..take]); pos += take; remaining -= take; } var trailer: [4]u8 = undefined; const nt = file.pread(pos, &trailer) catch return error.Storage; if (nt < trailer.len) return .torn; if (crc.final() != std.mem.readInt(u32, &trailer, .little)) return error.Corrupt; return .{ .frame = .{ .hash = fh.hash, .hash_len = fh.hash_len, .frame_len = frame_len, .payload_len = fh.payload_len, .kind = fh.kind, .delta_of = fh.delta_of } }; } /// Rebuilds `self.entries`/`self.index` by scanning frames from /// `header_len`. With `recover_torn`, a torn tail truncates to the last valid /// boundary; otherwise it is reported as corruption without mutating storage. /// A corrupt complete frame always propagates `error.Corrupt`. fn scan(self: *LogStore, recover_torn: bool) OpenError!void { const total = self.file.size() catch return error.Storage; const max_frames = (total - header_len) / min_frame + 1; var offset: u64 = header_len; var frames: u64 = 0; while (offset < total) { frames += 1; std.debug.assert(frames <= max_frames); switch (try scanOneFrame(self.file, offset, total)) { .torn => { if (!recover_torn) return error.Corrupt; self.file.truncate(offset) catch return error.Storage; self.end = offset; return; }, .frame => |f| { // Every complete frame — content or config — advances the // cache-key tail bookkeeping. self.frame_count += 1; self.last_frame_offset = offset; if (f.kind == .config) { // Config is not interned: it never enters entries/index, // and "latest wins" is just the last one scan reaches (§E). if (self.config_offset) |prev| std.debug.assert(offset > prev); self.config_offset = offset; } else if (f.kind == .shun) { // A shun row is not interned either (`ShunRecord`): it // never enters entries/index, so `lookup` never resolves a // tombstone. Unlike config there is no "latest wins" — the // set accumulates every row in scan order, which IS // persisted order. // // `scanOneFrame` streams the payload through the crc // without allocating, so the row's metadata is re-read here // directly. That read is stack-bounded and needs no // allocation because the declared length is checked against // `max_shun_payload` first — which is also where the // `max_shun_reason` bound is enforced on the READ side. // // The complete-digest key rule is enforced on BOTH sides: // `putShun` asserts it, and a frame arriving from the file // (hand-built, foreign, or written by a future kind) is // rejected outright here. Asserting only on the write path // would leave `open` able to rebuild a set containing an // abbreviated key — the same fail-OPEN hole, reached // through the file instead of the API. Fail-closed: a shun // frame we cannot key correctly voids the whole open. if (!isCompleteDigest(f.hash_len)) return error.Corrupt; if (f.payload_len > max_shun_payload) return error.Corrupt; const pl: usize = f.payload_len; var pbuf: [max_shun_payload]u8 = undefined; const payload_offset = offset + frameHeaderSpan(f.hash_len); const got = self.file.pread(payload_offset, pbuf[0..pl]) catch return error.Storage; // `scanOneFrame` already proved the whole frame present and // crc-clean, so a short read here is the file lying, not a // torn tail to recover. if (got < pl) return error.Corrupt; const meta = try parseShun(pbuf[0..pl]); // Not-interned, asserted rather than only tested. The whole // purpose of this arm is to bypass the `else` block below, // and nothing but its shape says so; a refactor folding the // two back together would make `lookup(hash)` resolve a // tombstone as content. Frame-level on purpose — the digest // may legitimately be in `index` already (shunning an // artifact this repo holds is the ordinary case), so what // must not move is the intern count. const entries_before = self.entries.items.len; try self.internShun(f.hash[0..f.hash_len], meta.epoch, meta.reason); std.debug.assert(self.entries.items.len == entries_before); } else { // Delta depth is recovered from the base frame, which — by // the append-only, strictly-backward `delta_of` invariant — // was scanned earlier and is already in `entries` (offset // order). A base whose offset is not found is a dangling // delta `content()` will reject; depth 0 is a safe placeholder. const depth: u8 = if (f.kind == .delta) self.depthForBaseOffset(f.delta_of) +| 1 else 0; const owned = try self.gpa.dupe(u8, f.hash[0..f.hash_len]); errdefer self.gpa.free(owned); try self.entries.append(self.gpa, .{ .hash = owned, .offset = offset, .payload_len = f.payload_len, .kind = f.kind, .phantom = false, .depth = depth }); errdefer _ = self.entries.pop(); const rid: Rid = self.entries.items.len; try self.index.put(self.gpa, owned, rid); } const next = offset + f.frame_len; std.debug.assert(next > offset); offset = next; }, } } self.end = offset; } /// Encodes one frame into `buf` (which must be at least the padded frame /// span) and returns the written slice. Shared by the production write /// path and by tests that hand-craft torn/corrupt fixtures. /// /// Layout (all little-endian): len(4) | flags(1) | hash_len(1) | /// prefix_crc32(4) = crc32(bytes[0..6)) | hash(hash_len) | delta_of(8) | /// header_crc32(4) = crc32(bytes[0..18+hash_len)) | payload(len) | /// frame_crc32(4) = crc32(bytes[0..22+hash_len+len)). `delta_of` is the /// base frame's byte offset for a `delta` frame, and 0 otherwise. fn encodeFrame(buf: []u8, hash: []const u8, payload: []const u8, flags: u8, delta_of: u64) []const u8 { std.debug.assert(hash.len >= min_hash_len and hash.len <= max_hash_len); std.debug.assert(payload.len <= max_payload); const hl: usize = hash.len; const pl: usize = payload.len; const hash_len: u8 = @intCast(hl); const payload_len: u32 = @intCast(pl); const unpadded: usize = frame_fixed_overhead + hl + pl; const frame_len: usize = @intCast(frameSpan(hl, pl)); std.debug.assert(buf.len >= frame_len); std.mem.writeInt(u32, buf[0..4], payload_len, .little); buf[4] = flags; buf[5] = hash_len; var pcrc = crc32.Crc32.init(); pcrc.update(buf[0..6]); std.mem.writeInt(u32, buf[6..][0..4], pcrc.final(), .little); @memcpy(buf[10..][0..hl], hash); std.mem.writeInt(u64, buf[10 + hl ..][0..8], delta_of, .little); var hcrc = crc32.Crc32.init(); hcrc.update(buf[0 .. 18 + hl]); std.mem.writeInt(u32, buf[18 + hl ..][0..4], hcrc.final(), .little); @memcpy(buf[22 + hl ..][0..pl], payload); var fcrc = crc32.Crc32.init(); fcrc.update(buf[0 .. 22 + hl + pl]); std.mem.writeInt(u32, buf[22 + hl + pl ..][0..4], fcrc.final(), .little); if (frame_len > unpadded) @memset(buf[unpadded..frame_len], 0); return buf[0..frame_len]; } fn alignUp8(comptime T: type, n: T) T { const a: T = frame_align; return (n + (a - 1)) / a * a; } /// The fixed per-frame overhead (bytes) that brackets the variable hash+payload /// on disk: len(4) + flags(1) + hash_len(1) + prefix_crc32(4) + delta_of(8) + /// header_crc32(4) + frame_crc32(4) = 26 (see `encodeFrame`'s layout). Named /// once so every on-disk span — `frameSpan`, `appendFrame`, and the decode/scan /// paths — derives the same number and pack's §6.1 monotonic floor can never /// silently drift from the span `appendFrame` commits. const frame_fixed_overhead = 26; /// The aligned on-disk span of a frame carrying `hash_len`-byte hash and /// `payload_len`-byte payload — the same `alignUp8(frame_fixed_overhead + hl + /// pl)` `appendFrame` commits. Used by `frameSpanOf` and pack's §6.1 /// monotonic-shrink guard to compare a re-encoding's byte cost against the frame /// it would replace, and by the scan/decode paths to advance frame-to-frame. fn frameSpan(hash_len: u64, payload_len: u64) u64 { return alignUp8(u64, frame_fixed_overhead + hash_len + payload_len); } /// The payload's offset relative to a frame's start: the fixed header window /// `encodeFrame` lays out ahead of it — len(4) + flags(1) + hash_len(1) + /// prefix_crc32(4) + hash(hash_len) + delta_of(8) + header_crc32(4). Named once /// so `readFrameHeader`'s window and the scan's direct payload read of a shun /// row derive the same number from the same layout. /// /// Carries its own domain check rather than inheriting each caller's: the result /// slices a `[22 + max_hash_len]u8` stack buffer and derives a file offset, so a /// future third call site reaching here before validating `hash_len` would slice /// out of bounds. Both current callers pass an already-validated width, which is /// exactly why the check belongs to the helper and not to them. fn frameHeaderSpan(hash_len: u8) usize { std.debug.assert(hash_len >= min_hash_len and hash_len <= max_hash_len); return 22 + @as(usize, hash_len); } /// Encodes `cfg` into `buf` as `[hash_policy u8][branch_len u16 LE][branch] /// [pid_len u16 LE][pid][pcode_len u16 LE][pcode][login_len u16 LE][login] /// [secret_len u16 LE][secret][url_len u16 LE][url]` and returns the written /// slice (§E fixed field order — deterministic, not a hash-map dump). Every /// string field is always written (length zero when empty), so decode /// consumption stays exact. Fixed bounds, no unbounded copy. fn encodeConfig(buf: []u8, cfg: Config) []const u8 { std.debug.assert(buf.len >= max_config_payload); std.debug.assert(cfg.default_branch.len <= max_config_str); std.debug.assert(cfg.project_id.len <= max_config_str); std.debug.assert(cfg.project_code.len <= max_config_str); std.debug.assert(cfg.login_user.len <= max_config_str); std.debug.assert(cfg.login_secret_hex.len <= max_config_str); std.debug.assert(cfg.remote_url.len <= max_config_str); const bl: usize = cfg.default_branch.len; const pl: usize = cfg.project_id.len; const cl: usize = cfg.project_code.len; const ul: usize = cfg.login_user.len; const sl: usize = cfg.login_secret_hex.len; const rl: usize = cfg.remote_url.len; buf[0] = cfg.hash_policy; std.mem.writeInt(u16, buf[1..3], @intCast(bl), .little); @memcpy(buf[3..][0..bl], cfg.default_branch); std.mem.writeInt(u16, buf[3 + bl ..][0..2], @intCast(pl), .little); @memcpy(buf[5 + bl ..][0..pl], cfg.project_id); std.mem.writeInt(u16, buf[5 + bl + pl ..][0..2], @intCast(cl), .little); @memcpy(buf[7 + bl + pl ..][0..cl], cfg.project_code); std.mem.writeInt(u16, buf[7 + bl + pl + cl ..][0..2], @intCast(ul), .little); @memcpy(buf[9 + bl + pl + cl ..][0..ul], cfg.login_user); std.mem.writeInt(u16, buf[9 + bl + pl + cl + ul ..][0..2], @intCast(sl), .little); @memcpy(buf[11 + bl + pl + cl + ul ..][0..sl], cfg.login_secret_hex); std.mem.writeInt(u16, buf[11 + bl + pl + cl + ul + sl ..][0..2], @intCast(rl), .little); @memcpy(buf[13 + bl + pl + cl + ul + sl ..][0..rl], cfg.remote_url); return buf[0 .. 13 + bl + pl + cl + ul + sl + rl]; } /// Parses an `encodeConfig` payload. Every LE length read is bounds-checked /// against the remaining slice before use, each length is capped at /// `max_config_str`, and total consumption must be exact (trailing bytes are /// corruption). Borrows `payload`. All wire-derived, validated before use. fn parseConfig(payload: []const u8) error{Corrupt}!Config { if (payload.len < 1) return error.Corrupt; // no hash_policy byte const hash_policy = payload[0]; var pos: usize = 1; if (payload.len - pos < 2) return error.Corrupt; const blen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (blen > max_config_str) return error.Corrupt; if (payload.len - pos < blen) return error.Corrupt; const branch = payload[pos..][0..blen]; pos += blen; if (payload.len - pos < 2) return error.Corrupt; const plen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (plen > max_config_str) return error.Corrupt; if (payload.len - pos < plen) return error.Corrupt; const pid = payload[pos..][0..plen]; pos += plen; if (payload.len - pos < 2) return error.Corrupt; const clen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (clen > max_config_str) return error.Corrupt; if (payload.len - pos < clen) return error.Corrupt; const pcode = payload[pos..][0..clen]; pos += clen; // sync §6.3 — the login credential fields are an optional trailing block: // a frame written before they existed ends exactly here, and parses with // an empty credential (forward-compatible), never as corruption. if (pos == payload.len) { return .{ .hash_policy = hash_policy, .default_branch = branch, .project_id = pid, .project_code = pcode }; } if (payload.len - pos < 2) return error.Corrupt; const ulen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (ulen > max_config_str) return error.Corrupt; if (payload.len - pos < ulen) return error.Corrupt; const login = payload[pos..][0..ulen]; pos += ulen; if (payload.len - pos < 2) return error.Corrupt; const slen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (slen > max_config_str) return error.Corrupt; if (payload.len - pos < slen) return error.Corrupt; const secret = payload[pos..][0..slen]; pos += slen; // sync §4/§6 — the saved remote URL is another optional trailing block: // a frame written before it existed ends exactly here and parses with an // empty remote (forward-compatible), never as corruption. if (pos == payload.len) { return .{ .hash_policy = hash_policy, .default_branch = branch, .project_id = pid, .project_code = pcode, .login_user = login, .login_secret_hex = secret }; } if (payload.len - pos < 2) return error.Corrupt; const rlen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (rlen > max_config_str) return error.Corrupt; if (payload.len - pos < rlen) return error.Corrupt; const url = payload[pos..][0..rlen]; pos += rlen; if (pos != payload.len) return error.Corrupt; // no trailing bytes return .{ .hash_policy = hash_policy, .default_branch = branch, .project_id = pid, .project_code = pcode, .login_user = login, .login_secret_hex = secret, .remote_url = url, }; } /// One shun row's payload fields, borrowing the buffer they were parsed from. /// The digest is NOT here: it lives in the frame's own `hash` field, which is /// what makes a row self-describing on a raw byte scan. const ShunMeta = struct { epoch: i64, reason: []const u8 }; /// Encodes a shun row's payload into `buf` as `[epoch i64 LE][reason_len u16 /// LE][reason]` and returns the written slice — the same fixed-field-order, /// deterministic shape as `encodeConfig`. `epoch` is written signed and read /// back signed; storage never interprets it. /// /// The `reason` bound stays an assert here — and only here — because its one /// production caller, `putShun`, rejects an over-long reason UNCONDITIONALLY /// before reaching this (`error.ReasonTooLong`). That is deliberate: the copy /// below is into a fixed `max_shun_payload` buffer, so an unchecked over-long /// reason is a stack overrun, not a formatting defect, and a bound that has to /// survive `ReleaseFast` cannot live in an assert. Truncating here instead /// would be worse than rejecting: the durable row and the in-memory row would /// then disagree about the operator's own text. fn encodeShun(buf: []u8, epoch: i64, reason: []const u8) []const u8 { std.debug.assert(buf.len >= max_shun_payload); std.debug.assert(reason.len <= max_shun_reason); const rl: usize = reason.len; std.mem.writeInt(i64, buf[0..8], epoch, .little); std.mem.writeInt(u16, buf[8..10], @intCast(rl), .little); @memcpy(buf[10..][0..rl], reason); return buf[0 .. 10 + rl]; } /// Parses an `encodeShun` payload. The length word is bounds-checked against /// the remaining slice before use, capped at `max_shun_reason`, and total /// consumption must be exact (trailing bytes are corruption) — the same /// discipline `parseConfig` applies field by field. Borrows `payload`. fn parseShun(payload: []const u8) error{Corrupt}!ShunMeta { if (payload.len < 8) return error.Corrupt; // no epoch const epoch = std.mem.readInt(i64, payload[0..8], .little); var pos: usize = 8; if (payload.len - pos < 2) return error.Corrupt; const rlen: usize = std.mem.readInt(u16, payload[pos..][0..2], .little); pos += 2; if (rlen > max_shun_reason) return error.Corrupt; if (payload.len - pos < rlen) return error.Corrupt; const reason = payload[pos..][0..rlen]; pos += rlen; if (pos != payload.len) return error.Corrupt; // no trailing bytes return .{ .epoch = epoch, .reason = reason }; } /// Map the canonical §4.1 decode error set (`codec.DecodeError`) into this /// layer's `ContentError`: a malformed container — short header, a declared /// capacity over `max_output`, or a corrupt RFC1950 stream — is `Corrupt`; /// OOM propagates. The §4.1 codec itself is the leaf `codec` module /// (`container.zig`), shared with `repostate` and `sync`. fn decodeContainer(gpa: std.mem.Allocator, wire: []const u8, max_output: usize) ContentError![]u8 { return codec.decode(gpa, wire, max_output) catch |e| switch (e) { error.OutOfMemory => error.OutOfMemory, error.TruncatedHeader, error.InvalidStream, error.OutputTooLarge => error.Corrupt, }; } // --------------------------------------------------------------------------- // Framing tests (no RFC section behind this file; the oracle is the module // doc's crash-recovery invariants above). // --------------------------------------------------------------------------- fn openTmp(tmp: anytype, io: std.Io, name: []const u8) !storage.FileStorage { const raw = try tmp.dir.createFile(io, name, .{ .read = true }); return storage.FileStorage.init(io, raw, 0); } fn writeRawHeader(file: storage.StorageFile, magic_bytes: []const u8, ver: u16, hk: u8, bad_crc: bool) !void { std.debug.assert(magic_bytes.len == 4); var buf: [header_len]u8 = [_]u8{0} ** header_len; @memcpy(buf[0..4], magic_bytes); std.mem.writeInt(u16, buf[4..6], ver, .little); buf[6] = hk; var crc = crc32.Crc32.init(); crc.update(buf[0..60]); var crc_val = crc.final(); if (bad_crc) crc_val ^= 1; std.mem.writeInt(u32, buf[60..64], crc_val, .little); try file.append(&buf); } test "logstore §C: header round-trips magic/version/hash_kind through open" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store1 = try LogStore.open(std.testing.allocator, file, 7); store1.deinit(std.testing.allocator); var raw: [header_len]u8 = undefined; const n = try file.pread(0, &raw); try std.testing.expectEqual(@as(usize, header_len), n); try std.testing.expectEqualStrings(magic, raw[0..4]); try std.testing.expectEqual(@as(u16, version), std.mem.readInt(u16, raw[4..6], .little)); try std.testing.expectEqual(@as(u8, 7), raw[6]); for (raw[7..60]) |b| try std.testing.expectEqual(@as(u8, 0), b); var crc = crc32.Crc32.init(); crc.update(raw[0..60]); try std.testing.expectEqual(crc.final(), std.mem.readInt(u32, raw[60..64], .little)); var store2 = try LogStore.open(std.testing.allocator, file, 0); // hint ignored on reopen defer store2.deinit(std.testing.allocator); try std.testing.expectEqual(@as(u8, 7), store2.hash_kind); } test "logstore: logKey + validateTail track the append-only tail in O(1)" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); // Empty log: no complete frames, end at the header boundary; validateTail // accepts the empty key exactly at that length. logKey is defined over an // already-scanned store, so `open` (which scans, and for an empty file just // writes the header) is the store the key is read from. const empty = blk: { var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const k = try store.logKey(); break :blk k; }; try std.testing.expectEqual(@as(u64, 0), empty.frame_count); try std.testing.expectEqual(@as(u64, header_len), empty.end); try std.testing.expect(LogStore.validateTail(file, empty)); // Append two frames through a writable store, then read the key from a // freshly-scanned store — the exact shape production keys on (`repocache` // computes the LogKey over an `openReadOnly` scan, never a mid-append // writable store, whose tail bookkeeping is only established by a scan). { var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); _ = try store.put(&([_]u8{1} ** 32), "payload one"); _ = try store.put(&([_]u8{2} ** 32), "payload two two"); try store.commit(); } var scan2 = try LogStore.openReadOnly(std.testing.allocator, file); const k2 = try scan2.logKey(); scan2.deinit(std.testing.allocator); try std.testing.expectEqual(@as(u64, 2), k2.frame_count); try std.testing.expect(LogStore.validateTail(file, k2)); // still fresh try std.testing.expect(!LogStore.validateTail(file, empty)); // stale (shorter) key is a miss // Appending one frame advances the key: the old key no longer validates. { var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); _ = try store.put(&([_]u8{3} ** 32), "third"); try store.commit(); } try std.testing.expect(!LogStore.validateTail(file, k2)); // grown log ⇒ miss var scan3 = try LogStore.openReadOnly(std.testing.allocator, file); const k3 = try scan3.logKey(); scan3.deinit(std.testing.allocator); try std.testing.expectEqual(@as(u64, 3), k3.frame_count); try std.testing.expect(k3.end > k2.end); try std.testing.expect(LogStore.validateTail(file, k3)); } test "logstore §3.6/§8.3-8: resolveHexPrefix resolves a unique prefix, refuses an ambiguous one, misses an unknown" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); // Two artifacts sharing the "abcd" nibble prefix and one distinct, so // "abcd" is ambiguous while "abcd00"/"ef" each name exactly one. const h_shared_a = [_]u8{ 0xab, 0xcd, 0x00 } ++ [_]u8{0} ** 29; // abcd00... const h_shared_b = [_]u8{ 0xab, 0xcd, 0x11 } ++ [_]u8{0} ** 29; // abcd11... const h_other = [_]u8{ 0xef, 0x01 } ++ [_]u8{0} ** 30; // ef01... { var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); _ = try store.put(&h_shared_a, "alpha"); _ = try store.put(&h_shared_b, "bravo"); _ = try store.put(&h_other, "other"); try store.commit(); } var store = try LogStore.openReadOnly(std.testing.allocator, file); defer store.deinit(std.testing.allocator); // Unique 6-char prefix → the exact rid; the full name round-trips. const rid_a = (try store.resolveHexPrefix("abcd00")).?; try std.testing.expectEqualSlices(u8, &h_shared_a, store.hashOf(rid_a)); // A distinct 4-char prefix names the third artifact uniquely. const rid_e = (try store.resolveHexPrefix("ef01")).?; try std.testing.expectEqualSlices(u8, &h_other, store.hashOf(rid_e)); // Ambiguous 4-char prefix → refused, never one silently picked. try std.testing.expectError(error.Ambiguous, store.resolveHexPrefix("abcd")); // A prefix no artifact carries → null (the not-found family, not an error). try std.testing.expectEqual(@as(?Rid, null), try store.resolveHexPrefix("dead")); } test "logstore §C: open rejects a corrupt file-header crc" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); try writeRawHeader(file, magic, version, 0, true); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C: open rejects a version newer than this reader" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); try writeRawHeader(file, magic, version + 1, 0, false); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C: the frame header crc covers len/flags/hash_len/hash/delta_of" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // header only var buf: [64]u8 = undefined; const frame = encodeFrame(&buf, "aaaa", "payload", @intFromEnum(FrameKind.raw), 0); var corrupted: [64]u8 = undefined; @memcpy(corrupted[0..frame.len], frame); corrupted[10] ^= 0xff; // inside the hash bytes, part of the fixed header try file.append(corrupted[0..frame.len]); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C(inv2): a genuinely torn (prefix-truncated) final frame truncates to exactly the last valid boundary" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store = try LogStore.open(std.testing.allocator, file, 0); _ = try store.put("frame-one-hash", "payload-one"); try store.commit(); const boundary = store.size(); store.deinit(std.testing.allocator); var buf: [96]u8 = undefined; const frame2 = encodeFrame(&buf, "frame-two-hash", "payload-two", @intFromEnum(FrameKind.raw), 0); try file.append(frame2[0..10]); // short of even the fixed header+crc window var reopened = try LogStore.open(std.testing.allocator, file, 0); defer reopened.deinit(std.testing.allocator); try std.testing.expectEqual(boundary, reopened.size()); try std.testing.expect(reopened.lookup("frame-two-hash") == null); } test "logstore §C(inv2): a torn payload/crc tail truncates exactly" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store = try LogStore.open(std.testing.allocator, file, 0); _ = try store.put("frame-a-hash-here", "hello-payload"); try store.commit(); const boundary = store.size(); store.deinit(std.testing.allocator); var buf: [128]u8 = undefined; const frame2 = encodeFrame(&buf, "frame-b-hash-value", "a longer second payload for this test", @intFromEnum(FrameKind.raw), 0); const header_window = 22 + "frame-b-hash-value".len; try file.append(frame2[0 .. header_window + 3]); // full header+crc landed; payload/crc badly short var reopened = try LogStore.open(std.testing.allocator, file, 0); defer reopened.deinit(std.testing.allocator); try std.testing.expectEqual(boundary, reopened.size()); } test "logstore §C(inv4): a bitflipped header on a durable non-final frame surfaces error.Corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); var buf1: [64]u8 = undefined; const f1 = encodeFrame(&buf1, "frame-one-abc", "payload-one", @intFromEnum(FrameKind.raw), 0); var corrupted: [64]u8 = undefined; @memcpy(corrupted[0..f1.len], f1); corrupted[10] ^= 0xff; try file.append(corrupted[0..f1.len]); var buf2: [64]u8 = undefined; const f2 = encodeFrame(&buf2, "frame-two-def", "payload-two", @intFromEnum(FrameKind.raw), 0); try file.append(f2); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C(inv4): a bitflipped complete final frame is error.Corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); var buf: [64]u8 = undefined; const f = encodeFrame(&buf, "final-frame-hash", "payload-for-final", @intFromEnum(FrameKind.raw), 0); var corrupted: [64]u8 = undefined; @memcpy(corrupted[0..f.len], f); corrupted[6] ^= 0xff; try file.append(corrupted[0..f.len]); // complete, fully landed, nothing follows try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C(inv4): a corrupted payload on a complete frame surfaces error.Corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); var buf: [96]u8 = undefined; const f = encodeFrame(&buf, "payload-hash-value", "some payload bytes here", @intFromEnum(FrameKind.raw), 0); var corrupted: [96]u8 = undefined; @memcpy(corrupted[0..f.len], f); const header_window = 22 + "payload-hash-value".len; corrupted[header_window] ^= 0xff; // inside the payload, not the fixed header try file.append(corrupted[0..f.len]); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C(inv4): a bitflipped hash_len that would need more bytes than the file has surfaces error.Corrupt, not a torn tail" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // hash_len=1, payload="" -> frame_len = align8(26+1+0) = 32, fully // landed, nothing follows. Corrupting hash_len to an in-range value // (20) makes the naive "read 18+hash_len bytes, short = torn" reading // demand 38 bytes from a 32-byte frame -- exactly the false-torn gap // the prefix checksum closes: it fails on this corrupted byte before // any window-size decision is made from it. var buf: [64]u8 = undefined; const f = encodeFrame(&buf, "h", "", @intFromEnum(FrameKind.raw), 0); var corrupted: [64]u8 = undefined; @memcpy(corrupted[0..f.len], f); corrupted[5] = 20; try file.append(corrupted[0..f.len]); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C: a crc-valid frame with an unknown flag kind (not raw/zlib/delta/config/shun) is rejected on open" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // The first flag byte `FrameKind` does not name: both crcs are computed // correctly over it, so the frame is byte-intact -- it is simply an // encoding this reader does not know, which must fail closed rather than be // interpreted. Taken from `frame_kind_count` rather than written as 5, so // adding a kind moves this fixture to the next unknown byte instead of // asserting that a now-KNOWN kind is rejected. try std.testing.expectEqual(@as(usize, 5), frame_kind_count); // raw|zlib|delta|config|shun var buf: [64]u8 = undefined; const f = encodeFrame(&buf, "unknown-flag-hash", "some payload bytes", @intCast(frame_kind_count), 0); try file.append(f); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } // --------------------------------------------------------------------------- // zlib and delta frames (container sync §4.1; verbatim serve sync §7.2; // delta apply delta §4). // --------------------------------------------------------------------------- /// Appends one hand-crafted frame (valid crcs) to `file`, for fixtures that /// need a payload/flag/delta_of a normal put would never produce. fn appendCrafted(gpa: std.mem.Allocator, file: storage.StorageFile, hash: []const u8, payload: []const u8, kind: FrameKind, delta_of: u64) !void { const frame_len: usize = @intCast(frameSpan(hash.len, payload.len)); const buf = try gpa.alloc(u8, frame_len); defer gpa.free(buf); const f = encodeFrame(buf, hash, payload, @intFromEnum(kind), delta_of); try file.append(f); } test "logstore §C: content on a malformed zlib container is error.Corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // A crc-valid zlib frame whose payload is not a valid container: two bytes // is short of even the 4-byte length header, so decodeContainer rejects it. const hash = "malformed-zlib-container-hash-01"; try appendCrafted(std.testing.allocator, file, hash, "\x00\x00", .zlib, 0); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const rid = store.lookup(hash).?; try std.testing.expectError(error.Corrupt, store.content(std.testing.allocator, rid)); } test "logstore §C/sync §4.1: a zlib container whose declared length exceeds max_payload is rejected without a huge allocation" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // Declared length 0xFFFFFFFF is far past max_payload; decodeContainer must // reject on the declared value before inflating anything. const hash = "oversize-declared-length-hash-01"; try appendCrafted(std.testing.allocator, file, hash, "\xff\xff\xff\xff\x78\x9c", .zlib, 0); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const rid = store.lookup(hash).?; try std.testing.expectError(error.Corrupt, store.content(std.testing.allocator, rid)); } test "logstore §C/delta §4: content on a delta whose inflated payload is not a valid delta is error.Corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // A valid raw base, then a crc-valid delta frame whose container inflates // to bytes that are not a conforming delta -> delta.apply rejects -> Corrupt. try appendCrafted(std.testing.allocator, file, "garbage-delta-base-hash-00000001", "base content here", .raw, 0); const container = try codec.encode(std.testing.allocator, "this is not a delta at all", .default); defer std.testing.allocator.free(container); try appendCrafted(std.testing.allocator, file, "garbage-delta-hash-value-0000001", container, .delta, header_len); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const rid = store.lookup("garbage-delta-hash-value-0000001").?; try std.testing.expectError(error.Corrupt, store.content(std.testing.allocator, rid)); } test "logstore §C: content on a delta whose delta_of is out of range or mid-frame is error.Corrupt" { const cases = [_]struct { name: []const u8, delta_of: u64 }{ .{ .name = "below-header", .delta_of = 0 }, .{ .name = "forward-ref", .delta_of = 1 << 20 }, .{ .name = "mid-base-frame", .delta_of = header_len + 3 }, }; for (cases) |c| { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); try appendCrafted(std.testing.allocator, file, "bad-delta-of-base-hash-000000001", "base content bytes", .raw, 0); const container = try codec.encode(std.testing.allocator, "1\ndelta-ish payload; only delta_of matters here", .default); defer std.testing.allocator.free(container); try appendCrafted(std.testing.allocator, file, "bad-delta-of-hash-value-00000001", container, .delta, c.delta_of); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const rid = store.lookup("bad-delta-of-hash-value-00000001").?; try std.testing.expectError(error.Corrupt, store.content(std.testing.allocator, rid)); } } // Seeded deterministic fuzz: decode/apply on adversarial wire bytes must // yield a value or a declared error, never a crash or a hang. test "logstore §C/sync §4.1 fuzz: a garbage zlib container never crashes content" { var prng = std.Random.DefaultPrng.init(0x2172B); const random = prng.random(); var iter: usize = 0; while (iter < 150) : (iter += 1) { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store = try LogStore.open(std.testing.allocator, file, 0); store.deinit(std.testing.allocator); // header only; craft the frame directly // A crc-valid zlib frame with a random payload must never crash content. var garbage: [64]u8 = undefined; const gn = random.uintLessThan(usize, garbage.len + 1); random.bytes(garbage[0..gn]); var ghash: [9]u8 = undefined; ghash[0] = 'g'; std.mem.writeInt(u64, ghash[1..9], iter, .little); try appendCrafted(std.testing.allocator, file, &ghash, garbage[0..gn], .zlib, 0); var reopened = try LogStore.open(std.testing.allocator, file, 0); defer reopened.deinit(std.testing.allocator); const grid = reopened.lookup(&ghash).?; if (reopened.content(std.testing.allocator, grid)) |maybe| { if (maybe) |b| std.testing.allocator.free(b); } else |err| switch (err) { error.OutOfMemory, error.Storage, error.Corrupt => {}, } } } test "logstore §C/delta §4 fuzz: a garbage delta payload or delta_of never crashes content" { var prng = std.Random.DefaultPrng.init(0xD317A); const random = prng.random(); var iter: usize = 0; while (iter < 150) : (iter += 1) { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store = try LogStore.open(std.testing.allocator, file, 0); store.deinit(std.testing.allocator); // header only; craft the frame directly // A crc-valid delta frame with a random container payload and a random // delta_of must never crash content (Corrupt/OOM/value only). var garbage: [64]u8 = undefined; const gn = random.uintLessThan(usize, garbage.len + 1); random.bytes(garbage[0..gn]); var ghash: [9]u8 = undefined; ghash[0] = 'x'; std.mem.writeInt(u64, ghash[1..9], iter, .little); try appendCrafted(std.testing.allocator, file, &ghash, garbage[0..gn], .delta, random.int(u64)); var reopened = try LogStore.open(std.testing.allocator, file, 0); defer reopened.deinit(std.testing.allocator); const grid = reopened.lookup(&ghash).?; if (reopened.content(std.testing.allocator, grid)) |maybe| { if (maybe) |b| std.testing.allocator.free(b); } else |err| switch (err) { error.OutOfMemory, error.Storage, error.Corrupt => {}, } } } test "logstore §C: a config frame is not interned — rids stay content-only and contiguous" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const r1 = try store.put("config-interleave-hash-one-0001", "one"); try store.putConfig(.{ .hash_policy = 0, .default_branch = "trunk", .project_id = "pid" }); const r2 = try store.put("config-interleave-hash-two-0001", "two"); // The config frame consumed a byte range but no rid: content rids stay // dense and contiguous, and the sentinel hash is never an index key. try std.testing.expectEqual(r1 + 1, r2); try std.testing.expect(store.lookup(config_sentinel) == null); } test "logstore §C: a delta base offset pointing at a config frame is rejected as corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // header only // A config frame at header_len, then a delta whose (valid) container base // offset points back at it: resolving the base hits the .config arm. var cbuf: [max_config_payload]u8 = undefined; const cfg_payload = encodeConfig(&cbuf, .{ .hash_policy = 4, .default_branch = "trunk", .project_id = "pid" }); try appendCrafted(std.testing.allocator, file, config_sentinel, cfg_payload, .config, 0); const container = try codec.encode(std.testing.allocator, "delta-ish bytes; only the base offset matters here", .default); defer std.testing.allocator.free(container); try appendCrafted(std.testing.allocator, file, "delta-over-config-base-hash-0001", container, .delta, header_len); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const rid = store.lookup("delta-over-config-base-hash-0001").?; try std.testing.expectError(error.Corrupt, store.content(std.testing.allocator, rid)); } test "logstore §E: parseConfig rejects malformed config payloads" { var over: [3]u8 = undefined; over[0] = 0; // hash_policy std.mem.writeInt(u16, over[1..3], @intCast(max_config_str + 1), .little); // branch_len > cap const cases = [_][]const u8{ &[_]u8{}, // (a) empty: no hash_policy byte &[_]u8{ 0x07, 0x05 }, // (b) truncated branch_len (1 of 2 bytes) &over, // (c) branch_len exceeds max_config_str &[_]u8{ 0x00, 0x05, 0x00, 'a', 'b' }, // (d) branch_len 5 exceeds remaining &[_]u8{ 0x00, 0x01, 0x00, 'b', 0x02 }, // (e) truncated pid_len after branch &[_]u8{ 0x00, 0x01, 0x00, 'b', 0x00, 0x00, 0x09 }, // (f) trailing byte after pid }; for (cases) |c| try std.testing.expectError(error.Corrupt, parseConfig(c)); } test "logstore §E/§6.3: parseConfig forward-compat-decodes a legacy pre-credential frame" { // A frame written before the login credential block existed carries only // the three string fields (branch, project_id, project_code) and ends // exactly after project_code — encodeConfig can no longer emit this shape // (it always writes the two trailing length words), so the bytes are hand- // built. parseConfig's forward-compat arm must accept it with the credential // fields defaulting empty, never as corruption. const legacy = [_]u8{ 0x04, // hash_policy 0x05, 0x00, 't', 'r', 'u', 'n', 'k', // branch_len=5, "trunk" 0x03, 0x00, 'p', 'i', 'd', // pid_len=3, "pid" 0x05, 0x00, 'c', 'o', 'd', 'e', '9', // pcode_len=5, "code9" }; const cfg = try parseConfig(&legacy); try std.testing.expectEqual(@as(u8, 4), cfg.hash_policy); try std.testing.expectEqualStrings("trunk", cfg.default_branch); try std.testing.expectEqualStrings("pid", cfg.project_id); try std.testing.expectEqualStrings("code9", cfg.project_code); try std.testing.expectEqual(@as(usize, 0), cfg.login_user.len); try std.testing.expectEqual(@as(usize, 0), cfg.login_secret_hex.len); try std.testing.expectEqual(@as(usize, 0), cfg.remote_url.len); } test "config logstore §E: encodeConfig/parseConfig round-trips remote_url" { // sync §4/§6 — the saved remote is the sixth trailing block; encoding a // full config and decoding it must recover the URL verbatim with every // other field intact (the fixed §E field order is a lossless round-trip). var buf: [max_config_payload]u8 = undefined; const wire = encodeConfig(&buf, .{ .hash_policy = 7, .default_branch = "trunk", .project_id = "pid7", .project_code = "code7", .login_user = "alice", .login_secret_hex = "0123456789abcdef0123456789abcdef01234567", // 40 hex .remote_url = "http://h.example/xfer", }); const cfg = try parseConfig(wire); try std.testing.expectEqual(@as(u8, 7), cfg.hash_policy); try std.testing.expectEqualStrings("trunk", cfg.default_branch); try std.testing.expectEqualStrings("pid7", cfg.project_id); try std.testing.expectEqualStrings("code7", cfg.project_code); try std.testing.expectEqualStrings("alice", cfg.login_user); try std.testing.expectEqualStrings("0123456789abcdef0123456789abcdef01234567", cfg.login_secret_hex); try std.testing.expectEqualStrings("http://h.example/xfer", cfg.remote_url); } test "config logstore §E: a pre-remote frame decodes with empty remote_url" { // A frame written before the remote-URL block existed carries the five // pre-URL fields (branch, pid, pcode, login, secret) and ends exactly after // the secret — encodeConfig can no longer emit this shape (it always writes // the URL length word), so the bytes are hand-built. parseConfig's // forward-compat arm must accept it with remote_url empty, never Corrupt. const pre_remote = [_]u8{ 0x04, // hash_policy 0x05, 0x00, 't', 'r', 'u', 'n', 'k', // branch_len=5 0x03, 0x00, 'p', 'i', 'd', // pid_len=3 0x05, 0x00, 'c', 'o', 'd', 'e', '9', // pcode_len=5 0x03, 0x00, 'b', 'o', 'b', // login_len=3 0x02, 0x00, 'a', 'b', // secret_len=2 (parseConfig caps, does not fix at 40) }; const cfg = try parseConfig(&pre_remote); try std.testing.expectEqualStrings("bob", cfg.login_user); try std.testing.expectEqualStrings("ab", cfg.login_secret_hex); try std.testing.expectEqual(@as(usize, 0), cfg.remote_url.len); } test "config logstore §E: remote_url over max_config_str is rejected" { // A url_len declaring more than the per-field cap is corruption: the same // guard every other length field carries (parity, not a special case). const pre_remote = [_]u8{ 0x04, // hash_policy 0x05, 0x00, 't', 'r', 'u', 'n', 'k', // branch 0x03, 0x00, 'p', 'i', 'd', // pid 0x05, 0x00, 'c', 'o', 'd', 'e', '9', // pcode 0x03, 0x00, 'b', 'o', 'b', // login 0x02, 0x00, 'a', 'b', // secret }; // url_len = max_config_str + 1 = 256 = 0x0100 (LE 0x00, 0x01). const oversized = pre_remote ++ [_]u8{ 0x00, 0x01 }; try std.testing.expectError(error.Corrupt, parseConfig(&oversized)); } test "logstore §B: a torn config-frame write reverts to the prior committed config" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var store = try LogStore.open(std.testing.allocator, file, 0); try store.putConfig(.{ .hash_policy = 9, .default_branch = "trunk", .project_id = "keep" }); try store.commit(); store.deinit(std.testing.allocator); // Append a torn config B frame (prefix only): scan truncates it away, so // the surviving latest config is still A. var pbuf: [max_config_payload]u8 = undefined; const cfgB = encodeConfig(&pbuf, .{ .hash_policy = 1, .default_branch = "x", .project_id = "y" }); var fbuf: [64]u8 = undefined; const frameB = encodeFrame(&fbuf, config_sentinel, cfgB, @intFromEnum(FrameKind.config), 0); try file.append(frameB[0..10]); // torn: short of the fixed header+crc window var reopened = try LogStore.open(std.testing.allocator, file, 0); defer reopened.deinit(std.testing.allocator); var oc = (try reopened.readConfig(std.testing.allocator)).?; defer oc.deinit(std.testing.allocator); try std.testing.expectEqual(@as(u8, 9), oc.config.hash_policy); try std.testing.expectEqualStrings("trunk", oc.config.default_branch); try std.testing.expectEqualStrings("keep", oc.config.project_id); } // Seeded deterministic fuzz: parseConfig on adversarial wire bytes yields a // value or error.Corrupt, never a crash. test "logstore §E fuzz: parseConfig never crashes on adversarial payloads" { var prng = std.Random.DefaultPrng.init(0xC04F16); const random = prng.random(); var iter: usize = 0; while (iter < 400) : (iter += 1) { var garbage: [64]u8 = undefined; const gn = random.uintAtMost(usize, garbage.len); random.bytes(garbage[0..gn]); if (parseConfig(garbage[0..gn])) |_| {} else |err| switch (err) { error.Corrupt => {}, } } } test "logstore §5: deltaProfitable is the exact <75% integer boundary" { // 75/100 is 75% exactly → NOT strictly under → reject; 74/100 accepts. try std.testing.expect(!deltaProfitable(75, 100)); try std.testing.expect(deltaProfitable(74, 100)); // Small-integer edges, where a rounded/float ratio would smear the line: // 3/4 == 75% rejects; 2/3 < 75% accepts; 1/1 (4 < 3) rejects. try std.testing.expect(!deltaProfitable(3, 4)); try std.testing.expect(deltaProfitable(2, 3)); try std.testing.expect(!deltaProfitable(1, 1)); // Degenerate ends: a zero-length delta beats any non-empty target; an // empty target admits no profitable delta. try std.testing.expect(deltaProfitable(0, 1)); try std.testing.expect(!deltaProfitable(0, 0)); } // --------------------------------------------------------------------------- // Shun frames. Tests reaching file-private decls (`encodeFrame`, // `appendCrafted`, `encodeShun`) stay here per the kind-separation convention; // the public-surface shun tests live in `logstore_test.zig`. // --------------------------------------------------------------------------- test "logstore §C: a delta base offset pointing at a shun frame is rejected as corrupt" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // header only // A shun frame at header_len, then a delta whose (valid) container base // offset points back at it: resolving the base hits the .shun arm. The // twin of the config case — a tombstone is not content and never a basis. var sbuf: [max_shun_payload]u8 = undefined; const shun_payload = encodeShun(&sbuf, 1700000000, "takedown"); try appendCrafted(std.testing.allocator, file, "shunned-artifact-hash-0000000001", shun_payload, .shun, 0); const container = try codec.encode(std.testing.allocator, "delta-ish bytes; only the base offset matters here", .default); defer std.testing.allocator.free(container); try appendCrafted(std.testing.allocator, file, "delta-over-shun-base-hash-000001", container, .delta, header_len); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); const rid = store.lookup("delta-over-shun-base-hash-000001").?; try std.testing.expectError(error.Corrupt, store.content(std.testing.allocator, rid)); } test "logstore §C(inv2): a torn shun frame truncates to exactly the last valid boundary" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); const durable = "s" ** 32; const torn = "t" ** 32; var store = try LogStore.open(std.testing.allocator, file, 0); _ = try store.put("frame-one-hash", "payload-one"); try store.putShun(durable, "durable tombstone", 42); try store.commit(); const boundary = store.size(); store.deinit(std.testing.allocator); var pbuf: [max_shun_payload]u8 = undefined; const torn_payload = encodeShun(&pbuf, 99, "never lands"); var fbuf: [128]u8 = undefined; const frame = encodeFrame(&fbuf, torn, torn_payload, @intFromEnum(FrameKind.shun), 0); try file.append(frame[0..10]); // short of the fixed header+crc window var reopened = try LogStore.open(std.testing.allocator, file, 0); defer reopened.deinit(std.testing.allocator); try std.testing.expectEqual(boundary, reopened.size()); try std.testing.expect(reopened.isShunned(durable)); // committed tombstone stands try std.testing.expect(!reopened.isShunned(torn)); // the torn one never existed try std.testing.expectEqual(@as(usize, 1), reopened.shunList().len); } test "logstore §C: a shun frame over the max_shun_reason bound is rejected on open" { // (a) a declared reason length past the cap, inside an otherwise plausible // payload — `parseShun`'s per-field guard (parity with `parseConfig`); // (b) a payload longer than any conforming shun record — rejected on the // frame's declared length, before a byte of it is parsed. var over_cap: [13]u8 = undefined; std.mem.writeInt(i64, over_cap[0..8], 5, .little); std.mem.writeInt(u16, over_cap[8..10], @intCast(max_shun_reason + 1), .little); @memcpy(over_cap[10..13], "abc"); var over_frame: [max_shun_payload + 1]u8 = [_]u8{0} ** (max_shun_payload + 1); std.mem.writeInt(u16, over_frame[8..10], @intCast(max_shun_reason + 1), .little); const cases = [_][]const u8{ &over_cap, &over_frame }; for (cases) |payload| { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // header only try appendCrafted(std.testing.allocator, file, "oversize-shun-reason-hash-000001", payload, .shun, 0); try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } } test "logstore §C: a shun frame keyed by an incomplete digest is rejected on open" { // The complete-artifact-name key rule on the READ path. `putShun` rejects // such a key with `error.BadKey`, but a frame that is already in the file // never went through `putShun` — it was hand-built, written by a foreign // implementation, or written by a future kind — so `scan` states the rule // too. Without this the invariant would hold only for logs this binary // wrote, and `open` could rebuild a set holding an abbreviated key — the // same fail-open hole, reached through the file instead of the API. // // 31 bytes is inside `min_hash_len .. max_hash_len` — a width `put` accepts // happily — so this pins the shun-specific rule, not the generic bound. const incomplete = "shun-key-that-is-31-bytes-long!"; try std.testing.expectEqual(@as(usize, 31), incomplete.len); try std.testing.expect(!isCompleteDigest(incomplete.len)); var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // header only var sbuf: [max_shun_payload]u8 = undefined; const payload = encodeShun(&sbuf, 1700000000, "prefix key"); try appendCrafted(std.testing.allocator, file, incomplete, payload, .shun, 0); // Fail-CLOSED: the whole open fails rather than the row being skipped. try std.testing.expectError(error.Corrupt, LogStore.open(std.testing.allocator, file, 0)); } test "logstore §C: two shun frames naming the same digest collapse to one set entry" { var tmp = std.testing.tmpDir(.{}); defer tmp.cleanup(); const io = std.testing.io; var fs = try openTmp(&tmp, io, "log.bin"); defer fs.file.close(io); const file = fs.storageFile(); var seed = try LogStore.open(std.testing.allocator, file, 0); seed.deinit(std.testing.allocator); // header only // A log CAN carry a repeat — a concurrent writer, or a hand-built fixture // like this one. `putShun`'s idempotence check cannot prevent it, so the // scan must: the first record stands and the duplicate's owned copies are // freed (`std.testing.allocator` fails the test on the leak otherwise). const dup = "d" ** 32; var b1: [max_shun_payload]u8 = undefined; try appendCrafted(std.testing.allocator, file, dup, encodeShun(&b1, 11, "first"), .shun, 0); var b2: [max_shun_payload]u8 = undefined; try appendCrafted(std.testing.allocator, file, dup, encodeShun(&b2, 22, "second"), .shun, 0); var store = try LogStore.open(std.testing.allocator, file, 0); defer store.deinit(std.testing.allocator); try std.testing.expect(store.isShunned(dup)); const list = store.shunList(); try std.testing.expectEqual(@as(usize, 1), list.len); try std.testing.expectEqual(@as(i64, 11), list[0].epoch); try std.testing.expectEqualStrings("first", list[0].reason); } test "logstore §E/shun: encodeShun/parseShun round-trip, and parseShun rejects malformed payloads" { var buf: [max_shun_payload]u8 = undefined; const wire = encodeShun(&buf, -1, "a" ** max_shun_reason); const rec = try parseShun(wire); try std.testing.expectEqual(@as(i64, -1), rec.epoch); // epoch is signed and round-trips try std.testing.expectEqualStrings("a" ** max_shun_reason, rec.reason); const empty = encodeShun(&buf, 0, ""); const rec2 = try parseShun(empty); try std.testing.expectEqual(@as(i64, 0), rec2.epoch); try std.testing.expectEqual(@as(usize, 0), rec2.reason.len); var trailing: [11]u8 = [_]u8{0} ** 11; // epoch + reason_len=0 + one stray byte // (e) an over-cap reason that is FULLY BACKED: reason_len = max_shun_reason // + 1 with that many real bytes behind it, so every other guard in // `parseShun` is satisfied — the short-payload check (`payload.len - pos >= // rlen`) and the exact-consumption check both pass, and only the per-field // `rlen > max_shun_reason` cap can reject it. Deleting that one line leaves // every other shun test green (the frame-level `max_shun_payload` guard // catches the oversize-FRAME case, not this one), so without this case the // cap is untested and free to rot when `max_shun_payload` is widened. var backed_over_cap: [8 + 2 + max_shun_reason + 1]u8 = [_]u8{'a'} ** (8 + 2 + max_shun_reason + 1); std.mem.writeInt(i64, backed_over_cap[0..8], 5, .little); std.mem.writeInt(u16, backed_over_cap[8..10], @intCast(max_shun_reason + 1), .little); const cases = [_][]const u8{ &[_]u8{}, // (a) empty: no epoch &[_]u8{ 1, 2, 3, 4, 5, 6, 7, 8 }, // (b) epoch only, no reason_len &[_]u8{ 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 'a' }, // (c) reason_len 5 exceeds remaining &trailing, // (d) trailing byte after an empty reason &backed_over_cap, // (e) over-cap reason with the bytes to back it }; for (cases) |c| try std.testing.expectError(error.Corrupt, parseShun(c)); }