Artifact 576549f588124d55efb870df97d2578973027e34cd6a6342ecf669fbe7ce1aed:
- File
src/transport.zig
— part of check-in
[b4637e26df]
at
2026-07-31 21:51:35
on branch issue22-diagnostics
— sync: name the HTTP status the peer answered with
"the peer answered with an unexpected HTTP status" sends an investigation nowhere. 500 (the server could not build a reply), 413 (the request exceeded the body ceiling), 400 (a malformed container) and 417 (a rejected Expect: 100-continue) are four different bugs, and the message distinguished none of them. Issue #22 has been open on exactly that sentence.
transport.zig discarded the code at the status check because Zig errors carry no payload and the closed error set is deliberate. Rather than widen TransportError with a variant per status, the status is left on the Transport as last_status for the caller to read after the error, and syncRunFail names it: "the peer answered HTTP 500 (Internal Server Error)".
Immediately useful: re-running the issue #22 gate against the fossil corpus with this build turns its failure_reason from the old sentence into HTTP 500, which -- together with the reply-size arithmetic -- narrows the cause to OutOfMemory or ServerStorage and eliminates every 4xx path. The issue's own hypothesis, a per-round shun-snapshot readiness problem, is not among the candidates: that arm is documented unreachable and the code agrees.
Also corrects three comments Slice 2 falsified, all on the same read path:
logstore's open no longer CRC-verifies payloads, so the serve snapshot's contentImpl/wireImpl read faults are no longer "post-scan EIO or bitrot after a verified scan" -- ordinary stored corruption now reaches serve there, on the read. That is a better shape (one bad frame fails the round that touches it rather than making the server refuse to serve at all) but it makes that arm live traffic, and anything reasoning about how often it fires must start from that.
runPack and runShun both claimed their read-only open was an "integrity pre-flight" that aborted a corrupt source before a byte was written. It no longer verifies payloads. The guarantee still holds -- packLog pulls every artifact through srcContent, which propagates error.Corrupt rather than skipping the frame, and readFrameAt CRC-checks every read, so a rotted payload still aborts with the temp deleted and .zit untouched -- but it is now collected during the copy loop instead of at open. Both verbs RENAME OVER the original, so runPack's comment now states the consequence plainly: if srcContent is ever changed to answer null on a corrupt read, the rewrite would silently omit the damaged artifact and then delete the only copy of it.
Verified: fmt clean, test 2327/2327, gate, e2e 671/671. The new message checked against a stub server answering 500. (user: dmestas, size: 25249) [annotate] [blame] [check-ins using] [more...]
//! CLI-layer HTTP transport for the Fossil sync protocol (sync §4). //! //! This is the sole module in which `std.http` and real network I/O appear. //! `sync.zig` stays sans-IO: it composes request bytes (`Client.buildRequest`) //! and consumes reply bytes (`Client.consumeReply`) without ever holding a //! socket. This module carries those opaque byte strings over an HTTP POST — //! framing the request as the §4.1 Fossil zlib container, POSTing it as //! `application/x-fossil`, decoding the reply container, and honoring the §4 //! redirect and HTTPS-downgrade rules — so the deterministic client can be //! exercised over the wire without any IO leaking into its state machine. //! //! The §4 policy primitives (media-type table, redirect limit, `/xfer` strip, //! downgrade refusal, the zlib container codec) are pure functions already //! defined and clause-tested in `sync.zig`; this module is a thin, IO-bearing //! seam over them. Every value that crosses the wire is `[]const u8` — Fossil //! card streams legally embed NUL, so no sentinel-terminated type is used. const std = @import("std"); const sync = @import("zit").sync; const codec = @import("zit").container; const assert = std.debug.assert; const http = std.http; /// Closed error set at the transport boundary: the CLI can `switch` these /// exhaustively (no inferred `!`, no `anyerror`). Container-codec and /// redirect-policy causes are /// mapped from the pure `sync` error sets into transport-level names; the /// `Connection*`/`Protocol*` causes belong to the live POST seam. pub const TransportError = error{ /// The output buffer could not be grown. OutOfMemory, /// §4.1 — the request body exceeds the container's 4-byte length field. RequestTooLarge, /// §4.1 — a reply container shorter than its 4-byte length prefix. ReplyTruncated, /// §4.1 — the reply's zlib stream is malformed or its Adler-32 fails. ReplyMalformed, /// §4 r9 / §4.1 — the reply body or its declared length exceeds the /// 512 MiB cap (`sync.reply_body_cap`). ReplyTooLarge, /// §4 r10 — a twentieth redirect was reached for a single exchange. RedirectLimit, /// §4 r12 — a redirect would downgrade HTTPS to plaintext HTTP. InsecureRedirect, /// §4 r13 — a redirect appeared on an SSH or local-file transport. RedirectNotAllowed, /// A URL (initial target or redirect Location) did not parse. InvalidUrl, /// A URL named a scheme other than http/https. UnsupportedScheme, /// A 3xx redirect carried no Location header. MissingLocation, /// The server answered with a status that is neither 200 nor a redirect. ProtocolError, /// Any socket/TLS/HTTP-framing failure on the live exchange. ConnectionFailed, }; // --------------------------------------------------------------------------- // §4.1 — request framing / reply deframing (the Fossil zlib container) // --------------------------------------------------------------------------- /// §4.1 — wrap plaintext sync bytes in the Fossil zlib container (4-byte /// big-endian uncompressed length + an RFC1950 zlib stream) at the default /// compression level, ready to POST as `application/x-fossil`. A thin /// error-mapping wrapper over `codec.encode`. pub fn frame(gpa: std.mem.Allocator, plaintext: []const u8) TransportError![]u8 { const container = codec.encode(gpa, plaintext, .default) catch |err| switch (err) { error.OutOfMemory => return error.OutOfMemory, error.InputTooLarge => return error.RequestTooLarge, }; // §4.1 — every container opens with the 4-byte big-endian length prefix, so // a framed body is never shorter than that header. assert(container.len >= 4); return container; } /// §4.1 (as corrected by the §4.1 compression-length erratum) — decode a /// Fossil zlib container back to plaintext. The 4-byte length field is a /// capacity bound, not an exact length: a stream that decodes within that /// capacity is accepted and the *actual* decoded bytes are returned, which /// may be fewer than declared (the overdeclared-sender case the reference /// receiver tolerates). `sync.reply_body_cap` caps the declared length before /// any decompression, so a lying header cannot drive unbounded allocation. pub fn unframe(gpa: std.mem.Allocator, container: []const u8) TransportError![]u8 { const plaintext = codec.decode(gpa, container, sync.reply_body_cap) catch |err| switch (err) { error.OutOfMemory => return error.OutOfMemory, error.TruncatedHeader => return error.ReplyTruncated, error.InvalidStream => return error.ReplyMalformed, error.OutputTooLarge => return error.ReplyTooLarge, }; // §4 r9 / §4.1 — the decode is capacity-capped at `reply_body_cap`, so an // accepted reply's plaintext never exceeds the ceiling (`OutputTooLarge` // rejected an over-cap declared length before any decompression). assert(plaintext.len <= sync.reply_body_cap); return plaintext; } // --------------------------------------------------------------------------- // §4 r10/r12/r13 — redirect policy (pure; the tested core of B1) // --------------------------------------------------------------------------- /// The base URL and transport a redirect resolves to, after §4 processing. /// `base_url` is a subslice of the Location passed to `resolveRedirect` (the /// `/xfer`-stripped prefix); it borrows that memory and does not own it. pub const RedirectStep = struct { transport: sync.Transport, base_url: []const u8, }; /// §4 r10/r12/r13 (redirect erratum) — resolve one redirect hop. `from` is the /// transport of the request that received the 3xx; `location` is its Location /// header; `followed` is the number of redirects already consumed by this /// exchange. Order: enforce the redirect limit (r10), reject a redirect on a /// non-HTTP transport (r13), strip any trailing `/xfer` from the Location /// (redirect erratum — the origin URL's path is never consulted), reparse the /// scheme, then refuse an HTTPS->HTTP downgrade (r12). Pure and total over its /// inputs; the returned `base_url` aliases `location`. pub fn resolveRedirect( from: sync.Transport, location: []const u8, followed: usize, ) TransportError!RedirectStep { // §4 r10 — the twentieth redirect (followed == max_redirects) aborts. sync.redirectAllowed(followed) catch return error.RedirectLimit; // §4 r13 — SSH and local-file transports never follow a redirect. if (!sync.redirectAllowedForTransport(from)) return error.RedirectNotAllowed; // Redirect erratum — strip trailing "/xfer" (repeatably) to recover the // base repository URL; no other path transformation. const base = sync.stripRedirectXfer(location); assert(base.len <= location.len); const to = try schemeTransport(base); // §4 r12 — a redirect MUST NOT drop TLS. sync.checkRedirectDowngrade(from, to) catch return error.InsecureRedirect; // Negative space: the resolved transport is always one of the two that // carry redirects; SSH/file were rejected above and no other scheme // survives `schemeTransport`. assert(to == .http or to == .https); return .{ .transport = to, .base_url = base }; } /// Map a URL's scheme to its `sync.Transport`. Only http/https are carried by /// this module; every other scheme (or an unparseable URL) is rejected. fn schemeTransport(url: []const u8) TransportError!sync.Transport { const uri = std.Uri.parse(url) catch return error.InvalidUrl; if (std.ascii.eqlIgnoreCase(uri.scheme, "https")) return .https; if (std.ascii.eqlIgnoreCase(uri.scheme, "http")) return .http; return error.UnsupportedScheme; } /// §4 — the four redirect statuses an HTTP exchange follows. fn isRedirectStatus(status: http.Status) bool { return switch (status) { .moved_permanently, .found, .temporary_redirect, .permanent_redirect => true, else => false, }; } /// §4.1 — whether a reply body carries an outer zlib container. `application/ /// x-fossil` replies do; the `application/x-fossil-uncompressed` clone-reply /// media type does not. An absent or unrecognized Content-Type is treated as /// the compressed default, matching §4.1's "used for all requests and for /// replies to pull, push, and sync operations." fn replyOuterCompressed(content_type: ?[]const u8) bool { const ct = content_type orelse return true; const media = sync.parseMediaType(ct) orelse return true; return sync.media_props.get(media).outer_compressed; } // --------------------------------------------------------------------------- // The live HTTP seam (§4) — the only IO in the module. // --------------------------------------------------------------------------- /// A connection to one sync peer. Concrete: this is the only transport today, /// so it carries no vtable and no speculative abstraction. The open / /// exchange / close shape means a future `union(enum){ http, pipe, local }` /// for SSH and local-file transports is a mechanical wrap touching zero call /// sites: the round-loop driver (plan B2) calls only `exchange`. /// /// `exchange` is the load-bearing boundary — it takes the UNCOMPRESSED /// protocol request bytes `sync.Client.buildRequest` emits and returns the /// UNCOMPRESSED reply bytes `sync.Client.consumeReply` wants, hiding entirely /// inside: the §4.1 zlib container framing, the `application/x-fossil` POST, /// §4 redirect following (with `/xfer` stripping and HTTPS-downgrade refusal), /// the §4 r8/r9 size cap, and cookie carry. Callers never see compression or /// HTTP. pub const Transport = struct { /// Closed error set at the boundary (namespaced alias of `TransportError`). pub const Error = TransportError; client: http.Client, /// The current base URL, owned; a followed redirect rewrites it in place so /// the next round targets the resolved endpoint (§4 redirect semantics). base_url: []u8, /// http or https — the transport scheme of `base_url`, tracked so a /// redirect's HTTPS->HTTP downgrade can be refused (§4 r12). scheme: sync.Transport, /// §6.5 — the preformed `x-f-l-c` cookie-login value to attach as /// `Cookie: x-f-l-c=<value>`, or null. The §6.5 eligibility gate and the /// login-card encoding (which depend on the peer's recorded version and on /// the body the nonce covers) are the sync layer's responsibility; the /// round-loop driver sets this per round when the gate selects the cookie /// transport. The §3.6/§4.3 *state* cookie is a body card threaded by /// `sync.Client` itself, carried opaquely in the exchange bytes — transport /// takes no action for it. login_cookie: ?[]const u8 = null, /// The status that produced the most recent `error.ProtocolError`, so the /// CLI can NAME it. Zig errors carry no payload and the closed error set is /// deliberate, so the code is left here for the caller to read after the /// error rather than widened into a variant per status. /// /// This exists because "the peer answered with an unexpected HTTP status" /// is nearly useless in a bug report: 500 (the server failed to build a /// reply), 413 (the request exceeded the body ceiling), 400 (malformed /// container) and 417 (a rejected Expect: 100-continue) are four different /// investigations, and the message named none of them. Only ever written /// immediately before returning `ProtocolError`; a stale value is /// unreadable because the caller only consults it on that error. last_status: ?http.Status = null, /// Resolve `url` and prepare an HTTP client. The URL scheme MUST be http or /// https; SSH and local-file peers are a later transport variant. pub fn open(io: std.Io, gpa: std.mem.Allocator, url: []const u8) Error!Transport { const scheme = try schemeTransport(url); const base = gpa.dupe(u8, url) catch return error.OutOfMemory; return .{ .client = .{ .allocator = gpa, .io = io }, .base_url = base, .scheme = scheme, }; } /// Release the owned base URL and tear down the HTTP client. pub fn close(self: *Transport) void { self.client.allocator.free(self.base_url); self.client.deinit(); } /// §4 — carry one sync round: frame `request` (§4.1), POST it as /// `application/x-fossil`, follow §4 redirects (`redirect_behavior = /// .unhandled` so `resolveRedirect` — not std.http's generic handling — /// governs `/xfer` stripping, the 19-redirect limit, and downgrade /// refusal), bound the reply by `sync.reply_body_cap` (§4 r8/r9), and /// deframe it. Returns the decoded plaintext reply (caller owns the slice). /// /// Exercised end-to-end against `zit serve` later (plan B4); the framing /// and redirect logic it composes are unit-tested directly below. pub fn exchange(self: *Transport, gpa: std.mem.Allocator, request: []const u8) Error![]u8 { const a = self.client.allocator; const framed = try frame(gpa, request); defer gpa.free(framed); var cookie_value: ?[]u8 = null; defer if (cookie_value) |cv| gpa.free(cv); if (self.login_cookie) |lc| { cookie_value = std.fmt.allocPrint(gpa, "x-f-l-c={s}", .{lc}) catch return error.OutOfMemory; } // §4 r10 — at most max_redirects hops, then one terminal response. The // 20th redirect is rejected by `resolveRedirect`; this loop bound is // the structural backstop. var followed: usize = 0; while (followed <= sync.max_redirects) : (followed += 1) { const uri = std.Uri.parse(self.base_url) catch return error.InvalidUrl; const cookie_headers: []const http.Header = if (cookie_value) |cv| &.{.{ .name = "Cookie", .value = cv }} else &.{}; var req = self.client.request(.POST, uri, .{ .redirect_behavior = .unhandled, .keep_alive = false, .headers = .{ .content_type = .{ .override = sync.media_props.get(.x_fossil).wire }, .accept_encoding = .omit, }, .extra_headers = cookie_headers, }) catch return error.ConnectionFailed; defer req.deinit(); req.transfer_encoding = .{ .content_length = framed.len }; var body = req.sendBodyUnflushed(&.{}) catch return error.ConnectionFailed; body.writer.writeAll(framed) catch return error.ConnectionFailed; body.end() catch return error.ConnectionFailed; req.connection.?.flush() catch return error.ConnectionFailed; var response = req.receiveHead(&.{}) catch return error.ConnectionFailed; const status = response.head.status; if (isRedirectStatus(status)) { const location = response.head.location orelse return error.MissingLocation; const step = try resolveRedirect(self.scheme, location, followed); const next = a.dupe(u8, step.base_url) catch return error.OutOfMemory; a.free(self.base_url); self.base_url = next; self.scheme = step.transport; // §4 r12/r13 — a followed redirect only ever lands on an // HTTP(S) transport; `resolveRedirect` rejected every other // scheme and any downgrade, so the next round targets a // carriable endpoint. assert(self.scheme == .http or self.scheme == .https); continue; } if (status != .ok) { self.last_status = status; return error.ProtocolError; } // §4.1 — decide the reply's outer framing from the Content-Type // BEFORE initializing the body stream. `response.reader` reuses the // connection buffer that backs `head.content_type` (std.http: // "Pointers in this struct are invalidated when the response body // stream is initialized"), so the slice must be read first — an // `application/x-fossil-uncompressed` clone reply (§8.1) would // otherwise be misread as the compressed default and double-decoded. const outer_compressed = replyOuterCompressed(response.head.content_type); var transfer_buf: [4096]u8 = undefined; const reader = response.reader(&transfer_buf); const raw = reader.allocRemaining(gpa, .limited(sync.reply_body_cap)) catch |err| switch (err) { error.OutOfMemory => return error.OutOfMemory, error.StreamTooLong => return error.ReplyTooLarge, // §4 r9 error.ReadFailed => return error.ConnectionFailed, // socket/framing fault }; // §4 r9 — explicit cap citation; `.limited` already enforced it. sync.replyCapCheck(raw.len) catch { gpa.free(raw); return error.ReplyTooLarge; }; // §4 r9 — both returns below are within the ceiling: the raw clone // body passed `replyCapCheck` just above, and `unframe`'s decode is // itself capacity-capped at `reply_body_cap`. assert(raw.len <= sync.reply_body_cap); if (!outer_compressed) return raw; // §4.1 clone reply (x-fossil-uncompressed) defer gpa.free(raw); return try unframe(gpa, raw); } return error.RedirectLimit; // §4 r10 structural backstop } }; // =========================================================================== // Tests — expected values come from sync §4/§4.1 and the two §4 errata, never // from running fossil; the RFC text is the sole oracle. // =========================================================================== const testing = std.testing; test "transport §4.1: frame then unframe round-trips the plaintext" { const gpa = testing.allocator; const cases = [_][]const u8{ "", "hello", "pragma client-version 2 22700 20250727 110500\n", &[_]u8{ 0x00, 0x0a, 0x00, 0xff, 'x', 0x00 }, // NUL/LF-bearing wire data }; for (cases) |plaintext| { const container = try frame(gpa, plaintext); defer gpa.free(container); const decoded = try unframe(gpa, container); defer gpa.free(decoded); try testing.expectEqualSlices(u8, plaintext, decoded); } } test "transport §4.1: a container compressed at a different level still decodes (level-independent)" { // §4.1 L1204-1208 — "implementations MUST NOT rely on a particular // compression level ... the decompressor is level-independent." `frame` // encodes at the default level; a level-9 container must decode identically. const gpa = testing.allocator; const plaintext = "the quick brown fox jumps over the lazy dog, repeatedly, for entropy."; const level9 = try codec.encode(gpa, plaintext, .best); defer gpa.free(level9); const decoded = try unframe(gpa, level9); defer gpa.free(decoded); try testing.expectEqualSlices(u8, plaintext, decoded); } test "transport §4.1 errata: an overdeclared length field is tolerated (capacity, not exact length)" { // §4.1 compression-length erratum — the 4-byte field is an allocation // capacity, not the exact decoded length; a decode that finishes within // capacity is accepted even when the declared length exceeds it. Build a // conformant container, then overdeclare its length prefix. const gpa = testing.allocator; const plaintext = "capacity-bound decode"; const container = try frame(gpa, plaintext); defer gpa.free(container); std.mem.writeInt(u32, container[0..4], @intCast(plaintext.len + 500), .big); const decoded = try unframe(gpa, container); defer gpa.free(decoded); // The actual decoded bytes are returned — fewer than the declared field. try testing.expectEqualSlices(u8, plaintext, decoded); } test "transport §4.1: a truncated container header is rejected cleanly" { const gpa = testing.allocator; try testing.expectError(error.ReplyTruncated, unframe(gpa, &[_]u8{ 0x00, 0x00 })); } test "transport §4.1: a malformed zlib stream is rejected cleanly" { // §4.1 — "A receiver MUST reject the envelope when the zlib stream is // malformed." A valid 4-byte length prefix over a non-zlib body (all-ones // bytes: BFINAL=1, BTYPE=11 is the reserved/invalid DEFLATE block type). const gpa = testing.allocator; var envelope: [16]u8 = undefined; std.mem.writeInt(u32, envelope[0..4], 8, .big); @memset(envelope[4..], 0xff); try testing.expectError(error.ReplyMalformed, unframe(gpa, &envelope)); } test "transport §4 redirect erratum: a trailing /xfer is stripped to recover the base URL" { // Redirect erratum — strip trailing "/xfer" (repeatably); the origin URL's // path is never consulted. const one = try resolveRedirect(.https, "https://h.example/repo/xfer", 0); try testing.expectEqual(sync.Transport.https, one.transport); try testing.expectEqualStrings("https://h.example/repo", one.base_url); const two = try resolveRedirect(.http, "http://h.example/proj/xfer/xfer", 0); try testing.expectEqualStrings("http://h.example/proj", two.base_url); } test "transport §4 redirect erratum: a Location without /xfer is used verbatim" { // Redirect erratum divergence row — origin path is never consulted; a // Location that does not end in /xfer passes through unchanged. const step = try resolveRedirect(.http, "http://h.example/mirror/proj", 0); try testing.expectEqualStrings("http://h.example/mirror/proj", step.base_url); } test "transport §4 r10: the redirect limit is reached on the twentieth hop" { // §4 r10 — an exchange follows at most 19 redirects; the 20th aborts. Drive // a self-referential redirect and count hops to the fatal limit. const loc = "https://loop.example/repo/xfer"; var followed: usize = 0; while (true) { const step = resolveRedirect(.https, loc, followed) catch |err| { try testing.expectEqual(TransportError.RedirectLimit, err); break; }; try testing.expectEqualStrings("https://loop.example/repo", step.base_url); followed += 1; try testing.expect(followed <= sync.max_redirects); } try testing.expectEqual(sync.max_redirects, followed); } test "transport §4 r12: an https->http redirect is refused" { try testing.expectError(error.InsecureRedirect, resolveRedirect(.https, "http://h.example/repo", 0)); // The safe directions are permitted. _ = try resolveRedirect(.https, "https://h.example/repo", 0); _ = try resolveRedirect(.http, "http://h.example/repo", 0); _ = try resolveRedirect(.http, "https://h.example/repo", 0); // http->https upgrade is fine } test "transport §4 r13: a redirect on an SSH or file transport is rejected" { try testing.expectError(error.RedirectNotAllowed, resolveRedirect(.ssh, "https://h.example/repo", 0)); try testing.expectError(error.RedirectNotAllowed, resolveRedirect(.file, "https://h.example/repo", 0)); } test "transport §4: a redirect to a non-http(s) scheme is rejected" { try testing.expectError(error.UnsupportedScheme, resolveRedirect(.http, "ftp://h.example/repo", 0)); } test "transport §4.1 fuzz: unframe on arbitrary bytes stays bounded and never crashes" { // Deterministic seeded fuzz: decode must terminate with a clean // TransportError or a bounded slice for any input, never UB or a hang. const gpa = testing.allocator; var prng = std.Random.DefaultPrng.init(0x5eed_7a17); const rand = prng.random(); var buf: [512]u8 = undefined; var i: usize = 0; while (i < 2000) : (i += 1) { const len = rand.uintLessThan(usize, buf.len + 1); const slice = buf[0..len]; rand.bytes(slice); if (unframe(gpa, slice)) |decoded| { gpa.free(decoded); // a random stream that happens to decode: bounded output } else |_| { // Any TransportError is acceptable; the point is termination // without UB or an unbounded loop, not a specific rejection cause. } } } test "transport §4.1: frame and unframe survive every allocation-failure point" { // Every public allocating function carries an allocation-failure test. const Ctx = struct { fn run(a: std.mem.Allocator, plaintext: []const u8) !void { const container = try frame(a, plaintext); defer a.free(container); const decoded = try unframe(a, container); defer a.free(decoded); try testing.expectEqualSlices(u8, plaintext, decoded); } }; try testing.checkAllAllocationFailures(testing.allocator, Ctx.run, .{"allocation-failure sweep payload"}); }