Timeline

* fix(sync): answer a client clone_seqno with bad command; make Clone's phantom pull opt-in

Two deliberate #255 deviations from sync-protocol-01 move to conformance.

SP-5.2-6: the server now answers a client-sent clone_seqno with "error bad command" and processes the cards after it, as Section 5.2 requires for a reply-only card and as fossil's page_xfer does.

SP-8.2-5: Clone now sends nothing after clone_seqno 0 and its cleanup round, so by default it leaves the phantoms a clone produces in place, as fossil(1)'s clone does. The follow-up pull that fetched them is now opt-in through CloneOpts.PullPhantoms, whose doc comment says enabling it departs from Section 8.2.

Both cases leave the sync deviations map. Tracked in #424.

* fix(sync): quote a client clone_seqno as received in bad command; keep DST coverage of the opt-in pull

Tracked in #424. check-in: d3fb5397c8 user: [email protected] tags: main, trunk

Check the remaining partial and uncovered sync rows, and fix the client private-gimme answer, the unversioned-file NAME encoding, the check-in lock table, the uv-hash name form and the uvgimme round limit they exposed. Tracked in #392. check-in: 161d6ab68f user: [email protected] tags: main, trunk

sync-protocol-01 Section 5.1 (erratum s5.1-truncated-payload-receiver) makes a file, cfile or uvfile payload that the body ends inside a validation failure of the card's handler, never a stored artifact. The decoder failed the whole message instead, so a server answered HTTP 400 and a client dropped every card of the reply.

The decoder now yields such a card as a BadFileCard carrying the handler's malformed-line text and stops, as it already does for a line it cannot frame. The server answers 200 with that error and keeps the cards before it; the client keeps the prefix it processed. fossil 2.28 stores the bytes present; -01 is followed here. check-in: 648f5d4479 user: [email protected] tags: main, trunk

fossil's client moves its login card from the body into the x-f-l-c cookie once the server's server-version passes xfer_xflc_check (2.27, 2025-07-27 11:05 or later), and fossil's check_login, when a stored password is not 40 characters, tries the signature again against the shared secret that cleartext password derives (sections 6.4, 6.5). The client always logged in in the body, and the server checked only the stored value.

A request marks itself LoginCookieOK once the gate passes; Message.SplitLoginCookie moves its login card out of the body, and the HTTP transports (sync.HTTPTransport and the public NewHTTPTransport through the adapter) send it as the cookie, on every redirect too. The nonce is unchanged: it already covers what follows the login line, which in cookie form is the whole body. Transports that cannot set a cookie send the login in the body as before. The server side of the cookie landed with the server-version exchange.

HandleOpts gains LoginCookie, so an embedder serving HandleSyncWithOpts over its own HTTP handler can pass the cookie through; XferHandler and ServeHTTP read it themselves. check-in: 9d35c01fb3 user: [email protected] tags: main, trunk

fossil's xfer_accept_unversioned_file stores a received file under the HASH its card gave, takes only the mtime when the content is the same and newer (status 2), and stores an omitted payload as NULL with size 0; send_unversioned_file sends nothing for a name it does not hold; and the client seeds uv_tosend from rows whose hash is not NULL (sections 7.3, 8.5.1). Both sides stored a received file under a hash computed by local policy and dropped status 2 and omitted-content cards; the server answered an absent name with a tombstone; the client queued its tombstones.

uv.Receive now owns the acceptance rule, which the server and the client share instead of each re-deriving it from uv.Read, uv.Status and uv.Write. check-in: d33a37fffd user: [email protected] tags: main, trunk

fossil's send_compressed_file sends a cfile as the blob table stores it and pads a payload that does not end in LF; page_xfer sends a version 3 clone's reply as application/x-fossil-uncompressed, sends the unversioned catalog with a clone when uv-sync is on, and refuses a clone with a reset, the push card and "error not authorized to clone". The server recompressed each cfile, never padded, compressed the reply, sent no catalog, appended its refusal to the reply, and restarted a clone whose SEQNO was negative.

CFileCard's StoredBlob, set by the clone path from the blob table, is now what the encoder sends; xfer.Message.Uncompressed tells the HTTP server to send a reply as card text. The catalog goes to a requester with Read, as for uv-hash: fossil lets Clone imply Read there, which this server does not. A negative SEQNO, which section 8.1 exempts from its fatal, now ends the clone with clone_seqno 0, and the unit test that expected a restart expects that. check-in: 6c69fc1231 user: [email protected] tags: main, trunk

fossil's page_xfer answers a client-version that carries DATE and TIME with pragma server-version, and both sides record the peer's release; send_file, send_compressed_file and send_unversioned_file then append "error Fossil version 2.0 or later required." in place of SHA3-named content for a peer older than 2.0, with a server treating a client that announced no release as one (sections 7.6, 9.1). The server sent no server-version and neither side kept or gated on the peer's release.

The server gates gimme, clone and uvgimme; the client gates its answer to a gimme. Because a server-version from a current release turns fossil's client to logging in through the x-f-l-c cookie (section 6.5), the HTTP server now reads that cookie: xfer.DecodeRequest puts its login card first with the nonce bound to the whole body, as page_xfer does. Without it a real fossil client's later rounds arrived as nobody. That moves sync.6.5.server-cookie-login, marked for #304, here. check-in: 6963b0a9dc user: [email protected] tags: main, trunk

The server refused application/x-fossil-debug with HTTP 415, where fossil dispatches it to sync like the other x-fossil types. The client read any 2xx as a sync reply and decoded a reply as compressed only when typed exactly application/x-fossil, so a reply with no Content-Type was read as plain cards and a proxy's HTML page was parsed as cards.

Replies now go through xfer.DecodeReply, which applies section 4's reply rule by case-insensitive prefix: no type is the compressed container, an x-fossil-debug or x-fossil-uncompressed prefix is card text, another x-fossil prefix is compressed, and anything else is refused. Only status 200 is a reply. check-in: 47dc6363c0 user: [email protected] tags: main, trunk

The card decoder refused a card by failing the entire message: the server answered HTTP 400 where §5.2 has it emit "bad command" and parse on, and the client dropped a whole reply including its valid prefix.

The decoder now yields a bad-command card for a line no rule accepts (an unknown operator, a bare pragma, a gimme with a surplus token, a config with a signed size) and a bad-file card for a file/cfile/uvfile line it cannot frame. The server resets its reply, emits the matching error (bad command, or the handler's "malformed file line" and the like) and either parses on (bad command) or stops (bad file), as fossil's page_xfer does; the client ends the reply-card loop, keeping the cards decoded before the bad one; a clone treats it as a non-retryable protocol error.

Tokens are split on ASCII whitespace only, within fossil's six-token window (blob_tokenize over aToken[6]): a no-break space no longer separates tokens, and a seventh token is ignored rather than rejected, so igot and private tolerate a surplus token. A full cfile's USIZE is carried through as metadata, not checked against the inflated size (the hash binds the content instead), and the byte after a config payload is consumed whatever it is, both matching fossil's xfer_accept_compressed_file and page_xfer.

This also resolves the bad-command cases tracked in #300 (sync.5.2.bad-command, .bad-command-resets, .unknown-prefix, sync.10.1.bad-command-keeps-mutations), whose marks are removed here. check-in: ddc4b5396d user: [email protected] tags: main, trunk

* fix(xfer): reject non-decimal card sizes and bound payload memory

A payload card's size was parsed with strconv.Atoi, so a peer sending `file HASH -1` reached readPayload's negative-size assertion and panicked the client or server decoding the message, and a signed size such as `+5` was accepted. Sizes now go through parseSize, which follows the sync draft's `decimal = 1*DIGIT` grammar and fossil's 32-bit limit and returns a decoding error for anything else. uvfile FLAGS use the same parser, so negative FLAGS are rejected as the draft requires. The assertion stays, now unreachable from peer input.

readPayload no longer allocates the declared size up front: its buffer grows with the bytes read, so a large size over a short message fails after allocating about what was sent.

A cfile payload may inflate only to its USIZE (a margin above it for a delta) and not past its 4-byte size prefix, and all cfile payloads in one message share a MaxCFileInflateBytes (1 GiB) budget, so a few megabytes of zlib cannot inflate to tens of gigabytes.

FuzzDecode exercises the decoder over both framings.

* fix(sync): refuse malformed xrow payloads and cap HTTP reply bodies

An xrow payload or xdelete PKData from a peer was decoded into a map and used as is. JSON null gave a nil map that reached coerceJSONNumbers' assertion, and a primary-key value of the wrong type or a nested value reached normalizeValue's, panicking the client or server. Both directions now decode through decodeRowJSON: one JSON object, nothing after it, and repo.ValidateRow's rules, which require scalar values and primary-key values that PKHash can normalize. A client refuses an xrow for a table it has not registered. Column types match case-insensitively, as table creation accepts them.

Both HTTP transports read replies through xfer.ReadBody, which stops at MaxBodyBytes (MaxDecompressedBytes plus framing headroom) and returns ErrBodyTooLarge, untagged for retry, instead of reading an endless body into memory.

FuzzTableSyncPayload drives the server handler and client over arbitrary payloads.

* fix(sync): check synced-table rows and identifiers before building SQL

repo.UpsertXRow wrote a row's map keys into its INSERT as column names, and the rows of an xrow payload or xdelete PKData come from the peer, so a key such as "val, mtime) VALUES(...) --" chose the statement's columns and values, and a key could select from other tables into a synced one. This held on the server for pushed rows and tombstones, and on the client for a server's rows.

A peer's row must now use exactly the table's declared columns: repo.ValidateRow refuses any other key, which also keeps a peer from setting _owner. UpsertXRow no longer trusts its caller: it reads the table's registered schema and refuses any key it does not declare, apart from _owner on an ownership-mode table. ListXRows, LookupXRowOwner and DeleteXRowByPKHash validate the table name and the column names they interpolate, so no synced-table identifier reaches SQL text unchecked.

* fix(sync): answer reqconfig only with fossil's transferable config

The sync server answered reqconfig NAME with any row of the config table, for any requester, including one with no capabilities, so project-code, server-code and every local setting could be read by anyone who could reach the sync endpoint.

handleReqConfig now follows fossil's page_xfer and configure.c. It answers only a /GROUP name, only for a requester with Read, matches the group by prefix in fossil's table order, and sends only the names fossil transfers in that group, in fossil's encoding. The user and subscriber tables also need Admin, which counts only for a verified login, and the concealed table needs RdAddr. A refused request gets no cards and no error.

The groups, their names and their capability rules live in one place, config_groups.go. A differential test answers the same requests with fossil's own server and with HandleSync on the same repository and requires identical config cards.

* test(sync): keep fossil http from leaving a backoffice process

* fix(sync): check the login nonce against the request body

The server verified a login card's signature over its nonce but never checked that the nonce was the SHA-1 of the request body after the login line, as fossil's check_tail_hash does (draft-fossil-sync-protocol-00 §6.2). A login card captured from one request therefore authenticated any other request body.

The decoder now records, for each login card, the SHA-1 of the decompressed bytes after the card's line, and VerifyLogin refuses a card whose nonce does not match it. A login card that was never decoded has no tail and is refused.

Login handling otherwise follows fossil's page_xfer and check_login: - a refused login ends the exchange with "error login failed" instead of continuing as nobody; - a second login card is refused with "multiple login cards"; - a successful login adds the user's capabilities to nobody's instead of replacing them; - nobody and anonymous log in without a password, and the reserved developer and reader logins and accounts with an empty password cannot be logged into.

Fixes GHSA-2g5p-q5f3-477p.

* fix(repo): seed nobody with fossil's default capabilities

A new repository seeded the nobody user with "cghijknorswy", so every request without a login card could push check-ins and unversioned files, write wiki pages and tickets, and held Setup. Seed the four built-in users `fossil new` creates, with fossil's capabilities: nobody "gjorz", anonymous "hz", developer "ei" and reader "kptw".

Setup and Admin now include check-in, read and clone, as they do in fossil, so the creating user can sync once given a password. A login to an account with no stored password is refused, as fossil's check_login refuses it; the creator starts with none, and the signature would otherwise be the SHA-1 of the nonce alone.

Tests that push without credentials grant nobody push explicitly. Existing repositories keep their stored capabilities.

* test(auth): reserved-login test updates the seeded developer and reader rows

New repositories now seed developer and reader as fossil does, so the test gives the existing rows a password instead of inserting them.

* docs(changelog): group the sync hardening fixes under Security

* test(sync): reqconfig tests sign their logins over the request body

The login nonce now has to cover the request body, so the reqconfig tests build their admin requests with signedRequest instead of a bare login card. check-in: 3a6126c3e9 user: [email protected] tags: trunk

* fix(blob): always inflate blob.content on load (#235)

blob.Load decided whether a row was stored raw by comparing len(content) with blob.size. That guess is unsound for delta rows: blob.size is the expanded artifact's length, while content is the compressed delta, and the two can be equal by coincidence. When they are, Load handed back the still-compressed bytes, and the delta applier failed with "expected integer at pos 0".

Go 1.27 replaced compress/flate's encoder, so the compressed size of any given delta shifted. The fixture in TestExpand_SingleDelta and TestIsAvailable_GroundedChain now compresses to exactly the target size (62 bytes), which exposed the bug. Any real repository could hit it under either Go version.

Fossil itself never makes this guess: content_of_blob (src/content.c:218-230) inflates blob.content unconditionally, and every writer here stores the compressed form. Load now does the same. TestLoadInflatesWhenStoredLengthEqualsSize forces the coincidence directly so it fails on the old code under any Go version.

Separately, TestDecodeRejectsOversizeDeclaredLength built its container from a 64-byte payload. Go 1.27's flate skips match-finding for inputs under 128 bytes, so that payload came out as a stored block larger than its input and tripped the helper's did-it-compress guard. The test now uses 1 KiB. The property it asserts is unchanged.

* docs(changelog): note the blob.Load inflate fix (#235) check-in: 60ef41b5b1 user: [email protected] tags: trunk

* fix: make authenticated sync interoperate with canonical fossil (#203)

Every credentialed sync against a real fossil server was answered "login failed", whether or not the password was right. Push always needs a login, so push was unusable against any real server, as was pull from anything that did not grant anonymous read.

The login algorithm was never the problem. Two wire-level defects were, and each on its own produces that identical, password-independent rejection.

The push and pull cards carried their two codes in the wrong order. The canonical form is `push SERVERCODE PROJECTCODE` — canonical's own client writes `blob_appendf(&send, "push %s %s\n", zSCode, zPCode)` and both ends read the project code out of the second argument (src/xfer.c). We emitted and parsed project code first. A clone therefore adopted the server's server-code as its project code, and since a login signature is salted with the project code, every signature was derived from the wrong value. The same inversion put the server code where a server validates the project code, so an authenticated push was refused with "wrong project" even once the login worked.

The nonce covered a random comment that was never transmitted. The nonce is the hash of every byte following the login card, and the server re-hashes those bytes to check it (src/xfer.c check_tail_hash) — so the comment supplying the randomness has to be on the wire, not merely in the hash. We appended it to the hashed payload only, producing a nonce the server could not reproduce; it rejected the login before comparing the password at all, which is why a wrong password looked exactly like a right one. Comment lines now have a card type, so they can be sent as well as hashed.

Two smaller faults on the same path: a clone treated the server's round-0 "not authorized to clone" as fatal, ending the session before credentials were ever offered, where canonical tolerates that error on the first cycle precisely so the client can retry with the login card it can only build once it knows the project code; and the sync path signed with SyncOpts.ProjectCode rather than the code it had already resolved from the repository, so a caller that let the repo supply its project code — as the push and pull cards allow — panicked instead of authenticating.

The password derivation itself, sha1("<project-code>/<login>/<password>"), was correct and is unchanged. It is now pinned two ways: against the hash the fossil binary actually stores in user.pw, and against a login card captured from fossil 2.28 on the wire. Canonical's rule that an already hashed 40-hex password is used as-is is now honoured too.

Regression coverage stands up a canonical fossil server with anonymous capabilities removed, so a login card is the only way in, and asserts that a credentialed clone transfers content, a wrong password is refused, and a local commit pushed under authentication lands in the server's blob table and on its timeline. Nothing in the suite authenticated against real fossil before; clone was only ever exercised against servers granting anonymous read, where the login card is never sent, which is how this shipped.

Four xfer codec expectations that pinned the inverted argument order are corrected to the canonical bytes.

* fix(sync): correct the nonce doc to match what the server recomputes

The comment described the nonce as covering the encoded non-login cards and the signature as using the stored password hash. Both readings are wrong in ways that matter to anyone changing this code: the nonce is defined over the bytes that actually follow the login card on the wire, because the server recomputes it from what it received (src/xfer.c check_tail_hash), and the signature is salted with the project code rather than derived from user.pw directly. Hashing bytes the message does not carry produces a nonce the peer cannot reproduce, which is exactly the password-independent rejection this fix was chasing. check-in: a745f4f2ac user: [email protected] tags: trunk

encodeCFile emitted the cfile payload as bare zlib. Fossil's on-disk cfile format, produced by blob_compress() in src/blob.c, is a 4-byte big-endian uncompressed-size prefix followed by the zlib stream, and fossil's receiver (content_put_ex) stores the wire payload verbatim into blob.content, later reading it back through blob_uncompress(), which reads that prefix to size its output before inflating. Without the prefix a real fossil client that clones from us stores every full (non-delta) artifact unreadable: fossil clone exits 0 but rebuild lands zero check-ins and manifests decode to garbage.

Prepend the 4-byte prefix in encodeCFile and count it in CSIZE, so the payload matches blob_compress byte-for-byte. The prefix value is len(Content) (the bytes being compressed -- a delta payload for delta cards), which stays distinct from the USIZE header field (the full reconstructed size). Our internal storage already used fossil's framing (blob.Compress), and the decoder already accepted both framings and rebuilt StoredBlob, so this is a wire-encode-only fix; receive and on-disk paths are unchanged.

TestCloneRealFossilWithDeltaChain now asserts six materialized check-ins instead of skipping against this issue, and a new TestCloneRealFossilFullContent drives a delta-free corpus through a real fossil clone+rebuild, asserting the check-in count and that a cloned manifest decodes to parseable manifest cards. check-in: 00e69bc9e0 user: [email protected] tags: trunk

* fix: select xfer framing by Content-Type (#106); bound clone rounds by the client's decode limit (#109)

seeing which parsed, which made a decompression failure indistinguishable from "wrong framing" (the root of #104). Select framing from the §4 Content-Type instead: application/x-fossil is the §4.1 compressed container, every other type (including application/x-fossil-uncompressed, used by clone-v3 replies) is plain card text — the same rule canonical fossil applies. Verified against fossil 2.28: pull replies arrive as the prefixed container, clone replies as plain text, and no reply is ever an unprefixed raw zlib stream. The former unprefixed-zlib framing, checked first on the unverified belief that fossil emits it, was removed along with the length-prefix/zlib-header aliasing guard that only existed to disambiguate the two compressed framings trial-dispatch conflated. A compressed body that fails to inflate is now reported as the fault it is, never re-fed to the card parser. xfer.Decode takes the Content-Type; byte-oriented callers with no header pass the compressed type explicitly.

crossing one whole, so a round carried budget + one whole artifact. When sub-budget filler preceded a large artifact, that sum exceeded what the client could decode (xfer.MaxDecompressedBytes) and the clone failed with a spurious "declared container length exceeds" error whose threshold depended on incidental filler size. The server now flushes the round and sends an over-bound artifact alone when adding it would cross the client's decode bound, so clonability is a property of the artifact alone. The 2*budget compile-time guard now certifies what it appears to.

* fix: reject unrecognized xfer Content-Type with 415 (#106)

A POST whose Content-Type was absent or unrecognized fell through to the plaintext card decoder, which is exactly the #104 failure shape: possibly- compressed binary fed to the card parser. Request bodies come from untrusted peers, so a body carrying no framing signal is rejected rather than guessed at. Real fossil always sets the header; no conforming peer is affected.

xfer.KnownContentType shares the media-type parsing with the framing check, so the server and Decode agree on what a type means.

Also runs the #109 filler subtests in parallel — serially they pushed internal/sync past the 120s per-package timeout CI uses. check-in: f1236bd031 user: [email protected] tags: trunk

* fix: clone emits cfile with stored content per sec7.2, not expanded file

Our clone server was building xfer.FileCard from cache.Expand(...) -- every artifact fully decompressed, and every delta materialized into full content -- where sec7.2 requires cfile cards carrying the repository's stored compressed content retransmitted verbatim. On real repositories this ran 80-220x more bytes on the wire than the protocol intends, and was the real blocker behind #88's clone-batching fix being unable to bound repositories much past libfossil's own size.

emitCloneBatch now reads blob.Load's stored bytes directly (full content for a whole artifact, delta bytes against DeltaSource for a deltified one) and emits xfer.CFileCard instead of xfer.FileCard, with DeltaSrc and USize (the full reconstructed size, from blob.size) populated correctly for delta rows. content.DeltaSource is exported so this and other callers outside internal/content can ask whether a rid is stored as a delta without duplicating the query.

No client changes were needed: session.go already routes CFileCard through the same handleFileCard path as FileCard, and storeDeltaContent already stores a received delta exactly as given via a create-phantom-if-missing lookup on its source, resolving delta-before-base arrival the way #53's phantom storage model was built for.

DefaultCloneBatchBytes is left at its current value rather than retuned toward canonical's 5,000,000 -- that constant's tuning is #109's territory now that bytesSent tracks stored bytes instead of expanded ones, and retuning it here risks colliding with that work.

* fix: expand deltified rows during clone instead of sending delta cfile

Code review reproduced a real interop regression: emitCloneBatch sends cards in ascending-rid order, but content.Deltify always deltifies the older artifact against the newer one, so a delta's DeltaSource almost always names a rid greater than the delta card's own rid. That means a delta cfile routinely reaches the client before the very source card it names. libfossil's own client tolerates the forward reference (storeDeltaContent phantoms the unseen source and resolves it later), but real fossil 2.28 does not -- the source stays a genuine, unfilled phantom, and content_get() on it during the client's post-clone rebuild pass returns without initializing its output blob, tripping an assertion in blob_copy (blob.c:397) and aborting the client mid-rebuild. Verified independently against a real fossil 2.28 binary cloning a libfossil server holding a delta chain.

emitCloneBatch now expands a deltified row to full content before sending it as a plain (non-delta) cfile, rather than forwarding the stored delta bytes with DeltaSrc set. The card is still zlib-compressed, so the wire-size win over the old FileCard/Expand path survives for every row; only the additional delta-retransmission savings are given up, and only for deltified rows. Re-enabling that needs either a source-before-dependent send order or confirmed forward-reference tolerance in canonical, and is left as a follow-up rather than blocking this fix.

TestEmitCloneBatchExpandsDeltifiedRows (renamed from TestEmitCloneBatchRetransmitsDeltaAsDelta) now pins the new contract: no DeltaSrc, full-sized payload. TestCloneRealFossilWithDeltaChain adds an end-to-end check against a real fossil binary, when one is on PATH. check-in: 0ca5133c62 user: [email protected] tags: trunk

* fix: store received wire frames verbatim instead of re-encoding

Clone reproduced content correctly but not bytes: the receive path decompressed every cfile frame and re-compressed it with blob.Compress's unparameterized zlib writer, so a clone's blob.content never matched its source's — about 0.5% larger on the wapp corpus, purely from re-encoding frames that were already correctly encoded on the wire.

xfer's cfile decoder now hands back StoredBlob alongside the decompressed Content: the same bytes recast into Fossil's on-disk blob format ([4-byte BE size][zlib data]) without a second zlib pass — verbatim when the wire already sent that format (Fossil's send_compressed_file case), or the received zlib stream with a length prefix bolted on when it didn't (our own encoder's plain-zlib case). storeReceivedFile/storeDeltaContent/ storeResolvedContent and blob.StoreDeltaRaw thread that through via a new blob.EncodeForStorage helper, which writes the verbatim bytes when given and only falls back to Compress when there's nothing to preserve — covering both the clone/pull and server-push receive paths.

Locally authored content (never received over the wire) always takes the Compress fallback, so #71's deltification policy is untouched. The clone BUGGIFY corruption test drops storedBlob alongside the corrupted content it's exercising, so verbatim storage can't mask the corruption it exists to catch.

Adds a corpus-level regression test cloning a real multi-revision fossil repo and asserting sum(length(content)) matches between source and clone.

* test: use RequireFossilBin instead of removed HasFossil helper

testutil.HasFossil was removed upstream in favor of RequireFossilBin; this test's skip guard needed the same update to compile against current main. check-in: cc42ad8829 user: [email protected] tags: trunk

* test: route fossil-binary tests through one REQUIRE_FOSSIL_BIN helper

REQUIRE_FOSSIL_BIN=1 turns a missing fossil binary into a test failure instead of an invisible skip, but only internal/manifest honored it. The six other fossil-dependent sites each did their own exec.LookPath + unconditional t.Skip, so on those a run with no fossil was byte-identical to a passing one.

Add testutil.RequireFossilBin, which resolves the binary the same way FossilBinary does (FOSSIL_BIN override, then PATH) and either skips or fails based on REQUIRE_FOSSIL_BIN. Route testutil.NewTestRepo, both internal/delta CLI round-trips, the four internal/merge validation tests, the internal/xfer sanity check, and the ls-branch-tip repro through it, deleting their bespoke lookup+skip logic. CI already sets REQUIRE_FOSSIL_BIN=1 on the unit-test steps, so this makes all seven sites fail loud on absence while local runs without fossil stay opt-in.

* test: route every fossil-binary gate through RequireFossilBin

The first pass left six sites still deciding for themselves whether a missing fossil binary is a skip or a failure. Three of them (internal/sync's startFossilServer and its two integration tests, manifest's two rebuild tests, the root TestPhaseB_Integration, and hash's TestSHA1_FossilValidation) skipped unconditionally, so with REQUIRE_FOSSIL_BIN=1 and no fossil on PATH those packages still reported ok -- the exact silent pass the flag exists to prevent. The other three had hand-rolled copies of the flag check, which is the duplication that let the original drift happen.

All of them now call testutil.RequireFossilBin, and HasFossil is gone: it was a second, skip-only way to ask the same question, and it is what produced the sites that ignored the flag.

Also validate the FOSSIL_BIN override with LookPath, so a stale path reports "fossil not found" instead of surfacing later as a raw fork/exec error.

* test: resolve fossil binary via RequireFossilBin at remaining call sites

* fix: correct stale module path missed by rebase auto-merge

Three imports added by earlier commits in this branch predated the rebase target and had no corresponding line on main to conflict against, so git's auto-merge carried the old module path through verbatim. check-in: f804878bd3 user: [email protected] tags: trunk

* fix: degrade unparseable clone VERSION/SEQNO instead of erroring (#95)

parseClone routed both tokens through strconv.Atoi before the §8.1 digit-only rule ever got a chance to apply, so a token Atoi could not parse at all (not just one that fails digit-only) hard-errored the whole message instead of degrading gracefully. Carry the raw tokens forward as an UnknownCard on parse failure, matching how UnknownCard already handles a non-decimal clone_seqno, so a later card in the same message still decodes.

* fix: reject an oversized §4.1 declared length before trying raw zlib (#110)

xfer.Decode tried the unprefixed-zlib framing before ever reading the §4.1 container's declared length prefix. A peer declaring a length at or above 0x08000000 produces a prefix whose first two bytes present as a structurally valid zlib CMF/FLG header (RFC 1950), so a large declared length aliases as the start of a raw zlib stream and gets first refusal on a legitimate compressed container. Read the declared length whenever the body opens with a valid zlib header and reject it against MaxDecompressedBytes before attempting format 1 at all. check-in: 2b9694f525 user: [email protected] tags: trunk

* fix: correct go.mod module path to go-libfossil (#76)

The module directive still declared github.com/danmestas/libfossil after the repository was renamed to go-libfossil. pkg.go.dev/godoc scraping matches the declared module path against its hosting path, so the stale path broke SDK doc generation.

Rename the module path across all four in-repo modules (root, the two db/driver submodules, observer/otel, and docs/site) and every internal import, plus the release workflow, the doc-generation script, and the docs.

No tag has been published under the go-libfossil path yet, so the cross-module requires cannot resolve from the proxy. Add replace directives pointing the in-repo requires at their local directories so GOWORK=off builds and the CI matrix resolve without a published version. These replaces are ignored by downstream consumers and can be dropped once new-path tags are cut.

* Guard releases against replace directives; regenerate SDK docs

The go-libfossil rename bridges the in-repo modules with `replace` directives. Go refuses a main module that carries them, so a tag cut while they are present breaks `go install .../cmd/libfossil@<tag>` and every `@version` fetch of the driver submodules. Add a Release workflow step that fails the job before anything is published if any tracked go.mod still contains a replace directive, so the bridge is a CI-enforced TODO rather than a silent landmine. The step comment records what unblocks a release: re-tag the submodules with the corrected module paths, bump the root requires, then drop the replace blocks.

Regenerate the SDK reference docs. The previous pass only substituted URLs over the pre-existing generated files, leaving stale signatures (Repo.Timeline was still documented as taking LogOpts rather than TimelineOpts). Note that gomarkdoc must be run outside a git worktree — inside one it silently strips every source link.

Also reword the README rename note, which read as though the import path had not changed, and re-sort the ncruces require block. check-in: 6273a0750d user: [email protected] tags: trunk

* fix: never parse a compressed body as cards (#104)

A clone served by libfossil failed in libfossil's own client with

xfer: decode card 16303: xfer: empty line after split

on a response that contained 202 cards. The index, the card, and the fault were all fictional.

xfer.Decode tried three body framings in order -- raw zlib, the §4.1 4-byte-length container, then uncompressed card text -- and treated any decompression failure as "this is not that framing". decompressBounded failed a well-formed container purely because its decompressed length passed a 50 MB bound, so the fallback chain handed the still-compressed bytes to the card parser. The parser read compressed noise as cards for sixteen thousand iterations before one of them split into no fields, and reported that as the fault. Captured at offset 15,084,752 of the failing body, the "card" the parser was reading is DEFLATE output:

"...\xd9\x34\x50\xe7\xcf\xa0\xf8\xd5\xd5\x87l\xff\t\xde\xfb\xe3..."

Two changes. Once a body is recognized as a zlib stream, a failure to decode it is reported rather than reinterpreted -- oversize, truncated, or corrupt, the error now names the transport failure instead of an arbitrary card position. Both compressed framings are still attempted, since a length prefix can coincidentally parse as a zlib header, and uncompressed bodies still reach the card parser; only compressed ones no longer can.

The bound itself was also wrong, and is the reason this was reachable at all. §4.1 places no limit on the decompressed size of the container; the 50 MB was a local memory guard that sat below what this implementation's own server emits in one clone round, so two libfossil peers could not clone from each other even though each could clone from fossil. It is now 64 MiB and coupled to sync.DefaultCloneBatchBytes by a compile-time guard that fails the build, rather than the clone, if the budget is raised past half of it.

The defect predates the size-bounded clone batching of #100. #100 shrank rounds enough to hide it on the corpus that exposed it; it did not touch the decoder.

Regression coverage, in two layers. internal/xfer pins the decode behaviour directly: an oversize container, a truncated one, a large in-bounds one, and an uncompressed body. internal/sync adds the configuration no existing test covered -- libfossil serving and libfossil cloning across a real HTTP encode/decode, on a corpus of multi-megabyte incompressible artifacts that crosses the server's per-round budget. Every prior clone test was either served by fossil or wired through MockTransport, which passes *xfer.Message values across untouched and never encodes a wire body at all. Artifact size is the other half of the reproduction: this failed on wapp, whose artifacts are large, and not on pikchr, whose artifacts are small.

Verified against the wapp corpus: the round trip that failed now completes with 1038 of 1038 artifacts on both ends.

* fix: make the #104 fixture reach the bound, and pin the bound from both sides

Review found the flagship regression test passed against pre-fix code. It did. Its corpus was 21.6 MB spread over twelve artifacts, and pagination keeps every round near the 16 MB budget however large such a corpus grows, so the round never came near the 50 MiB bound it was supposed to cross. The 25-line rationale reasoned about crossing DefaultCloneBatchBytes, which is not the quantity that matters and never was; a permanent comment asserting a mechanism that does not exist is worse than no comment.

The quantity that matters is one artifact. A round is charged against the budget before each artifact and sends whole the artifact that crosses it, so a round carries budget plus one entire artifact of unbounded size. A single artifact larger than bound-minus-budget is the only corpus that can exceed the bound, and it is also the one path by which a round can still exceed the current bound -- so the fixture guards the live risk and reproduces the historical one with the same bytes. It is now one incompressible artifact of 7/8ths of MaxDecompressedBytes, sized against the constant so it tracks it, behind testing.Short.

Verified against a pre-fix tree with both test files dropped in:

TestCloneSelfRoundTripOversizeArtifact FAIL Clone: handleFile aad935c2...: UUID mismatch for received file: expected aad935c2..., got 501b4814...

which is the sharper end of this defect: pre-fix did not merely misreport, it accepted the compressed bytes as cards and produced an artifact whose content did not hash to its name.

The cheap end-to-end round trip is kept as a separate test and described honestly as a coverage plank -- the first test anywhere here to have libfossil decode a body libfossil produced -- rather than as a guard on anything.

Second finding, message.go: the claim that format 1 is the "Fossil HTTP sync wire format" is false. Every application/x-fossil body carries the §4.1 length prefix, request side and reply side, both through blob_compress. Format 1 matches nothing canonical emits and is kept as a tolerance, now labelled as one.

Its safety was also load-bearing on a constant with no stated relationship to it. For a container of decompressed length N, prefix byte 0 is N>>24, and a zlib CMF requires CM==8 in the low nibble, so no container under 128 MiB can present a valid header at offset 0. Raising MaxDecompressedBytes past that point would leave decompressContainer's prefer-what-decodes ordering as the only remaining defense -- a narrower margin than this code was written against, and #98 is exactly the change that might want a different bound. That is now a build failure rather than a trap, matching the guard already on the other side. Verified to fire at 0x08000000 and to permit 0x07FFFFFF.

Also: the MaxDecompressedBytes doc claimed to stop a hostile body from exhausting an embedding process's memory. It does not -- the transport reads the body whole before Decode sees it. It bounds expansion only, and now says so.

Test hygiene: the server handshake polls for a live listener instead of sleeping 200ms and dropping ServeHTTP's error, and the 32 MiB and 64 MiB xfer allocations skip under -short like their sync counterpart.

* fix: correct three claims about what the #104 guards cover

The fix is unchanged. Three statements about it were wrong, and a comment that misstates a mechanism is worse than no comment because the next reader trusts it.

The ordering claim was the serious one. decompressContainer was described as "preferring whichever framing actually decodes, so a spurious header match still loses to a real container". It does not. Format 1 returns immediately on success and format 2 is reached only when format 1 fails; the error comparison at the end selects which failure to report when both have already failed, and protects nothing when format 1 spuriously succeeds. Probed: a bare zlib body returns from format 1 with format 2 never attempted.

That makes the aliasing guard weaker than its comment claimed, and in a direction neither review had reached: the aliasing byte is the top byte of the length the PEER declared, and this decoder never reads that field -- it slices past it and counts bytes actually inflated. No value of MaxDecompressedBytes constrains what a peer declares. A container declaring 0x081D0000 while carrying 46 bytes presents a parsing zlib header at offset 0, measured.

What actually saves a real container in that case is the stream, not arithmetic: after two prefix bytes are eaten as a header, inflate must still parse the remaining two plus the real stream as DEFLATE and match its Adler-32. Measured `flate: corrupt input` at all three aliasing lengths probed, so format 1 fails, format 2 runs, and the container decodes correctly.

The guard is kept because it states something true and useful -- Encode writes uint32(len(raw)) as the prefix, so holding the bound below the threshold means no body libfossil emits can carry an aliasing prefix -- but it is now documented as an invariant on our output, not a defence of our input, with the airtight alternatives named and left to their own issues.

Second, "a single artifact larger than this bound cannot be cloned" is true but not the operative threshold. A round already holding bytes has that much less room, so the ceiling is the bound minus what the round accumulated first. Measured on this branch with a 58,720,256-byte artifact: 8, 12 and 15 MB of filler ahead of it all fail, and 17 MB passes because it exhausts the budget and pushes the artifact into a round of its own. Non-monotonic in the filler, which is worth knowing. That is #109, a defect this fix exposes rather than causes; the fixture's claim to guard it is withdrawn and it now says plainly that it covers the leading-artifact case only.

Third, the changelog's "pin the bound from both sides" overclaimed. The guards pin the bound against the batch budget and against the aliasing threshold. Neither certifies a round.

Rebased onto 8b6da9a. The internal/content -race failure reported last round was not a flake and not unrelated -- this branch predated #107, which rewrote available.go. TestIsAvailable_CyclicChainTerminates goes from exceeding its 30s budget to 0.06s across that rebase.

---------

Co-authored-by: Dan Mestas <[email protected]> check-in: c892c7a622 user: [email protected] tags: trunk

* fix: clone_seqno is server-to-client only (#74)

The clone client transmitted a `clone_seqno` card to carry its pagination cursor. That card travels the other way: canonical's page_xfer() only ever writes it (src/xfer.c:1571) and has no parser for it, so a real fossil server answered `bad command: clone_seqno N` and the clone died on round two. Repositories that fit in a single batch never reached the pagination path, which is why wapp and pikchr cloned while fossil and sqlite did not.

The request cursor belongs on the clone card itself — `clone VERSION SEQNO`, read from token 2 at src/xfer.c:1553 — and the response cursor belongs on `clone_seqno`, which the client only consumes. Give each direction a single owner:

- The client sends its cursor on the clone card and never emits a clone_seqno card. It already consumed the server's card correctly; it now also rejects a negative value, matching src/xfer.c:2769. - The server reads its cursor from the clone card and the CloneSeqNoCard parser is gone, so there is exactly one way to set the cursor rather than two.

Aligning with canonical also corrects the cursor's meaning. Ours was the last rid sent, queried with `rid > cursor`; canonical's is the rid of the next blob to send, looped inclusively with `while( seqno<=max )`. Now that the value crosses to a real server it has to mean what canonical means, so emitCloneBatch queries `rid >= cursor` and reports the first rid it did not send, or 0 once the snapshot is exhausted. That collapses the old more/lastSentRID pair into a single nextRID and keeps the truncate hook honest by re-owing the dropped rid instead of skipping it.

TestCloneMultiBatchAgainstCanonicalServer drives the client against a stand-in server that implements the canonical contract and rejects a client-sent clone_seqno the way fossil does; it reproduces `bad command: clone_seqno` against the old client. TestHandleCloneCursorFromCloneCard pins the server half, including the inclusive cursor.

Verified against fossil 2.28 over a loopback server: fossil and sqlite now transfer every artifact (67,499 and 130,994 blobs), and wapp and pikchr still clone in 3 rounds.

* fix: conform clone cursor validation to the spec in both directions (#74)

Review caught two rules applied to the wrong side of the wire — the same axis error the original bug was.

The client rejected a negative clone_seqno as fatal, citing src/xfer.c:2769. That line is the server's rule for the SEQNO on a clone REQUEST. The reply parser is governed by §3.2 and §8.2: NEXT is recorded only when it is decimal, and a visible but non-decimal NEXT is a defined receiver-tolerance exception that leaves the recorded sequence unchanged and does not stop parsing later reply cards. Aborting the clone was the one thing the spec rules out. The rule belongs in the decoder, where a non-decimal NEXT now decodes to an unknown card: the token is preserved, no consumer records it, and following cards still parse. strconv.Atoi accepted "-1" off the wire, so this was reachable, not theoretical.

Conversely the server clamped `clone 3 0` to a cursor of 1 where §8.1 mandates a fatal: clear accumulated output, roll back, emit `error invalid clone sequence number`, and parse no later request card. Nothing is written before processDataCards, so returning early is what the rollback amounts to. Seeing this case at all needs a presence flag, because `clone` and `clone 3 0` both parsed to SeqNo 0 and only the former is exempt. §8.1 also keys on a digit-only SEQNO and withholds the fatal from wrong arity, a VERSION below 2, and a SEQNO that fails digit-only recognition, so the flag records digit-only presence rather than merely arity.

Both card rules now turn on the spec's `decimal = 1*DIGIT` production via a shared predicate instead of on whatever strconv.Atoi tolerates.

Also promotes emitCloneBatch's `count > 1` truncate guard to a documented liveness constraint. Under the old exclusive `rid > cursor` semantics `count > 0` was safe; under the inclusive cursor, dropping the only card in a batch reports a cursor equal to the one we entered with and the client re-requests forever. A check before the reply card now names a stalled round at its source instead of letting it surface as an opaque MaxRounds failure, and the cursor precondition became an error return rather than a panic — it is enforced ~350 lines away in a network-facing handler.

Records the deliberate asymmetry that our server ignores an inbound clone_seqno where canonical answers `bad command`, and renames TestHandleCloneSeqNo, which no longer sends that card.

* fix: name the clone SEQNO flag for the shape it records (#74)

HasSeqNo described presence, but the field records whether the SEQNO token is digit-only: `clone 3 -1` carries a SEQNO and sets it false. §8.1 keys its fatal on that shape rather than on the parsed value, which is why a plain int cannot stand in for it. SeqNoIsDecimal says so.

Reading it as presence would also collapse `clone 0 0` — a distinct wire form that parses to the same Version and SeqNo as a bare `clone` — back into `clone` on re-encode. encodeClone already guarded against that; a round-trip test over every clone wire form now pins it, and fails on exactly that input when the guard is removed.

Also records why the §8.1 rollback is currently vacuous: no card handled before processDataCards writes anything, so returning early leaves nothing to undo. Nothing enforces that ordering, and §3.5's ci-lock would break it, so the dependency is written down next to the code that relies on it.

Corrects a test comment that claimed a continuation property it did not exercise — the malformed card sits in the trailer, so nothing follows it. That half of §8.2 is covered by the follow-on card in TestDecode_CloneSeqNoNonDecimal. check-in: f2fa2cc751 user: [email protected] tags: trunk

The encoder emitted bare 'push\n' / 'pull\n' cards in code paths where the project code wasn't passed explicitly. The upstream Fossil C server's xfer parser requires at least the project code on push and both project + server codes on pull, rejecting 0-arg cards. This broke libfossil <-> fossil interop on the first sync round.

Make the project code a required parameter on the encoder and update call sites to thread it through. The libfossil server-side decoder is tightened to match — it previously accepted 0-arg cards, which masked the encoder bug during Go-to-Go testing. check-in: 4082bffaa5 user: [email protected] tags: trunk

* xfer: handle empty ServerCode/ProjectCode without arg-count error

Repo.Sync(ctx, transport, SyncOpts{Push: true, Pull: true}) with both codes empty previously wrote "push \n" to the wire. The decoder's strings.Fields collapsed adjacent spaces and failed with "xfer: pull requires 1-2 args, got 0".

Encoder now skips trailing empty args. Decoder accepts the verb with 0 args as "no specific code requested" — the same semantic as the existing 1- and 2-arg forms after fossil's normal arg interpretation.

Also fixes session.cardSummary's slice-out-of-range panic on empty codes by clamping the prefix length.

Adds a roundtrip test for the empty-codes path.

* sync: continue round-loop while either side has pending work

Sync's round-loop previously exited when the user passed Pull=false even if the peer was still requesting gimmes. The user's Pull flag controls whether we *request* new content; it should not gate whether we *service* the peer's gimmes. Symptom: agent-infra's Manager.Push had to pass Pull=true as a workaround to drive multi-round transfers.

Root cause: HandleSync.handleIGot gated gimme generation on pullOK alone. Push-only clients send igot cards from sendUnclustered every round (they always announce what they have), but the server would silently drop them because pullOK was false — leaving the server holding only what the client pushed proactively.

Continue the loop while any of: pending ungots to push, pending pulls to request, gimmes to service. Exit only when a round completes with no work in either direction. Concretely the gate is now (pushOK || pullOK), restoring parity with fossil-scm's client_sync continuation logic.

Adds an integration test (TestPushOnly_DrainsServerGimmes) that exercises the push-only path end-to-end and a unit test that asserts gimme emission under push-only handleIGot.

* sync: crosslink incoming manifests so hub-side state stays consistent

Repo.HandleSync previously stored manifest blobs in 'blob' but did not populate the relational tables (event, leaf, plink, mlink) that the timeline, fork detection, and clone protocol depend on. As a result:

- Leaves pulling from a hub got blobs but stale 'leaf' state, so WouldFork reported phantom siblings. - Verifier-clone of a hub repo failed with "sync.Clone: exceeded 100 rounds" because 'mlink' was empty and the clone protocol couldn't traverse manifest parents. - Hub timelines were always empty even after successful Push.

HandleSync now invokes manifest.Crosslink (the same scanner Clone and Sync run after convergence) when at least one file landed in the round. The scanner walks blobs that have no event/tagxref/ forumpost/attachment row yet, parses them, and populates the relational tables. Local commits already produced this state via manifest.Checkin's existing crosslink path; incoming syncs now match.

Coverage spans every artifact type the existing scanner already supports (ci, wiki, ticket, event, attachment, cluster, forum, control). The agent-infra trial that surfaced this only exercises 'ci' artifacts, so the divergence note in the prompt described the narrowest possible repair; reusing the existing scanner avoids the divergence and matches the C reference's full coverage.

Adds two tests: TestHandleSync_CrosslinksReceivedManifest verifies event/leaf/mlink rows after pushing a single checkin; the leaf- transition test verifies the parent leaves the leaf table when the child lands and that plink connects them. check-in: 3f290b159f user: [email protected] tags: trunk, v0.4.1

libfossil is a pure-Go implementation of the Fossil SCM, reading and writing the same .fossil SQLite repository format as upstream Fossil.

This is the first public release of code that previously lived in a private repository. The module path is github.com/danmestas/libfossil and the binary is named libfossil to avoid $PATH collisions with upstream fossil.

Included: - Repo lifecycle (create, clone), checkout/checkin, timeline, merge, diff, annotate, shun, tag, branch, bisect, stash, undo/redo - Sync-over-HTTP protocol (client and handler) - Two SQLite drivers: modernc (pure Go, default) and ncruces (wasm-capable) - Observer interfaces (SyncObserver, CheckoutObserver) with an optional OpenTelemetry implementation in observer/otel (separate submodule) - Deterministic simulation testing harness with BUGGIFY fault injection - Kong-based embeddable CLI under cli/ and a standalone binary under cmd/libfossil/ check-in: f5744359d3 user: [email protected] tags: trunk