Three tests started a real fossil server and killed it with Process.Kill() without waiting for the exit. Kill signals but does not reap, so the server could still be writing its -wal/-shm files while t.TempDir()'s cleanup walked the directory. A file appearing mid-walk fails RemoveAll with "directory not empty", which is how TestCloneFromRealFossilUncompressedReply failed on CI while passing locally -- it needs a loaded machine to lose the race.

internal/sync/integration_test.go already did this correctly, Kill followed by Wait, with a comment naming the same failure. This applies that established pattern at the three sites that lacked it.

Separately, the unit-test step ran with -timeout=120s, a per-package budget. CI runs the package binaries concurrently, so under contention internal/manifest and internal/sync exceeded it and panicked with "test timed out" despite finishing in 19s and 34s on an unloaded runner. Raised to 300s, matching the DST step, so the margin does not depend on how busy the runner is. check-in: 2256f5d7ce user: [email protected] tags: trunk

go.work.sum is only consulted for workspace-mode builds. CI never uses the workspace -- every step in test.yml runs GOWORK=off -- so the file earns nothing in the repository and re-churns on any local dependency change. Go regenerates it on demand for anyone building with the workspace active, so untracking it costs nothing and removes the churn.

docs/superpowers was added to .gitignore, but four files under it were already tracked; an ignore rule does not untrack what is already in the index, so the directory kept showing up in diffs regardless. Untrack it so the rule actually takes effect.

Both files stay on disk. Also terminates .gitignore with a newline. check-in: fd413ecc56 user: [email protected] tags: trunk

Keep validated receive bytes in a bounded session linker so clone does not expand and hash every artifact again during finalization. Preserve bounded retry, cancellation, replay, and residual-repair behavior.

Closes #214 check-in: 7b6a67fb81 user: [email protected] tags: trunk

Route direct check-ins and internal rebuilds through canonical mlink derivation. Update sparse-history consumers and preserve legacy conflict-fork rows with an explicit RID discriminator migration.\n\nCloses #213 check-in: 04f41a4c9c user: [email protected] tags: trunk

Cloning the Fossil SCM corpus (67.6k blobs) took 293-302 s at 1251-1287 MB against canonical fossil's 20.8 s at 205 MB. A CPU and allocation profile of one clone found the cost was not delta expansion or hash verification.

27.2% of all CPU was a single un-memoized statement. content.AvailabilityCache memoizes the delta-chain walk but not the "SELECT rid FROM blob WHERE uuid=?" that starts it, so manifest.missingCheckinRefs re-seeked the blob.uuid index once per F-card of every check-in -- the same file hash resolved again for every revision that did not change it. That index is far larger than SQLite's page cache, so each seek was three page reads: pread syscalls were 34.7% of total samples. The walk it guards was under 1% of its own cost.

Separately, materializing 8.27 GB of content allocated 360 GB. delta.ApplyInto starts at a 64 KiB cap deliberately -- the target length is a wire claim and a few bytes can declare 4 GiB -- and then climbs to the real size through append's ~1.25x large-slice growth, ~3.4x the target in discarded intermediates, leaving slack that content.Cache.clip copied away again (203.9 GB and 59.8 GB respectively).

Three changes:

- Memoize uuid -> rid in AvailabilityCache. Hits only. Within one crosslink run a hit cannot go stale, because a sweep and a cascade only ever add blob rows; a miss can, because ridOrPhantom and blob.StorePhantom create rows mid-run, and remembering an absence would leave an artifact deferred after the content it waits on arrived. The scope that makes this sound is now stated on the type, and TestAvailabilityCacheMustNotSpanAPurge pins its one boundary: shun.Purge deletes blob rows, so a cache must not span one.

- Size delta.ApplyInto's output exactly. scanOutputSize walks the command stream first without allocating, checking every copy against bytes the source holds and every insert against bytes the delta carries, so the size it returns is a measurement rather than the header's claim. Any stream it cannot walk to a clean terminator falls back to the existing growth path, so malformed and hostile deltas allocate and fail exactly as before. The buffer now arrives tightly sized, which also makes clip's copy a no-op.

- Cut crosslinkCacheBytes from 256 to 128 MiB. Each byte of that budget costs roughly five bytes of peak RSS -- GOGC doubles it into the heap goal, and the scavenger is paced off GC cycles, of which a faster sweep runs fewer. At 256 MiB the first two changes gave 205.7 s but 1473 MB, faster at the cost of memory. The measured curve (32/64/128/256 MiB: 369.6 s/318 MB, 292.3 s/447 MB, 249.4 s/812 MB, 205.7 s/1473 MB) puts 128 MiB where both axes improve.

Result on the same corpus: 245.4 s at 780 MB, from 293-302 s at 1251-1287 MB -- 17% less wall time and 39% less peak RSS, with instructions retired down from 3.37T to 2.48T. The gap to canonical narrows from 14.1x/6.3x to 11.8x/3.8x.

Derived tables are unchanged against an unmodified build cloning the same server: blob 67530, event 27099 (identical per type), mlink 104919, plink 23741, tagxref 51766, leaf 1426, filename 1496. The corpus parity gate passes.

What remains is structural rather than a missing optimization: a clone expands every blob twice, once in the receive-time phantom-fill cascade and once in the final whole-repository sweep, where canonical crosslinks each artifact from content it already holds in hand at receive time. Cache.Expand splits 45.4 s / 35.3 s across those two passes. Collapsing them is a change to the receive path and belongs in its own issue. check-in: 8b16c91f5c user: [email protected] tags: trunk

A full clone of the Fossil self-hosting repository disagreed with what canonical fossil derives from the SAME artifacts: mlink -3,351 rows, tagxref -632, plink -5, leaf -1. `fossil rebuild` on our own clone put every one of them back, so the artifact set was complete and the fault was entirely in crosslink-time derivation.

Six independent causes, each verified against `fossil rebuild` on a 67.6k-blob corpus clone:

mlink, -3,351. An mlink row records a parent-to-child TRANSITION, and a merge check-in has more than one parent. Canonical runs the same file-level diff once per parent -- primary, then each merge parent, then each cherrypick source -- and flags the extra rows isaux=1; we only ever diffed against the primary parent. add_mlink is now ported whole (internal/manifest/addmlink.go), which also brings the two passes the old code approximated: the delta-vs-delta revert/resurrect pass, and the baseline-child deletion pass with its mperm=0 delete rows. "Added by merge" (pid=-1) follows canonical's rule too -- a count over the rows a file produced, not a name lookup in the merge parents. A parent whose manifest cannot be read now yields NO rows rather than a whole tree of bogus "added" rows, and the rows are derived later from the parent's side once it arrives (linkPlinkChildren), so the result no longer depends on the order the sweep visits artifacts in.

tagxref, -612 of -632. A check-in's T-card does not have to name the check-in carrying it: `fossil merge --integrate` records the branch it closed as `T +closed <hash>` on the merge check-in itself. Canonical applies a check-in's, a control artifact's and a technote's T-cards through one code path; we applied only the self-referential ones on check-ins and none at all on technotes.

tagxref, the rest. A wiki-

or event-<id> tag's value is the W-card body measured after LEADING whitespace is skipped, not the raw card length.

Artifact identification. The sweep parses every blob in the repository, so what counts as an artifact decides which rows exist at all. Three gaps: a bare card letter with no arguments is legal (fossil's own first check-in spells its parentless-ness as a bare "P" card) and rejecting it dropped that check-in entirely; an N-card after a P-card is legal in a technote for historical compatibility, and rejecting it dropped real technote revisions with their event-<id> and bgcolor tags; and the check-in-versus-control decision turns on a comment, files or parents, which made `merge --integrate` check-ins read as control artifacts. Fossil's required-card table now gates linking (isArtifact), on both the sweep and the dephantomize cascade, so a file whose first line parses as a Z-card no longer links as a dateless check-in and a wiki revision missing its U-card no longer links at all.

plink, -5. The ancestry a check-in declares is a property of the check-in, not of what happens to be in the repository when it is crosslinked. A parent hash with no blob yet now reserves a phantom the way canonical's uuid_to_rid does, so the edge exists whether the parent arrives in a later round or never; skipping it lost the edge for good, because nothing revisits an already-linked artifact.

leaf, -1. Nothing calls leaf_rebuild() outside `fossil leaves --recompute`; the table is maintained by leaf_check() over the rids leaf_eventually_check() schedules -- each rid a branch tag lands on, plus that rid's parents. Drawing candidates from event instead both missed phantom parents (which fossil does list) and added check-ins that carry no branch tag and no ancestry (which it does not).

After: a fresh corpus clone and `fossil rebuild` of that clone agree row for row on mlink, plink, tagxref, leaf AND event -- 104,919 / 23,741 / 51,766 / 1,426 / 27,099 either way, and `fossil test-integrity` reports 0 errors.

Not addressed here: the local commit path (manifest.insertMlinks) still writes one mlink row per file rather than per change, and does not record merge-parent transitions. That is a separate divergence from the clone parity this issue is about.

Co-authored-by: Dan Mestas <[email protected]>

A pull into an already-up-to-date clone of the Fossil SCM repository took 2m44s and moved nothing. A CPU profile of that no-op pull puts 134.4s of 149.1s of samples (90%) inside manifest.CrosslinkContext, the whole-repository sweep Sync runs after the round loop. Nothing in the protocol path registers.

Two independent costs, both O(repository size) and neither bounded by what arrived:

collectCrosslinkCandidates 99.5s (67%) linkBatch -> Cache.Expand 34.9s (23%) of which hash.SHA3 26.0s (17%)

The query was quadratic. Its four exclusions were correlated NOT EXISTS subqueries, and tagxref is indexed on (rid, tagid) and (tagid, mtime) -- neither covers srcid -- so SQLite rescanned all 51,134 tagxref rows once per blob, 67,615 times. EXPLAIN QUERY PLAN showed the CORRELATED SCALAR SUBQUERY -> SCAN tx. Rewritten as NOT IN, each set materializes once and is probed: identical 39,727 rows, 47.7s -> 0.08s measured under canonical fossil's SQLite.

The candidate set never shrank. A blob qualifies when it has none of the derived rows a crosslink writes, and ordinary file content never gains those rows however often it is examined -- so 39,727 of the repository's 67,615 blobs were re-expanded (delta chain walked, SHA3 re-verified) and re-parsed on every sync. A probe run confirms the sweep linked exactly 0 artifacts on each of two consecutive runs over the converged repository, at 3m20s and 3m15s.

Canonical fossil has no equivalent cost because it never sweeps: xfer.c attempts one crosslink per blob as it arrives, artifact or not. This gives the sweep the same once-per-blob property via crosslink_nonartifact, an on-demand table recording blobs that expanded but hold no parseable artifact. Both facts are properties of immutable blob content, so the verdict cannot go stale; a blob that failed to expand, or that parsed but was deferred, is deliberately not recorded and is reconsidered next sweep.

Measured on a clone of the Fossil SCM repository (67,615 blobs), no-op pull:

before 2m43.9s after 0.40s

Clone is unchanged at 5m30s (it pays the one-time examination either way), and the two clones' derived tables are identical -- blob, event, mlink, plink, tagxref, filename and leaf totals all match row for row.

Fixes #202 check-in: 3748b95de7 user: [email protected] tags: trunk

* fix: make authenticated sync interoperate with canonical fossil (#203)

Every credentialed sync against a real fossil server was answered "login failed", whether or not the password was right. Push always needs a login, so push was unusable against any real server, as was pull from anything that did not grant anonymous read.

The login algorithm was never the problem. Two wire-level defects were, and each on its own produces that identical, password-independent rejection.

The push and pull cards carried their two codes in the wrong order. The canonical form is `push SERVERCODE PROJECTCODE` — canonical's own client writes `blob_appendf(&send, "push %s %s\n", zSCode, zPCode)` and both ends read the project code out of the second argument (src/xfer.c). We emitted and parsed project code first. A clone therefore adopted the server's server-code as its project code, and since a login signature is salted with the project code, every signature was derived from the wrong value. The same inversion put the server code where a server validates the project code, so an authenticated push was refused with "wrong project" even once the login worked.

The nonce covered a random comment that was never transmitted. The nonce is the hash of every byte following the login card, and the server re-hashes those bytes to check it (src/xfer.c check_tail_hash) — so the comment supplying the randomness has to be on the wire, not merely in the hash. We appended it to the hashed payload only, producing a nonce the server could not reproduce; it rejected the login before comparing the password at all, which is why a wrong password looked exactly like a right one. Comment lines now have a card type, so they can be sent as well as hashed.

Two smaller faults on the same path: a clone treated the server's round-0 "not authorized to clone" as fatal, ending the session before credentials were ever offered, where canonical tolerates that error on the first cycle precisely so the client can retry with the login card it can only build once it knows the project code; and the sync path signed with SyncOpts.ProjectCode rather than the code it had already resolved from the repository, so a caller that let the repo supply its project code — as the push and pull cards allow — panicked instead of authenticating.

The password derivation itself, sha1("<project-code>/<login>/<password>"), was correct and is unchanged. It is now pinned two ways: against the hash the fossil binary actually stores in user.pw, and against a login card captured from fossil 2.28 on the wire. Canonical's rule that an already hashed 40-hex password is used as-is is now honoured too.

Regression coverage stands up a canonical fossil server with anonymous capabilities removed, so a login card is the only way in, and asserts that a credentialed clone transfers content, a wrong password is refused, and a local commit pushed under authentication lands in the server's blob table and on its timeline. Nothing in the suite authenticated against real fossil before; clone was only ever exercised against servers granting anonymous read, where the login card is never sent, which is how this shipped.

Four xfer codec expectations that pinned the inverted argument order are corrected to the canonical bytes.

* fix(sync): correct the nonce doc to match what the server recomputes

The comment described the nonce as covering the encoded non-login cards and the signature as using the stored password hash. Both readings are wrong in ways that matter to anyone changing this code: the nonce is defined over the bytes that actually follow the login card on the wire, because the server recomputes it from what it received (src/xfer.c check_tail_hash), and the signature is salted with the project code rather than derived from user.pw directly. Hashing bytes the message does not carry produces a nonce the peer cannot reproduce, which is exactly the password-independent rejection this fix was chasing. check-in: a745f4f2ac user: [email protected] tags: trunk

AddTag, ApplyTag and ApplyTagWithTx each carried their own copy of the same type guard and eight-argument propagate call. #198 had to apply its mtime guard to all three, and a fourth site -- or one site updated while the others are missed -- reproduces exactly the shape that produced #197 (leaf derived two different ways) and #199 (project code resolved two different ways).

Extract propagateAfterInsert so the guard travels with the propagation instead of being restated at each caller, and cover the consolidated path. check-in: 4806de7907 user: [email protected] tags: trunk

Branch tag propagation left 1,155 of 20,859 check-ins on a clone of the fossil repository carrying a different branch value than canonical, and tagxref at 68,883 rows against canonical's 51,764. Two independent defects in the port, both confirmed against fossil's own source and a canonical rebuild of the same blobs.

tag_insert is mtime-guarded and our tag application was not. Canonical opens it with

SELECT 1 FROM tagxref WHERE tagid=%d AND rid=%d AND mtime>=:mtime

and returns without touching tagxref -- and without calling tag_propagate -- when it matches (src/tag.c:173-186). AddTag, ApplyTag and ApplyTagWithTx all wrote unconditionally, so a check-in's own inline T-cards clobbered a later control artifact that had retagged the same check-in, then re-propagated the stale value to every descendant. On the real corpus one branch rename did this 19,115 times: check-in ab0a0d76 declares branch dual-license, three control artifacts later rename it to clear-title, and we reinstated dual-license and flooded the repository with it. The guard makes the outcome independent of the order the sweep visits the two artifacts in.

repairTagPropagation seeded from the wrong node. tag_propagate's per-child test is strict --

coalesce(srcid=0 AND tagxref.mtime<:mtime, 1) AS doit

-- so a descendant that already carries the tag holds it at exactly the origin's mtime, doit is 0, and the walk neither retags that child nor queues it. Replaying from tag origins therefore stopped dead at the first already-tagged descendant, stranding any check-in linked by a later sweep. Canonical seeds from the primary parent instead, right after inserting a new check-in's plink edges (src/manifest.c:2300-2302 and 2467-2469), which only ever tests that parent's own children. Seed from every primary parent, which is the sweep-order-independent equivalent. Worth 693 missing propagated branch rows.

Measured on a clone of the fossil repository against a canonical rebuild of the same blobs:

divergent branch values 1,155 -> 1 (of 20,858 compared) tagxref rows 68,883 -> 51,132 (canonical 51,764) leaf rows 1,463 -> 1,425 (canonical 1,426)

The residual 632-row tagxref gap is 613 missing `closed` singletons plus small change; it measures identically before and after this change and is a separate defect. The one remaining divergent branch value is a NULL-vs-empty-string difference in cancel-tag values, also pre-existing.

repairTagPropagation still runs before repairLeafTable, as #197 requires. check-in: 7043d6e3a9 user: [email protected] tags: trunk

test.yml had zero references to wasm or GOOS=js, so the WebAssembly build was never compiled on any leg. Browser support is the reason the ncruces driver is in the tree, and a break there would have shipped undetected -- including in db/config_wasm.go, whose js-tagged wasmPragmaOverrides() no job compiled at all.

Adds a build-only job covering 31 of the main module's 33 js-resolvable packages plus the ncruces driver submodule. The package set comes from go list rather than a hand-written list so new packages are gated by default. Two main packages are excluded, both for the same structural reason: cmd/libfossil and examples/embed-repo-api import the modernc driver directly in their own main.go, and modernc.org/libc has no js/wasm build. internal/stash and internal/undo declare //go:build !js in source and are dropped by go list itself. Building under -tags test_ncruces keeps internal/testdriver in the gate by pointing it at the driver that can actually target WASM.

Running tests under WASM needs a browser harness (wasmbrowsertest) and is deliberately out of scope; compilation catches the realistic failure mode. check-in: 10abca922c user: [email protected] tags: trunk

* fix(sync): derive the project code from the repo instead of panicking (#199)

Supplying User/Password made SyncOpts.ProjectCode mandatory in practice: buildLoginCard passed the caller's raw value straight to computeLogin, which asserts it is non-empty, so an authenticated Sync on a repo whose project code the caller had not repeated took the process down. Nothing in the public API signalled that a password made the field required, and a server embedding the library died with it.

buildRequest already fell back to the repo's stored project-code for the push/pull cards; that fallback was local to one function, which is exactly how the login path drifted out of step with it. Resolve the code once when the session is built — caller value first, the repo's own config second — and let buildRequest and buildLoginCard both read it. A code that genuinely cannot be determined now returns a descriptive error from the site that needs it rather than panicking.

Sync's deferred result-not-nil assertion fired during panic unwinding, where err and result are both nil, replacing the original cause with its own message. The invariant only holds on a normal return, so it no longer asserts while a panic is in flight.

* fix(sync): write received artifacts' visibility inside the storing transaction (#200)

Receiving an artifact cost three write transactions, not one. The blob was stored in a transaction, its dephantomize cascade committed another, and then applyPrivateStatus re-derived the rid on the pool and issued a bare DELETE FROM private -- an implicit transaction of its own, with its own BEGIN IMMEDIATE and fsync, for a row that a transaction microseconds earlier had guaranteed did not exist. Every igot card announcing an artifact already held cost one more. Measured against a real fossil peer: 1509 statements executed outside a transaction to receive 1504 artifacts.

That is minutes of pure commit overhead on a large repository -- roughly 200,000 write transactions for a 67k-artifact pull -- and it is fragile as well as slow. Each of those writes contests the database's single write lock independently, so N artifacts means N chances to lose that contest and fail outright rather than one chance to wait at a single BEGIN.

Visibility now travels with the content it describes: storeReceivedFile takes the visibility a private card announced and records it inside the transaction that stores the blob, using the rid that transaction already has. The status is now atomic with the content rather than a separate write that could be interrupted between the two. The igot path buffers its updates and writes them once per round. The same three-transaction shape on the server's receive path goes with it.

The same real-fossil pull now executes 2 statements outside a transaction for those 1504 artifacts, down from 1509. db.DB.PoolExecs makes that budget assertable, which is how the regression test pins O(rounds) rather than O(artifacts) without needing a large corpus.

On the failure originally reported: the "database is locked" error came from a harness that copied a live SQLite database out from under a running server without its WAL, so that specific lock holder is not a product defect and this does not claim to fix it. The transaction storm above is a real performance and robustness defect on its own evidence, and is what this changes.

* test(sync): pin the transaction budget, not just writes outside a transaction

The budget tests measured PoolExecs, which counts statements issued outside a transaction. That pins one half of the defect and misses the other: moving a per-artifact write from the pool into its own WithTx leaves PoolExecs unchanged while every artifact still pays a BEGIN IMMEDIATE and an fsync, and still contests the single write lock on its own. That is the cost issue #200 actually measured -- roughly three write transactions per received file, about 203,000 for one clone-sized pull.

Verified by mutation: reverting the batched visibility flush to one transaction per update left both budget tests green. They now count transactions too, via a new db.DB.WriteTxns counter, and the same mutation fails with "began 51 write transactions for 50 igot cards over 1 rounds; want <= 4".

Also stop the contention test from starving the pull it is meant to observe. The rival connection re-acquired the write lock immediately on release, so on a loaded machine it could hold the pull's single BEGIN past busy_timeout and fail the test even when the batching under test is perfect -- which is what happened on CI. It now yields between bursts, modelling another process doing occasional writes rather than testing SQLite's fairness. check-in: 18bfbc166c user: [email protected] tags: trunk

* fix: derive leaf from fossil's branch rule and bound the deferral rollup

Leaf derivation dropped 66% of rows on the real Fossil repository (487 of 1,426) because both leaf paths treated any parent link as retiring a leaf. Canonical fossil retires a check-in only when a child carries the *same* branch name, with an absent branch tag reading as "trunk" (src/leaf.c: is_a_leaf / leaf_check / leaf_rebuild all share that predicate and none consult plink.isprim). Verified against fossil 2.28 itself: after a branch commit off the trunk tip, both the incremental table and `fossil rebuild` keep the trunk tip as a leaf even though its only child is its primary child, and a branch tip merged into trunk stays a leaf as well.

Two code paths were wrong, in the same way:

- repairLeafTable excluded every `pid` in plink, so merged branch tips disappeared -- the class that accounts for 939 of the corpus's leaves. - Checkin deleted every parent from leaf, so committing onto a new branch silently retired the branch point.

Both now share leafCheck/repairLeafTable in internal/manifest/leaf.go. The recompute also has to run after tag propagation rather than before it: a check-in that declares no branch of its own only has a branch value once propagation has written it, and comparing against a missing value made every inherited-branch tip look like a leaf.

Applying the new predicate to the real repository's own event/plink/tagxref yields 1,428 against fossil's 1,426, and every one of the six differing rows is fossil's incremental bookkeeping rather than its predicate: 2 are phantom rids fossil's leaf_check inserts as plink parents of check-ins it linked, and 4 are parentless check-ins with no branch tag, which nothing ever enqueues for a leaf check. Running fossil's own leaf_rebuild() on that repository gives 1,494, so it is leaf_check, not leaf_rebuild, that the shipped table reflects.

Also caps the deferral rollup's identifier lists (issue #194). A clone deferring 180 check-ins on 417 missing blobs rendered a single ~28 KB log line; the counts stay exact at INFO and the lists print eight entries plus an "...and N more" so nothing is dropped without saying so.

Fixes #189 Fixes #194

* chore: retrigger code scanning after branch restore check-in: df495ba69e user: [email protected] tags: trunk

A clone of a large repository is a conversation of dozens of rounds spanning minutes. A single transient disconnect anywhere in it aborted the whole operation with "http transport: Post ...: EOF" and produced no store at all: there was no retry, no reconnect, and minutes of transfer were discarded for one dropped packet. Reported failures landed at rounds 6, 11 and 18 across otherwise identical runs, which is the signature of independent random drops rather than a deterministic defect.

Both round loops now send a failed round again after a bounded exponential backoff — four attempts, 250 ms doubling to a 2 s cap — but only when the failure was the network dropping the exchange. The classification is an allow-list, not a deny-list: a reply that arrived whole and would not decode, a card the peer rejected, a 4xx, or an unrecognised error from a Transport this package does not know all still fail on their first occurrence. Retrying those would turn a reproducible defect into an intermittent one, which is worse than the failure retrying exists to prevent.

Telling the two apart cannot be done reliably after the fact — a truncated zlib body and a truncated wire read both surface as io.ErrUnexpectedEOF — so the transports tag their own wire stages via the optional TransportFault interface, in the same shape as FramedTransport. Non-2xx replies are now reported as such instead of being handed to the decoder, which also stops a peer's outage from being misreported as a framing defect.

The caller's context governs throughout: a cancellation or deadline aborts immediately, including during a backoff wait, so retrying cannot extend an operation past the bound its caller set (#166, #167, #183).

Resuming a partially-completed clone across process exit is deliberately not included: it needs the clone cursor persisted, a staleness policy for the server's snapshot bound, and a separate entry point, and within-session retry already preserves the expensive part. check-in: b2928e9629 user: [email protected] tags: trunk

All three submodules required the root module at v0.6.4-0.20260724230440-48111c1c984d. That pseudo-version names a commit pushed on the #118 PR branch, which was squash-merged — rewriting the work as a new commit and orphaning 48111c1. `git branch -r --contains` finds it on no branch; it survives only in proxy.golang.org's cache.

CI never noticed because it resolves through the proxy. Direct-from-git resolution fails outright:

$ GOMODCACHE=$(mktemp -d) GOWORK=off go mod download github.com/danmestas/go-libfossil go: github.com/danmestas/[email protected]: invalid version: unknown revision 48111c1c984d

v0.7.0 is tagged and reachable, so the bootstrap constraint that forced a pseudo-version is gone. Repoint all three at the tag and tidy. The root go.mod hash is unchanged between the orphaned commit and v0.7.0, so this is a pure provenance fix with no dependency movement.

Add a release guard alongside the existing `replace` guard: any in-repo module required at a pseudo-version now fails the release before the tag ships. The pattern anchors on the 14-digit timestamp and 12-hex hash, and accepts either separator before the timestamp so the vX.Y.Z-0.<ts>-<hash> form is caught as well as v0.0.0-<ts>-<hash>. Third-party pseudo-versions such as ncruces/go-sqlite3-wasm are deliberately untouched. check-in: 9d35407ecb user: [email protected] tags: trunk

* fix: write the ticket event row with its tag (#184), quiet the cascade (#186)

crosslinkTicket applied the tkt-<uuid> tag and returned the event row as a pending item for a second pass that was a no-op stub, so the row was never written at all. collectCrosslinkCandidates excludes any blob that already has a tagxref row naming its own rid as srcid, so the tag the incomplete operation wrote also hid the artifact from every later sweep. Cloning fossil's own repository produced 96 event rows of type 't' against canonical fossil's 3,808 -- 100% of the D,J,K,U,Z ticket-change shape lost, with nothing logged and test-integrity still clean, because only derived tables were wrong.

Implement the ticket rebuild instead of reordering the writes: there was no event write to reorder. ticketRebuildEntry replays every artifact carrying the ticket's tag in mtime order and derives its ticket, ticketchng and event rows, mirroring fossil's ticket_rebuild_entry, ticket_insert and manifest_ticket_event. It runs inline in the transaction that writes the tag, so the marker meaning "already linked" and the rows that make that true commit together or not at all. Replaying the whole tag set per artifact also makes the result independent of arrival order.

The event wording, the ticket/ticketchng column mapping including '+field' append semantics, the %h escaping and the two hyperlink hash lengths were all derived from the fossil binary rather than assumed, and are pinned by TestFossilBinaryTicketParity: a corpus built with `fossil ticket add`/`change` is crosslinked by us, rebuilt by fossil, and the two derivations compared row for row. TestCrosslink_TicketArtifactsProduceEventRows is the count assertion whose absence let this ship.

Audited the other three exclusion predicates for the same split-write shape: forumpost and attachment are each written by the same function, in the same transaction, as the event row that accompanies them, and propagate errors, so neither can diverge. No change made to them.

Secondary (#186): AfterDephantomize warned once per blob that failed to parse as a manifest, which is the ordinary outcome for file content and what the whole-repository sweep has always skipped silently -- ~39k lines on a fossil clone, burying the failures that matter. It now skips them the way linkBatch does. A blob that parses but fails to link is still a real fault and still warns.

* docs: record the ticket rebuild's measured cost and why it stays inline

The inline rebuild is quadratic in one ticket's artifact count. The cheaper deferred shape stays atomic but needs two call sites to remember to invoke it, and a forgotten call reintroduces #184 silently. Write the measurements and that reasoning down at the function, so a later profile of a repository with one hot ticket does not turn into an optimization back into the bug. check-in: 39f646e0ae user: [email protected] tags: trunk

The per-package parity tests compare our derivation against fossil's for fixtures built in-test, so they only cover the shapes someone thought to build. Issue #184 -- a silent 100% loss of the ticket-change artifact shape (D,J,K,U,Z) -- passed that suite throughout and was found by an external benchmark cloning the real Fossil repository.

Add TestCloneCorpusParity, which builds a repository with the canonical binary, serves it over loopback, clones it with both fossil and libfossil.Clone, and compares what each derived.

The comparison is per-event-type, not just totals. On the real Fossil repository the totals read 23,396 vs 27,099 -- a 14% shortfall that reads like rounding -- while the per-type breakdown put ticket events at 96 vs 3,808. Totals for blob, event, mlink, plink, tagxref and leaf are compared too, with deltas reported in both directions: our tagxref ran +17,119 over fossil's there, which is what writing a tag without finishing the work it marks looks like.

The corpus is built by the canonical binary at test time rather than downloaded -- seconds and under a megabyte, against the ~10 minutes and ~1 GB the real repository costs (#187) -- and covers check-ins with a branch and a merge, ticket creations and updates, a wiki page, a technote, an attachment, and a tag control artifact. requireCorpusShapes classifies every blob by its card set and fails if a shape falls below its floor, so a corpus that quietly stopped producing tickets cannot let the gate pass while covering nothing.

Gated behind LIBFOSSIL_CORPUS_PARITY=1 in addition to the existing fossil binary requirement, so the default suite is unchanged in runtime and outcome.

The gate fails on this commit, which is correct: #184 is unfixed, and it also surfaces a second divergence where a merged branch tip is missing from leaf.

Forum posts are not covered -- fossil exposes no CLI to create them. check-in: d01b94f4c1 user: [email protected] tags: trunk

The whole-repository crosslink sweep holds back a checkin whose referenced blobs (F-cards or B-card baseline) have not arrived locally, via deferCheckin, so a later sweep can rediscover and complete it once the blob shows up. The mid-round phantom-fill cascade (afterDephantomize) had no equivalent guard and linked such a checkin unconditionally, writing its event row despite insertCheckinMlinks silently skipping the missing-blob F-card.

Because collectCrosslinkCandidates only reselects a rid with no event row, that half-linked checkin was permanently stranded: no future sweep, no matter how many sync rounds ran or when the missing blob eventually arrived, would ever revisit it. A clone that terminated before the blob arrived could leave the repository with a leaf checkin selected by CalcUpdateVersion whose Checkout.Update would fail with "blob not found" -- with no recovery path.

Factor the sweep's deferral bookkeeping (availability cache, deferred rids, missing blobs) into checkinDeferralGuard, shared by both linkState (the sweep) and cascadeLinker (the cascade). The cascade now checks shouldDefer before opening a checkin's savepoint, so a deferred checkin writes nothing to the batch -- matching the sweep's defer semantics exactly, and preserving the invariant both paths depend on: a deferred rid must carry no event row, or the sweep's own candidate query can never find it again.

deferCheckin's sweep-facing behavior, and linkBatch's call site, are unchanged -- it is now a thin wrapper over the shared guard. check-in: 36d61bd582 user: [email protected] tags: trunk

TestChainBoundBitesAtSameLength inserted its fixture a statement at a time: one blob row per chain node and one delta row per edge, across two arms sized maxDeltaChainDepth and maxDeltaChainDepth+1. At the current constant that is ~65k statements, each in its own implicit transaction and each paying a commit.

The chain is incidental setup; the subject of the test is the bound. But the setup cost dominated the runtime and was sensitive to how loaded the machine was, so on a contended runner it consumed the package's 120s budget and failed a PR that touched only CI YAML.

Wrap both loops in a single db.WithTx so the chain commits once. This follows buildBackwardChain in chain_depth_bench_test.go, which already builds its chain this way in the same package.

The assertions are untouched, byte for byte: the two walks must still agree at maxDeltaChainDepth and disagree one past it. Verified by mutation -- widening one walk's loop bound by a single node still fails the test with the same diagnostic it was written to produce.

Measured on this machine, three runs each, whole-package wall clock:

modernc before 19.7s / 25.0s / 7.1s after 1.19s / 1.36s / 0.80s ncruces before 3.5s / 10.0s / 7.5s after 1.27s / 1.59s / 1.13s

The spread collapsing matters more than the median: the old fixture's cost tracked machine load, which is what made it a timeout risk rather than merely slow.

Refs #179 check-in: 39f9ea4d8a user: [email protected] tags: trunk

- Convert [Unreleased] to [0.7.0] and complete with remaining 67 commits - Add sections for v0.2.0 through v0.6.3 with commit-derived summaries - Add empty [Unreleased] section for future work - Highlight module path breaking change (libfossil → go-libfossil) - Maintain Keep a Changelog + SemVer format and existing prose style check-in: 35b79c90c6 user: [email protected] tags: trunk

emitMatches walked a hash bucket's entire collision chain at every target position with no limit. rollingHash is a weak multiply-accumulate over a 16-byte window, so low-entropy content — repeated tokens, runs of whitespace, ordinary SQL and C boilerplate — funnels an enormous number of source windows into one bucket, making Create quadratic in target length. Measured on the issue's repro (repeated 8-byte token, one 10-byte edit): 32 KiB 0.133s, 64 KiB 0.496s, 128 KiB 1.950s, 256 KiB 7.765s — ~3.9x per doubling — against 0.013s for a 128 KiB high-entropy control.

Bound the walk at maxChainCandidates, matching the `int limit = 250` that fossil's delta_create (src/delta.c) puts on the identical loop. The same walk also computed each candidate's match length twice, once to test it against the nHash threshold and again to read the value; compute it once.

Extract the walk into bestMatch so the bound is testable on a candidate count rather than on wall-clock time, which cannot flake on loaded CI.

Delta output changes, as fossil's own limit makes it change: a capped search can settle for a shorter match, so the encoder emits more copy commands. This lands our output on canonical fossil's — 128 KiB: 266 bytes vs fossil's 260; 1 MiB: 2142 vs 2120 — where before the unbounded walk produced 44 and 50 bytes and paid quadratic time for them.

After: 128 KiB 1.950s -> 0.018s, and 1 MiB 125.036s -> 0.062s.

Verified the deltas still apply byte-for-byte, by our Apply and by canonical fossil 2.28's test-delta-apply.

Co-authored-by: Dan Mestas <[email protected]> check-in: b1b6c09b41 user: [email protected] tags: trunk, v0.7.0

Fossil keeps a per-user global config database outside any repository and creates it lazily on first use. The fossil-gated tests invoke the binary with inherited env -- nothing sets HOME -- and go test runs packages concurrently, several of which shell out to fossil. On a fresh runner that database does not exist yet, so two test processes can race to create it: one wins and the others fail with "table global_config already exists".

Create it once, single-threaded, after the install and version check and before any build or test step, so no test process is ever the one to create it. fossil version does not open that database, so the existing version check does not incidentally cover this; fossil all list does, and is silent.

The warm-up command's exit status is ignored on purpose -- it exists to touch a file, not to assert anything. The file check that follows is the part that matters: a warm-up that silently stopped creating the database would leave the race live while looking like protection, so its absence fails the job loudly.

Refs #174

Co-authored-by: Dan Mestas <[email protected]> check-in: bba767af03 user: [email protected] tags: trunk

afterDephantomize crosslinked every artifact it discovered through crosslinkSingle, which opened its own write transaction per artifact and passed an explicit nil content cache into crosslinkCheckin. Both choices suit an isolated artifact and neither suits the cascade: one phantom fill unblocks hundreds or thousands of related artifacts that share delta bases and parent manifests, so the walk paid a transaction commit per artifact and re-expanded content a sibling had just expanded. A CPU profile of round 7 of a fossil-corpus clone showed exactly that pair -- _vdbeCommit plus expandChain/decompress reached through crosslinkSingle -- as the remaining 300+ seconds after the statement-cache fix.

crosslinkSingle had no callers outside the cascade, so it is replaced rather than kept alongside a second strategy. cascadeLinker buffers the artifacts the walk discovers and links them through linkArtifact -- the same type switch the whole-repository sweep uses, so the two paths no longer carry near-duplicate switches -- on one content.Cache for the whole cascade, committing once per crosslinkBatchSize artifacts instead of once per artifact. The sweep's own batching, cache, and tuning constants are untouched.

The walk's delta-child query filters on committed mlink rows, so deferring writes past it could change which artifacts it visits. Orphans are the only discovery that query can observe -- delta.rid is a primary key, so every other buffered artifact was itself produced by that query and cannot come back -- so the buffer flushes after an orphan batch and before the query, preserving the original order: orphans crosslinked, orphan rows cleared, children queried.

Failure handling keeps what per-artifact transactions bought. An artifact that cannot be expanded, parsed, or linked is logged with the same wording and skipped while the rest of the batch commits, and each artifact's writes are wrapped in a savepoint so a link that fails partway leaves no partial event/plink/mlink/tagxref behind. Cancellation is unchanged in the walk and newly observed inside a flush on the sweep's stride; an open batch is abandoned rather than committed, so committed batches stand and the next Crosslink sweep re-selects whatever still has no event row.

Tests: a characterization test pins the cascade's derived rows and visit count over a graph covering both sides of the mlink filter -- an orphan that writes mlink rows and one that does not, and so is still visited twice -- against the dump the per-artifact shape produced. A regression guard asserts one commit and one expansion per artifact with the parent manifest served from cache for a 300-artifact burst, plus tests for whole-batch flushing, savepoint isolation, and mid-flush cancellation. check-in: bfb2ad3aa4 user: [email protected] tags: trunk

Both test jobs installed the canonical fossil binary with an unpinned `apt-get install -y fossil`. On the Ubuntu runner that resolves to 2.23, while all local real-fossil parity work is validated against 2.28 -- so CI was checking our output against a different reference implementation than development does. The distro package is not stable even at a fixed version string: it tracks different upstream check-ins over time, so two runs at nominally the same version can behave differently.

Install the official upstream release tarball instead, pinned by version and by SHA-256 so it is a fixed artifact rather than a moving target. The version lives in one workflow-level env value both jobs reference, and a verification step asserts the installed binary actually reports it, so an upstream change fails loudly instead of drifting silently.

This also explains the intermittent `table global_config already exists` failure recorded in #162. `global_config` is not part of the repository schema at all -- it is the only table in `~/.fossil`, fossil's per-user global config database, created lazily on first use. Under concurrent fossil invocations against a HOME that has no `~/.fossil` yet (a fresh CI runner, never a dev machine) 2.23 races on that CREATE TABLE; 2.28 does not. Reproduced at 24-way concurrency on a fresh HOME: 2.23 hit the error in 30/30 trials, 2.28 in 0/30, and 0/60 at 48-way.

Our schema creation is not implicated. 1500 iterations of TestCreateRepoSchema_FossilValidation against 2.28 (500 in-process plus 200 separate processes per driver variant) and 100 full `./db/...` runs passed with zero failures, and 40 independently created schema artifacts across both drivers produced byte-identical DDL.

Closes #162 check-in: 9ff6e07d74 user: [email protected] tags: trunk

The hot per-artifact queries in the phantom-fill crosslink cascade — the content/blob-by-rid lookup and the per-node orphan/delta-child lookups in afterDephantomize — re-tokenized, re-parsed, and re-compiled their SQL text on every call because db.Querier delegated straight to the raw driver each time with no statement reuse. On a large repository (67k+ artifacts) the SQLite parser/compiler dominated CPU during the slow clone stretch.

Add a statement cache to both Querier implementations, keyed by exact query text, with lifetimes matched to each backing type:

- *DB caches *sql.Stmt prepared from its long-lived *sql.DB pool (safe for concurrent use, manages its own connections). Statements are prepared lazily on first use, reused thereafter, and closed in DB.Close. - *Tx caches *sql.Stmt scoped to a single *sql.Tx; the cache is torn down (all statements closed) when the transaction commits or rolls back, so no statement outlives its transaction or leaks across a clone/sync session.

Exec/QueryRow/Query transparently get-or-prepare the cached statement — no signature or caller-visible behavior change. QueryRow preserves database/sql's deferred-error contract by falling back to the raw path on prepare failure so the error still surfaces from Scan. Multi-statement strings (DDL setup batches) bypass the cache and run through the raw driver path, since a prepared statement compiles only one statement.

Benchmark (Apple M4 Pro, modernc driver): the blob-by-rid lookup drops from 4779 ns/op to 2289 ns/op (~2.1x). Cache reuse, transaction-boundary cleanup, and multi-statement bypass are covered by new tests in db/stmtcache_test.go. check-in: 218ebaec74 user: [email protected] tags: trunk

storeReceivedFile's mid-round AfterDephantomize crosslink cascade became context-aware in #166, and the clone and server-push receive paths already pass their real deadline context into it. The third caller -- the general push/pull path -- did not: (s *session) handleFileCard hardcoded context.Background(), so an ordinary Sync could not interrupt a long-running crosslink cascade via its caller's deadline the way a Clone now can.

Thread the sync loop's own context the one remaining hop:

- processResponse gains a ctx parameter and passes it to handleFileCard. - handleFileCard gains a ctx parameter and hands it to storeReceivedFile instead of context.Background(). - Sync's round loop passes its ctx into processResponse.

Existing processResponse callers in tests pass context.Background(), the convention already used elsewhere. A new regression test drives the general sync path (processResponse) with a context cancelled mid-cascade against a fanout large enough to cross crosslinkCancelCheckStride and asserts the cascade stops early rather than crosslinking every child. check-in: 5ef8deb561 user: [email protected] tags: trunk

AfterDephantomize crosslinks a just-filled phantom and walks the whole orphan/delta graph reachable from it. It runs synchronously mid-round as a clone stores received files (sync.storeReceivedFile), before the round returns and long before the post-round CrosslinkContext sweep that already observes the deadline. That walk took no context and was bounded only by a 1,000,000- iteration safety cap, so on a large graph a clone could run for many multiples of its configured deadline with the process busy inside this exact call chain and no cancellation ever reaching it (#166).

Thread context.Context through AfterDephantomize, afterDephantomize, and crosslinkSingle, and poll ctx.Done() inside the work-stack loop every crosslinkCancelCheckStride pops -- the same stride-batched check linkBatch already uses in the whole-repository sweep. On cancellation the walk returns what it has crosslinked so far, matching how the batch sweep abandons its remaining candidates. The clone path (cloneSession.handleFile) and server push handler (handler.handleFile) pass their real request context through; the general sync loop, whose deadline is not threaded to handleFileCard, passes context.Background() to keep its behavior unchanged.

This is an additive, disjoint cancellation path: the post-round batch sweep and its existing cancellation tests are untouched.

Regression test TestAfterDephantomizeObservesCancellationMidWalk builds a root-plus-fanout delta graph larger than the stride and drives the walk with a context that goes live -> cancelled mid-walk, asserting the cascade stops early instead of crosslinking every node. check-in: a633f335c9 user: [email protected] tags: trunk

The public byte-oriented Transport discarded the reply's Content-Type, and the adapter bridging it to the internal xfer decoder hardcoded the §4.1 compressed container framing unconditionally. That assumption holds only when both peers are this implementation (Message.Encode always emits the compressed container). Against a peer that legitimately replies uncompressed -- a real Fossil clone-v3 reply uses plain card text -- the adapter read the first four bytes of the card text (`prag` from `pragma server-version`) as a big-endian length prefix and failed on exchange round 0 with "declared container length ... exceeds ... bytes".

Add an optional FramedTransport interface a Transport may also implement to report the framing signal (e.g. an HTTP Content-Type) it observed on a reply, via a per-call RoundTripFramed return value -- no shared mutable state, so a reused transport stays safe. httpTransport implements it by returning the HTTP response Content-Type. transportAdapter.Exchange decodes by that signal when the transport reports one and otherwise falls back to the compressed framing, so MockTransport / TransportFunc and any fixed-framing peer transport keep working unchanged.

The low-level xfer.Decode framing selection is unchanged. No trial-and-error framing detection is reintroduced. check-in: 21a447e620 user: [email protected] tags: trunk

* chore: bump driver submodule requires, drop root replace directives

The db/driver submodules now have real tags under the corrected go-libfossil module path (v0.3.0 for modernc and ncruces). Root no longer needs to resolve them from local directories via replace.

go mod tidy also cleared the stale old-path (danmestas/libfossil) lines the earlier replace bridge left behind in go.sum.

* chore: drop submodule replace directives, pin root require to a pseudo-version

The three submodules required the root module (which they import directly or transitively) but no root tag has ever existed under the corrected go-libfossil path -- every prior tag (v0.1.0 through v0.6.3) still declares the old danmestas/libfossil path in its own go.mod, so pinning to any of them would reproduce the exact declares-as-X-required-as-Y failure the module rename was fixing.

Pinned each submodule's require on root to a pseudo-version of the just-pushed go.mod cleanup commit (48111c1) instead of a tag -- the standard resolution for this bootstrapping case. This is a stopgap: once the next root release tag exists, these can be bumped to point at it directly instead of the pseudo-version, but there's no functional difference in the meantime.

go mod tidy on each submodule also dropped now-unneeded indirect requires (modernc/ncruces don't actually import each other; the requires existed only to satisfy the local replace directives) and cleared the stale old-path go.sum lines.

All four go.mod files are now free of replace directives. check-in: 2de18e7747 user: [email protected] tags: trunk

insertCheckinMlinks walked only the child's F-cards, so two cases diverged from canonical Fossil's add_mlink:

1. A file present in the primary parent but dropped by OMISSION from a full (non-delta) child manifest got no mlink row at all. Canonical emits a delete row (fid=0) for it. A new second pass, insertOmittedDeletions, compares the primary parent's file set against the child's effective tree and emits a fid=0 delete row for every parent file the child no longer carries and named in no F-card.

2. A rename resolved its pid against the NEW path, which is absent from the parent, so pid defaulted to 0 as if the file were a fresh add -- losing pre-rename ancestry and defeating content_deltify. The F-card walk now resolves the parent lookup against the OLD path (OldName) for a rename, so pid is the parent's pre-rename blob rid.

Verified byte-for-byte against the fossil 2.28 binary (TestFossilBinaryMlinkParity_DeletionAndRename): canonical emits BOTH a rename row keyed on the new name (pfnid = old name) AND a delete row for the vacated old name, both carrying pid = the old-path blob rid. The omission pass reproduces that pair, and applyFileCard now vacates a rename's prior name from the effective tree so the old name is correctly seen as dropped in delta manifests too. check-in: 9fed05730c user: [email protected] tags: trunk

VerifyZ validated a control artifact by hashing everything up to the last 35 bytes and comparing against the Z card it expected to find there. When a manifest arrived PGP-clearsigned, the raw blob carried a trailing "-----BEGIN PGP SIGNATURE-----" ... "-----END PGP SIGNATURE-----" block after the Z card, so the last-35-byte window landed inside the signature framing instead of on the Z card, and verification failed. deck.Parse propagated that failure, so ~95 of Fossil's own 2007-era clearsigned check-ins could not be parsed -- and any child whose primary parent was one of them lost its parent's file tree in loadPrimaryParentFiles, cascading into all-added mlink data (~98 affected children on the real corpus).

Mirror fossil's src/manifest.c remove_pgp_signature: strip the armor header (up to the blank line that ends it) and the trailing signature block, isolating the inner content the Z card is actually computed over, then verify and parse that. The stripping is an identity on non-clearsigned input (the common case) and idempotent.

The artifact's content-addressed hash is taken over the raw blob elsewhere and is untouched by this change: a clearsigned manifest still resolves to the same UUID it always has. Verified against the real Fossil check-in 61829b07 (aku, 2007-11-29), whose raw-blob SHA-1 still equals its UUID and whose inner Z-card MD5 now validates; cross-checked byte-for-byte against fossil 2.28. check-in: c1cbd26795 user: [email protected] tags: trunk, observer/otel/v0.2.0, db/driver/ncruces/v0.3.0, db/driver/modernc/v0.3.0

* manifest: run leaf/tag repair on interrupted crosslink sweep

crosslinkSweep skipped the repairLeafTable/repairTagPropagation pass whenever linkCandidatesInOrder returned an error, including a context cancellation. After #89's batched sweep, batches commit as they go, so a sweep cancelled after committing at least one batch left the plink edges and tagxref origins those batches wrote without their leaf/tag consequences -- stale until a later sweep linked something, and a later sweep whose remaining candidates all link zero artifacts skips the same gate and leaves the window open longer than one sweep.

Gate the repair on linked > 0 rather than on sweep success and return the sweep's own error after repairing, so an interrupted-but-committed sweep still recovers derived leaf/tag state on the way out. The repair runs in its own context-free transaction, so a cancelled ctx interrupts the candidate sweep but not the recovery, preserving the batched commit/rollback consistency guarantees.

Closes #143.

* manifest: join repair error with sweep error on interrupted sweep

When both the sweep is interrupted and the deferred leaf/tag repair pass fails, the repair error was dropped entirely -- only the sweep error was returned. A real repair failure (disk full, corruption, constraint violation) during a cancelled sweep would then persist the exact stale leaf/tag window this fix closes with no signal.

Join both with errors.Join so the repair failure is surfaced while errors.Is(err, context.Canceled) still holds for the three callers (clone.go, session.go, handler.go), all of which wrap with %w and none of which use exact equality. check-in: 7bc832f339 user: [email protected] tags: trunk

* manifest: batch crosslink, memoize availability, preload mlink parents (#89)

Three throughput fixes for the whole-repository crosslink sweep, none of which change its observable output. Profiling the sweep on the Fossil SCM corpus (67,503 blobs, 27,084 events) after the delta-chain-order work (#126) found the earlier bottlenecks had moved on:

- Per-candidate transactions made every SQL statement open its own WAL read-transaction, whose setup cost climbs as the uncheckpointed WAL grows -- throughput collapsed from ~200 events/sec to ~7 within a few seconds of a large sweep. The sweep now commits in batches of crosslinkBatchSize candidates so SQLite checkpoints and truncates the WAL between batches, holding read-transaction cost flat. Batch-granular durability is correct for derived data: an interrupted sweep leaves committed batches and the candidate query re-selects only what is still unlinked. CrosslinkContext is split into crosslinkSweep / collectCrosslinkCandidates / linkCandidatesInOrder / linkBatch, with the per-artifact dispatch pulled into linkArtifact and the deferral rollup into deferCheckin / logDeferredCheckins.

- missingCheckinRefs walked a blob's whole delta chain (content.IsAvailable) once per F-card of every check-in to decide deferral -- ~40% of the sweep, re-walking the same file chains over and over. content.AvailabilityCache memoizes the verdict per rid and stops a walk at the first already-decided ancestor, bounding total walk work by the number of distinct blobs rather than F-card references.

- resolveMlinkParent looked up each F-card's parent version with SELECT ... FROM mlink WHERE mid=? AND fnid=?, which SQLite answers via the fnid index -- returning a file's entire crosslinked history, so the lookup grew as the mlink table did. loadMlinkParents now reads each parent's file set once with a single mid-indexed scan and resolves every F-card from the in-memory map. It reads exactly the same rows, so the mlink output is unchanged.

The crosslink helpers now take the batch's *db.Tx directly (replacing r *repo.Repo + per-helper r.WithTx), matching the single-transaction shape of internal/verify.Rebuild and letting a whole batch commit atomically; tag application routes through tag.ApplyTagWithTx. updateAttachmentComments materializes its cursor before writing so a single connection never runs two statements at once. crosslinkSingle (dephantomize) wraps its one artifact in its own transaction.

Whole suite green (1235 tests, 35 packages, incl. dst and the fossil-binary comparison). This does not by itself land #89's "minutes not hours" target; the dominant remaining cost is a separate, pre-existing mlink over-insertion documented on the issue.

* manifest: diff crosslink mlink against the parent manifest (#89)

insertCheckinMlinks emitted one mlink row per F-card and read pmid/pid from the parent's mlink rows. On the Fossil SCM repository that is ~180x canonical's row count (a full sweep would write ~19M rows against fossil's own 104,880), and because delta-chain order (#102/#126) routinely crosslinks a child before its parent, the parent's mlink rows were usually absent, so ~97% of rows defaulted to pid=0 -- wrong parent-column data independent of the volume.

Rewrite it to diff each check-in's file list against its primary parent's expanded manifest -- the same diff canonical Fossil computes in add_mlink (src/manifest.c) and the rebuild path already does in internal/verify.rebuildMlinks -- and emit a row only for a file that was added, modified, deleted, or renamed. pmid/pid now come from the parent MANIFEST, which is always present once a check-in is eligible to link, so pid is correct regardless of crosslink order. The parent expansion runs through the sweep's shared content cache, so each parent's delta chain is walked at most once across all its children.

Measured on the Fossil SCM corpus: 127,775 mlink rows (canonical: 104,878) with pid populated on 74% of rows (was ~3%); full-repo crosslink completes in ~15 minutes rather than the prior ~hour. The residual gap is 95 PGP-clearsigned parent manifests deck.Parse cannot yet strip, whose ~98 children fall back to the old all-added behavior -- a pre-existing deck limitation tracked separately. check-in: 81fbc0be5f user: [email protected] tags: trunk

encodeCFile emitted the cfile payload as bare zlib. Fossil's on-disk cfile format, produced by blob_compress() in src/blob.c, is a 4-byte big-endian uncompressed-size prefix followed by the zlib stream, and fossil's receiver (content_put_ex) stores the wire payload verbatim into blob.content, later reading it back through blob_uncompress(), which reads that prefix to size its output before inflating. Without the prefix a real fossil client that clones from us stores every full (non-delta) artifact unreadable: fossil clone exits 0 but rebuild lands zero check-ins and manifests decode to garbage.

Prepend the 4-byte prefix in encodeCFile and count it in CSIZE, so the payload matches blob_compress byte-for-byte. The prefix value is len(Content) (the bytes being compressed -- a delta payload for delta cards), which stays distinct from the USIZE header field (the full reconstructed size). Our internal storage already used fossil's framing (blob.Compress), and the decoder already accepted both framings and rebuilt StoredBlob, so this is a wire-encode-only fix; receive and on-disk paths are unchanged.

TestCloneRealFossilWithDeltaChain now asserts six materialized check-ins instead of skipping against this issue, and a new TestCloneRealFossilFullContent drives a delta-free corpus through a real fossil clone+rebuild, asserting the check-in count and that a cloned manifest decodes to parseable manifest cards. check-in: 00e69bc9e0 user: [email protected] tags: trunk

* sync: correct the clone cfile comment to match the actual code path (#140)

The comment claimed clone "retransmits the repository's stored compressed content verbatim... blob.Load returns exactly that." That is not what the code does: blob.Load decompresses the stored blob and encodeCFile then re-compresses it with zlib before it goes on the wire. The wire result is correct because the decompress/re-compress round-trip is byte-deterministic, but nothing forwards the stored compressed bytes untouched. Correct the comment to describe the real mechanism (decompress via blob.Load, re-compress via encodeCFile) and state explicitly why the round-trip is still stable.

Documentation-only; no behavior change.

* sync: retransmit deltas during clone instead of expanding them (#141)

The clone send path expanded every deltified row to full content before putting it on the wire, giving up the delta-transmission bandwidth the xfer protocol is designed for. Restore delta transmission on the clone send path.

Send ordering. content.Deltify deltifies the OLDER artifact against the NEWER one, so a delta's source almost always has a greater rid than the delta and would not yet have been sent under the ascending-rid clone cursor. Rather than reorder the whole cursor, buildCloneArtifact walks each artifact's delta chain toward its full-content tip and emits the chain source-first, mirroring canonical fossil's recursive send_file: the tip (or the lowest still-sendable anchor) leads, each delta follows the source it names, and the requested rid is last. A per-round "sent" set skips any source a dependent's chain already pulled in ahead of its ascending position, matching fossil's per-request bag. The snapshot bound (#17) is preserved: a delta whose source lies outside the clone snapshot, is a phantom, or is private under a public clone degrades to full content so no out-of-snapshot row is ever leaked.

Wire format. A delta goes out as an uncompressed "file UUID DELTASRC SIZE" card, exactly as canonical fossil's send_delta_native does, not as a "cfile". Full content still rides a compressed cfile, preserving the #98/#113 wire-size win.

Scope. This is a bandwidth win, verified content-identical for libfossil<->libfossil clones by the self-round-trip tests, with the source-before-dependent invariant guarded by TestEmitCloneBatchSourcePrecedesDelta. The receive side already re-deltifies via content.Deltify on the crosslink path (#114), so store and tip fidelity are unchanged. Measured on a 120-artifact, 59-delta corpus, a full clone's reply bytes drop from 32,551 (expand-before-send) to 17,419, a 46% reduction, close to the 14,941-byte stored size.

This does NOT make a real fossil client's clone usable. Full content still rides a compressed cfile, which go-libfossil emits as bare zlib while fossil expects [4-byte size][zlib] framing, so a real fossil client decodes full content to garbage and rebuilds to zero check-ins. That is a separate, pre-existing bug (present on unmodified main), tracked as #152. TestCloneRealFossilWithDeltaChain asserts real usability and skips against

* test(sync): update real-fossil delta test doc to the #141 mechanism

* test(sync): prove clone usability, drop false real-fossil interop claim (#141)

TestCloneRealFossilWithDeltaChain asserted only that `fossil clone` exits 0, which a real fossil client does even when the clone is empty and its content is undecodable. Strengthen it to prove usability: rebuild the clone and count its check-ins. The count is zero -- a real fossil client decodes go-libfossil's full-content cfile payload (bare zlib) to garbage because fossil's on-disk cfile format is [4-byte size][zlib] -- so the test skips against the tracking issue (#152) instead of silently passing on corrupt content. When #152 lands the clone materializes all six check-ins and the test passes for real.

Also correct the doc comment and the handler.go send-path comment, which claimed #141's send-order + uncompressed-delta-card change keeps a real fossil 2.28 client's rebuild from failing. It does not: that change is a libfossil<->libfossil bandwidth win (verified content-identical by the self-round-trip tests); real-fossil interop for delta-bearing clones remains blocked on the separate cfile-framing bug (#152), present on unmodified main. check-in: 5f66319ea3 user: [email protected] tags: trunk

The buffer-reuse fix in #145/#150 cut the cost of expanding one delta chain once. It did nothing for callers that expand many versions of the same artifact in a single operation: each bare content.Expand still replayed the whole O(depth) chain, so a walk over N overlapping revisions cost O(N^2) blob reads even though content.Cache already existed for exactly this pattern.

Audited every non-test content.Expand caller. Two access patterns genuinely repeat over overlapping delta chains and are now routed through a per-operation content.Cache:

- annotate walks every revision of one file along the parent chain. content_deltify stores each older file blob as a delta against its successor, so the per-revision expansions are all walks of the same few chains. One cache threaded through the walk (file blob + manifest listing via new manifest.ListFilesCached/GetManifestCached, both nil-passthrough so existing callers are unchanged) turns it linear. Measured on a deltified 150-revision file: 1.60s -> 0.65s (2.5x), 764k -> 308k allocs; the win grows with history depth.

- verify.Verify and verify.Rebuild sweep every blob more than once (Rebuild: checkBlobs + rebuildManifests + rebuildTags x2). A single cache shared across the pass walks each chain once instead of once per sweep and amortizes overlap within each sweep. Measured on a 150-revision two-file repo: Rebuild 3.34s -> 0.068s (48x), 4.59M -> 98k allocs.

Callers that only ever touch the tip or make a single pass over distinct blobs (branch create, merge-config load, R-card / delta manifest build, MergeParentFiles, crosslinkSingle, shun undelta, pairwise Diff) are left on bare Expand, each with a comment stating why a cache cannot amortize their access.

No behavior change: expansion output is byte-identical, verified across the modernc and ncruces drivers with REQUIRE_FOSSIL_BIN=1 cross-checks and the DST suite. check-in: 0fd2a507b8 user: [email protected] tags: trunk

delta.Apply's one-time capacity upgrade (#127) only engages above the 64 KiB threshold, so it is a no-op for the typical source file under 64 KiB. But below the threshold Apply already sizes its output buffer exactly, so there was never a 5x overhead there to remove -- extending the threshold buys nothing. The real per-link cost of a chain walk was elsewhere, and a profile of a plain Expand of a deep sub-64 KiB chain shows it: a fresh ~32 KiB zlib history window allocated per delta load (~27% of bytes), one full-size Apply output buffer per link (~16%), and a fresh inflate buffer per link. All three are O(depth) allocations for a single cold Expand, which is O(depth), making a full-history walk O(n^2).

Attack the per-link constant inside the one choke point every expansion goes through, content.expandChain, leaving the on-disk chain shape untouched:

- delta.ApplyInto(dst, source, delta) reconstructs into a caller-supplied buffer, reusing its capacity. Apply is now ApplyInto(nil, ...), unchanged. A reused buffer's capacity is proven prior work, not a wire-header claim, so it is a safe starting size; the anti-DoS growth path (growOutputCap, keyed off len(delta) not the targetLen header) is preserved for buffers a target outgrows. - blob.Inflater loads a run of blobs through one zlib reader, reset per blob instead of rebuilt, paying the ~32 KiB window setup once per chain. Each Load still returns a freshly owned buffer, so no returned-slice-aliasing contract leaks to callers. - expandChain's plain-Expand replay (put == nil) ping-pongs two output buffers -- each link's target reuses the buffer from two links back, whose content was already consumed as the previous link's source, so dst and source never alias. The caching path (put != nil) is unchanged: put takes ownership of every materialized buffer, so those cannot be reused underneath it, but it still shares the reused zlib reader for its delta loads.

Measured on Apple M4 Pro, BenchmarkExpandOldestVersion, median of 3, 40x:

sub-64 KiB (~26 KB file), Expand of the oldest revision: depth 11: 883 KB -> 232 KB/op (3.8x), 1.12 ms -> 0.90 ms depth 49: 3.57 MB -> 341 KB/op (10.5x), 4.55 ms -> 2.03 ms depth 199: 14.2 MB -> 772 KB/op (18.4x), 14.84 ms -> 7.25 ms

>64 KiB (~206 KB file), no regression -- ping-pong + reader reuse help it too: depth 11: 4.11 MB -> 1.27 MB/op (3.2x), 3.40 ms -> 2.74 ms depth 49: 16.5 MB -> 1.81 MB/op (9.1x), 13.02 ms -> 7.69 ms depth 199: 65.2 MB -> 3.92 MB/op (16.6x), 46.3 ms -> 26.6 ms

Sub-64 KiB allocation is now near-flat in depth instead of linear in depth*targetLen. expand_reuse_test.go guards the reuse; delta and blob gain allocation-reduction unit tests. check-in: a2bcca2535 user: [email protected] tags: trunk

A stock fossil 2.28 client cloning a go-libfossil server transferred ~2.06x the repository's stored size, while the server's own response bodies measure 1.04x and a fossil-serving-fossil control measures 1.03x. The gap was a full second content-transfer pass, not a card-encoding problem.

Root cause is card order. Canonical's client re-issues `clone 3 SEQNO` every round it sees a server `push` card while its clone cursor is still positive (xfer.c:2706), and it learns the cursor reached zero only from the clone_seqno card. Canonical's server therefore emits the batch, then clone_seqno (xfer.c:1571), then push (xfer.c:1577). go-libfossil emitted push *first*, so the client queued one more clone request before it processed the terminal clone_seqno 0 — and the server served the entire repository a second time.

Relocate the clone-mode push-card emission to after processDataCards so it trails the batch's clone_seqno card, matching canonical's order. The clone now completes in a single content pass: re-measured end-to-end at 1.03x stored, identical to the fossil control (1.00x ratio). check-in: 1fd6c06a65 user: [email protected] tags: trunk

docs/site was the one module in the tree with no CI compile gate. It has its own go.mod but zero Go source of its own -- the module exists only to import the hextra theme via Hugo Modules -- so `go build ./...` there matches no packages and gates nothing (and `go vet ./...` fails on "no packages to vet"). The equivalent compile gate for a Hugo module is a Hugo build: it resolves the module import, compiles templates, and renders content, failing loud on a broken import or template.

Add a test-docs job that installs pinned Hugo Extended and runs `hugo --gc --minify` in docs/site, matching how the driver/observer submodules each get their own fail-loud step. check-in: be89e6397b user: [email protected] tags: trunk

* fix: emit renames as a single F-card with the "w" placeholder (#51)

buildCommitFiles seeded its working set from the parent manifest and layered vfile changes on top, but never consulted vfile.origname. A pure rename has identical content, so ScanChanges clears its chnged flag; the new name then never entered the working set while the old name carried forward from the parent, so the manifest emitted the file's content under its old pathname and dropped the rename entirely.

Drive renames off vfile.origname, mirroring canonical Fossil's manifest query (src/checkin.c:1939): applyRenames retires the old pathname and records it on the surviving new-name entry, so a rename is emitted exactly once. The unchanged-content pass then loads the blob and preserves OldName.

Serialize the rename card to match stock Fossil: force a "w" permission placeholder when a renamed file's perm would otherwise be empty, keeping the prior-name field in its 4th positional slot (src/checkin.c:1999). Thread OldName through manifest.File and storeFileBlobs so the commit path can carry it.

Verified the exact card "F greet.txt <uuid> w hello.txt" against a real fossil binary for the same rename scenario.

* fix: never emit a renamed-then-deleted file in the commit manifest

applyRenames enumerated every vfile row with origname != pathname, including rows marked deleted, and unconditionally seeded a new-name entry for them. A file that was renamed and then unmanaged therefore committed as a rename F-card under its new name. The deleted-files pass could not catch it: it only knows the file's current pathname, so deleting the new name from a map keyed on parent-manifest pathnames was a no-op and nothing retired the old name.

Canonical Fossil excludes deleted rows from its manifest query outright (WHERE NOT deleted OR NOT is_selected). Match that: a deleted renamed row now retires both names and emits nothing, keeping the existing guard that leaves the old name alone when a live vfile row has reclaimed it.

Also move the deck marshaller's prior-name field inside the UUID guard. The field is positional and only meaningful after a hash; a deletion card carrying one would serialize as "F name oldname", with the old name misread as the UUID.

Tests: rename-then-delete emits no card under either name; a rename chain that recycles a freed pathname emits both cards with the right prior names and content. check-in: 29c2c5c6f5 user: [email protected] tags: trunk

* content: add delta-chain depth benchmark for #85 re-measurement

BenchmarkExpandOldestVersion expands the oldest version of a ~26 KB, 400-line file at 12/50/200 revisions, mirroring the table in #85. The chain is built through content.Deltify so the on-disk shape matches what content_deltify produces: newest version whole, each older version a delta against the one newer.

Reports ns/op plus B/op and allocs/op so the per-link allocation profile the issue is about is directly observable.

* content: straddle delta.Apply's 64 KiB cap in the chain-depth benchmark

BenchmarkExpandOldestVersion measured a ~26 KB file, which sits below delta.Apply's deltaInitialCap: for targets that small the output buffer is allocated at exactly targetLen and the capped/growOutputCap path is never entered, so the arm cannot observe #81/#82 at all.

Parameterise the chain builder on file size and add a ~206 KB arm above the cap, so the two arms isolate the threshold rather than leaving it inferred from reading delta.go. Also move the chain-depth metric after the loop -- ResetTimer deletes user-reported metrics, so the depth reported before it never reached the output.

Refs #85

---------

Co-authored-by: Dan Mestas <[email protected]> check-in: c3338c4d38 user: [email protected] tags: trunk

The clone command hardcoded a 10-minute context deadline with no way to raise or disable it, and the deadline did not reliably interrupt a slow clone.

- RepoCloneCmd gains a --timeout flag (default 10m, matching the old behavior; 0 disables the deadline for very large repositories). cloneContext branches away from context.WithTimeout for a non-positive timeout -- WithTimeout(parent, 0) yields an already-expired deadline, which would abort instantly the very clone the flag exists to unbound.

- The whole-repository crosslink sweep now observes context cancellation. It runs after the exchange-round loop, so it had no round boundary of its own at which the deadline could fire -- on a large repository it could run long past the deadline uninterrupted. manifest.CrosslinkContext checks the context every 256 candidates; manifest.Crosslink stays a background-context wrapper so its ~30 existing callers are unchanged. processResponse gains the same batched check so a large single-round batch is interruptible too, and the sync session's and the serve handler's post-convergence sweeps are wired to their contexts for the same reason.

Each cancellation point is pinned by a test that fails without it: the crosslink and processResponse tests drive a context that survives the first poll and dies before the second, so only the batched check can observe it; the crosslink-sweep test cancels after a zero-card final round, so nothing before the sweep is left to notice.

Also adds a clone -> serve -> reclone fidelity regression guard (#114), built with the canonical fossil binary rather than this library's Commit. Commit does not deltify, so a corpus built through it holds zero delta rows and never reaches the serve side's deltified-row expansion branch -- a green result from such a corpus proves nothing about #114. The guard now asserts the source holds delta rows before measuring anything, and compares the second-generation clone's tip, artifact count, blob-store size, delta-row count and checked-out tree content against the source.

Measured against that corpus (71 artifacts, 54 delta rows), #114 does not reproduce: gen1 and gen2 both hold the same 54 delta rows, the same tip and artifact count, a 1.009 store ratio, identical checked-out content, and pass fossil test-integrity. The receive path re-deltifies what the serve side expanded, via content.Deltify on the crosslink mlink path, so the expansion the serve side performs does not survive into the store. check-in: a02c9572da user: [email protected] tags: trunk

Adds a wire-efficiency regression guard the issue flagged as missing: correctness-only round-trip tests never asserted card type or wire cost, so the file-card-instead-of-cfile amplification #98 fixed could silently return. TestCloneServesCompressedNotExpandedContent serves a small compressible corpus, clones it over the real HTTP transport, and fails if the server emits any expanded file card or if served bytes exceed 3x the repository's stored size. Card type is the load-bearing signal: the reply container is compressed, so a raw-file-card regression on compressible content hides inside a byte ratio -- the 3x bound is only a coarse backstop.

Also adds a WIRE_MEASURE-gated manual harness that re-measures serve-side amplification against a real fossil client through a byte-counting proxy, with a fossil-serving-fossil control. Current main: 2.06x stored end to end (server response bodies alone are 1.04x, matching the fossil control's 1.03x), down from the 17.9x this issue reported. check-in: 1a73119ed4 user: [email protected] tags: trunk

FossilEncode/FossilDecode implemented only 3 of the 8 escapes the artifact/sync format §4.3 mandates (\s, \n, \\). A field value carrying a real TAB, CR, VT, FF or NUL byte -- as canonical fossil writes it -- was mis-decoded and no longer re-marshalled to its source wire bytes, changing the artifact hash. Unrecognized escapes were also left literal instead of dropping the backslash per spec.

Add the remaining escapes (\t \r \v \f \0) to both directions and make an unrecognized escape drop its backslash, matching canonical fossil fossilize/defossilize (verified against fossil 2.28). Every caller -- A/C/H/L/J/F/T/U cards and the xfer cards -- shares these functions and gets the fix.

Closes #119 check-in: 1bfa6b2094 user: [email protected] tags: trunk

The test step runs `go test $(go list ./... | grep -v '/dst')`. When a package fails to compile, `go list` prints an error to stderr and omits that package from stdout, so it silently drops out of the test run rather than failing CI. `go build ./cmd/libfossil/` only covered the CLI binary's dependency closure, which missed internal/deck when it broke.

Add `go build ./...` ahead of the test step on both driver-tag legs (modernc and test_ncruces) so any package that fails to compile is a loud, fail-fast build error. This subsumes the narrower cmd/libfossil build, so that step is removed rather than kept alongside. check-in: fa558f254d user: [email protected] tags: trunk

The xfer HTTP server let net/http frame large clone/sync replies as Transfer-Encoding: chunked with no Content-Length. Release-tagged fossil <=2.23 reads the reply length only from Content-Length (src/http.c leaves iLength negative otherwise) and aborts with "server did not reply" on any chunked reply above ~256KB, so those clients could not clone a repository larger than that from a libfossil server.

writeXferResponse already materialises the whole response via Encode before the first write, so the exact body length is known for free. Set Content-Length explicitly; net/http then never falls back to chunked.

The fix is unconditional rather than negotiated. The triage proposed sniffing a client capability and keeping chunked for capable clients to avoid the buffering cost of Content-Length. That premise does not hold here: the server never streams -- Encode always returns a complete slice -- so chunked saves no memory over Content-Length, and #88's per-clone bound (DefaultCloneBatchBytes) still caps each reply either way. Content-Length is exactly what canonical fossil's own server emits, so it is strictly the more compatible framing for every client, including builds whose version banner cannot be trusted (#86).

Tests: a reconstructed <=2.23 client that requires a non-negative Content-Length and rejects chunked framing on a >256KB clone response (the discriminating test, since real fossil 2.28 tolerates chunked and cannot reproduce the failure), plus a modern-client no-regression clone of the same corpus. The existing real-fossil multi-batch clone interop test continues to pass unchanged.

Fixes #101 check-in: b3139353e1 user: [email protected] tags: trunk

* Wire manifest.AfterDephantomize into the phantom fill path

AfterDephantomize was a faithful, tested port of Fossil's after_dephantomize (content.c:389-456) that nothing in production ever called — it crosslinked correctly in isolation but sat disconnected from the real receive path.

storeResolvedContent fills a phantom row in place when the full content for a previously-unknown UUID arrives (matching canonical's content_put_ex filling a phantom at content.c:626), but stopped short of crosslinking it or any delta children waiting on it. Return the filled rid from storeResolvedContent and call AfterDephantomize on it from storeReceivedFile once the fill transaction has committed — outside the transaction, since AfterDephantomize opens its own connection via repo.Repo.DB() and calling it from inside the tx's closure would contend with the still-open write lock.

Added an end-to-end regression test that delivers a checkin manifest as a delta against a base that hasn't arrived yet, then delivers the base, and asserts the manifest is crosslinked (its event row exists) immediately — with no explicit call to manifest.Crosslink anywhere in the test. This proves AfterDephantomize fires on the real fill path, not only under its own unit tests.

* Log crosslink failures during AfterDephantomize instead of swallowing them

A crosslink error while filling a phantom blob (or one of its orphan/delta dependents) previously vanished silently, leaving storeReceivedFile report success with the artifact filled but un-crosslinked. Surface it via slog.Warn so real corruption on the fill path is at least visible, matching the non-blocking nit from review. check-in: 94919ca357 user: [email protected] tags: trunk

* fix: select xfer framing by Content-Type (#106); bound clone rounds by the client's decode limit (#109)

seeing which parsed, which made a decompression failure indistinguishable from "wrong framing" (the root of #104). Select framing from the §4 Content-Type instead: application/x-fossil is the §4.1 compressed container, every other type (including application/x-fossil-uncompressed, used by clone-v3 replies) is plain card text — the same rule canonical fossil applies. Verified against fossil 2.28: pull replies arrive as the prefixed container, clone replies as plain text, and no reply is ever an unprefixed raw zlib stream. The former unprefixed-zlib framing, checked first on the unverified belief that fossil emits it, was removed along with the length-prefix/zlib-header aliasing guard that only existed to disambiguate the two compressed framings trial-dispatch conflated. A compressed body that fails to inflate is now reported as the fault it is, never re-fed to the card parser. xfer.Decode takes the Content-Type; byte-oriented callers with no header pass the compressed type explicitly.

crossing one whole, so a round carried budget + one whole artifact. When sub-budget filler preceded a large artifact, that sum exceeded what the client could decode (xfer.MaxDecompressedBytes) and the clone failed with a spurious "declared container length exceeds" error whose threshold depended on incidental filler size. The server now flushes the round and sends an over-bound artifact alone when adding it would cross the client's decode bound, so clonability is a property of the artifact alone. The 2*budget compile-time guard now certifies what it appears to.

* fix: reject unrecognized xfer Content-Type with 415 (#106)

A POST whose Content-Type was absent or unrecognized fell through to the plaintext card decoder, which is exactly the #104 failure shape: possibly- compressed binary fed to the card parser. Request bodies come from untrusted peers, so a body carrying no framing signal is rejected rather than guessed at. Real fossil always sets the header; no conforming peer is affected.

xfer.KnownContentType shares the media-type parsing with the framing check, so the server and Decode agree on what a type means.

Also runs the #109 filler subtests in parallel — serially they pushed internal/sync past the 120s per-package timeout CI uses. check-in: f1236bd031 user: [email protected] tags: trunk

Issue #60's triage was stale: the live truncation bug it describes was already closed on main by 11967ff (#62), which added millisecond rounding to timeToJulian. What remained was a coverage gap, not a defect.

An earlier attempt on this branch also normalized the float64 branch onto the millisecond grid, on the theory that a driver-independent helper is strictly better. That was wrong, and this commit does not do it. A scanned float64 IS the value stored in the row, so snapping it can only move it away from what the row holds. Timeline's pagination cursor depends on the opposite property: its predicate `e.mtime < ? OR (e.mtime = ? AND b.rid < ?)` requires the cursor to compare exactly equal to the row it came from. SQLite's own julianday() computes iJD/86400000.0, a different expression from TimeToJulian's julianEpoch + ms/86400000.0, and the two land one ulp apart on a real fraction of instants — so normalizing alters mtimes written by upstream Fossil, the library's primary real-world input. Raised, the cursor row repeats as the next page's first row; lowered, rows are skipped. Both are silent wrong answers.

The float64 branch therefore stays verbatim, and its doc comment now records why that exactness is load-bearing rather than incidental. The same reasoning applies to tag propagation's `tagxref.mtime < ?`, where SQLite compares the bound value against raw stored floats: an exact bound is the correct one there too, so no normalized variant is introduced for it either.

The one genuine transform stays where it belongs, in the time.Time branch, which must round because ncruces hands back DATETIME columns carrying its own sub-millisecond conversion noise. Its comment now also records the limit of that recovery: an mtime written at finer than millisecond resolution cannot be reproduced through a time.Time at all, and rounding the float64 branch to match would discard precision on both drivers rather than recover any.

Non-functional cleanup: millisPerDay and julianDayFromMillis replace two open-coded copies of the same expression.

Coverage added (no build tag; these run under both drivers): - db: TestScanJulianDayFloatBranchIsBitExact sweeps julianday()-computed mtimes and asserts each scans back bit-for-bit, with a guard that fails the test if the sweep ever stops containing divergent values. Verified to fail against the normalizing version and pass against this one. - db: TestScanJulianDayIsDriverIndependent proves the float64 and time.Time representations of one millisecond-aligned write converge, with noise in both directions. TestScanJulianDaySentinelZero pins the "no mtime" sentinel that bisect's `mtime != 0` guard depends on. - tag: TestPropagateAllOrderingAdjacentMillisecond drives the `tagxref.mtime < ?` decision through ScanJulianDay with adjacent-millisecond mtimes; it fails under a truncating conversion and passes under the rounding one, and passes under modernc regardless — the reason this class went unnoticed twice.

Consumer classification (ScanJulianDay): - manifest/timeline.go NewCursor: ordering/equality, requires exactness - tag/propagate.go PropagateAll -> `tagxref.mtime < ?`: ordering (now covered) - tag/propagate.go childMtime -> mtimeQueue.Less: ordering, traversal-order only (visited set makes it result-neutral) - manifest/log.go, bisect/bisect.go: display-only

Known limitation, left alone deliberately: julianToTime truncates where timeToJulian rounds, so the two are not inverses at sub-millisecond resolution. Pre-existing, now documented; ScanTime's consumers are display paths with no ordering boundary. This commit does not widen the gap. check-in: 78a793dd83 user: [email protected] tags: trunk