Timeline

* manifest: batch crosslink, memoize availability, preload mlink parents (#89)

Three throughput fixes for the whole-repository crosslink sweep, none of which change its observable output. Profiling the sweep on the Fossil SCM corpus (67,503 blobs, 27,084 events) after the delta-chain-order work (#126) found the earlier bottlenecks had moved on:

- Per-candidate transactions made every SQL statement open its own WAL read-transaction, whose setup cost climbs as the uncheckpointed WAL grows -- throughput collapsed from ~200 events/sec to ~7 within a few seconds of a large sweep. The sweep now commits in batches of crosslinkBatchSize candidates so SQLite checkpoints and truncates the WAL between batches, holding read-transaction cost flat. Batch-granular durability is correct for derived data: an interrupted sweep leaves committed batches and the candidate query re-selects only what is still unlinked. CrosslinkContext is split into crosslinkSweep / collectCrosslinkCandidates / linkCandidatesInOrder / linkBatch, with the per-artifact dispatch pulled into linkArtifact and the deferral rollup into deferCheckin / logDeferredCheckins.

- missingCheckinRefs walked a blob's whole delta chain (content.IsAvailable) once per F-card of every check-in to decide deferral -- ~40% of the sweep, re-walking the same file chains over and over. content.AvailabilityCache memoizes the verdict per rid and stops a walk at the first already-decided ancestor, bounding total walk work by the number of distinct blobs rather than F-card references.

- resolveMlinkParent looked up each F-card's parent version with SELECT ... FROM mlink WHERE mid=? AND fnid=?, which SQLite answers via the fnid index -- returning a file's entire crosslinked history, so the lookup grew as the mlink table did. loadMlinkParents now reads each parent's file set once with a single mid-indexed scan and resolves every F-card from the in-memory map. It reads exactly the same rows, so the mlink output is unchanged.

The crosslink helpers now take the batch's *db.Tx directly (replacing r *repo.Repo + per-helper r.WithTx), matching the single-transaction shape of internal/verify.Rebuild and letting a whole batch commit atomically; tag application routes through tag.ApplyTagWithTx. updateAttachmentComments materializes its cursor before writing so a single connection never runs two statements at once. crosslinkSingle (dephantomize) wraps its one artifact in its own transaction.

Whole suite green (1235 tests, 35 packages, incl. dst and the fossil-binary comparison). This does not by itself land #89's "minutes not hours" target; the dominant remaining cost is a separate, pre-existing mlink over-insertion documented on the issue.

* manifest: diff crosslink mlink against the parent manifest (#89)

insertCheckinMlinks emitted one mlink row per F-card and read pmid/pid from the parent's mlink rows. On the Fossil SCM repository that is ~180x canonical's row count (a full sweep would write ~19M rows against fossil's own 104,880), and because delta-chain order (#102/#126) routinely crosslinks a child before its parent, the parent's mlink rows were usually absent, so ~97% of rows defaulted to pid=0 -- wrong parent-column data independent of the volume.

Rewrite it to diff each check-in's file list against its primary parent's expanded manifest -- the same diff canonical Fossil computes in add_mlink (src/manifest.c) and the rebuild path already does in internal/verify.rebuildMlinks -- and emit a row only for a file that was added, modified, deleted, or renamed. pmid/pid now come from the parent MANIFEST, which is always present once a check-in is eligible to link, so pid is correct regardless of crosslink order. The parent expansion runs through the sweep's shared content cache, so each parent's delta chain is walked at most once across all its children.

Measured on the Fossil SCM corpus: 127,775 mlink rows (canonical: 104,878) with pid populated on 74% of rows (was ~3%); full-repo crosslink completes in ~15 minutes rather than the prior ~hour. The residual gap is 95 PGP-clearsigned parent manifests deck.Parse cannot yet strip, whose ~98 children fall back to the old all-added behavior -- a pre-existing deck limitation tracked separately. check-in: 81fbc0be5f user: [email protected] tags: trunk

* sync: correct the clone cfile comment to match the actual code path (#140)

The comment claimed clone "retransmits the repository's stored compressed content verbatim... blob.Load returns exactly that." That is not what the code does: blob.Load decompresses the stored blob and encodeCFile then re-compresses it with zlib before it goes on the wire. The wire result is correct because the decompress/re-compress round-trip is byte-deterministic, but nothing forwards the stored compressed bytes untouched. Correct the comment to describe the real mechanism (decompress via blob.Load, re-compress via encodeCFile) and state explicitly why the round-trip is still stable.

Documentation-only; no behavior change.

* sync: retransmit deltas during clone instead of expanding them (#141)

The clone send path expanded every deltified row to full content before putting it on the wire, giving up the delta-transmission bandwidth the xfer protocol is designed for. Restore delta transmission on the clone send path.

Send ordering. content.Deltify deltifies the OLDER artifact against the NEWER one, so a delta's source almost always has a greater rid than the delta and would not yet have been sent under the ascending-rid clone cursor. Rather than reorder the whole cursor, buildCloneArtifact walks each artifact's delta chain toward its full-content tip and emits the chain source-first, mirroring canonical fossil's recursive send_file: the tip (or the lowest still-sendable anchor) leads, each delta follows the source it names, and the requested rid is last. A per-round "sent" set skips any source a dependent's chain already pulled in ahead of its ascending position, matching fossil's per-request bag. The snapshot bound (#17) is preserved: a delta whose source lies outside the clone snapshot, is a phantom, or is private under a public clone degrades to full content so no out-of-snapshot row is ever leaked.

Wire format. A delta goes out as an uncompressed "file UUID DELTASRC SIZE" card, exactly as canonical fossil's send_delta_native does, not as a "cfile". Full content still rides a compressed cfile, preserving the #98/#113 wire-size win.

Scope. This is a bandwidth win, verified content-identical for libfossil<->libfossil clones by the self-round-trip tests, with the source-before-dependent invariant guarded by TestEmitCloneBatchSourcePrecedesDelta. The receive side already re-deltifies via content.Deltify on the crosslink path (#114), so store and tip fidelity are unchanged. Measured on a 120-artifact, 59-delta corpus, a full clone's reply bytes drop from 32,551 (expand-before-send) to 17,419, a 46% reduction, close to the 14,941-byte stored size.

This does NOT make a real fossil client's clone usable. Full content still rides a compressed cfile, which go-libfossil emits as bare zlib while fossil expects [4-byte size][zlib] framing, so a real fossil client decodes full content to garbage and rebuilds to zero check-ins. That is a separate, pre-existing bug (present on unmodified main), tracked as #152. TestCloneRealFossilWithDeltaChain asserts real usability and skips against

* test(sync): update real-fossil delta test doc to the #141 mechanism

* test(sync): prove clone usability, drop false real-fossil interop claim (#141)

TestCloneRealFossilWithDeltaChain asserted only that `fossil clone` exits 0, which a real fossil client does even when the clone is empty and its content is undecodable. Strengthen it to prove usability: rebuild the clone and count its check-ins. The count is zero -- a real fossil client decodes go-libfossil's full-content cfile payload (bare zlib) to garbage because fossil's on-disk cfile format is [4-byte size][zlib] -- so the test skips against the tracking issue (#152) instead of silently passing on corrupt content. When #152 lands the clone materializes all six check-ins and the test passes for real.

Also correct the doc comment and the handler.go send-path comment, which claimed #141's send-order + uncompressed-delta-card change keeps a real fossil 2.28 client's rebuild from failing. It does not: that change is a libfossil<->libfossil bandwidth win (verified content-identical by the self-round-trip tests); real-fossil interop for delta-bearing clones remains blocked on the separate cfile-framing bug (#152), present on unmodified main. check-in: 5f66319ea3 user: [email protected] tags: trunk