| Comment: | sync: retransmit deltas during clone instead of expanding them (#154)
* sync: correct the clone cfile comment to match the actual code path (#140) The comment claimed clone "retransmits the repository's stored compressed content verbatim... blob.Load returns exactly that." That is not what the code does: blob.Load decompresses the stored blob and encodeCFile then re-compresses it with zlib before it goes on the wire. The wire result is correct because the decompress/re-compress round-trip is byte-deterministic, but nothing forwards the stored compressed bytes untouched. Correct the comment to describe the real mechanism (decompress via blob.Load, re-compress via encodeCFile) and state explicitly why the round-trip is still stable. Documentation-only; no behavior change. * sync: retransmit deltas during clone instead of expanding them (#141) The clone send path expanded every deltified row to full content before putting it on the wire, giving up the delta-transmission bandwidth the xfer protocol is designed for. Restore delta transmission on the clone send path. Send ordering. content.Deltify deltifies the OLDER artifact against the NEWER one, so a delta's source almost always has a greater rid than the delta and would not yet have been sent under the ascending-rid clone cursor. Rather than reorder the whole cursor, buildCloneArtifact walks each artifact's delta chain toward its full-content tip and emits the chain source-first, mirroring canonical fossil's recursive send_file: the tip (or the lowest still-sendable anchor) leads, each delta follows the source it names, and the requested rid is last. A per-round "sent" set skips any source a dependent's chain already pulled in ahead of its ascending position, matching fossil's per-request bag. The snapshot bound (#17) is preserved: a delta whose source lies outside the clone snapshot, is a phantom, or is private under a public clone degrades to full content so no out-of-snapshot row is ever leaked. Wire format. A delta goes out as an uncompressed "file UUID DELTASRC SIZE" card, exactly as canonical fossil's send_delta_native does, not as a "cfile". Full content still rides a compressed cfile, preserving the #98/#113 wire-size win. Scope. This is a bandwidth win, verified content-identical for libfossil<->libfossil clones by the self-round-trip tests, with the source-before-dependent invariant guarded by TestEmitCloneBatchSourcePrecedesDelta. The receive side already re-deltifies via content.Deltify on the crosslink path (#114), so store and tip fidelity are unchanged. Measured on a 120-artifact, 59-delta corpus, a full clone's reply bytes drop from 32,551 (expand-before-send) to 17,419, a 46% reduction, close to the 14,941-byte stored size. This does NOT make a real fossil client's clone usable. Full content still rides a compressed cfile, which go-libfossil emits as bare zlib while fossil expects [4-byte size][zlib] framing, so a real fossil client decodes full content to garbage and rebuilds to zero check-ins. That is a separate, pre-existing bug (present on unmodified main), tracked as #152. TestCloneRealFossilWithDeltaChain asserts real usability and skips against * test(sync): update real-fossil delta test doc to the #141 mechanism * test(sync): prove clone usability, drop false real-fossil interop claim (#141) TestCloneRealFossilWithDeltaChain asserted only that `fossil clone` exits 0, which a real fossil client does even when the clone is empty and its content is undecodable. Strengthen it to prove usability: rebuild the clone and count its check-ins. The count is zero -- a real fossil client decodes go-libfossil's full-content cfile payload (bare zlib) to garbage because fossil's on-disk cfile format is [4-byte size][zlib] -- so the test skips against the tracking issue (#152) instead of silently passing on corrupt content. When #152 lands the clone materializes all six check-ins and the test passes for real. Also correct the doc comment and the handler.go send-path comment, which claimed #141's send-order + uncompressed-delta-card change keeps a real fossil 2.28 client's rebuild from failing. It does not: that change is a libfossil<->libfossil bandwidth win (verified content-identical by the self-round-trip tests); real-fossil interop for delta-bearing clones remains blocked on the separate cfile-framing bug (#152), present on unmodified main. |
|---|---|
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA3-256: |
5f66319ea3a4b9d006e3f957e7013fd5 |
| User & Date: | [email protected] on 2026-07-24 16:07:43.000 |
| Other Links: | manifest | tags |
|
2026-07-24
| ||
| 17:06 |
fix: frame cfile wire payload as fossil on-disk blob [4-byte size][zlib] (#152) (#155)
encodeCFile emitted the cfile payload as bare zlib. Fossil's on-disk cfile format, produced by blob_compress() in src/blob.c, is a 4-byte big-endian uncompressed-size prefix followed by the zlib stream, and fossil's receiver (content_put_ex) stores the wire payload verbatim into blob.content, later reading it back through blob_uncompress(), which reads that prefix to size its output before inflating. Without the prefix a real fossil client that clones from us stores every full (non-delta) artifact unreadable: fossil clone exits 0 but rebuild lands zero check-ins and manifests decode to garbage. Prepend the 4-byte prefix in encodeCFile and count it in CSIZE, so the payload matches blob_compress byte-for-byte. The prefix value is len(Content) (the bytes being compressed -- a delta payload for delta cards), which stays distinct from the USIZE header field (the full reconstructed size). Our internal storage already used fossil's framing (blob.Compress), and the decoder already accepted both framings and rebuilt StoredBlob, so this is a wire-encode-only fix; receive and on-disk paths are unchanged. TestCloneRealFossilWithDeltaChain now asserts six materialized check-ins instead of skipping against this issue, and a new TestCloneRealFossilFullContent drives a delta-free corpus through a real fossil clone+rebuild, asserting the check-in count and that a cloned manifest decodes to parseable manifest cards. check-in: 00e69bc9e0 user: [email protected] tags: trunk | |
| 16:07 |
sync: retransmit deltas during clone instead of expanding them (#154)
* sync: correct the clone cfile comment to match the actual code path (#140) The comment claimed clone "retransmits the repository's stored compressed content verbatim... blob.Load returns exactly that." That is not what the code does: blob.Load decompresses the stored blob and encodeCFile then re-compresses it with zlib before it goes on the wire. The wire result is correct because the decompress/re-compress round-trip is byte-deterministic, but nothing forwards the stored compressed bytes untouched. Correct the comment to describe the real mechanism (decompress via blob.Load, re-compress via encodeCFile) and state explicitly why the round-trip is still stable. Documentation-only; no behavior change. * sync: retransmit deltas during clone instead of expanding them (#141) The clone send path expanded every deltified row to full content before putting it on the wire, giving up the delta-transmission bandwidth the xfer protocol is designed for. Restore delta transmission on the clone send path. Send ordering. content.Deltify deltifies the OLDER artifact against the NEWER one, so a delta's source almost always has a greater rid than the delta and would not yet have been sent under the ascending-rid clone cursor. Rather than reorder the whole cursor, buildCloneArtifact walks each artifact's delta chain toward its full-content tip and emits the chain source-first, mirroring canonical fossil's recursive send_file: the tip (or the lowest still-sendable anchor) leads, each delta follows the source it names, and the requested rid is last. A per-round "sent" set skips any source a dependent's chain already pulled in ahead of its ascending position, matching fossil's per-request bag. The snapshot bound (#17) is preserved: a delta whose source lies outside the clone snapshot, is a phantom, or is private under a public clone degrades to full content so no out-of-snapshot row is ever leaked. Wire format. A delta goes out as an uncompressed "file UUID DELTASRC SIZE" card, exactly as canonical fossil's send_delta_native does, not as a "cfile". Full content still rides a compressed cfile, preserving the #98/#113 wire-size win. Scope. This is a bandwidth win, verified content-identical for libfossil<->libfossil clones by the self-round-trip tests, with the source-before-dependent invariant guarded by TestEmitCloneBatchSourcePrecedesDelta. The receive side already re-deltifies via content.Deltify on the crosslink path (#114), so store and tip fidelity are unchanged. Measured on a 120-artifact, 59-delta corpus, a full clone's reply bytes drop from 32,551 (expand-before-send) to 17,419, a 46% reduction, close to the 14,941-byte stored size. This does NOT make a real fossil client's clone usable. Full content still rides a compressed cfile, which go-libfossil emits as bare zlib while fossil expects [4-byte size][zlib] framing, so a real fossil client decodes full content to garbage and rebuilds to zero check-ins. That is a separate, pre-existing bug (present on unmodified main), tracked as #152. TestCloneRealFossilWithDeltaChain asserts real usability and skips against * test(sync): update real-fossil delta test doc to the #141 mechanism * test(sync): prove clone usability, drop false real-fossil interop claim (#141) TestCloneRealFossilWithDeltaChain asserted only that `fossil clone` exits 0, which a real fossil client does even when the clone is empty and its content is undecodable. Strengthen it to prove usability: rebuild the clone and count its check-ins. The count is zero -- a real fossil client decodes go-libfossil's full-content cfile payload (bare zlib) to garbage because fossil's on-disk cfile format is [4-byte size][zlib] -- so the test skips against the tracking issue (#152) instead of silently passing on corrupt content. When #152 lands the clone materializes all six check-ins and the test passes for real. Also correct the doc comment and the handler.go send-path comment, which claimed #141's send-order + uncompressed-delta-card change keeps a real fossil 2.28 client's rebuild from failing. It does not: that change is a libfossil<->libfossil bandwidth win (verified content-identical by the self-round-trip tests); real-fossil interop for delta-bearing clones remains blocked on the separate cfile-framing bug (#152), present on unmodified main. check-in: 5f66319ea3 user: [email protected] tags: trunk | |
| 15:27 |
perf: route history-walking expansions through content.Cache (#151) (#153)
The buffer-reuse fix in #145/#150 cut the cost of expanding one delta chain once. It did nothing for callers that expand many versions of the same artifact in a single operation: each bare content.Expand still replayed the whole O(depth) chain, so a walk over N overlapping revisions cost O(N^2) blob reads even though content.Cache already existed for exactly this pattern. Audited every non-test content.Expand caller. Two access patterns genuinely repeat over overlapping delta chains and are now routed through a per-operation content.Cache: - annotate walks every revision of one file along the parent chain. content_deltify stores each older file blob as a delta against its successor, so the per-revision expansions are all walks of the same few chains. One cache threaded through the walk (file blob + manifest listing via new manifest.ListFilesCached/GetManifestCached, both nil-passthrough so existing callers are unchanged) turns it linear. Measured on a deltified 150-revision file: 1.60s -> 0.65s (2.5x), 764k -> 308k allocs; the win grows with history depth. - verify.Verify and verify.Rebuild sweep every blob more than once (Rebuild: checkBlobs + rebuildManifests + rebuildTags x2). A single cache shared across the pass walks each chain once instead of once per sweep and amortizes overlap within each sweep. Measured on a 150-revision two-file repo: Rebuild 3.34s -> 0.068s (48x), 4.59M -> 98k allocs. Callers that only ever touch the tip or make a single pass over distinct blobs (branch create, merge-config load, R-card / delta manifest build, MergeParentFiles, crosslinkSingle, shun undelta, pairwise Diff) are left on bare Expand, each with a comment stating why a cache cannot amortize their access. No behavior change: expansion output is byte-identical, verified across the modernc and ncruces drivers with REQUIRE_FOSSIL_BIN=1 cross-checks and the DST suite. check-in: 0fd2a507b8 user: [email protected] tags: trunk | |