Artifact 332947b8e582741e5fc01bb15435657c6d857b5a986c2340aebce3ad79f75172:
- File
internal/sync/clone_realfossil_delta_test.go
— part of check-in
[5f66319ea3]
at
2026-07-24 16:07:43
on branch trunk
— sync: retransmit deltas during clone instead of expanding them (#154)
* sync: correct the clone cfile comment to match the actual code path (#140)
The comment claimed clone "retransmits the repository's stored compressed content verbatim... blob.Load returns exactly that." That is not what the code does: blob.Load decompresses the stored blob and encodeCFile then re-compresses it with zlib before it goes on the wire. The wire result is correct because the decompress/re-compress round-trip is byte-deterministic, but nothing forwards the stored compressed bytes untouched. Correct the comment to describe the real mechanism (decompress via blob.Load, re-compress via encodeCFile) and state explicitly why the round-trip is still stable.
Documentation-only; no behavior change.
* sync: retransmit deltas during clone instead of expanding them (#141)
The clone send path expanded every deltified row to full content before putting it on the wire, giving up the delta-transmission bandwidth the xfer protocol is designed for. Restore delta transmission on the clone send path.
Send ordering. content.Deltify deltifies the OLDER artifact against the NEWER one, so a delta's source almost always has a greater rid than the delta and would not yet have been sent under the ascending-rid clone cursor. Rather than reorder the whole cursor, buildCloneArtifact walks each artifact's delta chain toward its full-content tip and emits the chain source-first, mirroring canonical fossil's recursive send_file: the tip (or the lowest still-sendable anchor) leads, each delta follows the source it names, and the requested rid is last. A per-round "sent" set skips any source a dependent's chain already pulled in ahead of its ascending position, matching fossil's per-request bag. The snapshot bound (#17) is preserved: a delta whose source lies outside the clone snapshot, is a phantom, or is private under a public clone degrades to full content so no out-of-snapshot row is ever leaked.
Wire format. A delta goes out as an uncompressed "file UUID DELTASRC SIZE" card, exactly as canonical fossil's send_delta_native does, not as a "cfile". Full content still rides a compressed cfile, preserving the #98/#113 wire-size win.
Scope. This is a bandwidth win, verified content-identical for libfossil<->libfossil clones by the self-round-trip tests, with the source-before-dependent invariant guarded by TestEmitCloneBatchSourcePrecedesDelta. The receive side already re-deltifies via content.Deltify on the crosslink path (#114), so store and tip fidelity are unchanged. Measured on a 120-artifact, 59-delta corpus, a full clone's reply bytes drop from 32,551 (expand-before-send) to 17,419, a 46% reduction, close to the 14,941-byte stored size.
This does NOT make a real fossil client's clone usable. Full content still rides a compressed cfile, which go-libfossil emits as bare zlib while fossil expects [4-byte size][zlib] framing, so a real fossil client decodes full content to garbage and rebuilds to zero check-ins. That is a separate, pre-existing bug (present on unmodified main), tracked as #152. TestCloneRealFossilWithDeltaChain asserts real usability and skips against
* test(sync): update real-fossil delta test doc to the #141 mechanism
* test(sync): prove clone usability, drop false real-fossil interop claim (#141)
TestCloneRealFossilWithDeltaChain asserted only that `fossil clone` exits 0, which a real fossil client does even when the clone is empty and its content is undecodable. Strengthen it to prove usability: rebuild the clone and count its check-ins. The count is zero -- a real fossil client decodes go-libfossil's full-content cfile payload (bare zlib) to garbage because fossil's on-disk cfile format is [4-byte size][zlib] -- so the test skips against the tracking issue (#152) instead of silently passing on corrupt content. When #152 lands the clone materializes all six check-ins and the test passes for real.
Also correct the doc comment and the handler.go send-path comment, which claimed #141's send-order + uncompressed-delta-card change keeps a real fossil 2.28 client's rebuild from failing. It does not: that change is a libfossil<->libfossil bandwidth win (verified content-identical by the self-round-trip tests); real-fossil interop for delta-bearing clones remains blocked on the separate cfile-framing bug (#152), present on unmodified main. (user: [email protected], size: 6102) [annotate] [blame] [check-ins using]
package sync_test import ( "context" "fmt" "os/exec" "path/filepath" "strconv" "strings" "testing" "github.com/danmestas/go-libfossil/internal/manifest" "github.com/danmestas/go-libfossil/simio" libfossil "github.com/danmestas/go-libfossil/internal/fsltype" "github.com/danmestas/go-libfossil/internal/repo" ) // TestCloneRealFossilWithDeltaChain drives a fixed corpus (a linear delta // chain) from this server through a real fossil binary and checks that the // clone is actually usable, not merely that the `fossil clone` subprocess // exits 0. It is skipped, not failed, when no fossil binary is on PATH. // // #141 changed how a deltified row is sent during clone: instead of expanding // the delta and shipping full content, it retransmits the row as a delta and // emits the delta's source ahead of it (buildCloneArtifact walks the chain // source-first). content.Deltify deltifies the OLDER artifact against the // NEWER one, so a delta's source almost always has a *greater* rid than the // delta itself and, under the send loop's ascending-rid order, would not yet // have been sent; walking source-first guarantees no delta forward-references // a card that has not arrived. The delta rides an uncompressed "file UUID // DELTASRC SIZE" card (not a compressed "cfile"), matching canonical fossil's // send_delta_native, so the receiver re-frames it into fossil's on-disk blob // format. That change is a bandwidth win and is verified content-identical for // libfossil<->libfossil clones by the self-round-trip tests. // // It does NOT, on its own, make a real fossil client's clone usable. Full // content still rides a compressed "cfile", and go-libfossil emits that cfile // as bare zlib while fossil's on-disk cfile format is [4-byte size][zlib] -- // so a real fossil client decodes full content to garbage and rebuilds to zero // check-ins. That is a separate, pre-existing bug (present on unmodified main), // tracked as #152. This test asserts real usability so the gap is visible: it // clones, rebuilds, and counts check-ins, and when the count is zero it skips // with a pointer to #152 rather than silently passing on corrupt content. When // #152 lands, the clone will materialize all six check-ins and this test starts // genuinely passing. func TestCloneRealFossilWithDeltaChain(t *testing.T) { bin, err := exec.LookPath("fossil") if err != nil { t.Skip("no fossil binary on PATH") } dir := t.TempDir() srcRepo, err := repo.Create(filepath.Join(dir, "source.fossil"), "testuser", simio.CryptoRand{}, "") if err != nil { t.Fatalf("repo.Create: %v", err) } defer srcRepo.Close() // Six revisions of one file, each a small edit of the last, chained via // Parent so mlink resolves a predecessor for content.Deltify to act on. // Small edits keep the delta well under deltifyMinRatio, so every // revision but the last is actually stored as a delta. const wantCheckins = 6 base := make([]byte, 4096) for i := range base { base[i] = byte(i % 251) } var parent libfossil.FslID for c := 0; c < wantCheckins; c++ { content := append([]byte(nil), base...) content[c] = byte(200 + c) content = append(content, []byte(fmt.Sprintf("\nrevision marker %d\n", c))...) mid, _, err := manifest.Checkin(srcRepo, manifest.CheckinOpts{ Comment: fmt.Sprintf("checkin %d", c), User: "testuser", Parent: parent, Files: []manifest.File{ {Name: "big.bin", Content: content}, }, }) if err != nil { t.Fatalf("Checkin %d: %v", c, err) } parent = mid } if _, err := manifest.Crosslink(srcRepo); err != nil { t.Fatalf("Crosslink: %v", err) } var deltaCount int if err := srcRepo.DB().QueryRow("SELECT count(*) FROM delta").Scan(&deltaCount); err != nil { t.Fatalf("count deltas: %v", err) } if deltaCount == 0 { t.Fatal("fixture bug: no deltas were created in the source repo -- this test needs a deltified chain to be meaningful") } ctx, cancel := context.WithCancel(context.Background()) defer cancel() url := serveRepo(ctx, t, srcRepo) clonePath := filepath.Join(dir, "clone.fossil") cmd := exec.Command(bin, "clone", url, clonePath) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("real fossil clone of a %d-deep delta chain failed: %v\n%s", deltaCount, err, out) } // A clone that exits 0 is necessary but not sufficient: a real fossil client // reports success and can still hold unusable content. Rebuild forces it to // decode and verify every artifact; counting check-ins afterward proves the // content actually materialized. Asserting only on the clone exit code (as // this test originally did) proves nothing but that the subprocess ran. if out, err := exec.Command(bin, "rebuild", clonePath).CombinedOutput(); err != nil { t.Fatalf("fossil rebuild of the clone failed: %v\n%s", err, out) } got := fossilCheckinCount(t, bin, clonePath) if got == 0 { t.Skipf("KNOWN BUG (#152): a real fossil client cloned 0 usable check-ins "+ "from this server. #141's delta retransmission is correct and verified "+ "for libfossil<->libfossil clones, but real-fossil interop is blocked on "+ "a separate cfile-framing bug: go-libfossil emits the cfile payload as "+ "bare zlib while fossil expects [4-byte size][zlib], so full content "+ "decodes to garbage and no check-in survives rebuild. Remove this skip "+ "when #152 lands (expect %d check-ins).", wantCheckins) } if got != wantCheckins { t.Fatalf("clone materialized %d check-ins, want %d", got, wantCheckins) } } // fossilCheckinCount returns the number of check-in events a real fossil repo // holds, via `fossil sql`. Used to prove a clone is actually populated. func fossilCheckinCount(t *testing.T, bin, repoPath string) int { t.Helper() out, err := exec.Command(bin, "sql", "-R", repoPath, "SELECT count(*) FROM event WHERE type='ci';").Output() if err != nil { t.Fatalf("counting check-ins in clone: %v", err) } n, err := strconv.Atoi(strings.TrimSpace(string(out))) if err != nil { t.Fatalf("parsing check-in count %q: %v", string(out), err) } return n }