Artifact 3f24f6504e3e3995f391894190abe83eb6bb03bc24ca104c3f2948d8bf074764:
- File
internal/sync/clone_realfossil_delta_test.go
— part of check-in
[0ca5133c62]
at
2026-07-23 10:36:32
on branch trunk
— clone: emit cfile with stored content per §7.2, not expanded file cards (#131)
* fix: clone emits cfile with stored content per sec7.2, not expanded file
Our clone server was building xfer.FileCard from cache.Expand(...) -- every artifact fully decompressed, and every delta materialized into full content -- where sec7.2 requires cfile cards carrying the repository's stored compressed content retransmitted verbatim. On real repositories this ran 80-220x more bytes on the wire than the protocol intends, and was the real blocker behind #88's clone-batching fix being unable to bound repositories much past libfossil's own size.
emitCloneBatch now reads blob.Load's stored bytes directly (full content for a whole artifact, delta bytes against DeltaSource for a deltified one) and emits xfer.CFileCard instead of xfer.FileCard, with DeltaSrc and USize (the full reconstructed size, from blob.size) populated correctly for delta rows. content.DeltaSource is exported so this and other callers outside internal/content can ask whether a rid is stored as a delta without duplicating the query.
No client changes were needed: session.go already routes CFileCard through the same handleFileCard path as FileCard, and storeDeltaContent already stores a received delta exactly as given via a create-phantom-if-missing lookup on its source, resolving delta-before-base arrival the way #53's phantom storage model was built for.
DefaultCloneBatchBytes is left at its current value rather than retuned toward canonical's 5,000,000 -- that constant's tuning is #109's territory now that bytesSent tracks stored bytes instead of expanded ones, and retuning it here risks colliding with that work.
* fix: expand deltified rows during clone instead of sending delta cfile
Code review reproduced a real interop regression: emitCloneBatch sends cards in ascending-rid order, but content.Deltify always deltifies the older artifact against the newer one, so a delta's DeltaSource almost always names a rid greater than the delta card's own rid. That means a delta cfile routinely reaches the client before the very source card it names. libfossil's own client tolerates the forward reference (storeDeltaContent phantoms the unseen source and resolves it later), but real fossil 2.28 does not -- the source stays a genuine, unfilled phantom, and content_get() on it during the client's post-clone rebuild pass returns without initializing its output blob, tripping an assertion in blob_copy (blob.c:397) and aborting the client mid-rebuild. Verified independently against a real fossil 2.28 binary cloning a libfossil server holding a delta chain.
emitCloneBatch now expands a deltified row to full content before sending it as a plain (non-delta) cfile, rather than forwarding the stored delta bytes with DeltaSrc set. The card is still zlib-compressed, so the wire-size win over the old FileCard/Expand path survives for every row; only the additional delta-retransmission savings are given up, and only for deltified rows. Re-enabling that needs either a source-before-dependent send order or confirmed forward-reference tolerance in canonical, and is left as a follow-up rather than blocking this fix.
TestEmitCloneBatchExpandsDeltifiedRows (renamed from TestEmitCloneBatchRetransmitsDeltaAsDelta) now pins the new contract: no DeltaSrc, full-sized payload. TestCloneRealFossilWithDeltaChain adds an end-to-end check against a real fossil binary, when one is on PATH. (user: [email protected], size: 3827) [annotate] [blame] [check-ins using]
package sync_test import ( "context" "fmt" "os/exec" "path/filepath" "testing" "github.com/danmestas/go-libfossil/internal/manifest" "github.com/danmestas/go-libfossil/simio" libfossil "github.com/danmestas/go-libfossil/internal/fsltype" "github.com/danmestas/go-libfossil/internal/repo" ) // TestCloneRealFossilWithDeltaChain requires a real fossil client to clone, // and successfully rebuild, a repository this server holds with deltified // content in the chain -- the interop regression a code review caught on // issue #98's first attempt. // // That attempt forwarded a deltified row's stored delta bytes verbatim // (correct per §7.2) but emitted cards in ascending-rid order for // cursor/pagination reasons unrelated to delta direction. content.Deltify // always deltifies the OLDER artifact against the NEWER one, so a delta's // DeltaSource almost always names a rid greater than the delta's own rid -- // meaning the delta card routinely reached the client before the very source // card it names. libfossil's own client tolerates that forward reference // (storeDeltaContent phantoms the unseen source and resolves it later), but // real fossil 2.28 does not: the source stayed a genuine, unfilled phantom, // and content_get() on it during the client's post-clone rebuild pass // returned without initializing its output blob, tripping an assertion in // blob_copy (blob.c:397) and aborting the client mid-rebuild. // // The fix in emitCloneBatch expands a deltified row to full content before // sending, rather than forwarding the stored delta bytes -- still // zlib-compressed as a cfile card, just no longer dependent on send order. // This test drives a fixed corpus (a linear 5-deep delta chain) through a // real fossil binary end to end; it is skipped, not failed, when no fossil // binary is on PATH. func TestCloneRealFossilWithDeltaChain(t *testing.T) { bin, err := exec.LookPath("fossil") if err != nil { t.Skip("no fossil binary on PATH") } dir := t.TempDir() srcRepo, err := repo.Create(filepath.Join(dir, "source.fossil"), "testuser", simio.CryptoRand{}, "") if err != nil { t.Fatalf("repo.Create: %v", err) } defer srcRepo.Close() // Six revisions of one file, each a small edit of the last, chained via // Parent so mlink resolves a predecessor for content.Deltify to act on. // Small edits keep the delta well under deltifyMinRatio, so every // revision but the last is actually stored as a delta. base := make([]byte, 4096) for i := range base { base[i] = byte(i % 251) } var parent libfossil.FslID for c := 0; c < 6; c++ { content := append([]byte(nil), base...) content[c] = byte(200 + c) content = append(content, []byte(fmt.Sprintf("\nrevision marker %d\n", c))...) mid, _, err := manifest.Checkin(srcRepo, manifest.CheckinOpts{ Comment: fmt.Sprintf("checkin %d", c), User: "testuser", Parent: parent, Files: []manifest.File{ {Name: "big.bin", Content: content}, }, }) if err != nil { t.Fatalf("Checkin %d: %v", c, err) } parent = mid } if _, err := manifest.Crosslink(srcRepo); err != nil { t.Fatalf("Crosslink: %v", err) } var deltaCount int if err := srcRepo.DB().QueryRow("SELECT count(*) FROM delta").Scan(&deltaCount); err != nil { t.Fatalf("count deltas: %v", err) } if deltaCount == 0 { t.Fatal("fixture bug: no deltas were created in the source repo -- this test needs a deltified chain to be meaningful") } ctx, cancel := context.WithCancel(context.Background()) defer cancel() url := serveRepo(ctx, t, srcRepo) clonePath := filepath.Join(dir, "clone.fossil") cmd := exec.Command(bin, "clone", url, clonePath) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { t.Fatalf("real fossil clone of a %d-deep delta chain failed: %v\n%s", deltaCount, err, out) } }