Artifact 86437a8cdffa99e703c01fe64a1de10ec7ba5db582037e3ba52df154fe0e97c5:
- File
internal/sync/handler.go
— part of check-in
[f2fa2cc751]
at
2026-07-19 22:42:56
on branch trunk
— fix: clone_seqno is server-to-client only (#96)
* fix: clone_seqno is server-to-client only (#74)
The clone client transmitted a `clone_seqno` card to carry its pagination cursor. That card travels the other way: canonical's page_xfer() only ever writes it (src/xfer.c:1571) and has no parser for it, so a real fossil server answered `bad command: clone_seqno N` and the clone died on round two. Repositories that fit in a single batch never reached the pagination path, which is why wapp and pikchr cloned while fossil and sqlite did not.
The request cursor belongs on the clone card itself — `clone VERSION SEQNO`, read from token 2 at src/xfer.c:1553 — and the response cursor belongs on `clone_seqno`, which the client only consumes. Give each direction a single owner:
- The client sends its cursor on the clone card and never emits a clone_seqno card. It already consumed the server's card correctly; it now also rejects a negative value, matching src/xfer.c:2769. - The server reads its cursor from the clone card and the CloneSeqNoCard parser is gone, so there is exactly one way to set the cursor rather than two.
Aligning with canonical also corrects the cursor's meaning. Ours was the last rid sent, queried with `rid > cursor`; canonical's is the rid of the next blob to send, looped inclusively with `while( seqno<=max )`. Now that the value crosses to a real server it has to mean what canonical means, so emitCloneBatch queries `rid >= cursor` and reports the first rid it did not send, or 0 once the snapshot is exhausted. That collapses the old more/lastSentRID pair into a single nextRID and keeps the truncate hook honest by re-owing the dropped rid instead of skipping it.
TestCloneMultiBatchAgainstCanonicalServer drives the client against a stand-in server that implements the canonical contract and rejects a client-sent clone_seqno the way fossil does; it reproduces `bad command: clone_seqno` against the old client. TestHandleCloneCursorFromCloneCard pins the server half, including the inclusive cursor.
Verified against fossil 2.28 over a loopback server: fossil and sqlite now transfer every artifact (67,499 and 130,994 blobs), and wapp and pikchr still clone in 3 rounds.
* fix: conform clone cursor validation to the spec in both directions (#74)
Review caught two rules applied to the wrong side of the wire — the same axis error the original bug was.
The client rejected a negative clone_seqno as fatal, citing src/xfer.c:2769. That line is the server's rule for the SEQNO on a clone REQUEST. The reply parser is governed by §3.2 and §8.2: NEXT is recorded only when it is decimal, and a visible but non-decimal NEXT is a defined receiver-tolerance exception that leaves the recorded sequence unchanged and does not stop parsing later reply cards. Aborting the clone was the one thing the spec rules out. The rule belongs in the decoder, where a non-decimal NEXT now decodes to an unknown card: the token is preserved, no consumer records it, and following cards still parse. strconv.Atoi accepted "-1" off the wire, so this was reachable, not theoretical.
Conversely the server clamped `clone 3 0` to a cursor of 1 where §8.1 mandates a fatal: clear accumulated output, roll back, emit `error invalid clone sequence number`, and parse no later request card. Nothing is written before processDataCards, so returning early is what the rollback amounts to. Seeing this case at all needs a presence flag, because `clone` and `clone 3 0` both parsed to SeqNo 0 and only the former is exempt. §8.1 also keys on a digit-only SEQNO and withholds the fatal from wrong arity, a VERSION below 2, and a SEQNO that fails digit-only recognition, so the flag records digit-only presence rather than merely arity.
Both card rules now turn on the spec's `decimal = 1*DIGIT` production via a shared predicate instead of on whatever strconv.Atoi tolerates.
Also promotes emitCloneBatch's `count > 1` truncate guard to a documented liveness constraint. Under the old exclusive `rid > cursor` semantics `count > 0` was safe; under the inclusive cursor, dropping the only card in a batch reports a cursor equal to the one we entered with and the client re-requests forever. A check before the reply card now names a stalled round at its source instead of letting it surface as an opaque MaxRounds failure, and the cursor precondition became an error return rather than a panic — it is enforced ~350 lines away in a network-facing handler.
Records the deliberate asymmetry that our server ignores an inbound clone_seqno where canonical answers `bad command`, and renames TestHandleCloneSeqNo, which no longer sends that card.
* fix: name the clone SEQNO flag for the shape it records (#74)
HasSeqNo described presence, but the field records whether the SEQNO token is digit-only: `clone 3 -1` carries a SEQNO and sets it false. §8.1 keys its fatal on that shape rather than on the parsed value, which is why a plain int cannot stand in for it. SeqNoIsDecimal says so.
Reading it as presence would also collapse `clone 0 0` — a distinct wire form that parses to the same Version and SeqNo as a bare `clone` — back into `clone` on re-encode. encodeClone already guarded against that; a round-trip test over every clone wire form now pins it, and fails on exactly that input when the guard is removed.
Also records why the §8.1 rollback is currently vacuous: no card handled before processDataCards writes anything, so returning early leaves nothing to undo. Nothing enforces that ordering, and §3.5's ci-lock would break it, so the dependency is written down next to the code that relies on it.
Corrects a test comment that claimed a continuation property it did not exercise — the malformed card sits in the trailer, so nothing follows it. That half of §8.2 is covered by the follow-on card in TestDecode_CloneSeqNoNonDecimal. (user: [email protected], size: 26110) [annotate] [blame] [check-ins using]
package sync import ( "context" "database/sql" "errors" "fmt" "strconv" "strings" "github.com/danmestas/libfossil/internal/auth" "github.com/danmestas/libfossil/internal/blob" "github.com/danmestas/libfossil/internal/content" "github.com/danmestas/libfossil/internal/manifest" "github.com/danmestas/libfossil/internal/repo" "github.com/danmestas/libfossil/internal/xfer" libfossil "github.com/danmestas/libfossil/internal/fsltype" ) // DefaultCloneBatchSize is the number of blobs sent per clone round. const DefaultCloneBatchSize = 200 // cloneCapPrefix tags a CookieCard value that carries the upper rid bound // of a clone session. The handler captures max(rid) on the first clone round // and echoes it via CookieCard; the client returns it on every subsequent // round so the (otherwise stateless) server can scope its blob query and // converge even if the hub is being written to during the clone session. const cloneCapPrefix = "lf-clone-cap:" // parseCloneCap returns the snapshot rid encoded in a CookieCard value, // or (0, false) if the value isn't a clone-cap cookie. Non-matching // cookies (e.g. arbitrary sync session cookies) leave the bound at zero, // which preserves pre-snapshot behavior for forward/back compat. func parseCloneCap(v string) (int, bool) { if !strings.HasPrefix(v, cloneCapPrefix) { return 0, false } rid, err := strconv.Atoi(v[len(cloneCapPrefix):]) if err != nil || rid < 0 { return 0, false } return rid, true } // HandleFunc is the server-side sync handler signature. // Transport listeners call this with decoded requests and write back the response. type HandleFunc func(ctx context.Context, r *repo.Repo, req *xfer.Message) (*xfer.Message, error) // HandleOpts configures optional behavior for HandleSync. type HandleOpts struct { Buggify BuggifyChecker // nil in production. Observer Observer // nil defaults to no-op. ContentCache *content.Cache // nil = no caching. } // HandleSync processes an incoming xfer request and produces a response. // Stateless per-round — the client drives convergence. func HandleSync(ctx context.Context, r *repo.Repo, req *xfer.Message) (*xfer.Message, error) { return HandleSyncWithOpts(ctx, r, req, HandleOpts{}) } // HandleSyncWithOpts processes an incoming xfer request with optional // fault injection. Used by DST harness; production callers use HandleSync. func HandleSyncWithOpts(ctx context.Context, r *repo.Repo, req *xfer.Message, opts HandleOpts) (*xfer.Message, error) { if r == nil { panic("sync.HandleSync: r must not be nil") } if req == nil { panic("sync.HandleSync: req must not be nil") } obs := resolveObserver(opts.Observer) ctx = obs.HandleStarted(ctx, HandleStart{ Operation: detectOperation(req), }) h := &handler{repo: r, buggify: opts.Buggify, cache: opts.ContentCache} resp, err := h.process(ctx, req) if err == nil && resp == nil { panic("sync.HandleSync: resp must not be nil on success") } obs.HandleCompleted(ctx, HandleEnd{ CardsProcessed: len(req.Cards), FilesSent: h.filesSent, FilesReceived: h.filesRecvd, Err: err, }) return resp, err } // detectOperation checks request cards to determine if this is a clone or sync. func detectOperation(req *xfer.Message) string { for _, c := range req.Cards { if _, ok := c.(*xfer.CloneCard); ok { return "clone" } } return "sync" } // remoteHasEntry records that the client announced a blob via igot. type remoteHasEntry struct { isPrivate bool // IsPrivate flag from the client's igot card } // handler holds per-request state while processing cards. type handler struct { repo *repo.Repo buggify BuggifyChecker resp []xfer.Card pushOK bool // client sent a valid push card pullOK bool // client sent a valid pull card cloneMode bool // client sent a clone card fatal bool // a card rule ended the request; resp holds only its error cloneSeq int // rid of the next blob to send, from the client's clone card cloneSnapMax int // upper rid bound for this clone session (from cookie); 0 = capture fresh in emitCloneBatch uvCatalogSent bool // true after sending UV catalog reqClusters bool // client sent pragma req-clusters filesSent int // files sent in response (for observer) filesRecvd int // files received from client (for observer) syncPrivate bool // true if pragma send-private was accepted nextIsPrivate bool // true if a private card precedes the next file/cfile syncedTables map[string]*SyncedTable // cached table definitions xrowsSent int // table sync rows sent xrowsRecvd int // table sync rows received cache *content.Cache // nil = passthrough to content.Expand remoteHas map[string]remoteHasEntry // UUIDs the client announced via igot (mirrors Fossil's onremote table) // Auth state user string // verified username ("nobody" if no login card) caps string // capability string from user table authed bool // whether login card was verified } func (h *handler) initAuth() { h.user = "nobody" h.caps = "" h.authed = false var caps string err := h.repo.DB().QueryRow("SELECT cap FROM user WHERE login='nobody'").Scan(&caps) if err == nil { h.caps = caps } } func (h *handler) handleLoginCard(c *xfer.LoginCard) { var projectCode string if err := h.repo.DB().QueryRow("SELECT value FROM config WHERE name='project-code'").Scan(&projectCode); err != nil { h.resp = append(h.resp, &xfer.ErrorCard{Message: "authentication failed"}) return } u, err := auth.VerifyLogin(h.repo.DB(), projectCode, c) if err != nil { h.resp = append(h.resp, &xfer.ErrorCard{Message: "authentication failed"}) return } h.user = u.Login h.caps = u.Cap h.authed = true } func (h *handler) process(_ context.Context, req *xfer.Message) (*xfer.Message, error) { // Initialize auth state from nobody user. h.initAuth() // Load synced tables. if err := h.loadSyncedTables(); err != nil { return nil, err } // First pass: resolve login cards before other control cards. for _, card := range req.Cards { if lc, ok := card.(*xfer.LoginCard); ok { h.handleLoginCard(lc) } } // Second pass: process other control cards with capability checks. for _, card := range req.Cards { if _, ok := card.(*xfer.LoginCard); ok { continue // Already processed. } h.handleControlCard(card) if h.fatal { return &xfer.Message{Cards: h.resp}, nil } } // Emit PushCard with project-code/server-code so the clone client can // identify the repo. Only in clone mode — sync clients already have // codes, and real Fossil treats server-sent "push" as unknown during sync. if h.cloneMode { var projectCode, serverCode string _ = h.repo.DB().QueryRow("SELECT value FROM config WHERE name='project-code'").Scan(&projectCode) _ = h.repo.DB().QueryRow("SELECT value FROM config WHERE name='server-code'").Scan(&serverCode) if projectCode != "" { h.resp = append(h.resp, &xfer.PushCard{ ProjectCode: projectCode, ServerCode: serverCode, }) } } // Process data cards and emit response blobs. if err := h.processDataCards(req.Cards); err != nil { return nil, err } // Crosslink any newly-received manifests into the relational tables // (event/leaf/plink/mlink/tagxref). Without this the receiving repo // stores blobs durably but exposes nothing on the timeline, fork // detection sees stale leaf state, and the clone protocol cannot // traverse manifest parents (mlink is empty) — symptoms documented // in agent-infra trial 2026-04-25 finding #3. // // Only walk the crosslink scanner when we accepted files this round; // pure pull/igot rounds add nothing relational to update. if h.filesRecvd > 0 { if _, err := manifest.Crosslink(h.repo); err != nil { return nil, fmt.Errorf("HandleSync: crosslink: %w", err) } } return &xfer.Message{Cards: h.resp}, nil } // processDataCards handles file, igot, gimme, and other data cards in the // correct order, then emits igot/clone batches. Extracted from process() to // keep each function under 70 lines. func (h *handler) processDataCards(cards []xfer.Card) error { // File cards (and private prefix) first so blobs are stored before // IGotCard checks blob.Exists. Without this, a request containing // both IGotCard and FileCard for the same blob produces a spurious // GimmeCard — the IGotCard runs before the FileCard stores it. for _, card := range cards { switch card.(type) { case *xfer.FileCard, *xfer.CFileCard, *xfer.PrivateCard: if err := h.handleDataCard(card); err != nil { return err } } } // Remaining data cards (igot, gimme, etc.). for _, card := range cards { switch card.(type) { case *xfer.FileCard, *xfer.CFileCard, *xfer.PrivateCard: continue // Already handled above. default: if err := h.handleDataCard(card); err != nil { return err } } } // If pull was requested, emit igot for all non-phantom blobs. if h.pullOK { if err := h.emitIGots(); err != nil { return err } if err := h.emitXIGots(); err != nil { return err } } // If clone, emit paginated file cards. if h.cloneMode { if err := h.emitCloneBatch(); err != nil { return err } } return nil } func (h *handler) handleControlCard(card xfer.Card) { switch c := card.(type) { case *xfer.LoginCard: return // Already processed in first pass (initAuth/handleLoginCard). case *xfer.PragmaCard: if c.Name == "uv-hash" && len(c.Values) >= 1 { if err := h.handlePragmaUVHash(c.Values[0]); err != nil { h.resp = append(h.resp, &xfer.ErrorCard{ Message: fmt.Sprintf("uv-hash: %v", err), }) } } else if c.Name == "xtable-hash" && len(c.Values) >= 2 { h.handlePragmaXTableHash(c.Values[0], c.Values[1]) } if c.Name == "req-clusters" { h.reqClusters = true } if c.Name == "ci-lock" && len(c.Values) >= 2 { fail := processCkinLock(h.repo.DB(), c.Values[0], c.Values[1], h.user, DefaultCkinLockTimeout) if fail != nil { h.resp = append(h.resp, &xfer.PragmaCard{ Name: "ci-lock-fail", Values: []string{fail.HeldBy, fmt.Sprintf("%d", fail.Since.Unix())}, }) } } if c.Name == "send-private" { if auth.CanSyncPrivate(h.caps) { h.syncPrivate = true } else { h.resp = append(h.resp, &xfer.ErrorCard{ Message: "not authorized to sync private content", }) } } // Acknowledge client-version, ignore other unknown pragmas. case *xfer.PushCard: if auth.CanPush(h.caps) { h.pushOK = true } else { h.resp = append(h.resp, &xfer.ErrorCard{ Message: "push denied: insufficient capabilities", }) } case *xfer.PullCard: if auth.CanPull(h.caps) { h.pullOK = true } else { h.resp = append(h.resp, &xfer.ErrorCard{ Message: "pull denied: insufficient capabilities", }) } case *xfer.CloneCard: if !auth.CanClone(h.caps) { h.resp = append(h.resp, &xfer.ErrorCard{ Message: "clone denied: insufficient capabilities", }) return } // §8.1: an otherwise authorized three-token clone with VERSION >= 2 // and a digit-only SEQNO of zero or less clears accumulated output, // rolls back, emits this error, and parses no later request card. // // The mandated rollback is satisfied vacuously: no card handled // before processDataCards writes anything, so returning here leaves // nothing to undo. That depends on an ordering nothing enforces — // the day a control card starts writing (§3.5's ci-lock is the // obvious candidate, since a lock taken ahead of this card would // survive the abort), this needs a real transaction instead. if c.SeqNoIsDecimal && c.Version >= 2 && c.SeqNo <= 0 { h.resp = []xfer.Card{&xfer.ErrorCard{Message: "invalid clone sequence number"}} h.fatal = true return } h.cloneMode = true // The pagination cursor rides on the clone card and nowhere else // (§8.1; canonical xfer.c:1553 reads token 2 of `clone VERSION // SEQNO`). A request carrying no usable cursor — a bare `clone`, or // a SEQNO that fails digit-only recognition — starts from the first // rid. §8.1 withholds the fatal above from both of those cases. h.cloneSeq = c.SeqNo if h.cloneSeq <= 0 { h.cloneSeq = 1 } case *xfer.CloneSeqNoCard: // Deliberately ignored. clone_seqno is a reply-only card, so no // conforming client sends one; canonical answers `bad command`. We // stay tolerant rather than strict, which is why the model server in // TestCloneMultiBatchAgainstCanonicalServer rejects it and we do not. case *xfer.CookieCard: // Clone clients echo the cap cookie on every round. Sync sessions also // use cookies opaquely; non-matching values leave cloneSnapMax at zero. if rid, ok := parseCloneCap(c.Value); ok { h.cloneSnapMax = rid } case *xfer.SchemaCard: h.handleSchemaCard(c) } } func (h *handler) handleDataCard(card xfer.Card) error { switch c := card.(type) { case *xfer.IGotCard: return h.handleIGot(c) case *xfer.GimmeCard: return h.handleGimme(c) case *xfer.FileCard: return h.handleFile(c.UUID, c.DeltaSrc, c.Content) case *xfer.CFileCard: return h.handleFile(c.UUID, c.DeltaSrc, c.Content) case *xfer.PrivateCard: if !auth.CanSyncPrivate(h.caps) { h.resp = append(h.resp, &xfer.ErrorCard{ Message: "not authorized to sync private content", }) h.nextIsPrivate = false } else { h.nextIsPrivate = true } return nil case *xfer.ReqConfigCard: return h.handleReqConfig(c) case *xfer.UVIGotCard: return h.handleUVIGot(c) case *xfer.UVGimmeCard: return h.handleUVGimme(c) case *xfer.UVFileCard: return h.handleUVFile(c) case *xfer.XIGotCard: return h.handleXIGot(c) case *xfer.XGimmeCard: return h.handleXGimme(c) case *xfer.XRowCard: return h.handleXRow(c) case *xfer.XDeleteCard: return h.handleXDelete(c) } return nil } func (h *handler) handleIGot(c *xfer.IGotCard) error { if c == nil { panic("handler.handleIGot: c must not be nil") } _, exists := blob.Exists(h.repo.DB(), c.UUID) if exists { // Record that the client has this blob so emitIGots can skip it. // Mirrors Fossil's remote_has() → onremote table (xfer.c:1471). if h.remoteHas == nil { h.remoteHas = make(map[string]remoteHasEntry) } h.remoteHas[c.UUID] = remoteHasEntry{isPrivate: c.IsPrivate} return nil } // Server requests the missing blob whenever either side has expressed // interest in transferring data. Pre-fix this gate was just !pullOK, // which silently dropped server gimmes when a client called // SyncOpts{Push:true, Pull:false}: the client emits igot cards from // sendUnclustered every round regardless of Pull, so the server // would see the announcements but never request the blobs and the // loop would converge with the server holding only what the client // pushed proactively. Mirrors fossil-scm/c xfer.c, which generates // gimmes from igot cards as long as either direction is active. if !h.pushOK && !h.pullOK { return nil } if c.IsPrivate && !h.syncPrivate { return nil // not authorized — don't request } h.resp = append(h.resp, &xfer.GimmeCard{UUID: c.UUID}) return nil } func (h *handler) handleGimme(c *xfer.GimmeCard) error { if c == nil { panic("handler.handleGimme: c must not be nil") } // BUGGIFY: 5% chance skip sending a file to test client retry. if h.buggify != nil && h.buggify.Check("handler.handleGimme.skip", 0.05) { return nil } rid, ok := content.AvailableByUUID(h.repo.DB(), c.UUID) if !ok { return nil // blob not found — not fatal, skip. } isPriv := content.IsPrivate(h.repo.DB(), int64(rid)) if isPriv && !h.syncPrivate { return nil // private blob, client not authorized — skip. } data, err := h.cache.Expand(h.repo.DB(), rid) if err != nil { h.resp = append(h.resp, &xfer.ErrorCard{ Message: fmt.Sprintf("expand %s: %v", c.UUID, err), }) return nil } if isPriv { // BUGGIFY: 5% chance skip the PrivateCard prefix — client should // treat the file as public; next sync round corrects the status. if h.buggify == nil || !h.buggify.Check("handler.handleGimme.dropPrivateCard", 0.05) { h.resp = append(h.resp, &xfer.PrivateCard{}) } } h.resp = append(h.resp, &xfer.FileCard{UUID: c.UUID, Content: data}) h.filesSent++ return nil } func (h *handler) handleFile(uuid, deltaSrc string, payload []byte) error { if uuid == "" { panic("handler.handleFile: uuid must not be empty") } if !h.pushOK { h.resp = append(h.resp, &xfer.ErrorCard{ Message: fmt.Sprintf("file %s rejected: no push card", uuid), }) return nil } // BUGGIFY: 3% chance reject a valid file to test client re-push. if h.buggify != nil && h.buggify.Check("handler.handleFile.reject", 0.03) { h.resp = append(h.resp, &xfer.ErrorCard{ Message: fmt.Sprintf("buggify: rejected file %s", uuid), }) return nil } if err := storeReceivedFile(h.repo, uuid, deltaSrc, payload); err != nil { h.resp = append(h.resp, &xfer.ErrorCard{ Message: fmt.Sprintf("storing %s: %v", uuid, err), }) return nil } rid, ok := blob.Exists(h.repo.DB(), uuid) if h.buggify != nil && h.buggify.Check("handler.handleFile.missingAfterStore", 0.01) { ok = false } if !ok { h.resp = append(h.resp, &xfer.ErrorCard{ Message: fmt.Sprintf("blob %s missing after store", uuid), }) return nil } if h.nextIsPrivate { if err := content.MakePrivate(h.repo.DB(), int64(rid)); err != nil { return fmt.Errorf("handler: MakePrivate %s: %w", uuid, err) } h.nextIsPrivate = false } else { if err := content.MakePublic(h.repo.DB(), int64(rid)); err != nil { return fmt.Errorf("handler: MakePublic %s: %w", uuid, err) } } h.filesRecvd++ return nil } func (h *handler) handleReqConfig(c *xfer.ReqConfigCard) error { if c == nil { panic("handler.handleReqConfig: c must not be nil") } var val string err := h.repo.DB().QueryRow( "SELECT value FROM config WHERE name = ?", c.Name, ).Scan(&val) if errors.Is(err, sql.ErrNoRows) { return nil // config key not found — expected, not fatal. } if err != nil { return fmt.Errorf("handler: config query %q: %w", c.Name, err) } h.resp = append(h.resp, &xfer.ConfigCard{ Name: c.Name, Content: []byte(val), }) return nil } func (h *handler) emitIGots() error { // Emit igot for all non-phantom blobs so the client can discover // everything the server has. Cluster generation is a client-side // optimization for push; the server always advertises all blobs. rows, err := h.repo.DB().Query(` SELECT uuid FROM blob WHERE size >= 0 AND NOT EXISTS(SELECT 1 FROM shun WHERE uuid=blob.uuid) AND NOT EXISTS(SELECT 1 FROM private WHERE rid=blob.rid)`, ) if err != nil { return fmt.Errorf("handler: listing blobs: %w", err) } defer rows.Close() var uuids []string for rows.Next() { var uuid string if err := rows.Scan(&uuid); err != nil { return err } // remoteHas is populated from client igot cards in handleIGot. // Skip if the client already has this blob as public (non-private). // If the client has it as private, we still emit the public igot so // the client can clear its private status (private→public transition). if e, ok := h.remoteHas[uuid]; ok && !e.isPrivate { continue } uuids = append(uuids, uuid) } if err := rows.Err(); err != nil { return err } // BUGGIFY: 10% chance truncate igot list to test multi-round convergence. if h.buggify != nil && h.buggify.Check("handler.emitIGots.truncate", 0.10) && len(uuids) > 1 { uuids = uuids[:len(uuids)/2] } for _, uuid := range uuids { h.resp = append(h.resp, &xfer.IGotCard{UUID: uuid}) } if h.syncPrivate { if err := h.emitPrivateIGots(); err != nil { return err } } return nil } // emitPrivateIGots emits igot cards with IsPrivate=true for all blobs in // the private table. Only called when the client sent pragma send-private // and has the 'x' capability. func (h *handler) emitPrivateIGots() error { rows, err := h.repo.DB().Query( "SELECT b.uuid FROM private p JOIN blob b ON p.rid=b.rid WHERE b.size >= 0", ) if err != nil { return fmt.Errorf("handler: listing private blobs: %w", err) } defer rows.Close() var uuids []string for rows.Next() { var uuid string if err := rows.Scan(&uuid); err != nil { return err } // Skip if the client already has this blob as private. // If the client has it as public, we still emit the private igot so // the client can update its private status (public→private transition). if e, ok := h.remoteHas[uuid]; ok && e.isPrivate { continue } uuids = append(uuids, uuid) } if err := rows.Err(); err != nil { return err } // BUGGIFY: 10% chance truncate private igot list to test multi-round convergence. if h.buggify != nil && h.buggify.Check("handler.emitPrivateIGots.truncate", 0.10) && len(uuids) > 1 { uuids = uuids[:len(uuids)/2] } for _, uuid := range uuids { h.resp = append(h.resp, &xfer.IGotCard{UUID: uuid, IsPrivate: true}) } return nil } // sendAllClusters emits igot cards for all cluster artifacts that are // not still in unclustered (i.e., already fully clustered themselves). func (h *handler) sendAllClusters() error { rows, err := h.repo.DB().Query(` SELECT b.uuid FROM tagxref tx JOIN blob b ON tx.rid = b.rid WHERE tx.tagid = 7 AND NOT EXISTS (SELECT 1 FROM unclustered WHERE rid = b.rid) AND NOT EXISTS (SELECT 1 FROM phantom WHERE rid = b.rid) AND NOT EXISTS (SELECT 1 FROM shun WHERE uuid = b.uuid) AND NOT EXISTS (SELECT 1 FROM private WHERE rid = b.rid) AND b.size >= 0 `) if err != nil { return fmt.Errorf("handler: listing clusters: %w", err) } defer rows.Close() for rows.Next() { var uuid string if err := rows.Scan(&uuid); err != nil { return err } // Clusters are always public (query excludes private table). if e, ok := h.remoteHas[uuid]; ok && !e.isPrivate { continue } h.resp = append(h.resp, &xfer.IGotCard{UUID: uuid}) } return rows.Err() } func (h *handler) emitCloneBatch() error { // Capture the snapshot bound on the first round of a clone session. // Without an upper bound the batch query keeps picking up blobs that other // writers commit between rounds, the completion sentinel // CloneSeqNoCard{SeqNo:0} never fires, and the client loops to MaxRounds // (issue #17). The bound rides back to the client via CookieCard and // gets echoed on every subsequent round. if h.cloneSnapMax == 0 { var maxRid int if err := h.repo.DB().QueryRow("SELECT COALESCE(MAX(rid), 0) FROM blob WHERE size >= 0").Scan(&maxRid); err != nil { return fmt.Errorf("handler: capture clone snapshot: %w", err) } h.cloneSnapMax = maxRid h.resp = append(h.resp, &xfer.CookieCard{Value: fmt.Sprintf("%s%d", cloneCapPrefix, maxRid)}) } batchSize := DefaultCloneBatchSize // BUGGIFY: 10% chance reduce batch size to 1 to stress pagination. if h.buggify != nil && h.buggify.Check("handler.emitCloneBatch.smallBatch", 0.10) { batchSize = 1 } truncate := h.buggify != nil && h.buggify.Check("clone.emitCloneBatch.truncate", 0.10) // cloneSeq is the rid of the next blob to send and is inclusive, matching // §8.2's "iterate from supplied RID through current maximum" and // canonical's `while( seqno<=max )` loop in page_xfer(). handleControlCard // is the sole writer and floors it at 1, but that is enforced far from // here with nothing compile-time behind it, so this stays a check rather // than an assumption. cursorAtEntry := h.cloneSeq if cursorAtEntry <= 0 { return fmt.Errorf("handler: clone cursor must be positive, got %d", cursorAtEntry) } rows, err := h.repo.DB().Query( "SELECT rid, uuid FROM blob WHERE rid >= ? AND rid <= ? AND size >= 0 ORDER BY rid", h.cloneSeq, h.cloneSnapMax, ) if err != nil { return fmt.Errorf("handler: clone batch: %w", err) } defer rows.Close() count := 0 var lastSentRID int // nextRID is the cursor reported back to the client: the rid of the first // blob this batch did not send, or 0 once the snapshot is exhausted. nextRID := 0 for rows.Next() { var rid int var uuid string if err := rows.Scan(&rid, &uuid); err != nil { return err } isPriv := content.IsPrivate(h.repo.DB(), int64(rid)) if isPriv && !h.syncPrivate { continue // skip private blob, don't count toward batch } if count >= batchSize { nextRID = rid break } data, err := h.cache.Expand(h.repo.DB(), libfossil.FslID(rid)) if err != nil { return fmt.Errorf("handler: expanding rid %d: %w", rid, err) } if isPriv { h.resp = append(h.resp, &xfer.PrivateCard{}) } h.resp = append(h.resp, &xfer.FileCard{UUID: uuid, Content: data}) h.filesSent++ lastSentRID = rid count++ } if err := rows.Err(); err != nil { return err } // BUGGIFY: truncate — remove last file card to simulate incomplete batch. // `count > 1` is a liveness constraint, not just a chaos-injection guard. // Dropping the only card in a batch would set nextRID to lastSentRID, // which equals the cursor we entered with, and the client would re-request // the same batch until MaxRounds. Under the old exclusive `rid > cursor` // semantics `count > 0` was safe here; it no longer is. if truncate && count > 1 { for i := len(h.resp) - 1; i >= 0; i-- { if _, ok := h.resp[i].(*xfer.FileCard); ok { h.resp = append(h.resp[:i], h.resp[i+1:]...) h.filesSent-- count-- // The dropped card held lastSentRID, so that rid becomes the // next one owed to the client rather than being skipped. nextRID = lastSentRID break } } } // The cursor must strictly advance or signal exhaustion. Catching a stall // here names the round that caused it; letting it through only surfaces // later as an opaque MaxRounds failure on the client. if nextRID != 0 && nextRID <= cursorAtEntry { return fmt.Errorf( "handler: clone cursor did not advance: next %d, cursor %d", nextRID, cursorAtEntry) } // SeqNo 0 signals completion so the client stops requesting. h.resp = append(h.resp, &xfer.CloneSeqNoCard{SeqNo: nextRID}) return nil }