Artifact 5ad73cf018a38c1f4092ba351dfbcee223b843b47a5ac04a577e9094e806d9bb:
- File
internal/manifest/manifest.go
— part of check-in
[81fbc0be5f]
at
2026-07-24 17:23:36
on branch trunk
— manifest: fix crosslink's WAL/availability/mlink working-set problems (#158)
* manifest: batch crosslink, memoize availability, preload mlink parents (#89)
Three throughput fixes for the whole-repository crosslink sweep, none of which change its observable output. Profiling the sweep on the Fossil SCM corpus (67,503 blobs, 27,084 events) after the delta-chain-order work (#126) found the earlier bottlenecks had moved on:
- Per-candidate transactions made every SQL statement open its own WAL read-transaction, whose setup cost climbs as the uncheckpointed WAL grows -- throughput collapsed from ~200 events/sec to ~7 within a few seconds of a large sweep. The sweep now commits in batches of crosslinkBatchSize candidates so SQLite checkpoints and truncates the WAL between batches, holding read-transaction cost flat. Batch-granular durability is correct for derived data: an interrupted sweep leaves committed batches and the candidate query re-selects only what is still unlinked. CrosslinkContext is split into crosslinkSweep / collectCrosslinkCandidates / linkCandidatesInOrder / linkBatch, with the per-artifact dispatch pulled into linkArtifact and the deferral rollup into deferCheckin / logDeferredCheckins.
- missingCheckinRefs walked a blob's whole delta chain (content.IsAvailable) once per F-card of every check-in to decide deferral -- ~40% of the sweep, re-walking the same file chains over and over. content.AvailabilityCache memoizes the verdict per rid and stops a walk at the first already-decided ancestor, bounding total walk work by the number of distinct blobs rather than F-card references.
- resolveMlinkParent looked up each F-card's parent version with SELECT ... FROM mlink WHERE mid=? AND fnid=?, which SQLite answers via the fnid index -- returning a file's entire crosslinked history, so the lookup grew as the mlink table did. loadMlinkParents now reads each parent's file set once with a single mid-indexed scan and resolves every F-card from the in-memory map. It reads exactly the same rows, so the mlink output is unchanged.
The crosslink helpers now take the batch's *db.Tx directly (replacing r *repo.Repo + per-helper r.WithTx), matching the single-transaction shape of internal/verify.Rebuild and letting a whole batch commit atomically; tag application routes through tag.ApplyTagWithTx. updateAttachmentComments materializes its cursor before writing so a single connection never runs two statements at once. crosslinkSingle (dephantomize) wraps its one artifact in its own transaction.
Whole suite green (1235 tests, 35 packages, incl. dst and the fossil-binary comparison). This does not by itself land #89's "minutes not hours" target; the dominant remaining cost is a separate, pre-existing mlink over-insertion documented on the issue.
* manifest: diff crosslink mlink against the parent manifest (#89)
insertCheckinMlinks emitted one mlink row per F-card and read pmid/pid from the parent's mlink rows. On the Fossil SCM repository that is ~180x canonical's row count (a full sweep would write ~19M rows against fossil's own 104,880), and because delta-chain order (#102/#126) routinely crosslinks a child before its parent, the parent's mlink rows were usually absent, so ~97% of rows defaulted to pid=0 -- wrong parent-column data independent of the volume.
Rewrite it to diff each check-in's file list against its primary parent's expanded manifest -- the same diff canonical Fossil computes in add_mlink (src/manifest.c) and the rebuild path already does in internal/verify.rebuildMlinks -- and emit a row only for a file that was added, modified, deleted, or renamed. pmid/pid now come from the parent MANIFEST, which is always present once a check-in is eligible to link, so pid is correct regardless of crosslink order. The parent expansion runs through the sweep's shared content cache, so each parent's delta chain is walked at most once across all its children.
Measured on the Fossil SCM corpus: 127,775 mlink rows (canonical: 104,878) with pid populated on 74% of rows (was ~3%); full-repo crosslink completes in ~15 minutes rather than the prior ~hour. The residual gap is 95 PGP-clearsigned parent manifests deck.Parse cannot yet strip, whose ~98 children fall back to the old all-added behavior -- a pre-existing deck limitation tracked separately. (user: [email protected], size: 13794) [annotate] [blame] [check-ins using]
package manifest import ( "fmt" "time" "github.com/danmestas/go-libfossil/db" "github.com/danmestas/go-libfossil/internal/blob" "github.com/danmestas/go-libfossil/internal/content" "github.com/danmestas/go-libfossil/internal/deck" libfossil "github.com/danmestas/go-libfossil/internal/fsltype" "github.com/danmestas/go-libfossil/internal/hash" "github.com/danmestas/go-libfossil/internal/repo" "github.com/danmestas/go-libfossil/internal/tag" ) type CheckinOpts struct { Files []File Comment string User string Parent libfossil.FslID MergeParents []libfossil.FslID // additional parents beyond Parent (for merge commits) Delta bool Time time.Time Tags []deck.TagCard } type File struct { Name string Content []byte Perm string OldName string // prior pathname when this file card records a rename } func Checkin(r *repo.Repo, opts CheckinOpts) (manifestRid libfossil.FslID, manifestUUID string, err error) { if r == nil { panic("manifest.Checkin: r must not be nil") } if len(opts.Files) == 0 { panic("manifest.Checkin: opts.Files must not be empty") } if opts.User == "" { panic("manifest.Checkin: opts.User must not be empty") } defer func() { // If a panic is already unwinding the stack (e.g. a lower-level // invariant violation inside the transaction body), that panic is // the real diagnosis and must propagate unchanged. Without this // check, the postcondition assertion below fires unconditionally // during unwind too — manifestRid is still its zero value because // the panicking call never returned — replacing the original // panic with an unrelated "manifestRid must be positive" message // and masking the actual cause (issue #68). if p := recover(); p != nil { panic(p) } if err == nil && manifestRid <= 0 { panic("manifest.Checkin: manifestRid must be positive on success") } }() if opts.Time.IsZero() { opts.Time = time.Now().UTC() } var inlineTCards []deck.TagCard err = r.WithTx(func(tx *db.Tx) error { fCards, fileRids, txErr := storeFileBlobs(tx, opts.Files) if txErr != nil { return txErr } d, txErr := buildCheckinDeck(tx, opts, fCards) if txErr != nil { return txErr } inlineTCards = d.T // capture for post-tx tag processing manifestRid, manifestUUID, txErr = insertCheckinBlob(tx, d) if txErr != nil { return txErr } if txErr := insertMlinks(tx, opts, manifestRid); txErr != nil { return txErr } // Store the parent check-in manifest as a delta against this one, // the same backwards direction used for file content. Canonical does // this at the end of commit (src/checkin.c:3234, // `content_deltify(vid, &nvid, 1, 0)`). File content is handled by // insertMlinks above; this covers the manifests themselves. if opts.Parent > 0 { if _, txErr := content.Deltify(tx, opts.Parent, manifestRid); txErr != nil { return fmt.Errorf("deltify parent manifest: %w", txErr) } } if txErr := markLeafAndEvent(tx, opts, manifestRid); txErr != nil { return txErr } // Mark file blobs as unsent so sync pushes them. // (unclustered is handled by blob.Store automatically.) for _, frid := range fileRids { if _, err := tx.Exec("INSERT OR IGNORE INTO unsent(rid) VALUES(?)", frid); err != nil { return fmt.Errorf("unsent file: %w", err) } } return nil }) if err != nil { return 0, "", fmt.Errorf("manifest.Checkin: %w", err) } // Process inline T-cards (branch, sym-trunk, etc.) after the transaction // completes so the event/plink tables are populated for tag propagation. for _, tc := range inlineTCards { if tc.UUID != "*" { continue // non-self T-cards are for control artifacts } var tagType int switch tc.Type { case deck.TagPropagating: tagType = tag.TagPropagating case deck.TagSingleton: tagType = tag.TagSingleton case deck.TagCancel: tagType = tag.TagCancel default: continue } if err := tag.ApplyTag(r, tag.ApplyOpts{ TargetRID: manifestRid, SrcRID: manifestRid, TagName: tc.Name, TagType: tagType, Value: tc.Value, MTime: libfossil.TimeToJulian(opts.Time), }); err != nil { return 0, "", fmt.Errorf("manifest.Checkin inline tag %q: %w", tc.Name, err) } } // Inherit propagating tags (branch, sym-*, bgcolor) from the primary parent. // Without this, a new commit on an existing branch would not carry its // parent's branch tag — breaking branch-aware queries. A local commit is // always linked immediately after its parent exists, so calling this // here (rather than deferring to a repair pass) is safe; crosslink's // sync-import path defers the equivalent work to repairTagPropagation // instead, since candidates there are not visited in parent-before-child // order. Both converge on the same tagxref state. if opts.Parent > 0 { if err := tag.PropagateAll(r.DB(), opts.Parent); err != nil { return 0, "", fmt.Errorf("manifest.Checkin propagate from parent: %w", err) } } return manifestRid, manifestUUID, nil } func storeFileBlobs(tx *db.Tx, files []File) ([]deck.FileCard, []libfossil.FslID, error) { fCards := make([]deck.FileCard, len(files)) var rids []libfossil.FslID for i, f := range files { rid, uuid, err := blob.Store(tx, f.Content) if err != nil { return nil, nil, fmt.Errorf("storing file %q: %w", f.Name, err) } fCards[i] = deck.FileCard{Name: f.Name, UUID: uuid, Perm: f.Perm, OldName: f.OldName} rids = append(rids, rid) } return fCards, rids, nil } func buildCheckinDeck(tx *db.Tx, opts CheckinOpts, fCards []deck.FileCard) (*deck.Deck, error) { d := &deck.Deck{ Type: deck.Checkin, C: opts.Comment, D: opts.Time, F: fCards, U: deck.User(opts.User), } // Parents: primary parent first, then any merge parents. Fossil renders // this as a space-separated P-card; plink marks i==0 as the primary. if opts.Parent > 0 { var parentUUID string if err := tx.QueryRow("SELECT uuid FROM blob WHERE rid=?", opts.Parent).Scan(&parentUUID); err != nil { return nil, fmt.Errorf("parent uuid: %w", err) } d.P = []string{parentUUID} for _, mp := range opts.MergeParents { if mp <= 0 || mp == opts.Parent { continue } var mpUUID string if err := tx.QueryRow("SELECT uuid FROM blob WHERE rid=?", mp).Scan(&mpUUID); err != nil { return nil, fmt.Errorf("merge parent uuid: %w", err) } d.P = append(d.P, mpUUID) } } // Tags: use custom tags if provided, otherwise default trunk tags for initial checkin if len(opts.Tags) > 0 { d.T = opts.Tags } else if opts.Parent == 0 { d.T = []deck.TagCard{ {Type: deck.TagPropagating, Name: "branch", UUID: "*", Value: "trunk"}, {Type: deck.TagSingleton, Name: "sym-trunk", UUID: "*"}, } } // Delta manifest support if opts.Delta && opts.Parent > 0 { if err := applyDelta(tx, d, fCards, opts.Parent); err != nil { return nil, err } } // R-card (always over full file set) rDeck := &deck.Deck{F: fCards} // Bare Expand: computing one checkin's R-card expands each file in the // tree exactly once. Distinct blobs, single pass, no overlap to amortize. getContent := func(uuid string) ([]byte, error) { rid, ok := content.AvailableByUUID(tx, uuid) if !ok { return nil, fmt.Errorf("blob not found: %s", uuid) } return content.Expand(tx, rid) } rHash, err := rDeck.ComputeR(getContent) if err != nil { return nil, fmt.Errorf("R-card: %w", err) } d.R = rHash return d, nil } func insertCheckinBlob(tx *db.Tx, d *deck.Deck) (libfossil.FslID, string, error) { manifestBytes, err := d.Marshal() if err != nil { return 0, "", fmt.Errorf("marshal: %w", err) } rid, uuid, err := blob.Store(tx, manifestBytes) if err != nil { return 0, "", fmt.Errorf("store manifest: %w", err) } return rid, uuid, nil } func insertMlinks(tx *db.Tx, opts CheckinOpts, manifestRid libfossil.FslID) error { parents, err := loadMlinkParents(tx, opts.Parent, opts.MergeParents) if err != nil { return err } for _, f := range opts.Files { fnid, err := ensureFilename(tx, f.Name) if err != nil { return fmt.Errorf("filename %q: %w", f.Name, err) } fileUUID := hash.SHA1(f.Content) fileRid, _ := blob.Exists(tx, fileUUID) if err := insertMlinkRow(tx, manifestRid, int64(fileRid), fnid, "", f.Perm, parents); err != nil { return err } } return nil } func markLeafAndEvent(tx *db.Tx, opts CheckinOpts, manifestRid libfossil.FslID) error { // plink: one row per parent; primary (isprim=1) is opts.Parent, merge // parents are secondary (isprim=0). Matches the sync-path layout in // crosslink.go so merged history looks identical however it arrived. mtime := libfossil.TimeToJulian(opts.Time) allParents := collectParents(opts) for i, pid := range allParents { isPrim := 0 if i == 0 { isPrim = 1 } if _, err := tx.Exec( "INSERT INTO plink(pid, cid, isprim, mtime) VALUES(?, ?, ?, ?)", pid, manifestRid, isPrim, mtime, ); err != nil { return fmt.Errorf("plink: %w", err) } } // event if _, err := tx.Exec( "INSERT INTO event(type, mtime, objid, user, comment) VALUES('ci', ?, ?, ?, ?)", libfossil.TimeToJulian(opts.Time), manifestRid, opts.User, opts.Comment, ); err != nil { return fmt.Errorf("event: %w", err) } // leaf: this checkin is new; remove every parent from leaf so merge // commits correctly supersede both sides. if _, err := tx.Exec("INSERT OR IGNORE INTO leaf(rid) VALUES(?)", manifestRid); err != nil { return fmt.Errorf("leaf insert: %w", err) } for _, pid := range allParents { if _, err := tx.Exec("DELETE FROM leaf WHERE rid=?", pid); err != nil { return fmt.Errorf("leaf delete parent: %w", err) } } // Mark manifest as unsent so sync pushes it (unclustered is handled by blob.Store). if _, err := tx.Exec("INSERT OR IGNORE INTO unsent(rid) VALUES(?)", manifestRid); err != nil { return fmt.Errorf("unsent: %w", err) } return nil } // collectParents returns the primary parent followed by any merge parents, // skipping zero entries and de-duplicating. Used for both the plink insert // loop and the leaf cleanup so both see the same set. func collectParents(opts CheckinOpts) []libfossil.FslID { if opts.Parent <= 0 { return nil } seen := map[libfossil.FslID]bool{opts.Parent: true} out := []libfossil.FslID{opts.Parent} for _, mp := range opts.MergeParents { if mp <= 0 || seen[mp] { continue } seen[mp] = true out = append(out, mp) } return out } func ensureFilename(tx *db.Tx, name string) (int64, error) { if tx == nil { panic("manifest.ensureFilename: tx must not be nil") } if name == "" { panic("manifest.ensureFilename: name must not be empty") } var fnid int64 err := tx.QueryRow("SELECT fnid FROM filename WHERE name=?", name).Scan(&fnid) if err == nil { return fnid, nil } result, err := tx.Exec("INSERT INTO filename(name) VALUES(?)", name) if err != nil { return 0, err } return result.LastInsertId() } func GetManifest(r *repo.Repo, rid libfossil.FslID) (*deck.Deck, error) { return GetManifestCached(r, rid, nil) } // GetManifestCached is GetManifest that serves the manifest's own delta-chain // expansion through cache. A nil cache behaves exactly like GetManifest: // content.Cache's nil receiver falls through to content.Expand. Passing a live // cache pays off only for a caller that reads many manifests whose delta chains // overlap -- annotate walking one file's whole history re-expands the same // chain interiors at every revision -- so a single-manifest read gains nothing // and should pass nil. func GetManifestCached(r *repo.Repo, rid libfossil.FslID, cache *content.Cache) (result *deck.Deck, err error) { if r == nil { panic("manifest.GetManifest: r must not be nil") } if rid <= 0 { panic("manifest.GetManifest: rid must be positive") } defer func() { if err == nil && result == nil { panic("manifest.GetManifest: result must not be nil on success") } }() data, err := cache.Expand(r.DB(), rid) if err != nil { return nil, fmt.Errorf("manifest.GetManifest: %w", err) } return deck.Parse(data) } func applyDelta(tx *db.Tx, d *deck.Deck, fullFCards []deck.FileCard, parentRid libfossil.FslID) error { // Bare Expand: building one delta manifest reads its parent and baseline // once each. This is the commit path, not a history sweep -- no repeated // overlapping-chain access for a cache to pay off against. parentData, err := content.Expand(tx, parentRid) if err != nil { return fmt.Errorf("expand parent: %w", err) } parentDeck, err := deck.Parse(parentData) if err != nil { return fmt.Errorf("parse parent: %w", err) } baselineUUID := parentDeck.B if baselineUUID == "" { var puuid string tx.QueryRow("SELECT uuid FROM blob WHERE rid=?", parentRid).Scan(&puuid) baselineUUID = puuid } baseRid, ok := content.AvailableByUUID(tx, baselineUUID) if !ok { return fmt.Errorf("baseline %s not found", baselineUUID) } baseData, err := content.Expand(tx, baseRid) if err != nil { return fmt.Errorf("expand baseline: %w", err) } baseDeck, err := deck.Parse(baseData) if err != nil { return fmt.Errorf("parse baseline: %w", err) } baseFiles := make(map[string]string) for _, f := range baseDeck.F { baseFiles[f.Name] = f.UUID } var deltaFCards []deck.FileCard currentFiles := make(map[string]bool) for _, f := range fullFCards { currentFiles[f.Name] = true if baseUUID, exists := baseFiles[f.Name]; !exists || baseUUID != f.UUID { deltaFCards = append(deltaFCards, f) } } for name := range baseFiles { if !currentFiles[name] { deltaFCards = append(deltaFCards, deck.FileCard{Name: name}) } } if len(deltaFCards) < len(fullFCards) { d.B = baselineUUID d.F = deltaFCards } return nil }