Artifact faefba0cc2d8299156e160025b7ad281026c1898a98f2f5ae8fad171f0500c84:
- File
internal/verify/verify.go
— part of check-in
[8b16c91f5c]
at
2026-07-28 04:03:58
on branch trunk
— perf: stop clone re-seeking blob.uuid per F-card and over-allocating delta output (#187) (#212)
Cloning the Fossil SCM corpus (67.6k blobs) took 293-302 s at 1251-1287 MB against canonical fossil's 20.8 s at 205 MB. A CPU and allocation profile of one clone found the cost was not delta expansion or hash verification.
27.2% of all CPU was a single un-memoized statement. content.AvailabilityCache memoizes the delta-chain walk but not the "SELECT rid FROM blob WHERE uuid=?" that starts it, so manifest.missingCheckinRefs re-seeked the blob.uuid index once per F-card of every check-in -- the same file hash resolved again for every revision that did not change it. That index is far larger than SQLite's page cache, so each seek was three page reads: pread syscalls were 34.7% of total samples. The walk it guards was under 1% of its own cost.
Separately, materializing 8.27 GB of content allocated 360 GB. delta.ApplyInto starts at a 64 KiB cap deliberately -- the target length is a wire claim and a few bytes can declare 4 GiB -- and then climbs to the real size through append's ~1.25x large-slice growth, ~3.4x the target in discarded intermediates, leaving slack that content.Cache.clip copied away again (203.9 GB and 59.8 GB respectively).
Three changes:
- Memoize uuid -> rid in AvailabilityCache. Hits only. Within one crosslink run a hit cannot go stale, because a sweep and a cascade only ever add blob rows; a miss can, because ridOrPhantom and blob.StorePhantom create rows mid-run, and remembering an absence would leave an artifact deferred after the content it waits on arrived. The scope that makes this sound is now stated on the type, and TestAvailabilityCacheMustNotSpanAPurge pins its one boundary: shun.Purge deletes blob rows, so a cache must not span one.
- Size delta.ApplyInto's output exactly. scanOutputSize walks the command stream first without allocating, checking every copy against bytes the source holds and every insert against bytes the delta carries, so the size it returns is a measurement rather than the header's claim. Any stream it cannot walk to a clean terminator falls back to the existing growth path, so malformed and hostile deltas allocate and fail exactly as before. The buffer now arrives tightly sized, which also makes clip's copy a no-op.
- Cut crosslinkCacheBytes from 256 to 128 MiB. Each byte of that budget costs roughly five bytes of peak RSS -- GOGC doubles it into the heap goal, and the scavenger is paced off GC cycles, of which a faster sweep runs fewer. At 256 MiB the first two changes gave 205.7 s but 1473 MB, faster at the cost of memory. The measured curve (32/64/128/256 MiB: 369.6 s/318 MB, 292.3 s/447 MB, 249.4 s/812 MB, 205.7 s/1473 MB) puts 128 MiB where both axes improve.
Result on the same corpus: 245.4 s at 780 MB, from 293-302 s at 1251-1287 MB -- 17% less wall time and 39% less peak RSS, with instructions retired down from 3.37T to 2.48T. The gap to canonical narrows from 14.1x/6.3x to 11.8x/3.8x.
Derived tables are unchanged against an unmodified build cloning the same server: blob 67530, event 27099 (identical per type), mlink 104919, plink 23741, tagxref 51766, leaf 1426, filename 1496. The corpus parity gate passes.
What remains is structural rather than a missing optimization: a clone expands every blob twice, once in the receive-time phantom-fill cascade and once in the final whole-repository sweep, where canonical crosslinks each artifact from content it already holds in hand at receive time. Cache.Expand splits 45.4 s / 35.3 s across those two passes. Collapsing them is a change to the receive path and belongs in its own issue. (user: [email protected], size: 3280) [annotate] [blame] [check-ins using]
// Package verify provides comprehensive repository verification and rebuild. package verify import ( "time" "github.com/danmestas/go-libfossil/internal/content" libfossil "github.com/danmestas/go-libfossil/internal/fsltype" "github.com/danmestas/go-libfossil/internal/repo" ) // verifyCacheBytes bounds the expanded content one Verify or Rebuild pass keeps // live. Both passes sweep every blob more than once -- Verify expands each blob // in checkBlobs and again in checkCheckins; Rebuild expands each in checkBlobs, // rebuildManifests, and twice in rebuildTags -- so a single cache shared across // the pass walks each blob's delta chain once instead of once per sweep, and // amortizes overlapping chains within each sweep. A miss costs throughput, not // correctness; blob content is immutable and neither pass rewrites it, so no // cached entry can go stale. // // It was once the same number as manifest.crosslinkCacheBytes and no longer // is: that budget was measured down against clone wall time and peak RSS // (see its comment), and this pass has not been measured the same way. Sharing // a number is not a reason to inherit someone else's measurement. const verifyCacheBytes = 256 << 20 // IssueKind categorizes the type of verification issue found. type IssueKind int const ( IssueHashMismatch IssueKind = iota IssueBlobCorrupt IssueDeltaDangling IssuePhantomOrphan IssueEventMissing IssueEventMismatch IssueMlinkMissing IssuePlinkMissing IssueTagxrefMissing IssueFilenameMissing IssueLeafIncorrect IssueMissingReference ) // Issue represents a single verification problem found in the repository. type Issue struct { Kind IssueKind RID libfossil.FslID UUID string Table string Message string } // Report contains the results of a repository verification pass. type Report struct { Issues []Issue BlobsChecked int BlobsOK int BlobsFailed int BlobsSkipped int MissingRefs int TablesRebuilt []string Duration time.Duration } // OK returns true if no issues were found during verification. func (r *Report) OK() bool { return len(r.Issues) == 0 } // addIssue appends a new issue to the report. func (r *Report) addIssue(issue Issue) { r.Issues = append(r.Issues, issue) } // Verify performs comprehensive repository verification. // It checks blob integrity, delta chains, phantom records, and derived tables. // Returns a report of all issues found. Never stops early - reports all problems. func Verify(r *repo.Repo) (*Report, error) { if r == nil { panic("verify: nil repo") } start := time.Now() report := &Report{} // One expansion cache for the whole pass; see verifyCacheBytes. cache := content.NewCache(verifyCacheBytes) // Phase 1: Blob integrity (content hash verification) if err := checkBlobs(r, report, cache); err != nil { return nil, err } // Phase 2: Structural integrity (delta chains, phantoms) if err := checkDeltaChains(r, report); err != nil { return nil, err } if err := checkPhantoms(r, report); err != nil { return nil, err } // Phase 3: Derived tables (event, mlink, plink, tagxref, filename, leaf) if err := checkDerived(r, report, cache); err != nil { return nil, err } report.Duration = time.Since(start) return report, nil }