| Comment: | manifest: run leaf/tag repair on interrupted crosslink sweep (#159)
* manifest: run leaf/tag repair on interrupted crosslink sweep crosslinkSweep skipped the repairLeafTable/repairTagPropagation pass whenever linkCandidatesInOrder returned an error, including a context cancellation. After #89's batched sweep, batches commit as they go, so a sweep cancelled after committing at least one batch left the plink edges and tagxref origins those batches wrote without their leaf/tag consequences -- stale until a later sweep linked something, and a later sweep whose remaining candidates all link zero artifacts skips the same gate and leaves the window open longer than one sweep. Gate the repair on linked > 0 rather than on sweep success and return the sweep's own error after repairing, so an interrupted-but-committed sweep still recovers derived leaf/tag state on the way out. The repair runs in its own context-free transaction, so a cancelled ctx interrupts the candidate sweep but not the recovery, preserving the batched commit/rollback consistency guarantees. Closes #143. * manifest: join repair error with sweep error on interrupted sweep When both the sweep is interrupted and the deferred leaf/tag repair pass fails, the repair error was dropped entirely -- only the sweep error was returned. A real repair failure (disk full, corruption, constraint violation) during a cancelled sweep would then persist the exact stale leaf/tag window this fix closes with no signal. Join both with errors.Join so the repair failure is surfaced while errors.Is(err, context.Canceled) still holds for the three callers (clone.go, session.go, handler.go), all of which wrap with %w and none of which use exact equality. |
|---|---|
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA3-256: |
7bc832f339c221119bda1e9edf6255ec |
| User & Date: | [email protected] on 2026-07-24 17:51:14.000 |
| Other Links: | manifest | tags |
|
2026-07-24
| ||
| 22:17 |
deck: strip PGP/SSH clearsign framing before VerifyZ (#156) (#160)
VerifyZ validated a control artifact by hashing everything up to the last 35 bytes and comparing against the Z card it expected to find there. When a manifest arrived PGP-clearsigned, the raw blob carried a trailing "-----BEGIN PGP SIGNATURE-----" ... "-----END PGP SIGNATURE-----" block after the Z card, so the last-35-byte window landed inside the signature framing instead of on the Z card, and verification failed. deck.Parse propagated that failure, so ~95 of Fossil's own 2007-era clearsigned check-ins could not be parsed -- and any child whose primary parent was one of them lost its parent's file tree in loadPrimaryParentFiles, cascading into all-added mlink data (~98 affected children on the real corpus). Mirror fossil's src/manifest.c remove_pgp_signature: strip the armor header (up to the blank line that ends it) and the trailing signature block, isolating the inner content the Z card is actually computed over, then verify and parse that. The stripping is an identity on non-clearsigned input (the common case) and idempotent. The artifact's content-addressed hash is taken over the raw blob elsewhere and is untouched by this change: a clearsigned manifest still resolves to the same UUID it always has. Verified against the real Fossil check-in 61829b07 (aku, 2007-11-29), whose raw-blob SHA-1 still equals its UUID and whose inner Z-card MD5 now validates; cross-checked byte-for-byte against fossil 2.28. check-in: c1cbd26795 user: [email protected] tags: trunk, observer/otel/v0.2.0, db/driver/ncruces/v0.3.0, db/driver/modernc/v0.3.0 | |
| 17:51 |
manifest: run leaf/tag repair on interrupted crosslink sweep (#159)
* manifest: run leaf/tag repair on interrupted crosslink sweep crosslinkSweep skipped the repairLeafTable/repairTagPropagation pass whenever linkCandidatesInOrder returned an error, including a context cancellation. After #89's batched sweep, batches commit as they go, so a sweep cancelled after committing at least one batch left the plink edges and tagxref origins those batches wrote without their leaf/tag consequences -- stale until a later sweep linked something, and a later sweep whose remaining candidates all link zero artifacts skips the same gate and leaves the window open longer than one sweep. Gate the repair on linked > 0 rather than on sweep success and return the sweep's own error after repairing, so an interrupted-but-committed sweep still recovers derived leaf/tag state on the way out. The repair runs in its own context-free transaction, so a cancelled ctx interrupts the candidate sweep but not the recovery, preserving the batched commit/rollback consistency guarantees. Closes #143. * manifest: join repair error with sweep error on interrupted sweep When both the sweep is interrupted and the deferred leaf/tag repair pass fails, the repair error was dropped entirely -- only the sweep error was returned. A real repair failure (disk full, corruption, constraint violation) during a cancelled sweep would then persist the exact stale leaf/tag window this fix closes with no signal. Join both with errors.Join so the repair failure is surfaced while errors.Is(err, context.Canceled) still holds for the three callers (clone.go, session.go, handler.go), all of which wrap with %w and none of which use exact equality. check-in: 7bc832f339 user: [email protected] tags: trunk | |
| 17:23 |
manifest: fix crosslink's WAL/availability/mlink working-set problems (#158)
* manifest: batch crosslink, memoize availability, preload mlink parents (#89) Three throughput fixes for the whole-repository crosslink sweep, none of which change its observable output. Profiling the sweep on the Fossil SCM corpus (67,503 blobs, 27,084 events) after the delta-chain-order work (#126) found the earlier bottlenecks had moved on: - Per-candidate transactions made every SQL statement open its own WAL read-transaction, whose setup cost climbs as the uncheckpointed WAL grows -- throughput collapsed from ~200 events/sec to ~7 within a few seconds of a large sweep. The sweep now commits in batches of crosslinkBatchSize candidates so SQLite checkpoints and truncates the WAL between batches, holding read-transaction cost flat. Batch-granular durability is correct for derived data: an interrupted sweep leaves committed batches and the candidate query re-selects only what is still unlinked. CrosslinkContext is split into crosslinkSweep / collectCrosslinkCandidates / linkCandidatesInOrder / linkBatch, with the per-artifact dispatch pulled into linkArtifact and the deferral rollup into deferCheckin / logDeferredCheckins. - missingCheckinRefs walked a blob's whole delta chain (content.IsAvailable) once per F-card of every check-in to decide deferral -- ~40% of the sweep, re-walking the same file chains over and over. content.AvailabilityCache memoizes the verdict per rid and stops a walk at the first already-decided ancestor, bounding total walk work by the number of distinct blobs rather than F-card references. - resolveMlinkParent looked up each F-card's parent version with SELECT ... FROM mlink WHERE mid=? AND fnid=?, which SQLite answers via the fnid index -- returning a file's entire crosslinked history, so the lookup grew as the mlink table did. loadMlinkParents now reads each parent's file set once with a single mid-indexed scan and resolves every F-card from the in-memory map. It reads exactly the same rows, so the mlink output is unchanged. The crosslink helpers now take the batch's *db.Tx directly (replacing r *repo.Repo + per-helper r.WithTx), matching the single-transaction shape of internal/verify.Rebuild and letting a whole batch commit atomically; tag application routes through tag.ApplyTagWithTx. updateAttachmentComments materializes its cursor before writing so a single connection never runs two statements at once. crosslinkSingle (dephantomize) wraps its one artifact in its own transaction. Whole suite green (1235 tests, 35 packages, incl. dst and the fossil-binary comparison). This does not by itself land #89's "minutes not hours" target; the dominant remaining cost is a separate, pre-existing mlink over-insertion documented on the issue. * manifest: diff crosslink mlink against the parent manifest (#89) insertCheckinMlinks emitted one mlink row per F-card and read pmid/pid from the parent's mlink rows. On the Fossil SCM repository that is ~180x canonical's row count (a full sweep would write ~19M rows against fossil's own 104,880), and because delta-chain order (#102/#126) routinely crosslinks a child before its parent, the parent's mlink rows were usually absent, so ~97% of rows defaulted to pid=0 -- wrong parent-column data independent of the volume. Rewrite it to diff each check-in's file list against its primary parent's expanded manifest -- the same diff canonical Fossil computes in add_mlink (src/manifest.c) and the rebuild path already does in internal/verify.rebuildMlinks -- and emit a row only for a file that was added, modified, deleted, or renamed. pmid/pid now come from the parent MANIFEST, which is always present once a check-in is eligible to link, so pid is correct regardless of crosslink order. The parent expansion runs through the sweep's shared content cache, so each parent's delta chain is walked at most once across all its children. Measured on the Fossil SCM corpus: 127,775 mlink rows (canonical: 104,878) with pid populated on 74% of rows (was ~3%); full-repo crosslink completes in ~15 minutes rather than the prior ~hour. The residual gap is 95 PGP-clearsigned parent manifests deck.Parse cannot yet strip, whose ~98 children fall back to the old all-added behavior -- a pre-existing deck limitation tracked separately. check-in: 81fbc0be5f user: [email protected] tags: trunk | |