Timeline

The delta format holds integers below 2^32 only, so a producer must not write a target length or copy offset at or past that. Create now returns ErrTooLarge for such inputs; Deltify then stores the content whole, as fossil does when a delta is not worthwhile, and the other callers propagate the error. Apply's header check becomes >= so it refuses exactly what Create will not write. The bound is tested on lengths, as a direct test needs a 4 GiB input. check-in: 38eddf43f1 user: [email protected] tags: main, trunk

Cloning the Fossil SCM corpus (67.6k blobs) took 293-302 s at 1251-1287 MB against canonical fossil's 20.8 s at 205 MB. A CPU and allocation profile of one clone found the cost was not delta expansion or hash verification.

27.2% of all CPU was a single un-memoized statement. content.AvailabilityCache memoizes the delta-chain walk but not the "SELECT rid FROM blob WHERE uuid=?" that starts it, so manifest.missingCheckinRefs re-seeked the blob.uuid index once per F-card of every check-in -- the same file hash resolved again for every revision that did not change it. That index is far larger than SQLite's page cache, so each seek was three page reads: pread syscalls were 34.7% of total samples. The walk it guards was under 1% of its own cost.

Separately, materializing 8.27 GB of content allocated 360 GB. delta.ApplyInto starts at a 64 KiB cap deliberately -- the target length is a wire claim and a few bytes can declare 4 GiB -- and then climbs to the real size through append's ~1.25x large-slice growth, ~3.4x the target in discarded intermediates, leaving slack that content.Cache.clip copied away again (203.9 GB and 59.8 GB respectively).

Three changes:

- Memoize uuid -> rid in AvailabilityCache. Hits only. Within one crosslink run a hit cannot go stale, because a sweep and a cascade only ever add blob rows; a miss can, because ridOrPhantom and blob.StorePhantom create rows mid-run, and remembering an absence would leave an artifact deferred after the content it waits on arrived. The scope that makes this sound is now stated on the type, and TestAvailabilityCacheMustNotSpanAPurge pins its one boundary: shun.Purge deletes blob rows, so a cache must not span one.

- Size delta.ApplyInto's output exactly. scanOutputSize walks the command stream first without allocating, checking every copy against bytes the source holds and every insert against bytes the delta carries, so the size it returns is a measurement rather than the header's claim. Any stream it cannot walk to a clean terminator falls back to the existing growth path, so malformed and hostile deltas allocate and fail exactly as before. The buffer now arrives tightly sized, which also makes clip's copy a no-op.

- Cut crosslinkCacheBytes from 256 to 128 MiB. Each byte of that budget costs roughly five bytes of peak RSS -- GOGC doubles it into the heap goal, and the scavenger is paced off GC cycles, of which a faster sweep runs fewer. At 256 MiB the first two changes gave 205.7 s but 1473 MB, faster at the cost of memory. The measured curve (32/64/128/256 MiB: 369.6 s/318 MB, 292.3 s/447 MB, 249.4 s/812 MB, 205.7 s/1473 MB) puts 128 MiB where both axes improve.

Result on the same corpus: 245.4 s at 780 MB, from 293-302 s at 1251-1287 MB -- 17% less wall time and 39% less peak RSS, with instructions retired down from 3.37T to 2.48T. The gap to canonical narrows from 14.1x/6.3x to 11.8x/3.8x.

Derived tables are unchanged against an unmodified build cloning the same server: blob 67530, event 27099 (identical per type), mlink 104919, plink 23741, tagxref 51766, leaf 1426, filename 1496. The corpus parity gate passes.

What remains is structural rather than a missing optimization: a clone expands every blob twice, once in the receive-time phantom-fill cascade and once in the final whole-repository sweep, where canonical crosslinks each artifact from content it already holds in hand at receive time. Cache.Expand splits 45.4 s / 35.3 s across those two passes. Collapsing them is a change to the receive path and belongs in its own issue. check-in: 8b16c91f5c user: [email protected] tags: trunk

emitMatches walked a hash bucket's entire collision chain at every target position with no limit. rollingHash is a weak multiply-accumulate over a 16-byte window, so low-entropy content — repeated tokens, runs of whitespace, ordinary SQL and C boilerplate — funnels an enormous number of source windows into one bucket, making Create quadratic in target length. Measured on the issue's repro (repeated 8-byte token, one 10-byte edit): 32 KiB 0.133s, 64 KiB 0.496s, 128 KiB 1.950s, 256 KiB 7.765s — ~3.9x per doubling — against 0.013s for a 128 KiB high-entropy control.

Bound the walk at maxChainCandidates, matching the `int limit = 250` that fossil's delta_create (src/delta.c) puts on the identical loop. The same walk also computed each candidate's match length twice, once to test it against the nHash threshold and again to read the value; compute it once.

Extract the walk into bestMatch so the bound is testable on a candidate count rather than on wall-clock time, which cannot flake on loaded CI.

Delta output changes, as fossil's own limit makes it change: a capped search can settle for a shorter match, so the encoder emits more copy commands. This lands our output on canonical fossil's — 128 KiB: 266 bytes vs fossil's 260; 1 MiB: 2142 vs 2120 — where before the unbounded walk produced 44 and 50 bytes and paid quadratic time for them.

After: 128 KiB 1.950s -> 0.018s, and 1 MiB 125.036s -> 0.062s.

Verified the deltas still apply byte-for-byte, by our Apply and by canonical fossil 2.28's test-delta-apply.

Co-authored-by: Dan Mestas <[email protected]> check-in: b1b6c09b41 user: [email protected] tags: trunk, v0.7.0

delta.Apply's one-time capacity upgrade (#127) only engages above the 64 KiB threshold, so it is a no-op for the typical source file under 64 KiB. But below the threshold Apply already sizes its output buffer exactly, so there was never a 5x overhead there to remove -- extending the threshold buys nothing. The real per-link cost of a chain walk was elsewhere, and a profile of a plain Expand of a deep sub-64 KiB chain shows it: a fresh ~32 KiB zlib history window allocated per delta load (~27% of bytes), one full-size Apply output buffer per link (~16%), and a fresh inflate buffer per link. All three are O(depth) allocations for a single cold Expand, which is O(depth), making a full-history walk O(n^2).

Attack the per-link constant inside the one choke point every expansion goes through, content.expandChain, leaving the on-disk chain shape untouched:

- delta.ApplyInto(dst, source, delta) reconstructs into a caller-supplied buffer, reusing its capacity. Apply is now ApplyInto(nil, ...), unchanged. A reused buffer's capacity is proven prior work, not a wire-header claim, so it is a safe starting size; the anti-DoS growth path (growOutputCap, keyed off len(delta) not the targetLen header) is preserved for buffers a target outgrows. - blob.Inflater loads a run of blobs through one zlib reader, reset per blob instead of rebuilt, paying the ~32 KiB window setup once per chain. Each Load still returns a freshly owned buffer, so no returned-slice-aliasing contract leaks to callers. - expandChain's plain-Expand replay (put == nil) ping-pongs two output buffers -- each link's target reuses the buffer from two links back, whose content was already consumed as the previous link's source, so dst and source never alias. The caching path (put != nil) is unchanged: put takes ownership of every materialized buffer, so those cannot be reused underneath it, but it still shares the reused zlib reader for its delta loads.

Measured on Apple M4 Pro, BenchmarkExpandOldestVersion, median of 3, 40x:

sub-64 KiB (~26 KB file), Expand of the oldest revision: depth 11: 883 KB -> 232 KB/op (3.8x), 1.12 ms -> 0.90 ms depth 49: 3.57 MB -> 341 KB/op (10.5x), 4.55 ms -> 2.03 ms depth 199: 14.2 MB -> 772 KB/op (18.4x), 14.84 ms -> 7.25 ms

>64 KiB (~206 KB file), no regression -- ping-pong + reader reuse help it too: depth 11: 4.11 MB -> 1.27 MB/op (3.2x), 3.40 ms -> 2.74 ms depth 49: 16.5 MB -> 1.81 MB/op (9.1x), 13.02 ms -> 7.69 ms depth 199: 65.2 MB -> 3.92 MB/op (16.6x), 46.3 ms -> 26.6 ms

Sub-64 KiB allocation is now near-flat in depth instead of linear in depth*targetLen. expand_reuse_test.go guards the reuse; delta and blob gain allocation-reduction unit tests. check-in: a2bcca2535 user: [email protected] tags: trunk

* fix: bound delta.Apply growth per-command and cut ~5x allocation overhead (#81, #82)

Two related fixes to the same command loop in delta.Apply:

- #82: check len(output) against the declared target length after every copy/insert command, not just at the ';' terminator. A malformed delta with many copy commands could previously grow output well past both targetLen and maxDeltaTargetSize before anything noticed, since the only comparison happened at the end.

- #81: once a command has actually executed (proving the delta contains real work, not just an inflated header claim), upgrade output's capacity in a single step sized off len(delta) rather than relying on append's own ~1.25x large-slice growth factor to climb from the 64 KiB initial cap to the final size. That climb was landing near 5x the target size in cumulative allocation; the single upgrade restores near-exact sizing for honest deltas while keeping the tiny-payload/ huge-header attack closed, since a delta with no real commands never reaches the upgrade.

Added TestApply_LoopBoundsAgainstTargetDuringCommands (regression for #82: many copy commands exceeding a small declared target are rejected early, with allocation growth bounded near the declared target rather than the full command sequence) and TestApply_LargeTargetAllocationBounded plus BenchmarkApply_LargeTarget (regression/measurement for #81: a 1 MiB target now allocates ~1.06x its size instead of the prior ~5x).

Cross-checked against fossil 2.28 for wire-format compatibility; full non-dst suite and dst simulation suite (90 scenarios) pass.

* fix: address delta.Apply review nits from #81/#82

Rebuild the #81 regression test/benchmark around a many-small-copy delta shape (4096 copy commands into a 1 MiB target) instead of a Create()-derived single-big-insert delta. The single-insert shape allocates ~1.06x on both fixed and unfixed code (grows once), so it neither exercised the ~5x append-growth-climb regression nor would have caught a future reintroduction of it. Verified against a reverted delta.go: unfixed code now measures ~5.29x on this benchmark, fixed code ~1.06x.

Also move the len(output) > targetLen bound checks ahead of their append calls in both the copy and insert command branches, so an over-target command is rejected before its bytes are appended rather than after -- matches the #82 issue wording exactly, closing the (non-exploitable, since a single command's overshoot is already bounded and resident) gap the reviewer flagged as a nit. check-in: 355e7d9910 user: [email protected] tags: trunk

* test: route fossil-binary tests through one REQUIRE_FOSSIL_BIN helper

REQUIRE_FOSSIL_BIN=1 turns a missing fossil binary into a test failure instead of an invisible skip, but only internal/manifest honored it. The six other fossil-dependent sites each did their own exec.LookPath + unconditional t.Skip, so on those a run with no fossil was byte-identical to a passing one.

Add testutil.RequireFossilBin, which resolves the binary the same way FossilBinary does (FOSSIL_BIN override, then PATH) and either skips or fails based on REQUIRE_FOSSIL_BIN. Route testutil.NewTestRepo, both internal/delta CLI round-trips, the four internal/merge validation tests, the internal/xfer sanity check, and the ls-branch-tip repro through it, deleting their bespoke lookup+skip logic. CI already sets REQUIRE_FOSSIL_BIN=1 on the unit-test steps, so this makes all seven sites fail loud on absence while local runs without fossil stay opt-in.

* test: route every fossil-binary gate through RequireFossilBin

The first pass left six sites still deciding for themselves whether a missing fossil binary is a skip or a failure. Three of them (internal/sync's startFossilServer and its two integration tests, manifest's two rebuild tests, the root TestPhaseB_Integration, and hash's TestSHA1_FossilValidation) skipped unconditionally, so with REQUIRE_FOSSIL_BIN=1 and no fossil on PATH those packages still reported ok -- the exact silent pass the flag exists to prevent. The other three had hand-rolled copies of the flag check, which is the duplication that let the original drift happen.

All of them now call testutil.RequireFossilBin, and HasFossil is gone: it was a second, skip-only way to ask the same question, and it is what produced the sites that ignored the flag.

Also validate the FOSSIL_BIN override with LookPath, so a stale path reports "fossil not found" instead of surfacing later as a raw fork/exec error.

* test: resolve fossil binary via RequireFossilBin at remaining call sites

* fix: correct stale module path missed by rebase auto-merge

Three imports added by earlier commits in this branch predated the rebase target and had no corresponding line on main to conflict against, so git's auto-merge carried the old module path through verbatim. check-in: f804878bd3 user: [email protected] tags: trunk

* fix: correct go.mod module path to go-libfossil (#76)

The module directive still declared github.com/danmestas/libfossil after the repository was renamed to go-libfossil. pkg.go.dev/godoc scraping matches the declared module path against its hosting path, so the stale path broke SDK doc generation.

Rename the module path across all four in-repo modules (root, the two db/driver submodules, observer/otel, and docs/site) and every internal import, plus the release workflow, the doc-generation script, and the docs.

No tag has been published under the go-libfossil path yet, so the cross-module requires cannot resolve from the proxy. Add replace directives pointing the in-repo requires at their local directories so GOWORK=off builds and the CI matrix resolve without a published version. These replaces are ignored by downstream consumers and can be dropped once new-path tags are cut.

* Guard releases against replace directives; regenerate SDK docs

The go-libfossil rename bridges the in-repo modules with `replace` directives. Go refuses a main module that carries them, so a tag cut while they are present breaks `go install .../cmd/libfossil@<tag>` and every `@version` fetch of the driver submodules. Add a Release workflow step that fails the job before anything is published if any tracked go.mod still contains a replace directive, so the bridge is a CI-enforced TODO rather than a silent landmine. The step comment records what unblocks a release: re-tag the submodules with the corrected module paths, bump the root requires, then drop the replace blocks.

Regenerate the SDK reference docs. The previous pass only substituted URLs over the pre-existing generated files, leaving stale signatures (Repo.Timeline was still documented as taking LogOpts rather than TimelineOpts). Note that gomarkdoc must be run outside a git worktree — inside one it silently strips every source link.

Also reword the README rename note, which read as though the import path had not changed, and re-sort the ncruces require block. check-in: 6273a0750d user: [email protected] tags: trunk

* fix: store delta content against a phantomized base instead of discarding it

A delta artifact arriving before its base is normal steady-state behavior during a Fossil transfer, not a fault. The receive path used to discard such deltas outright and phantomize the target instead, so a phantom only ever cleared when its base happened to already be present -- resolution advanced one delta-chain hop per round, making clones of deep-history repos provably unable to converge within MaxRounds.

Match canonical Fossil's content_put_ex/rid_from_uuid model: resolve the delta's source via a create-phantom-if-missing lookup, persist the delta bytes unconditionally (reading the target size straight out of the delta header, so the base never needs to be readable to store it), and record the delta-to-source link immediately. Availability resolves lazily via the existing content.IsAvailable chain walk and manifest.Crosslink's deferred retry -- both already correct and untouched here.

storeReceivedFile is the single call site shared by clone, sync-pull, and server-push, so fixing it here also closes out the delta-before-base sentinel handling on the pull and push paths (issue #57, folded into this fix): neither path needs to special-case the condition anymore because it stops being an error.

Verified against the pikchr corpus via a loopback fossil server: rounds dropped from 100 (exhausted) to 3, files received from 63,155 (22.2x) to 2,980 (1.05x), and the phantom table empties out completely.

* fix: converge delta storage on canonical, close integrity gap, harden wire parsing

Three defects from review, all in the raw-delta path added for the delta-before-base fix, all reachable from an untrusted peer:

1. delta.getInt accumulated v = v*64 + digit with no overflow check. A twelve-byte header of maximum-value digits wraps to exactly math.MaxUint64, which casts to int64(-1) -- the same sentinel blob.size uses for "phantom". That silently corrupted the row: real content reporting as a phantom, with no phantom-table entry to make it re-requestable. getInt now rejects overflow instead of wrapping, and parseTargetLen (shared by Apply and OutputSize) bounds the parsed target length to a sane maximum before it ever reaches an allocation or a size column.

2. StoreDeltaRaw panicked on a zero-length delta payload. storeReceivedFile guarded payload == nil but not len(payload) == 0, and nothing between wire decode and storage checked for it -- a peer sending an empty delta file card crashed the process. Wire data gets an error return, not an assertion; OutputSize and StoreDeltaRaw are now consistent with delta.Apply's existing "empty delta is an error" contract.

3. The receive-time split (store raw only when the base happens to be unavailable, otherwise eagerly expand and verify) left a gap an attacker chooses the timing of: withhold the base, and a delta claiming any UUID goes down the unverified raw path. Nothing downstream checked it either -- content.Expand applied deltas with no hash check, and content.Verify was reachable only from repo rebuild and one integration test.

Closed by converging fully on canonical's model: every delta is now stored raw and linked via a create-phantom-if-missing lookup, regardless of whether its base is currently available -- receive time no longer branches on availability at all. Verification moves to the one choke point every expansion path already goes through: content.Expand now hashes its result against the row's claimed UUID before returning it, catching a mismatch regardless of which caller or how the chain assembled. content.Verify becomes a thin wrapper over the same check.

Storage-layer and end-to-end regression tests added for all three, including the exact claim-one-UUID/ship-a-delta-for-another/withhold-the- base reproduction, guarded against degenerating into a no-op (delta must actually compress, base must be genuinely absent, rejection must be a real error rather than merely different bytes).

Re-verified against pikchr (2,842 artifacts) and wapp (1,039): pikchr converges in 3 rounds, 2,980 files, 0 phantoms; wapp converges in 3 rounds, 1,115 files, 1 phantom outstanding (pre-existing, unrelated -- clone.go has no DB-phantom rescan the way the pull path's loadDBPhantoms does; separately tracked). On-disk repository size is close to but slightly smaller than storing everything as raw delta bytes should produce (pikchr 6.81MB -> 6.73MB, wapp 5.64MB -> 5.46MB): zlib on full text already captures most of the redundancy an explicit delta would, so the win here is mainly integrity and convergence, not size.

* fix: bound delta allocation to actual work, validate delta source hex

Two defects from round-two review, plus a stray comment.

1. delta.Apply preallocated its output buffer directly from the delta header's claimed target length (make([]byte, 0, targetLen)). That claim only had to clear maxDeltaTargetSize (4 GiB) to be trusted as an allocation size, so a handful of wire bytes declaring a near-4 GiB target triggered an allocation sized to the claim, not to any work actually done. Harmless as a one-off parse spike; durable once a delta can be stored unverified (see blob.StoreDeltaRaw, this branch) -- a hostile peer plants one such delta and every later Expand of that row repeats the attempt. The initial capacity is now a small fixed bound; append grows it normally for legitimately large content instead of trusting the header's own claim about itself.

2. storeReceivedFile validated the target UUID's hex-ness (hash.IsValidHash) but not the delta source's. The blob table's CHECK constraint only bounds uuid length, not hex-ness, so a non-hex string of valid length passed straight through to blob.StorePhantom and became a permanent, unfillable row -- one the pull path's loadDBPhantoms would then re-request every sync round, forever. deltaSrc now gets the same check as the target uuid, at the same place.

Also deleted a stray doc-comment line left over from an earlier refactor.

Regression tests for both, RED-verified against reverted code before being fixed: the allocation test had to be corrected mid-way through -- delta headers encode their integer in Fossil's own base-64 alphabet, not decimal, so an initial fmt.Sprintf("%d", ...) fixture parsed as a different (still large but differently-bounded) value and passed vacuously; appendInt is the real encoder and is used directly now, with a round-trip sanity check against OutputSize guarding the fixture itself. check-in: 89535a6c42 user: [email protected] tags: trunk

libfossil is a pure-Go implementation of the Fossil SCM, reading and writing the same .fossil SQLite repository format as upstream Fossil.

This is the first public release of code that previously lived in a private repository. The module path is github.com/danmestas/libfossil and the binary is named libfossil to avoid $PATH collisions with upstream fossil.

Included: - Repo lifecycle (create, clone), checkout/checkin, timeline, merge, diff, annotate, shun, tag, branch, bisect, stash, undo/redo - Sync-over-HTTP protocol (client and handler) - Two SQLite drivers: modernc (pure Go, default) and ncruces (wasm-capable) - Observer interfaces (SyncObserver, CheckoutObserver) with an optional OpenTelemetry implementation in observer/otel (separate submodule) - Deterministic simulation testing harness with BUGGIFY fault injection - Kong-based embeddable CLI under cli/ and a standalone binary under cmd/libfossil/ check-in: f5744359d3 user: [email protected] tags: trunk