Timeline

* fix: zero-length file commit panic and stale missing-file carry-forward

Two related bugs in the checkin path:

#68 - Committing a zero-length file panicked in blob.Store, which treated an empty artifact (a normal, well-known Fossil blob) as invalid input. Worse, manifest.Checkin's postcondition defer fired unconditionally during the panic's stack unwind and re-panicked with its own unrelated "manifestRid must be positive" message, masking the real cause. Fixed both: blob.Store now accepts zero-length (and nil) content, and Checkin's defer re-panics with the original panic value instead of asserting a postcondition that was never going to hold mid-unwind. Fixing the panic surfaced a second bug in the same family: buildCommitFiles used len(Content) > 0 to detect "already loaded from disk," which can't distinguish a genuinely empty file from one not yet loaded, so a newly added empty file failed with "no RID for unchanged file." Replaced the length sentinel with an explicit loaded-set.

#79 - A tracked file deleted from disk without Unmanage/Remove was silently carried forward into the new manifest with its last-committed content, rather than being detected. Verified against fossil 2.28: canonical fails a commit when a missing tracked file is in scope (either the implicit "commit everything" case, or explicitly named), but leaves an out-of-scope missing file alone. buildCommitFiles now stats every in-scope tracked file before treating it as unchanged and errors clearly, naming the file, when it's gone; a file outside the commit's scope (relevant only to an explicit Enqueue) is left untouched exactly as before. Updated the two existing tests that had documented the old silent-carry-forward behavior as deliberately deferred to this issue.

* fix: add CHANGELOG entry and unwrap-safe stat checks for #68/#79

Adds the CHANGELOG entries the #79 acceptance criteria required for a user-visible behavior change (missing tracked file now hard-fails a commit instead of silently carrying stale content forward) and for the #68 zero-length-file panic fix.

Also switches checkin.go's two os.IsNotExist(err) checks to errors.Is(err, os.ErrNotExist), which correctly unwraps the %w-wrapped os.ErrNotExist that MemStorage.Stat returns. Production was already correct (OSStorage.Stat returns a raw *PathError), but the tests were only exercising a generic fallback error branch instead of the intended "tracked file missing from disk" message. check-in: 0351167ae5 user: [email protected] tags: trunk

* fix: clone emits cfile with stored content per sec7.2, not expanded file

Our clone server was building xfer.FileCard from cache.Expand(...) -- every artifact fully decompressed, and every delta materialized into full content -- where sec7.2 requires cfile cards carrying the repository's stored compressed content retransmitted verbatim. On real repositories this ran 80-220x more bytes on the wire than the protocol intends, and was the real blocker behind #88's clone-batching fix being unable to bound repositories much past libfossil's own size.

emitCloneBatch now reads blob.Load's stored bytes directly (full content for a whole artifact, delta bytes against DeltaSource for a deltified one) and emits xfer.CFileCard instead of xfer.FileCard, with DeltaSrc and USize (the full reconstructed size, from blob.size) populated correctly for delta rows. content.DeltaSource is exported so this and other callers outside internal/content can ask whether a rid is stored as a delta without duplicating the query.

No client changes were needed: session.go already routes CFileCard through the same handleFileCard path as FileCard, and storeDeltaContent already stores a received delta exactly as given via a create-phantom-if-missing lookup on its source, resolving delta-before-base arrival the way #53's phantom storage model was built for.

DefaultCloneBatchBytes is left at its current value rather than retuned toward canonical's 5,000,000 -- that constant's tuning is #109's territory now that bytesSent tracks stored bytes instead of expanded ones, and retuning it here risks colliding with that work.

* fix: expand deltified rows during clone instead of sending delta cfile

Code review reproduced a real interop regression: emitCloneBatch sends cards in ascending-rid order, but content.Deltify always deltifies the older artifact against the newer one, so a delta's DeltaSource almost always names a rid greater than the delta card's own rid. That means a delta cfile routinely reaches the client before the very source card it names. libfossil's own client tolerates the forward reference (storeDeltaContent phantoms the unseen source and resolves it later), but real fossil 2.28 does not -- the source stays a genuine, unfilled phantom, and content_get() on it during the client's post-clone rebuild pass returns without initializing its output blob, tripping an assertion in blob_copy (blob.c:397) and aborting the client mid-rebuild. Verified independently against a real fossil 2.28 binary cloning a libfossil server holding a delta chain.

emitCloneBatch now expands a deltified row to full content before sending it as a plain (non-delta) cfile, rather than forwarding the stored delta bytes with DeltaSrc set. The card is still zlib-compressed, so the wire-size win over the old FileCard/Expand path survives for every row; only the additional delta-retransmission savings are given up, and only for deltified rows. Re-enabling that needs either a source-before-dependent send order or confirmed forward-reference tolerance in canonical, and is left as a follow-up rather than blocking this fix.

TestEmitCloneBatchExpandsDeltifiedRows (renamed from TestEmitCloneBatchRetransmitsDeltaAsDelta) now pins the new contract: no DeltaSrc, full-sized payload. TestCloneRealFossilWithDeltaChain adds an end-to-end check against a real fossil binary, when one is on PATH. check-in: 0ca5133c62 user: [email protected] tags: trunk

* fix: bound delta.Apply growth per-command and cut ~5x allocation overhead (#81, #82)

Two related fixes to the same command loop in delta.Apply:

- #82: check len(output) against the declared target length after every copy/insert command, not just at the ';' terminator. A malformed delta with many copy commands could previously grow output well past both targetLen and maxDeltaTargetSize before anything noticed, since the only comparison happened at the end.

- #81: once a command has actually executed (proving the delta contains real work, not just an inflated header claim), upgrade output's capacity in a single step sized off len(delta) rather than relying on append's own ~1.25x large-slice growth factor to climb from the 64 KiB initial cap to the final size. That climb was landing near 5x the target size in cumulative allocation; the single upgrade restores near-exact sizing for honest deltas while keeping the tiny-payload/ huge-header attack closed, since a delta with no real commands never reaches the upgrade.

Added TestApply_LoopBoundsAgainstTargetDuringCommands (regression for #82: many copy commands exceeding a small declared target are rejected early, with allocation growth bounded near the declared target rather than the full command sequence) and TestApply_LargeTargetAllocationBounded plus BenchmarkApply_LargeTarget (regression/measurement for #81: a 1 MiB target now allocates ~1.06x its size instead of the prior ~5x).

Cross-checked against fossil 2.28 for wire-format compatibility; full non-dst suite and dst simulation suite (90 scenarios) pass.

* fix: address delta.Apply review nits from #81/#82

Rebuild the #81 regression test/benchmark around a many-small-copy delta shape (4096 copy commands into a 1 MiB target) instead of a Create()-derived single-big-insert delta. The single-insert shape allocates ~1.06x on both fixed and unfixed code (grows once), so it neither exercised the ~5x append-growth-climb regression nor would have caught a future reintroduction of it. Verified against a reverted delta.go: unfixed code now measures ~5.29x on this benchmark, fixed code ~1.06x.

Also move the len(output) > targetLen bound checks ahead of their append calls in both the copy and insert command branches, so an over-target command is rejected before its bytes are appended rather than after -- matches the #82 issue wording exactly, closing the (non-exploitable, since a single command's overshoot is already bounded and resident) gap the reviewer flagged as a nit. check-in: 355e7d9910 user: [email protected] tags: trunk

* manifest: create forumpost on demand, matching canonical rebuild

Crosslink's candidate query names the forumpost table unconditionally, but canonical fossil creates it lazily -- only once a forum artifact needs it -- and fossil rebuild drops it for a repository that never populated it. Crosslink now recreates the table itself before the sweep runs, matching canonical's on-demand behavior instead of erroring with "no such table: forumpost" against a rebuilt repo.

Audited the rest of the on-demand-in-canonical set (ticket, ticketchng) against the current schema: both are created eagerly by canonical at init and survive rebuild, so they don't share this problem.

Added TestCrosslinkForumPostTableDropped (drops the table directly) and TestCrosslinkAfterFossilRebuild (drives an actual fossil rebuild first) to cover the regression, and updated the existing fossilbin comparison test's comment now that the rebuild-last ordering is no longer required.

* manifest: visit crosslink candidates in delta-chain order

Crosslink swept candidates in ascending rid order, but content_deltify stores a blob's older versions as deltas against its newer ones, so delta chains run the opposite direction: high-rid root down to low-rid ancestors. An ascending sweep's first candidate of a chain therefore materializes the whole chain, and the rest are consumed arbitrarily far in the future -- the working set of a whole-repository sweep was the entire repository expanded (8.2 GiB for the Fossil SCM repository).

Candidates are now topologically sorted so a delta's base is always visited before its dependents (deltaChainOrder), matching Fossil's own rebuild_step shape: root first, each descendant one delta application from an already-expanded predecessor. The working set this sweep needs is now bounded by how many chains are in flight at once, not by repository size.

That reordering also changes the order artifact side effects get written in, since content expansion and artifact crosslinking share one loop. Most of what Crosslink writes turned out to already be a pure function of its own artifact (event/plink/mlink/tagxref-origin rows) and needed no change. Two things generalize across the whole plink graph rather than any one artifact and were genuinely order-sensitive:

- leaf table bookkeeping, which inserted the new checkin and deleted its parent -- correct only when parents are linked before their children, which delta-chain order no longer guarantees. - tag propagation, which ran once per checkin from its immediate parent -- reachable only to children plink already held at that moment.

Both are now full, order-independent repair passes (repairLeafTable, repairTagPropagation) run once after the sweep, mirroring Fossil's own manifest_crosslink_end + leaf_rebuild() shape: defer the order-sensitive work and repair it once rather than trying to preserve order through the sweep. tag.propagate's mtime-gated, plink-live-at-call-time design turned out to already be confluent -- replaying every self-declared tag origin once, in any order, converges on the same tagxref state -- so the repair is just doing that instead of depending on visiting order to get there implicitly. Tech-note event replacement and forum thread titles turned out to already have the same confluent shape and needed no change; left comments explaining why.

Also fixes a tag.PropagateAll crash on any origin with a NULL tagxref value (e.g. CrosslinkCluster's cluster tag), surfaced by the new propagation repair actually reaching those rows for the first time.

Regression coverage: a real three-generation checkin chain, forced into a genuine manifest delta chain, that would have left two of three checkins untagged and all three marked leaf under the old per-checkin propagation and incremental leaf bookkeeping.

* fix: address review nits on crosslink order-sensitivity and delta cycles

- deltaChainOrder now returns an error instead of panicking when the candidate delta graph fails to drain. The delta table is on-disk data that can arrive over sync from an untrusted peer, so a cycle (corrupt or hostile repo) should surface as an error rather than crash the process.

- snapshotDerived's digest now covers tagxref (rid/tagid/tagtype/ srcid/origid/value), and TestFossilBinaryReadsCrosslinkedRepo compares it against canonical fossil rebuild -- closing the gap the test previously called out by name ("tagxref is the notable gap").

- Added TestFossilBinaryReadsCrosslinkedRepoBranchAndMerge, exercising the order-sensitive paths incrementalHistory's single-branch history cannot reach: a feature branch diverging from trunk, and a merge commit whose own branch tag must come from its primary parent, not the merged-in one. event/plink/tagxref match canonical fossil row-for-row on this topology.

leaf and mlink are deliberately excluded from that comparison: the fixture surfaced that both have pre-existing gaps against canonical fossil (repairLeafTable counts any plink edge rather than only primary-parent edges; insertCheckinMlinks writes a row for every F-card rather than only changed ones) that predate this fix and are out of scope for #102/#103.

* fix: store received wire frames verbatim instead of re-encoding

Clone reproduced content correctly but not bytes: the receive path decompressed every cfile frame and re-compressed it with blob.Compress's unparameterized zlib writer, so a clone's blob.content never matched its source's — about 0.5% larger on the wapp corpus, purely from re-encoding frames that were already correctly encoded on the wire.

xfer's cfile decoder now hands back StoredBlob alongside the decompressed Content: the same bytes recast into Fossil's on-disk blob format ([4-byte BE size][zlib data]) without a second zlib pass — verbatim when the wire already sent that format (Fossil's send_compressed_file case), or the received zlib stream with a length prefix bolted on when it didn't (our own encoder's plain-zlib case). storeReceivedFile/storeDeltaContent/ storeResolvedContent and blob.StoreDeltaRaw thread that through via a new blob.EncodeForStorage helper, which writes the verbatim bytes when given and only falls back to Compress when there's nothing to preserve — covering both the clone/pull and server-push receive paths.

Locally authored content (never received over the wire) always takes the Compress fallback, so #71's deltification policy is untouched. The clone BUGGIFY corruption test drops storedBlob alongside the corrupted content it's exercising, so verbatim storage can't mask the corruption it exists to catch.

Adds a corpus-level regression test cloning a real multi-revision fossil repo and asserting sum(length(content)) matches between source and clone.

* test: use RequireFossilBin instead of removed HasFossil helper

testutil.HasFossil was removed upstream in favor of RequireFossilBin; this test's skip guard needed the same update to compile against current main. check-in: cc42ad8829 user: [email protected] tags: trunk

* test: route fossil-binary tests through one REQUIRE_FOSSIL_BIN helper

REQUIRE_FOSSIL_BIN=1 turns a missing fossil binary into a test failure instead of an invisible skip, but only internal/manifest honored it. The six other fossil-dependent sites each did their own exec.LookPath + unconditional t.Skip, so on those a run with no fossil was byte-identical to a passing one.

Add testutil.RequireFossilBin, which resolves the binary the same way FossilBinary does (FOSSIL_BIN override, then PATH) and either skips or fails based on REQUIRE_FOSSIL_BIN. Route testutil.NewTestRepo, both internal/delta CLI round-trips, the four internal/merge validation tests, the internal/xfer sanity check, and the ls-branch-tip repro through it, deleting their bespoke lookup+skip logic. CI already sets REQUIRE_FOSSIL_BIN=1 on the unit-test steps, so this makes all seven sites fail loud on absence while local runs without fossil stay opt-in.

* test: route every fossil-binary gate through RequireFossilBin

The first pass left six sites still deciding for themselves whether a missing fossil binary is a skip or a failure. Three of them (internal/sync's startFossilServer and its two integration tests, manifest's two rebuild tests, the root TestPhaseB_Integration, and hash's TestSHA1_FossilValidation) skipped unconditionally, so with REQUIRE_FOSSIL_BIN=1 and no fossil on PATH those packages still reported ok -- the exact silent pass the flag exists to prevent. The other three had hand-rolled copies of the flag check, which is the duplication that let the original drift happen.

All of them now call testutil.RequireFossilBin, and HasFossil is gone: it was a second, skip-only way to ask the same question, and it is what produced the sites that ignored the flag.

Also validate the FOSSIL_BIN override with LookPath, so a stale path reports "fossil not found" instead of surfacing later as a raw fork/exec error.

* test: resolve fossil binary via RequireFossilBin at remaining call sites

* fix: correct stale module path missed by rebase auto-merge

Three imports added by earlier commits in this branch predated the rebase target and had no corresponding line on main to conflict against, so git's auto-merge carried the old module path through verbatim. check-in: f804878bd3 user: [email protected] tags: trunk

* fix: degrade unparseable clone VERSION/SEQNO instead of erroring (#95)

parseClone routed both tokens through strconv.Atoi before the §8.1 digit-only rule ever got a chance to apply, so a token Atoi could not parse at all (not just one that fails digit-only) hard-errored the whole message instead of degrading gracefully. Carry the raw tokens forward as an UnknownCard on parse failure, matching how UnknownCard already handles a non-decimal clone_seqno, so a later card in the same message still decodes.

* fix: reject an oversized §4.1 declared length before trying raw zlib (#110)

xfer.Decode tried the unprefixed-zlib framing before ever reading the §4.1 container's declared length prefix. A peer declaring a length at or above 0x08000000 produces a prefix whose first two bytes present as a structurally valid zlib CMF/FLG header (RFC 1950), so a large declared length aliases as the start of a raw zlib stream and gets first refusal on a legitimate compressed container. Read the declared length whenever the body opens with a valid zlib header and reject it against MaxDecompressedBytes before attempting format 1 at all. check-in: 2b9694f525 user: [email protected] tags: trunk

The internal/hash import was added by the #93 T-card validation fix on a branch cut before the go-libfossil module rename merged. The rename's automated rewrite only touched import lines that existed at the time, so this newly-added import kept the old path and broke go build ./... on main after both merged.

CI's go build ./cmd/libfossil/ and go test $(go list ./... | grep -v /dst) didn't catch it: go list ./... drops a package that fails to compile from its output rather than failing the command, so the broken package silently dropped out of the test run instead of failing the build. check-in: 7764ecf6ff user: [email protected] tags: trunk

* fix: J card decodes value, stores name verbatim (#90)

The J-card parser had the escape direction inverted: it decoded the field name and exposed the value with its escape sequences intact. Both the spec (4.7.8) and canonical fossil (manifest.c) do the opposite -- defossilize only the value, compare the raw name.

Swap both the parse and marshal directions so the value round-trips decoded and the name stays verbatim.

* fix: T card escape-encodes the tag name on marshal (#91)

A tag/branch name containing a space was written verbatim, so 'my branch' produced 'T *sym-my branch *' -- a structurally valid but silently wrong card that re-parses as two tokens.

Align with canonical fossil: escape-encode the name on marshal and escape-decode it on parse, leaving the target token verbatim in both directions (4.7.16). TagCard.Name now holds the decoded form, so tagOrderKey reattaches the sign without re-decoding.

* fix: enforce two skipped T-card §4.7.16 checks on parse (#93)

parseTCard validated the sign and ordering key but accepted two malformed shapes from untrusted peers:

- a name that after the sign is entirely hex digits, indistinguishable from a hash target; - a target that is neither a valid artifact hash nor the literal '*'.

Reject both with errors (never a panic). hash.IsValidHash reuses the same target check the primary uuid path uses; the '*' and non-empty guards run first since it panics on an empty string.

* fix: T card decodes and encodes the tag value (#91), guard hex names on marshal (#93)

§4.7.16 applies the same escape rule to all three T-card tokens: the name and value are escape-decoded on parse and escape-encoded on marshal, while the target is stored verbatim. Only the name was handled; the value was passed through raw in both directions.

The read side was the real damage. `fossil branch new 'my branch'` writes

T *branch * my\sbranch T *sym-my\sbranch *

so importing an ordinary fossil repo with a spaced branch name produced a branch literally named "my\sbranch" through crosslink. Verified against the fossil binary: the artifact for that command now parses to value "my branch" and re-marshals byte-for-byte identical.

Marshal also gains the mirror of the hex-only tag name check parse already enforces. Without it a caller could build and store an artifact this repository could never read back; canonical fossil never emits one either, since it prefixes branch and user tags with "sym-". check-in: 0e579e24a4 user: [email protected] tags: trunk

* fix: correct go.mod module path to go-libfossil (#76)

The module directive still declared github.com/danmestas/libfossil after the repository was renamed to go-libfossil. pkg.go.dev/godoc scraping matches the declared module path against its hosting path, so the stale path broke SDK doc generation.

Rename the module path across all four in-repo modules (root, the two db/driver submodules, observer/otel, and docs/site) and every internal import, plus the release workflow, the doc-generation script, and the docs.

No tag has been published under the go-libfossil path yet, so the cross-module requires cannot resolve from the proxy. Add replace directives pointing the in-repo requires at their local directories so GOWORK=off builds and the CI matrix resolve without a published version. These replaces are ignored by downstream consumers and can be dropped once new-path tags are cut.

* Guard releases against replace directives; regenerate SDK docs

The go-libfossil rename bridges the in-repo modules with `replace` directives. Go refuses a main module that carries them, so a tag cut while they are present breaks `go install .../cmd/libfossil@<tag>` and every `@version` fetch of the driver submodules. Add a Release workflow step that fails the job before anything is published if any tracked go.mod still contains a replace directive, so the bridge is a CI-enforced TODO rather than a silent landmine. The step comment records what unblocks a release: re-tag the submodules with the corrected module paths, bump the root requires, then drop the replace blocks.

Regenerate the SDK reference docs. The previous pass only substituted URLs over the pre-existing generated files, leaving stale signatures (Repo.Timeline was still documented as taking LogOpts rather than TimelineOpts). Note that gomarkdoc must be run outside a git worktree — inside one it silently strips every source link.

Also reword the README rename note, which read as though the import path had not changed, and re-sort the ncruces require block. check-in: 6273a0750d user: [email protected] tags: trunk