Timeline

The clone command hardcoded a 10-minute context deadline with no way to raise or disable it, and the deadline did not reliably interrupt a slow clone.

- RepoCloneCmd gains a --timeout flag (default 10m, matching the old behavior; 0 disables the deadline for very large repositories). cloneContext branches away from context.WithTimeout for a non-positive timeout -- WithTimeout(parent, 0) yields an already-expired deadline, which would abort instantly the very clone the flag exists to unbound.

- The whole-repository crosslink sweep now observes context cancellation. It runs after the exchange-round loop, so it had no round boundary of its own at which the deadline could fire -- on a large repository it could run long past the deadline uninterrupted. manifest.CrosslinkContext checks the context every 256 candidates; manifest.Crosslink stays a background-context wrapper so its ~30 existing callers are unchanged. processResponse gains the same batched check so a large single-round batch is interruptible too, and the sync session's and the serve handler's post-convergence sweeps are wired to their contexts for the same reason.

Each cancellation point is pinned by a test that fails without it: the crosslink and processResponse tests drive a context that survives the first poll and dies before the second, so only the batched check can observe it; the crosslink-sweep test cancels after a zero-card final round, so nothing before the sweep is left to notice.

Also adds a clone -> serve -> reclone fidelity regression guard (#114), built with the canonical fossil binary rather than this library's Commit. Commit does not deltify, so a corpus built through it holds zero delta rows and never reaches the serve side's deltified-row expansion branch -- a green result from such a corpus proves nothing about #114. The guard now asserts the source holds delta rows before measuring anything, and compares the second-generation clone's tip, artifact count, blob-store size, delta-row count and checked-out tree content against the source.

Measured against that corpus (71 artifacts, 54 delta rows), #114 does not reproduce: gen1 and gen2 both hold the same 54 delta rows, the same tip and artifact count, a 1.009 store ratio, identical checked-out content, and pass fossil test-integrity. The receive path re-deltifies what the serve side expanded, via content.Deltify on the crosslink mlink path, so the expansion the serve side performs does not survive into the store. check-in: a02c9572da user: [email protected] tags: trunk

Adds a wire-efficiency regression guard the issue flagged as missing: correctness-only round-trip tests never asserted card type or wire cost, so the file-card-instead-of-cfile amplification #98 fixed could silently return. TestCloneServesCompressedNotExpandedContent serves a small compressible corpus, clones it over the real HTTP transport, and fails if the server emits any expanded file card or if served bytes exceed 3x the repository's stored size. Card type is the load-bearing signal: the reply container is compressed, so a raw-file-card regression on compressible content hides inside a byte ratio -- the 3x bound is only a coarse backstop.

Also adds a WIRE_MEASURE-gated manual harness that re-measures serve-side amplification against a real fossil client through a byte-counting proxy, with a fossil-serving-fossil control. Current main: 2.06x stored end to end (server response bodies alone are 1.04x, matching the fossil control's 1.03x), down from the 17.9x this issue reported. check-in: 1a73119ed4 user: [email protected] tags: trunk

FossilEncode/FossilDecode implemented only 3 of the 8 escapes the artifact/sync format §4.3 mandates (\s, \n, \\). A field value carrying a real TAB, CR, VT, FF or NUL byte -- as canonical fossil writes it -- was mis-decoded and no longer re-marshalled to its source wire bytes, changing the artifact hash. Unrecognized escapes were also left literal instead of dropping the backslash per spec.

Add the remaining escapes (\t \r \v \f \0) to both directions and make an unrecognized escape drop its backslash, matching canonical fossil fossilize/defossilize (verified against fossil 2.28). Every caller -- A/C/H/L/J/F/T/U cards and the xfer cards -- shares these functions and gets the fix.

Closes #119 check-in: 1bfa6b2094 user: [email protected] tags: trunk

The test step runs `go test $(go list ./... | grep -v '/dst')`. When a package fails to compile, `go list` prints an error to stderr and omits that package from stdout, so it silently drops out of the test run rather than failing CI. `go build ./cmd/libfossil/` only covered the CLI binary's dependency closure, which missed internal/deck when it broke.

Add `go build ./...` ahead of the test step on both driver-tag legs (modernc and test_ncruces) so any package that fails to compile is a loud, fail-fast build error. This subsumes the narrower cmd/libfossil build, so that step is removed rather than kept alongside. check-in: fa558f254d user: [email protected] tags: trunk

The xfer HTTP server let net/http frame large clone/sync replies as Transfer-Encoding: chunked with no Content-Length. Release-tagged fossil <=2.23 reads the reply length only from Content-Length (src/http.c leaves iLength negative otherwise) and aborts with "server did not reply" on any chunked reply above ~256KB, so those clients could not clone a repository larger than that from a libfossil server.

writeXferResponse already materialises the whole response via Encode before the first write, so the exact body length is known for free. Set Content-Length explicitly; net/http then never falls back to chunked.

The fix is unconditional rather than negotiated. The triage proposed sniffing a client capability and keeping chunked for capable clients to avoid the buffering cost of Content-Length. That premise does not hold here: the server never streams -- Encode always returns a complete slice -- so chunked saves no memory over Content-Length, and #88's per-clone bound (DefaultCloneBatchBytes) still caps each reply either way. Content-Length is exactly what canonical fossil's own server emits, so it is strictly the more compatible framing for every client, including builds whose version banner cannot be trusted (#86).

Tests: a reconstructed <=2.23 client that requires a non-negative Content-Length and rejects chunked framing on a >256KB clone response (the discriminating test, since real fossil 2.28 tolerates chunked and cannot reproduce the failure), plus a modern-client no-regression clone of the same corpus. The existing real-fossil multi-batch clone interop test continues to pass unchanged.

Fixes #101 check-in: b3139353e1 user: [email protected] tags: trunk

* Wire manifest.AfterDephantomize into the phantom fill path

AfterDephantomize was a faithful, tested port of Fossil's after_dephantomize (content.c:389-456) that nothing in production ever called — it crosslinked correctly in isolation but sat disconnected from the real receive path.

storeResolvedContent fills a phantom row in place when the full content for a previously-unknown UUID arrives (matching canonical's content_put_ex filling a phantom at content.c:626), but stopped short of crosslinking it or any delta children waiting on it. Return the filled rid from storeResolvedContent and call AfterDephantomize on it from storeReceivedFile once the fill transaction has committed — outside the transaction, since AfterDephantomize opens its own connection via repo.Repo.DB() and calling it from inside the tx's closure would contend with the still-open write lock.

Added an end-to-end regression test that delivers a checkin manifest as a delta against a base that hasn't arrived yet, then delivers the base, and asserts the manifest is crosslinked (its event row exists) immediately — with no explicit call to manifest.Crosslink anywhere in the test. This proves AfterDephantomize fires on the real fill path, not only under its own unit tests.

* Log crosslink failures during AfterDephantomize instead of swallowing them

A crosslink error while filling a phantom blob (or one of its orphan/delta dependents) previously vanished silently, leaving storeReceivedFile report success with the artifact filled but un-crosslinked. Surface it via slog.Warn so real corruption on the fill path is at least visible, matching the non-blocking nit from review. check-in: 94919ca357 user: [email protected] tags: trunk

* fix: select xfer framing by Content-Type (#106); bound clone rounds by the client's decode limit (#109)

seeing which parsed, which made a decompression failure indistinguishable from "wrong framing" (the root of #104). Select framing from the §4 Content-Type instead: application/x-fossil is the §4.1 compressed container, every other type (including application/x-fossil-uncompressed, used by clone-v3 replies) is plain card text — the same rule canonical fossil applies. Verified against fossil 2.28: pull replies arrive as the prefixed container, clone replies as plain text, and no reply is ever an unprefixed raw zlib stream. The former unprefixed-zlib framing, checked first on the unverified belief that fossil emits it, was removed along with the length-prefix/zlib-header aliasing guard that only existed to disambiguate the two compressed framings trial-dispatch conflated. A compressed body that fails to inflate is now reported as the fault it is, never re-fed to the card parser. xfer.Decode takes the Content-Type; byte-oriented callers with no header pass the compressed type explicitly.

crossing one whole, so a round carried budget + one whole artifact. When sub-budget filler preceded a large artifact, that sum exceeded what the client could decode (xfer.MaxDecompressedBytes) and the clone failed with a spurious "declared container length exceeds" error whose threshold depended on incidental filler size. The server now flushes the round and sends an over-bound artifact alone when adding it would cross the client's decode bound, so clonability is a property of the artifact alone. The 2*budget compile-time guard now certifies what it appears to.

* fix: reject unrecognized xfer Content-Type with 415 (#106)

A POST whose Content-Type was absent or unrecognized fell through to the plaintext card decoder, which is exactly the #104 failure shape: possibly- compressed binary fed to the card parser. Request bodies come from untrusted peers, so a body carrying no framing signal is rejected rather than guessed at. Real fossil always sets the header; no conforming peer is affected.

xfer.KnownContentType shares the media-type parsing with the framing check, so the server and Decode agree on what a type means.

Also runs the #109 filler subtests in parallel — serially they pushed internal/sync past the 120s per-package timeout CI uses. check-in: f1236bd031 user: [email protected] tags: trunk

Issue #60's triage was stale: the live truncation bug it describes was already closed on main by 11967ff (#62), which added millisecond rounding to timeToJulian. What remained was a coverage gap, not a defect.

An earlier attempt on this branch also normalized the float64 branch onto the millisecond grid, on the theory that a driver-independent helper is strictly better. That was wrong, and this commit does not do it. A scanned float64 IS the value stored in the row, so snapping it can only move it away from what the row holds. Timeline's pagination cursor depends on the opposite property: its predicate `e.mtime < ? OR (e.mtime = ? AND b.rid < ?)` requires the cursor to compare exactly equal to the row it came from. SQLite's own julianday() computes iJD/86400000.0, a different expression from TimeToJulian's julianEpoch + ms/86400000.0, and the two land one ulp apart on a real fraction of instants — so normalizing alters mtimes written by upstream Fossil, the library's primary real-world input. Raised, the cursor row repeats as the next page's first row; lowered, rows are skipped. Both are silent wrong answers.

The float64 branch therefore stays verbatim, and its doc comment now records why that exactness is load-bearing rather than incidental. The same reasoning applies to tag propagation's `tagxref.mtime < ?`, where SQLite compares the bound value against raw stored floats: an exact bound is the correct one there too, so no normalized variant is introduced for it either.

The one genuine transform stays where it belongs, in the time.Time branch, which must round because ncruces hands back DATETIME columns carrying its own sub-millisecond conversion noise. Its comment now also records the limit of that recovery: an mtime written at finer than millisecond resolution cannot be reproduced through a time.Time at all, and rounding the float64 branch to match would discard precision on both drivers rather than recover any.

Non-functional cleanup: millisPerDay and julianDayFromMillis replace two open-coded copies of the same expression.

Coverage added (no build tag; these run under both drivers): - db: TestScanJulianDayFloatBranchIsBitExact sweeps julianday()-computed mtimes and asserts each scans back bit-for-bit, with a guard that fails the test if the sweep ever stops containing divergent values. Verified to fail against the normalizing version and pass against this one. - db: TestScanJulianDayIsDriverIndependent proves the float64 and time.Time representations of one millisecond-aligned write converge, with noise in both directions. TestScanJulianDaySentinelZero pins the "no mtime" sentinel that bisect's `mtime != 0` guard depends on. - tag: TestPropagateAllOrderingAdjacentMillisecond drives the `tagxref.mtime < ?` decision through ScanJulianDay with adjacent-millisecond mtimes; it fails under a truncating conversion and passes under the rounding one, and passes under modernc regardless — the reason this class went unnoticed twice.

Consumer classification (ScanJulianDay): - manifest/timeline.go NewCursor: ordering/equality, requires exactness - tag/propagate.go PropagateAll -> `tagxref.mtime < ?`: ordering (now covered) - tag/propagate.go childMtime -> mtimeQueue.Less: ordering, traversal-order only (visited set makes it result-neutral) - manifest/log.go, bisect/bisect.go: display-only

Known limitation, left alone deliberately: julianToTime truncates where timeToJulian rounds, so the two are not inverses at sub-millisecond resolution. Pre-existing, now documented; ScanTime's consumers are display paths with no ordering boundary. This commit does not widen the gap. check-in: 78a793dd83 user: [email protected] tags: trunk

* fix: return error when clone stalls with phantoms outstanding

Clone's convergence loop broke out of its round loop as soon as the phantom count stopped decreasing and returned success, even when the phantom table was still non-empty. That leaves the caller holding a repository missing content with no error and exit code 0.

Stopping on a stall is still correct — spinning forever would be worse — but reporting it as success is not. checkStop now returns a PhantomStallError when the loop halts with phantoms remaining, giving the caller a programmatically-reachable count plus a sample of the missing UUIDs for diagnosis. A clean clone still returns nil with an empty phantom table.

Fixes #64.

* fix: preserve partial CloneResult when Clone returns an error

Clone's named result return was never assigned on the error paths — cs.run's cloneResult stayed in a local variable, so a caller catching a PhantomStallError (or a crosslink failure) got a nil result even though a partial one, with the round/blob counts collected so far, was sitting right there. Assign it before returning on both paths.

* fix: make phantom stall error sample deterministic

newPhantomStallError previously took an arbitrary 10 entries from map iteration order before sorting, so the sample was nondeterministic across runs when more than 10 phantoms were outstanding. Sort all UUIDs first, then take the lexicographically-smallest 10, matching what the doc comment already implied.

Addresses a non-blocking nit from review on #64. check-in: e5be6a9199 user: [email protected] tags: trunk

* fix: zero-length file commit panic and stale missing-file carry-forward

Two related bugs in the checkin path:

#68 - Committing a zero-length file panicked in blob.Store, which treated an empty artifact (a normal, well-known Fossil blob) as invalid input. Worse, manifest.Checkin's postcondition defer fired unconditionally during the panic's stack unwind and re-panicked with its own unrelated "manifestRid must be positive" message, masking the real cause. Fixed both: blob.Store now accepts zero-length (and nil) content, and Checkin's defer re-panics with the original panic value instead of asserting a postcondition that was never going to hold mid-unwind. Fixing the panic surfaced a second bug in the same family: buildCommitFiles used len(Content) > 0 to detect "already loaded from disk," which can't distinguish a genuinely empty file from one not yet loaded, so a newly added empty file failed with "no RID for unchanged file." Replaced the length sentinel with an explicit loaded-set.

#79 - A tracked file deleted from disk without Unmanage/Remove was silently carried forward into the new manifest with its last-committed content, rather than being detected. Verified against fossil 2.28: canonical fails a commit when a missing tracked file is in scope (either the implicit "commit everything" case, or explicitly named), but leaves an out-of-scope missing file alone. buildCommitFiles now stats every in-scope tracked file before treating it as unchanged and errors clearly, naming the file, when it's gone; a file outside the commit's scope (relevant only to an explicit Enqueue) is left untouched exactly as before. Updated the two existing tests that had documented the old silent-carry-forward behavior as deliberately deferred to this issue.

* fix: add CHANGELOG entry and unwrap-safe stat checks for #68/#79

Adds the CHANGELOG entries the #79 acceptance criteria required for a user-visible behavior change (missing tracked file now hard-fails a commit instead of silently carrying stale content forward) and for the #68 zero-length-file panic fix.

Also switches checkin.go's two os.IsNotExist(err) checks to errors.Is(err, os.ErrNotExist), which correctly unwraps the %w-wrapped os.ErrNotExist that MemStorage.Stat returns. Production was already correct (OSStorage.Stat returns a raw *PathError), but the tests were only exercising a generic fallback error branch instead of the intended "tracked file missing from disk" message. check-in: 0351167ae5 user: [email protected] tags: trunk