Timeline

* fix(manifest): link check-ins with absent files; refuse absent baselines

The crosslink sweep held a check-in back until every file it names was stored. A root check-in, or one on a present parent, whose files never arrived got no event, plink, mlink, leaf or tag rows. fossil links it against phantom files (add_one_mlink, uuid_to_rid(..., 1)), and repo-state-01 Section 4.2 makes an unresolved reference a phantom, not an error. The sweep -- Crosslink, the close of every sync and clone, and verify.Rebuild -- now waits only on a delta manifest's baseline. The receive-time cascade and receive linker still wait for files while blobs are still arriving; the closing sweep bounds that wait.

verify.Rebuild linked a delta manifest whose baseline was absent, with no baseline id. fossil refuses it (fetch_baseline) and files it in orphan under a phantom for the baseline; artifact-format-01 Section 8.3 calls this a soft failure retried when the baseline arrives. Every path now does the same, so a sync requests the baseline and the phantom-fill cascade links the manifest when it lands. Rebuild clears and rederives orphan.

The gap case repostate.4.2.root-absent-file passes and is unmarked; new cases cover a present parent with an absent file and an absent or late baseline, each also through verify.Rebuild where fossil rebuild agrees.

Tracked in #406.

* fix(manifest): refuse a non-manifest baseline and drop the orphan row on link

Tracked in #406. check-in: 67fc6ced4b user: [email protected] tags: main, trunk

fossil's rebuild_db empties and re-derives the repository's tables in a single transaction. Since #285 Rebuild committed the emptied derived tables before the crosslink sweep, because the sweep found candidates outside any transaction and opened its own; a sweep that failed partway left event, plink, mlink, tagxref and leaf empty.

manifest.CrosslinkRebuild now runs inside the caller's transaction: candidate discovery, linking and the leaf/tag repair gate all on tx. Rebuild empties, re-derives and refills the bookkeeping tables in one WithTx, so a failure rolls back to the state it found. The incremental sweep keeps its batched commits; only the repair gate is factored out so both paths share it. check-in: 2832099f56 user: [email protected] tags: main, trunk

verify.Rebuild kept its own partial derivation of repository state: it wrote event/plink/mlink/tagxref rows only for check-ins and control artifacts, applied tags from artifacts the crosslink would refuse, skipped absent tag targets and parents instead of reserving phantoms, wrote no plink baseid, and computed leaves as "no child at all" regardless of branch. Against fossil rebuild it missed hundreds of event, plink, tagxref and leaf rows on real repositories.

Make Rebuild empty the crosslink-derived tables and run the same sweep a clone runs, so derived state has a single owner. The three partial derivation files (rebuild_tags.go, rebuild_manifest.go, rebuild_leaves.go) are deleted; Rebuild keeps only what the sweep does not do -- blob hash verification and the unclustered/unsent bookkeeping.

A rebuild is terminal: nothing more will arrive, so a check-in whose file or baseline content is absent must be linked with phantoms rather than deferred for a blob that never comes, as fossil's add_one_mlink does via uuid_to_rid(...,1). Add manifest.CrosslinkRebuild, a sweep that does not defer; on a complete repository it is identical to Crosslink. Rebuild's leaf rule now follows fossil's branch-aware sweep, so an untagged parentless check-in is no longer a leaf -- the same Section 7.3 deviation the sweep and per-artifact paths already carry. check-in: 33959a6e3e user: [email protected] tags: main, trunk

* fix(manifest): apply the parent (reparent) tag (#286)

A singleton parent tag relocates a check-in's ancestry. fossil's manifest_crosslink_end rewrites the tagged check-in's plink and mlink to the hashes the tag names, before it propagates tags, so the leaf recompute sees the new edges and an old parent with no remaining child becomes a leaf. ReparentCheckins does this over a deck whose P-card is the override, reusing the normal plink/mlink builders. It runs from the crosslink repair gate and from verify.Rebuild, which had no reparent step.

* test(manifest): reparent plink/mlink/leaf, idempotence, phantom target (#286)

Also fold the plink baseid resolution into a shared checkinBaseid helper used by both the normal linkage path and the reparent pass, and note that the reparent repair is O(reparented) per round like fossil's manifest_crosslink_end. check-in: 088f1d8a90 user: [email protected] tags: main, trunk

* fix(manifest): store check-in, link and tag rows as fossil does (#258)

A rebuilt or locally written repository differed from fossil's in three derived values: event and plink times in their last digits, event.omtime left NULL, and a valueless tag stored as '' instead of NULL.

Fossil stores a tag time as SQLite's julianday() bound as a double, but writes event, plink, attachment and forumpost times into SQL text with its own printf("%.17g"), which emits 16 digits in long double arithmetic. TimeToJulian is now julianday() exactly (whole milliseconds over 86400000) and TimeToTextJulian ports fossil's printf conversion, so both match fossil bit for bit. The conversion follows builds where long double is a double (macOS, Windows); fossil itself differs across platforms in the last digit.

Check-in events record omtime, tag writes bind NULL for an empty value, the readers of event and plink times under the ncruces driver rebuild the text form, and the CLI's tag listing reads NULL values.

* test(db): sweep both stored julian forms; scan the text form from driver times (#258)

* test(verify): compare text times bit for bit only where fossil computes them in double (#258) check-in: 2c2e5e74b6 user: [email protected] tags: trunk

Route direct check-ins and internal rebuilds through canonical mlink derivation. Update sparse-history consumers and preserve legacy conflict-fork rows with an explicit RID discriminator migration.\n\nCloses #213 check-in: 04f41a4c9c user: [email protected] tags: trunk

Cloning the Fossil SCM corpus (67.6k blobs) took 293-302 s at 1251-1287 MB against canonical fossil's 20.8 s at 205 MB. A CPU and allocation profile of one clone found the cost was not delta expansion or hash verification.

27.2% of all CPU was a single un-memoized statement. content.AvailabilityCache memoizes the delta-chain walk but not the "SELECT rid FROM blob WHERE uuid=?" that starts it, so manifest.missingCheckinRefs re-seeked the blob.uuid index once per F-card of every check-in -- the same file hash resolved again for every revision that did not change it. That index is far larger than SQLite's page cache, so each seek was three page reads: pread syscalls were 34.7% of total samples. The walk it guards was under 1% of its own cost.

Separately, materializing 8.27 GB of content allocated 360 GB. delta.ApplyInto starts at a 64 KiB cap deliberately -- the target length is a wire claim and a few bytes can declare 4 GiB -- and then climbs to the real size through append's ~1.25x large-slice growth, ~3.4x the target in discarded intermediates, leaving slack that content.Cache.clip copied away again (203.9 GB and 59.8 GB respectively).

Three changes:

- Memoize uuid -> rid in AvailabilityCache. Hits only. Within one crosslink run a hit cannot go stale, because a sweep and a cascade only ever add blob rows; a miss can, because ridOrPhantom and blob.StorePhantom create rows mid-run, and remembering an absence would leave an artifact deferred after the content it waits on arrived. The scope that makes this sound is now stated on the type, and TestAvailabilityCacheMustNotSpanAPurge pins its one boundary: shun.Purge deletes blob rows, so a cache must not span one.

- Size delta.ApplyInto's output exactly. scanOutputSize walks the command stream first without allocating, checking every copy against bytes the source holds and every insert against bytes the delta carries, so the size it returns is a measurement rather than the header's claim. Any stream it cannot walk to a clean terminator falls back to the existing growth path, so malformed and hostile deltas allocate and fail exactly as before. The buffer now arrives tightly sized, which also makes clip's copy a no-op.

- Cut crosslinkCacheBytes from 256 to 128 MiB. Each byte of that budget costs roughly five bytes of peak RSS -- GOGC doubles it into the heap goal, and the scavenger is paced off GC cycles, of which a faster sweep runs fewer. At 256 MiB the first two changes gave 205.7 s but 1473 MB, faster at the cost of memory. The measured curve (32/64/128/256 MiB: 369.6 s/318 MB, 292.3 s/447 MB, 249.4 s/812 MB, 205.7 s/1473 MB) puts 128 MiB where both axes improve.

Result on the same corpus: 245.4 s at 780 MB, from 293-302 s at 1251-1287 MB -- 17% less wall time and 39% less peak RSS, with instructions retired down from 3.37T to 2.48T. The gap to canonical narrows from 14.1x/6.3x to 11.8x/3.8x.

Derived tables are unchanged against an unmodified build cloning the same server: blob 67530, event 27099 (identical per type), mlink 104919, plink 23741, tagxref 51766, leaf 1426, filename 1496. The corpus parity gate passes.

What remains is structural rather than a missing optimization: a clone expands every blob twice, once in the receive-time phantom-fill cascade and once in the final whole-repository sweep, where canonical crosslinks each artifact from content it already holds in hand at receive time. Cache.Expand splits 45.4 s / 35.3 s across those two passes. Collapsing them is a change to the receive path and belongs in its own issue. check-in: 8b16c91f5c user: [email protected] tags: trunk

The buffer-reuse fix in #145/#150 cut the cost of expanding one delta chain once. It did nothing for callers that expand many versions of the same artifact in a single operation: each bare content.Expand still replayed the whole O(depth) chain, so a walk over N overlapping revisions cost O(N^2) blob reads even though content.Cache already existed for exactly this pattern.

Audited every non-test content.Expand caller. Two access patterns genuinely repeat over overlapping delta chains and are now routed through a per-operation content.Cache:

- annotate walks every revision of one file along the parent chain. content_deltify stores each older file blob as a delta against its successor, so the per-revision expansions are all walks of the same few chains. One cache threaded through the walk (file blob + manifest listing via new manifest.ListFilesCached/GetManifestCached, both nil-passthrough so existing callers are unchanged) turns it linear. Measured on a deltified 150-revision file: 1.60s -> 0.65s (2.5x), 764k -> 308k allocs; the win grows with history depth.

- verify.Verify and verify.Rebuild sweep every blob more than once (Rebuild: checkBlobs + rebuildManifests + rebuildTags x2). A single cache shared across the pass walks each chain once instead of once per sweep and amortizes overlap within each sweep. Measured on a 150-revision two-file repo: Rebuild 3.34s -> 0.068s (48x), 4.59M -> 98k allocs.

Callers that only ever touch the tip or make a single pass over distinct blobs (branch create, merge-config load, R-card / delta manifest build, MergeParentFiles, crosslinkSingle, shun undelta, pairwise Diff) are left on bare Expand, each with a comment stating why a cache cannot amortize their access.

No behavior change: expansion output is byte-identical, verified across the modernc and ncruces drivers with REQUIRE_FOSSIL_BIN=1 cross-checks and the DST suite. check-in: 0fd2a507b8 user: [email protected] tags: trunk

* fix: correct go.mod module path to go-libfossil (#76)

The module directive still declared github.com/danmestas/libfossil after the repository was renamed to go-libfossil. pkg.go.dev/godoc scraping matches the declared module path against its hosting path, so the stale path broke SDK doc generation.

Rename the module path across all four in-repo modules (root, the two db/driver submodules, observer/otel, and docs/site) and every internal import, plus the release workflow, the doc-generation script, and the docs.

No tag has been published under the go-libfossil path yet, so the cross-module requires cannot resolve from the proxy. Add replace directives pointing the in-repo requires at their local directories so GOWORK=off builds and the CI matrix resolve without a published version. These replaces are ignored by downstream consumers and can be dropped once new-path tags are cut.

* Guard releases against replace directives; regenerate SDK docs

The go-libfossil rename bridges the in-repo modules with `replace` directives. Go refuses a main module that carries them, so a tag cut while they are present breaks `go install .../cmd/libfossil@<tag>` and every `@version` fetch of the driver submodules. Add a Release workflow step that fails the job before anything is published if any tracked go.mod still contains a replace directive, so the bridge is a CI-enforced TODO rather than a silent landmine. The step comment records what unblocks a release: re-tag the submodules with the corrected module paths, bump the root requires, then drop the replace blocks.

Regenerate the SDK reference docs. The previous pass only substituted URLs over the pre-existing generated files, leaving stale signatures (Repo.Timeline was still documented as taking LogOpts rather than TimelineOpts). Note that gomarkdoc must be run outside a git worktree — inside one it silently strips every source link.

Also reword the README rename note, which read as though the import path had not changed, and re-sort the ncruces require block. check-in: 6273a0750d user: [email protected] tags: trunk

* fix: distinguish absent, empty, and valued U-cards in manifest parsing

Manifest parsing collapsed a wholly absent U-card and a present-but-empty U-card into the same stored value (empty string), so every crosslink call site that wrote the check-in user into event/attachment/forumpost rows lost the distinction between "no login recorded" (should be SQL NULL) and "login explicitly blanked" (canonical fossil resolves this to the literal "anonymous" at parse time — see src/manifest.c:1008-1016).

Change deck.Deck.U from string to *string so the three states are representable: nil (absent), non-nil "anonymous" (present-empty, resolved in parseUCard the same way fossil's own parser does), and non-nil with a value (present). Resolving once in the parser means every downstream crosslink/rebuild call site can keep binding d.U directly — Go's database/sql converts a nil *string argument to SQL NULL and dereferences a non-nil one automatically, so no per-call-site special casing is needed. The type change itself makes the old ambiguous string-binding path a compile error everywhere Deck.U is constructed, closing off the drift that caused issue #29.

Added deck.Str, a small pointer-literal helper, since Go can't take the address of a string literal directly; existing tests that built Deck{U: "..."} now go through it.

* fix: review nits — whitespace U-cards, rebuild coverage, tag.AddTag, naming

Four follow-ups from review of the U-card presence fix:

- deck.parseUCard now trims args before the emptiness check, matching canonical fossil's next_token() (skips leading whitespace) and every sibling card parser in this file. Previously "U \n" stored literal whitespace instead of resolving to "anonymous". Added a whitespace-only case to TestParseUCardStates.

- Added TestRebuild_UCardPresenceStates covering the rebuild path (rebuild_manifest.go's rebuildCheckin), mirroring TestCrosslink_UCardPresenceStates. The binding site was already correct (same nil-to-NULL pointer semantics as crosslink.go) but had no dedicated regression test.

- tag.AddTag now panics on an empty opts.User, matching the panic idiom CheckinOpts.User already establishes. Previously a tag created with an empty User silently crosslinked to user=''; that asymmetry with CheckinOpts was a wart worth closing rather than documenting as intentional.

- Renamed deck.Str to deck.User: the name should say what the helper constructs (a Deck.U value), not how it's implemented (a pointer take). check-in: 527759acd5 user: [email protected] tags: trunk

* fix: separate blob existence from blob availability (#43)

A phantom is a real blob row with size = -1 and NULL content, so blob.Exists — "SELECT rid FROM blob WHERE uuid=?" — answers true for it. Every guard built on Exists therefore admitted artifacts whose content cannot be read, and content.Expand aborted with "rid N is a phantom".

Canonical Fossil keeps two predicates and never substitutes one for the other: rid_from_uuid (existence, xfer.c:70) and content_is_available (availability, content.c:163). The latter is transitive — it follows delta.srcid up the chain, so a delta whose base is a phantom is unavailable even though every row in the chain exists.

Add content.IsAvailable / content.AvailableByUUID, ported from content_is_available including the chain walk, with an explicit depth bound so a cyclic delta table terminates instead of hanging. The walk lives in exactly one place; no caller touches the delta table.

blob.Exists keeps its behavior and gains a doc comment stating that it answers existence only, returns true for phantoms, and naming the availability predicate as the alternative.

manifest.blobPresent is deleted. It was a local patch at one call site that tested size >= 0 without the chain walk, so it still reported a delta-on-phantom as present — the same class of bug it was written to work around.

Every blob.Exists call site was classified as existence or availability and converted where it guards a content read. Sites that want rid identity — plink/mlink/tagxref population, cluster phantom creation, private-status bookkeeping, the post-store sanity check in handleFile, and phantom filling in storeReceivedFile — keep existence, since delta-before-base is a normal steady state during clone.

Verified against a loopback fossil 2.28 server: the 23-artifact repro that previously failed roughly half the time now clones 6 of 6.

* fix: IsAvailable must fail closed when the delta lookup errors

Only sql.ErrNoRows means the chain is grounded. Any other failure — cancelled context, closed connection, NULL srcid — says nothing about groundedness, so returning true there claims readable content that was never verified. That is the same class of bug this predicate exists to prevent, and it runs on data arriving over the wire during clone.

The blob-size lookup above already fails closed on the same pattern; this restores the symmetry. check-in: 670ca0e75f user: [email protected] tags: trunk

* feat: CreateOpts.ProjectCode — accept explicit project-code on Create

Downstream EdgeSync coordinates a shared project-code across sibling Fossil repos created on different machines so its NATS sync subject ("<prefix>.<project-code>.sync") matches across leaves. Today every Create generates a fresh project-code, so commits never propagate.

Add CreateOpts.ProjectCode: empty preserves current behavior; non-empty must be 40-char lowercase hex (^[0-9a-f]{40}$) — the same format generated project-codes use. Validation runs at the top of libfossil.Create before any disk I/O, so invalid input cannot leave a half-written file behind. Server-code stays auto-generated.

Threaded through repo.Create / repo.CreateWithEnv / db.SeedConfig. SeedConfig validates as defense-in-depth (public package, multiple callers possible).

Clone accepting ProjectCode is out of scope; will follow separately if EdgeSync needs it.

* test: round-trip explicit ProjectCode through fossil rebuild

Asserts that vanilla `fossil` (2.28) accepts a Go-side repo created with an explicit ProjectCode, and that `fossil sql` reads back the same value we set. Closes the gap where the existing TestCreate_FossilValidation only covered the generated-project-code path. check-in: 6b66ea6b2d user: [email protected] tags: trunk, v0.6.1

libfossil is a pure-Go implementation of the Fossil SCM, reading and writing the same .fossil SQLite repository format as upstream Fossil.

This is the first public release of code that previously lived in a private repository. The module path is github.com/danmestas/libfossil and the binary is named libfossil to avoid $PATH collisions with upstream fossil.

Included: - Repo lifecycle (create, clone), checkout/checkin, timeline, merge, diff, annotate, shun, tag, branch, bisect, stash, undo/redo - Sync-over-HTTP protocol (client and handler) - Two SQLite drivers: modernc (pure Go, default) and ncruces (wasm-capable) - Observer interfaces (SyncObserver, CheckoutObserver) with an optional OpenTelemetry implementation in observer/otel (separate submodule) - Deterministic simulation testing harness with BUGGIFY fault injection - Kong-based embeddable CLI under cli/ and a standalone binary under cmd/libfossil/ check-in: f5744359d3 user: [email protected] tags: trunk