Count the decoded file set a tree index keeps as the next difference's base toward memoTreeBytes, check a decoded id's gap without wrapping in uint64, and say what cloning the index on publish costs (about 0.5ms for fossil-scm.org's whole history, BenchmarkTreeIndexClone).

TestMemoWideTreesMatchFresh classifies every cone over wide trees that rename, delete and add files, in all three manifest forms, with chains of differences 15 long and with a limit that starts the index again, before and after the store gains check-ins; a branch heads the chains trunk trees continue, so a trunk horizon decodes trees whose chain starts outside its held set. check-in: 7ba1eb6915 user: [email protected] tags: main, perf/memo-tree-encoding, trunk

# Conflicts: # CHANGELOG.md check-in: a90035689e user: [email protected] tags: main, perf/memo-tree-encoding, trunk

feat(horizon): targeted demand fetch for reads below the horizon check-in: c63cab2f36 user: [email protected] tags: main, trunk

Review of the chunked-blob change found a missing last part read back short with no error, and a chunked write outside a transaction could leave a torn row that looked held.

A chunked row now keeps its stored length in blob.content as an INTEGER. Reads check the parts run unbroken from seq 0, are full-sized but for the last, and add up to that length; anything else is ErrPartsCorrupt, and a NULL content is no longer mistaken for a chunked one. StoredLenSQL reads the length off the row and needs no schema probe.

Chunked writes, and the phantom fill around them, run in a savepoint of the caller's transaction or a transaction of their own, so a failed part leaves nothing behind. A value needing more than 4096 parts is an error, not a panic, since stored values can come from peers. The cone and sparse local payload reads use ReadStoredExact, which loads a value only at the length already budgeted for.

MaxValueBytes is documented as process-wide, and unversioned content as not chunked. check-in: 0fc4d176c3 user: [email protected] tags: feat/chunked-blobs, main, trunk

A full read of every tree with the graph made a cold classification on fossil-scm.org's clone three to four times slower, which an origin that starts cold often cannot afford. Walks read a tree when they first need it, as before, and the memo keeps what they read as a treeIndex (interned ids, delta-varint sets, full manifests as differences), replaced on each publish so views read it without the lock. memoTreeBytes now bounds that encoding, which holds the whole history of fossil-scm.org in about 11 MiB; past it the index starts again. HZ-6.2-10 stays partial: trees are still read on first use, not with the graph. check-in: b53551c192 user: [email protected] tags: main, perf/memo-tree-encoding, trunk

- manifest.Placement computes the horizon's held set at most once per operation, so an annotate walk or a whole-check-in diff no longer rebuilds the check-in graph on every failed read. - AnnotateContext whose peer cannot supply a version returns the partial result, the lines past C marked BelowHorizon, as Annotate does without a peer; only a cancelled context fails it. - AnnotatedLine documents that BelowHorizon lines carry C's identity and that a partial annotate is signalled only by those marks. check-in: 02a2ea3214 user: [email protected] tags: feat/demand-fetch-reads, main, trunk

Review follow-ups on the demand fetch for reads below the horizon:

- A read is refused as below the horizon, and demand-fetched, only when its check-in lies outside the held set K; an unheld payload above C, such as one outside the store cone's paths, keeps its own error and is not fetched. - Annotate no longer refuses without a peer: on a content-horizon store it reports the lines it cannot attribute past C at the horizon, marked BelowHorizon, as a structural store does. AnnotateContext fetches each version as the walk reaches it, so nothing the walk does not read is fetched, up to 200 payloads; past that it returns the partial result. - diffWholeCheckin reports a failed structure classification, and the read paths take the caller's context through to the fetch. - Tests: misclassification above C, annotate over-fetch, the partial result over the limit, and a structural peer withholding the payload. check-in: 52f026832e user: [email protected] tags: feat/demand-fetch-reads, main, trunk

ReadFileContext, ReadFileAtContext, AnnotateContext and DiffContext take RequiredContentOpts and, when a read needs content a horizon store does not hold, fetch exactly the artifacts it names from the peer and read again (draft-fossil-horizon-00 4.4): the payload, and on a structural store first the check-in's manifest, baseline and nodes. The store cone and its horizon do not move; fetched artifacts are held content.

Reads now refuse with content.MissingBelowHorizon, which matches ErrBelowHorizon and names the horizon. Annotate gathers every missing version along its walk into one round and Diff both sides; a read over one round's gimme budget is refused naming DeepenHorizon.

Closes #489 leaf check-in: 8821596379 user: [email protected] tags: feat/demand-fetch-reads, main, trunk

A hosted SQLite that caps value size (a Durable Object's 2 MB) cannot hold a large file's stored blob in one blob.content value. With the new db.DriverConfig.MaxValueBytes set, a stored value over the limit is written in 1,000,000-byte parts to blobpart(rid, seq, content), created on first use, leaving blob.content NULL and blob.size the expanded length, so phantoms (size < 0) stay unambiguous. Chunking sits below compression: hashes, deltas and the wire format are unchanged.

internal/blob owns the seam (ReadStored, WriteStored, InsertStored, StoredHead, DropStored, SweepParts and two scan expressions), and every direct reader or writer of blob.content now goes through it: delta rewrite and undelta, receive, clone send and walk, compaction, sparse demotion, seed drop, shun purge, horizon cost and cone preparation.

Zero, the default, keeps every value inline and creates no table, so local stores stay byte-identical. A store with blobpart rows is not readable by stock fossil; enable it only for hosted backends.

Closes #459 check-in: 1fd12b3ed3 user: [email protected] tags: feat/chunked-blobs, main, trunk

conformance: HZ-4.1-1 scoped to zit and go-libfossil origins check-in: 1455a3803e user: [email protected] tags: main, trunk

The origin classification memo now reads the F cards of every tree in the store once, with its references, and extends them by appended check-ins (draft-fossil-horizon-00 §6.2, HZ-6.2-10). Blob names are interned to uint32 ids, each tree's file set is a sorted delta-varint id list, and a full manifest is kept as its symmetric difference from the previous full manifest read, at most 15 links from a piece kept whole, decoded through a 16-entry LRU per view. A store whose index would pass memoTreeBytes keeps none and falls back to the bounded cache of walked trees.

On a clone of fossil-scm.org the trees take 11 MiB (the walked-tree cache held 50 MiB for two horizons), the memo's retained heap halves and peak in-use heap drops from 345-524 MiB to about 170 MiB. Tracked in #463. check-in: 1cb9e11a65 user: [email protected] tags: main, perf/memo-tree-encoding, trunk

Drop the stock-fossil caveat from the row, so it reports COVERED, and probe zit's reply to req-clusters under a horizon pragma (after a dense control). leaf check-in: e9eb2cabac user: [email protected] tags: conf/hz-4-1-1-zit-scope, main, trunk

perf: carry a cone clone's forwarded delta chains across rounds check-in: b85673a6cb user: [email protected] tags: main, trunk

fix(content): classify a fossil-written store's file versions from its tables check-in: d3574f0c74 user: [email protected] tags: main, trunk

test: expect fossil's newest-first ticket history check-in: 3ef7f9fb1f user: [email protected] tags: main, trunk

The mlink-named, unclaimed versions are taken as file content in a store with the non-artifact record too, including this library's own between a Checkin that commits an artifact-shaped file and the sweep that links it. The comment now says so, and that the window closes as the sweep's batches add the claims the memo follows. check-in: dd3f875e99 user: [email protected] tags: main, perf/cone-clone-rounds, trunk

The re-deltified version now sits inside a carried run, and the unreached blob it is stored against is itself a delta of the run's tip, so the chain from the run's head keeps the run's length and differs only in its rids. Disabling the checksum in cloneRun now ends the clone with a phantom. check-in: caea59079f user: [email protected] tags: main, perf/cone-clone-resend, trunk

fossil ticket history orders changes by time, newest first (tkt.c, ORDER BY 1 DESC), and leaves a same-second tie unordered. The test expected the original ticket before its amendment, which held only when both landed in the same second and the tie broke that way; CI failed it whenever the clock ticked between them.

Stamp the amendment a minute later and expect it first. leaf check-in: 88082202ad user: [email protected] tags: test/ticket-history-order, trunk

seedcheck: drop a seed completed as a delta in a later round check-in: 93cf3b65f6 user: [email protected] tags: main, trunk

A delta on a refused seed now counts as a new phantom only when it makes one, so a peer resending the same delta no longer holds a pull open to the round cap. The phantom it leaves is private when the card was, so its name is never asked for in a public sync.

seedcheck's drop reuses db's savepoint instead of its own copy: db.Tx.WithSavepoint exposes what WithTx already does on a bound DB. check-in: f8446cdfcc user: [email protected] tags: fix/seed-delta-drop, main, trunk

A push into a store fossil wrote starts the non-artifact record with what arrived, and loadFiles then took only the record's versions as file content: every older version went back to be read raw, past the budget on a real repository. File versions are now the record's together with those mlink names and no claim table names (fileVersionsSQL), in a full load and in a memo's extension alike, and the memo's prefix counts the claims in every store, since file versions follow them whether or not a record exists. The horizon memo conformance case that pushes a manifest ahead of its blob into such a store now extends by the arrivals again. leaf check-in: 9cc2297ef8 user: [email protected] tags: main, perf/cone-clone-rounds, trunk

A request carries a cookie of any length, and splitting one on ':' or '.' before any bound allocated a string header per separator: 8 MB of dots cost hundreds of megabytes. cloneCapFields now refuses a cookie past maxCloneCookieBytes (64 KiB, well past the longest the server writes) before it splits, and splits with SplitN, as parseCloneRuns does.

A forged cookie could also name one deep chain maxCloneRuns times. The server now rebuilds each head once and at most maxCloneRunRIDs rids in all, newest runs first, and the cookie keeps the newest maxCloneRuns runs rather than the oldest, since an old run may never be reached.

A SHA3 artifact refused to an older peer was recorded as sent, and so carried in a run across rounds; it is now left out as a withheld tree is.

TestConeCloneSurvivesRedeltify re-deltifies a forwarded version against an unreached in-cone blob mid-clone and checks the clone ends whole; it fails with a phantom if the run's checksum is not checked. leaf check-in: 624df80df2 user: [email protected] tags: main, perf/cone-clone-resend, trunk

Without the non-artifact record, file versions are those mlink names and no claim table names, and a sweep linking an artifact late adds claims without adding a blob. The memo's store prefix now also counts the rids the claim tables name at or below its top (claimRows; distinct rids, so a check-in's propagated tags do not move it), which settled checks before extending, and an extension takes as file content the old versions the arrivals' mlink rows name that no claim table names, as it does with the record.

loadFiles' comment now says what the fallback decides rather than claiming a proof: an unlinked blob that parses as an artifact is the file mlink says it is, elided outside the cone, and a corrupt out-of-cone version is elided unread. Tests pin both. check-in: 40b018695a user: [email protected] tags: main, perf/cone-clone-rounds, trunk

With the store classified, the cone clone of fossil-scm.org's src takes 8 rounds and receives 57,041 blobs to hold 56,262, against 12 rounds and 83,521 without the carried chains. The chains also cover versions a merge check-in brings in from a branch, which made up most of the remaining repeats. check-in: 102dab787b user: [email protected] tags: main, perf/cone-clone-resend, trunk

Review follow-ups to the seed drop.

A delta that arrives on a seed already dropped could never be read: it was stored on a content-less row that is never filled or asked for. It is now not stored as a delta but left a phantom and asked for again in full, on a pull, a clone's phantom pull and a push to an origin. A seed that arrives in full after deltas built on it is stored just long enough to check those deltas, judge them and rewrite them as full text, and then dropped.

seedcheck.Drop runs in a savepoint (or a transaction of its own on a pool), so a failure partway, which the after-dephantomize cascade logs and commits past, leaves the seed and its dependents as they were.

An origin answering a push no longer asks for a refused seed's phantom, and a clone asking for what it left missing no longer counts one. fossil 2.28's rebuild re-adds every size<0 row to the phantom table and deletes the store-local seedrefused and seedpending tables; the CHANGELOG notes this and the one-time drop of refused seeds that older stores kept.

The client now adds to a reply's new-phantom count rather than replacing it, so a phantom made while receiving a file continues the exchange. leaf check-in: 41c092205e user: [email protected] tags: fix/seed-delta-drop, main, trunk

A store this library crosslinked records each blob it found to be no artifact, and classification skips the file versions among them. A store fossil wrote has no such record, so classification read every file version raw and exceeded its byte budget: on a copy of fossil-scm.org's repository it failed on every request. A cone clone's walk then kept no base it had not reached yet, sent every in-cone version whole, and fit two or three 2-4 MB sqlite3.c texts into a 16 MB round: about 390 rounds and 30 minutes, where a dense clone takes 8 rounds.

Without the record, a version that mlink names and no table an artifact's crosslink writes claims is file content too, the proof the cone walk's knownFile already relies on; a table the store lacks claims nothing. That clone now takes 12 rounds and 9 minutes, verifying the same.

Related to #465. check-in: 4536dae44c user: [email protected] tags: main, perf/cone-clone-rounds, trunk

The server's cone walk sends an in-cone file version as a delta with the chain of newer versions ahead of it as its bases. Each round restarts the walk from the clone's cursor, so every later round that reached one of those versions at its own check-in sent it again.

The walk's cookie now names each chain a send forwarded: its head, length and an FNV-64 checksum of its rids. The next round rebuilds the chain from the store's delta table, counts its rids as sent, and drops it when the checksum no longer matches, so a store re-deltified mid-clone can never pass off a base the client was not sent. A run loses the leading rids the walk reaches, and the cookie carries at most 1,024 runs; one dropped is only sent again.

On a copy of fossil-scm.org's repository a cone clone of src now receives 72,835 blobs to hold 56,262 (main: 100,212); the clones verify identically. check-in: 9d266a0f51 user: [email protected] tags: main, perf/cone-clone-resend, trunk

A horizon seed stored as a delta whose base arrived only in a later round (pull, clone) or a later request (origin push) was refused by its hash but kept in the store, and a pull or clone counted the delta as received on arrival, before it could be judged (draft-fossil-horizon-00 §5.2.3, HZ-5.2-14).

On a seed verdict seedcheck now rewrites each delta built on the seed as full text, then empties the seed's row: size -1, no content, no delta link, and kept out of the phantom table so no request asks for it. The rid stays held for anything a receive pass tracks by it, and the refused hash is still recorded in store-local state.

A pull or clone counts a delta it could not read on arrival only at the end of the exchange, once readable and not dropped. A round still treats it as progress. The receive pass treats a row without content as no basis even when its cache holds the old bytes, and the push handler's same-request seed check consults the refused hash, since the dropped row no longer reads.

HZ-5.2-14 is now covered. check-in: 917bcc9830 user: [email protected] tags: fix/seed-delta-drop, main, trunk