zit 0.7.0

zit 0.7.0 adds horizon clones: a clone that holds a repository's history from a chosen check-in forward, and fetches what lies below it only when asked. It ships the draft that specifies them, and it changes how a cone-scoped origin serves a delta that crosses the cone and how a cone clone is written to disk.

This release is from trunk. A store at a horizon cannot be opened by older releases; see Compatibility.

Horizon clones

A horizon is a check-in C and a rule for which check-ins count as at or after it. The held set is computed over P cards, never over log order or timestamps. Under the default rule, descendants, it is C and every check-in descended from C. Under leaves it also takes in, for every childless check-in that does not descend from C, everything descended from where its ancestry meets C's. That keeps the merge base of two live branches inside the clone, and on a repository with old open branches it costs a good deal more. Every artifact a horizon clone holds is the artifact the origin holds, under the same name. Nothing is rewritten, and what the clone lacks it knows by name as elided phantoms.

There are two kinds.

The content horizon keeps every check-in manifest, every control artifact, wiki, ticket, technote, forum and attachment artifact of every age. It keeps file content only for the check-ins in the held set. Timeline, leaves, branches, tags and merge ancestry are derived exactly as on a dense clone, because every input to the derive is present.

The structural horizon also leaves out the check-in manifests and tree nodes below C, keeping only those the held set reaches through P, B and S cards. A delta manifest's baseline below C is kept, because the manifest cannot be expanded without it. Control artifacts are still kept at every age. In place of the missing ancestry the origin supplies a seed (below), so branch and tag state above C still derives exactly. The timeline ends at C.

zit clone URL [DIR] --horizon=HASH [--horizon-rule=descendants|leaves]
zit clone URL [DIR] --horizon=HASH --structural

--horizon takes the full hash of C, and --structural is meaningless without it. A clone reports what the origin did:

cloned 2986 artifacts into usr.bin in 5 rounds (horizon clone; origin served: structural; nothing elided locally)

An operation that needs history below C says so and names the horizon, rather than treating the missing artifact as damage or as an empty file. cat, diff and annotate name the elision. zit update CHECKIN with a hash below C deepens: it moves C back, keeps the depth and rule, and pulls what the new held set adds from the saved remote. merge asks the origin for a merge base below C, deepens to it, and runs over real artifacts. A deepen with no known common ancestor of C and the target is refused, and C does not move. zit verify counts deliberately elided artifacts on their own line, distinct from faults. zit export-git refuses a horizon store, because a Git mirror needs every blob.

A new plumbing verb, zit merge-base A B, prints Fossil's merge base of two check-ins. An origin answers a horizon clone's merge-base question with the same computation.

Measured on NetBSD src (317,081 check-ins, 3,649 MB dense), served over loopback, every clone verifying with 0 faults: a structural clone of the whole tree at the trunk tip holds 182,735 artifacts and takes 189 s. On the pkgsrc sub-manifest store a structural clone is 61 MB against 632 MB dense.

Horizons and path cones

A horizon is a member of the store cone, so --horizon combines with --cone, --cone-not and --cone-file. A blob is kept when it is both at or after the horizon and named by an in-cone F card. The horizon alone decides which check-in manifests are held. Within the trees it holds, the origin withholds the sub-manifest nodes that no S-card slot reaching into the cone leads to. Under a content horizon that walk covers every check-in's tree, so the origin walks only the trees appended since it last classified the cone.

On NetBSD src at the trunk tip, with every clone verifying clean:

Clone	Wall
structural, --cone=usr.bin, cold	4.6 s
structural, --cone=usr.bin, warm	1.10 s
structural, --cone=sys	44.5 s
content, --cone=usr.bin	452 s, 449,113 artifacts, 359,052,984-byte store
content, --cone=sys	1611 s, 713,630 artifacts

The seed

The seed is a control artifact the origin mints from its own derivation. Its T cards apply directly to C the tags in force there: the branch tag with its value, every other propagating tag, the closed marker when C is closed, and every non-propagating tag on C. It does the same for each other check-in of the held set whose primary parent lies below the horizon, which is what a branch forked below C and later merged with C needs. The D card is the latest time among the tag events it summarizes, rounded up to a whole second, so a seed over unchanged state has the same bytes and the same name every time it is minted. The U card is zit-serve, not a user's login.

The clone holds the seed beside the store cone, outside the artifact store, and applies it at every open as an ordinary tag application. It never sends it to any peer. It names the seed it holds on every pull or sync, and the origin sends a new one only when its name differs. A repository with no tag rows in force has nothing to seed: the origin mints none and serves the request at the content level.

The seed feeds an artifact in one place. A check-in that creates a branch writes a cancel card for the parent's branch as the clone derived it. An origin checks that card at push. If the clone's seed was stale, it refuses the check-in and names the command that corrects it, and it admits the check-in once the same push carries that correction.

Against a stock Fossil server

Fossil ignores the horizon pragmas and serves densely. A horizon clone of a Fossil server classifies what arrives locally, keeps what the content horizon keeps and records the rest as elided, so it costs bandwidth and never correctness. A structural request has no seed to stand on, so the clone falls back to the content horizon and says so:

zit clone: the origin did not serve the structural horizon with a seed; keeping the content horizon instead

Fossil folds unclustered artifacts into a cluster and announces only the cluster, so a horizon clone pulling from Fossil requests what a cluster names and classifies it on arrival. Against a zit origin that says horizon-served, it requests nothing that only a cluster names.

Pulling and syncing from a horizon clone

pull and sync carry the store's horizon on every request. Pass no flag: the origin filters by the same held set, labels boundary parents, re-seeds when needed, and sends what new check-ins add. The origin hands back a cursor, so a later pull announces nothing and a push announces only what the clone appended. On the pkgsrc structural clone, a no-op pull is 431 bytes up and 237 down. The origin keeps no record of which clients hold what.

A structural store that pulls or syncs from a peer whose replies do not carry horizon-served structural, such as stock Fossil, falls back to the content horizon at the same C and rule. It drops its seed, fetches the manifests below C that the content horizon holds, and says so:

zit pull: the origin did not serve the structural horizon; keeping the content horizon instead, with every manifest below it

A horizon clone is not a complete origin. Served, it refuses a dense clone and any horizon clone it cannot cover, with one error card and no content. A stock Fossil clone of it ends in a refusal and leaves no repository behind. A dense peer that already holds the history below the horizon can still pull from it.

Pushing to stock Fossil

A check-in made in a horizon clone is an ordinary check-in. Its manifest names its parent and its files by real hash, and nothing in it reveals the horizon. Check-ins made in a content horizon clone of a Fossil server and in a structural horizon clone of a zit server, pushed to a stock Fossil server, arrive byte for byte as zit wrote them. A fresh Fossil clone of that server rebuilds and integrity-checks clean and checks each one out to the files zit committed.

Streaming persist for cone and horizon clones

A cone or horizon clone used to hold every round in memory until the exchange ended, then filter the whole transfer locally and write it. It now does that only when it must. When the origin's first reply says it filtered for this request, the rounds are written to the log as they arrive, the way a dense clone's are, and each round's bytes are released once written. For a horizon the origin says horizon-served at the depth asked for, and for a request naming paths it says cone-served. The cone is then classified once over the log. An origin that did not filter still gets the batched local filter.

Crossing deltas

An origin may store a kept blob as a delta against a base that the requester's cone elides. Served verbatim, it is a blob the recipient holds and can never expand. A cone-scoped or horizon-scoped origin now checks each served delta against the request's keep set and sends one whose base is outside it in full form, as sparse-cone section 6.2 and horizon section 6.2.1 require. Only deltas that cross are rebuilt. The recipient still treats the source of a retained delta as a transit phantom, so an origin that cannot rebuild one costs bandwidth, never a stuck artifact.

On a path-only --cone=usr.bin clone of NetBSD src, 0.6.6's origin sent 1563 out-of-cone delta bases that the client fetched and then dropped. 0.7.0 sends none.

Path-cone classification

The classifier's last step re-walked every tree in the store to find sub-manifest nodes it could elide by slot. A clone from an origin that sends every node has no such node, so the walk found nothing. It now runs only when some S card names a node the store does not hold.

The same path-only usr.bin clone of NetBSD src, 932,336 artifacts:

Build	Wall	Client user CPU	Store
zit 0.6.6	1826 s	1347 s	1,244,509,552 bytes
zit 0.7.0	1713 s	1343 s	1,241,666,920 bytes

In both, about 55% of the wall time is the cache derive after the transfer.

Cloning a cone from an older origin

An origin from 0.6.3 through 0.6.6 acknowledges a path cone with cone-served, so a 0.7.0 client streams its clone. That origin predates the crossing-delta rule: it sends a kept blob as a delta against an out-of-cone base, and answers the clone's request for the base. Streamed, that base would have been written to the store as it landed. What arrives after the clone sequence is now held back for the cone to judge. A base the cone does not keep is dropped, and the kept blob is stored in full form, as the batched path already did.

Wire capabilities

An origin that implements the horizon draft says pragma horizon-served content or pragma horizon-served structural in every reply to a request carrying a horizon, naming the deepest cut it served. It advertises structural only when the same reply carries the seed, or the client already holds the seed it would send. A client enters the structural horizon only on that word, and a structural store leaves it on any exchange that lacks it. cone-served, from 0.6.3, now also decides whether a path-cone clone streams. The other pragmas are horizon, horizon-structural, horizon-below, horizon-seed, horizon-mergebase and horizon-cursor. A merge-base query carries at most two secondaries, the checkout and one pending merge, and a merge that needs more across a horizon is refused.

Every one is a pragma, which Fossil and zit 0.6.x ignore. Against such a peer a horizon request degrades as described above. A path-cone request to an origin without cone-served is filtered locally, as before.

The horizon draft

The source package ships docs/rfcs/draft-fossil-horizon-00.txt. It specifies the horizon as a temporal member of the sparse cone, both cuts, the seed, boundary labels, re-seeding, commit and push across a stale seed, deepen, the sync pragmas, the origin's classification kept across requests, the cursor, and degradation against a peer that knows none of it. The sparse-cone draft states the crossing-delta rule and the member kinds a cone may carry. The sync-protocol draft lists every horizon pragma with its direction.

Compatibility

Dense and path-cone stores are unchanged and stay readable by every 0.6.x release. A store at a horizon carries a new store header version, written before the horizon policy and lowered only once no horizon is held. 0.6.x refuses such a store, and its derived image, rather than read it as dense. Deepening, re-seeding and pack keep the horizon version. The origin's persisted classification and structural graph are .zit-cache sidecars: deleting them costs time, never data.

On the wire, a 0.7.0 origin is an ordinary origin to stock Fossil and to zit 0.6.x. The horizon pragmas change only a reply to a request that carries one. A stock Fossil clone of a 0.7.0 server, and later pulls from it, produce the repository a Fossil server of the same content does. A 0.7.0 horizon clone works against Fossil or zit 0.6.x at the content level. It falls back from structural, and pushes check-ins Fossil stores byte for byte. A path-cone clone from 0.6.x's origin keeps nothing outside the cone. The Fossil interoperability smoke scenarios cover each direction.

Qualification

The release manifest records native qualification against the exact tagged source archive on macOS ARM64, Linux x86-64 and Linux ARM64, using the pinned Zig 0.17.0 toolchain. Publication is gated on unit and end-to-end tests, Git differentials, deterministic simulation, seeded fuzzing, and Fossil interoperability checks on every distributed target.

For the basis of the conformance claim, see the 0.1.0 release notes.