Skip to content
quarry Artifact [b81e5e6f63]
Login

Artifact [b81e5e6f63]

Check-ins Using Download Text

Artifact b81e5e6f634ef6cc1915478d0314c55df5c4a94c9418060a675f115d3de441bf:

Summary

Fossil keeps a second kind of content beside the artifact DAG: unversioned files, "files stored in a Fossil repository without history, meaning it retains the newest version of each such file, and that alone" (fossil www/unvers.wiki). They sync between repositories with fossil sync -u, the newest mtime wins, a deletion is a tombstone, a server accepts them from a user holding y and shows them to a user holding o, and the web serves them at /uv/<name>. Fossil and sqlite publish their own downloads this way, and zit's releases are published this way to a stock fossil server that quarry is meant to replace.

Today zit validates uvfile cards and does not act on them (zit#58), and quarry's sync gate fails closed on one (scanRequest in internal/xfer/container.go). A fossil repository mirrored onto quarry loses every download link in its own documentation.

This RFC puts the tier in:

Motivation

Three things want the same tier.

Mirrors break without it. zeitforge hosts fossil's and sqlite's own repositories as mirrors. Their documentation links /uv/download.html, /uv/fossil-linux-x64-2.25.tar.gz and the like; docs/fossil-paths.md already marks /uv/<name> as "the one with a clear path" and says what breaks: "the download links in that repository's own documentation".

zit's releases need a home that is not a stock fossil server. The release skill publishes with fossil uv add releases/X.Y.Z/<file> --as ..., serves download.html and download.js from the same store, and syncs with fossil sync -u. quarry exists to replace that server. A forge that cannot hold the release downloads of the tool it is a forge for has not replaced anything.

Releases that sync are not forge-only. quarry ticket #19 chose the opposite: "Release binaries are forge-hosted, not repository content ... upload through the UI, store the bytes the way attachments are already stored and served by hash, and serve them from a per-release download page", and called the unversioned tier "zit verb and protocol work quarry cannot do, and which is not wanted". This RFC supersedes that direction, decided by the owner, for the reasons above plus one: unversioned content travels with sync -u to every clone that asks, so a release published on quarry is also on every developer's machine and on every mirror, which a forge-only attachment never is. That is fossil's design and the reason fossil has the tier at all.

What #19 got right stands: checksums come from release-manifest.json and SHA256SUMS, there is a per-release page, the upload is a form on the forge for anyone holding the letter, and large binaries never enter the DAG. Unversioned content is outside the artifact graph by definition; the newest version replaces the last, so nothing accumulates.

Fossil's semantics, as relied on

Everything below is cited from fossil's www/sync.wiki §3.3.4, §3.6.1, §3.7.1, §3.11, §5.4, www/unvers.wiki, www/caps/ref.html, and the server's own source where the documents are silent (src/xfer.c, src/unversioned.c at trunk).

Fact Source
A file is name, mtime (seconds since 1970), content hash, size. Only the version with the largest mtime is kept. sync.wiki §5.4
uvfile NAME MTIME HASH SIZE FLAGS \n CONTENT, both directions. HASH is - for deleted content. Flag 0x0004: content omitted (too large, or only the mtime changed). The receiver accepts only if hash and size match the content and the mtime is newer than what it holds. sync.wiki §3.3.4
uvigot NAME MTIME HASH SIZE is the catalog, one per file the server holds; uvgimme NAME asks for one. §3.6.1, §3.7.1
The client sends pragma uv-hash H (a hash over every name, mtime and hash it holds). A server that holds something different answers pragma uv-push-ok when the user has y, else pragma uv-pull-only, then the catalog. Both replies need o (g.perm.Read); without it the server answers nothing and the client learns the server does not share. unvers.wiki says instead that "a user needs the 'y' capability on the remote before they can sync unversioned content"; the source is read as the authority for the read half, and the mirror test against a real fossil server (Testing) decides it before rung 8 ships. §3.11; xfer.c, the uv-hash handler; unvers.wiki
A server accepts a uvfile card only from a user holding y ("Write permissions for unversioned files missing"). §3.3.4; xfer.c xfer_accept_unversioned_file
y, WrUnver: "Push unversioned content". o, Read. a does not grant y; s does (RFC 0004 keeps this). "Even the all-powerful setup user does not get unversioned file sync capability by default." caps/ref.html; unvers.wiki
A deletion keeps the row: hash=NULL, content=NULL, sz=0, with a new mtime. That tombstone is what syncs. unversioned.c, uv rm
/uvlist needs o; /uv/<name> serves the content. unversioned.c, doc.c
sync -u and clone -u carry unversioned content; push and pull have no -u. uv sync moves only unversioned content. The uv-sync setting makes a server send its catalog to every clone unasked. unvers.wiki; db.c
Names: no whitespace, no complex paths, at most 500 bytes; a directory prefix like releases/1.2.3/ is allowed and is only a name. A file larger than SQLite's blob limit is refused. unversioned.c
The content hash is the repository's hash policy (SHA1 or SHA3-256). §3.6.1

Where the bytes live

In the zit store, not in a bucket. Fossil keeps unversioned content in the repository file beside the artifacts, and three things follow that nothing else gives:

  1. A stock fossil client syncing with a zit-served repository, with no quarry in front, finds the content. Compatibility "both ways" means the store answers, not the forge.
  2. A clone with -u carries it, and a mirror pull carries it, through the same protocol with the same credential. Nothing new travels.
  3. RFC 0001 §11 is passed for free: a re-ingest rebuilds the index's unversioned listing from the store with one zit uv list -l, the way it rebuilds everything else. Unversioned content is store state, which is exactly the one kind of state the forge may hold.

RFC 0016's bucket is the steward's own storage for attachments, and its reasons (custody, a 300 MB file in the DAG forever) do not apply here: unversioned content is not in the DAG, does not accumulate, and is meant to be in every clone that asks. docs/fossil-paths.md's note that the attachment store is "the machinery" is withdrawn by this RFC; the attachment binding (internal/oltp/bucket.go) is untouched.

Limits. Two, both forge settings a site administrator (s) sets, each shown on the repository's unversioned page:

The index folds zit uv list -l at ingest (name, mtime, hash, size, deleted) and nothing else reads the store on the listing path. The uv.changed event joins RFC 0005's vocabulary, on the repository, with the name and whether it was a deletion.

Who. Fossil records only the receive id. quarry keeps a trail row (uv_trail: repo, name, action, who, at, source) in OLTP for every write it admitted: the signed-in uploader, the sync login that pushed, or mirror:<host> for an import. The listing shows it; a re-ingest keeps it, because OLTP is authoritative non-DAG state (RFC 0001 §8). A row whose file the store no longer lists is shown struck through, so a tombstone that arrived by sync still says who removed what.

zit: verbs and wire

zit#58 chose "tolerate and report" for uvfile; this replaces it. Three issues, scoped so each lands alone:

zit-A, the store and the verbs. An unversioned relation in the store with fossil's columns (name, mtime, hash, sz, encoding, content, plus the receive id), fossil's name rules, the hash policy's hash. Verbs, matching fossil's unversioned help page where quarry depends on them:

verb does
zit uv add FILE... [--as NAME] [--mtime T] add or replace; --mtime defaults to now
zit uv list [-l] [--glob G] names; with -l mtime, hash, size, and deleted for a tombstone, one record per line, machine-readable
zit uv cat NAME..., zit uv export NAME OUT content
zit uv rm NAME... the tombstone (null content, size 0, new mtime)
zit uv hash the uv-hash value, so a caller can ask "anything to sync?" cheaply

import-fossil carries the unversioned table; export writes it back. Content hashes verify on every read.

zit-B, the wire. zit serve answers pragma uv-hash with uv-push-ok or uv-pull-only and the uvigot catalog, answers uvgimme with uvfile (or uvigot when the reply is already large, as fossil does), and accepts uvfile by fossil's rule: hash and size match, mtime newer, tombstones included. zit's serve knows no users, so which pragma to send is told by the fronting proxy, the way protected-branch rules already reach the child (ProtectHeader in internal/xfer/xfer.go): a request header X-Zit-UV: push | pull | none. push admits uvfile; pull answers the catalog and uvgimme and refuses uvfile with fossil's error text; none answers nothing to uv-hash, which is what a fossil server does for a user without o. Absent the header (a bare zit serve, no forge in front) the answer is push, which is the single-user default the sync gate already applies to pushes (RFC 0001 §7: a repository with zero sync users pushes anonymously).

On the client side: zit sync -u, zit clone -u, and zit uv sync URL [--pull-only]. --pull-only sends uv-hash and uvgimme and never uvfile; it is what a mirror uses, and it has no fossil equivalent (fossil's nearest is uv revert, which overwrites local content, the wrong thing for a mirror that may hold local writes).

zit-C, fossil-compatibility proof. A test in zit's own suite: a stock fossil binary clones -u from zit serve, adds a file, syncs -u back, removes it, syncs again; zit's store shows the content, then the tombstone, and fossil uv list on a second fossil clone agrees. The same test runs in quarry's suite through the sync front (Testing).

quarry: the sync front

scanRequest (internal/xfer/container.go) frames a request's cards to gate pushes and today returns failClosed() on uvfile, because the card's payload framing was outside the request vocabulary it knew. It learns the card: uvfile has five tokens, SIZE fourth, and SIZE payload bytes follow the line unless flag 0x0004 is set, in which case none do. The scan records three facts: the request carries a uvfile (a write), a uvgimme or uv-hash (a read of unversioned content), and the largest single uvfile size. Fail-closed stays for a card it cannot frame.

The gate then decides, before the child sees a byte:

request carries needs on failure
uvfile a verified login card whose account holds y on the repository, and the write guard (WriteGuard, the single-writer lease), and the size limits refused with the same shape as a push without i: error card naming the missing letter, the DAG part of the request is not applied either (fossil rejects the whole message)
uv-hash, uvgimme the read gate (ReadGate): o on a private repository, anyone on a public one the child is told X-Zit-UV: none and the client sees a server that does not share
neither unchanged unchanged

The header the child receives is push when the login holds y, pull when it may read, none otherwise. A tracking mirror (accept_writes off, internal/oltp/mirror.go) is pull for everyone, as it is for pushes: the forge's copy of someone else's repository takes no local writes, unversioned ones included. A mirror with accept_writes on takes them, and the next import applies fossil's rule: the newer mtime wins, whichever side wrote it.

Private repositories: MayMirror (internal/oltp/repo.go) already answers no for a private repository, and the read gate already admits a verified login that may read it. Unversioned content rides both and adds no rule.

Letters and roles

RFC 0004 §4 forbids a new letter and none is needed: y is in internal/oltp/capref.go already, listed as "Write-UV: Push unversioned content" and inert. It stops being inert. The op table gains:

op letter fossil surface
uv.list, uv.get o /uvlist, /uv/NAME, the catalog
uv.put, uv.rm y uvfile
the limits, per repository a sees, s sets the uv-sync setting's neighbourhood

RFC 0008's roles do not change. Admin stays "everything except setup, private branches, unversioned writes", which is fossil's rule and the right one: an administrator of settings is not thereby a publisher of binaries. An account that publishes is Custom, read by 0008 as "Maintainer, plus push unversioned content", and that sentence is the whole interface. s holds y everywhere, as it holds everything.

The forge's own release account (Releases) is an ordinary account with y on the quarry repository and nothing else.

The web

Listing, /r/{repo}/uv, visible to whoever may read the repository: name, size, hash (prefix, full on hover), mtime, who (from the trail), with tombstones in a collapsed "removed" section so a reader can see what used to be there and when it went. Sortable by name and by age, as fossil's /uvlist is. The page says how much of the repository's total is used.

Download, /r/{repo}/uv/{name}, with name allowed to contain /. Served with X-Content-Type-Options: nosniff, a Content-Security-Policy that permits nothing, and the type from the extension by a fixed table. HTML, SVG, JavaScript and anything else that would run on the forge's origin is served as Content-Disposition: attachment, the rule RFC 0016 applies to attachments. This differs from fossil, which renders /uv/download.html as a page. A forge serving user-published HTML on its own origin is one y grant from script on every reader's session; the releases page below is the replacement for a hand-written download page. ETag is the content hash, so a client that has it gets 304.

Fossil's paths: /uv/<name> and /uvlist join FossilRoute (docs/fossil-paths.md), so a link in a mirrored repository's wiki resolves. Its line moves from "Future work" to "Mapped".

Upload and remove, for holders of y: a plain multipart form on the listing page, name defaulting to the file's, optional prefix, replaced on the same name; a remove button per row. Both go through uv.put and uv.rm, which call zit uv add and zit uv rm under the write guard, record the trail, and run one ingest round, as every write op does. Uploads are not held or screened: y is the trust, and fossil's own posture ("uncommonly dangerous ... no history to revert to") is the reason the letter is granted by hand.

Releases, /r/{repo}/releases: every name under releases/<version>/ grouped by version, newest first by semantic order, with latest marked. When the group holds release-manifest.json, the page reads it: each artifact's target, size and sha256, the source check-in (linked), the date, and the qualification notes if present, in the schema zit's releases already use (schema_version: 1, keys artifacts, omitted_targets, qualification, release, source). Without a manifest, the group is a list of files with SHA256SUMS beside them, and the page says checksums come from the file. A CHANGELOG.md section whose heading names the version is shown beside it, the way /changelog renders the file today (internal/web/web.go, changelogPage). The repository tabs show Releases only when a releases/ prefix exists, the way they hide what a repository cannot use.

Mirrors

A mirror pulls with zit pull URL (Client.Pull in internal/zitcli/zitcli.go, called from PullMirror in internal/api/register.go) on its own interval, and the mirror table in internal/oltp/mirror.go holds repo, url, every, enabled and its history. Two additions:

uv, a switch per mirror, off by default, on the fleet page's mirror form (/fleet?tab=mirrors) where the address and interval are set, and shown read-only in the repository's own Mirror section with its last result. When on, each pull is followed by zit uv sync URL --pull-only. Latest mtime wins; a tombstone upstream becomes a tombstone here; a local file the upstream never had stays (relevant only with accept_writes). The round is recorded in mirror_pull beside the DAG pull with its own uv_changed and uv_err, so "the code is current but the downloads are three days old" is visible.

A credential per mirror, user and password for the upstream, sealed the way a bucket secret is (internal/oltp/bucket.go, secret_enc), never returned by an op. Today a mirror pulls as whoever the URL names, usually nobody. Reading the unversioned catalog needs o on the upstream (xfer.c: the uv-hash reply is behind g.perm.Read); fossil's stock nobody holds o, so most public fossils answer anonymously, and the credential is for the ones that do not and for upstreams that gate the DAG too. y is never needed on the upstream: a mirror only reads.

When the upstream does not share. zit's --pull-only reports "no unversioned catalog" when uv-hash draws no pragma. quarry records it as the round's uv_err, the Mirror section shows "upstream does not share its unversioned files with this account", and the DAG pull's result stands on its own. A mirror never fails because of its downloads. The same shape covers a file over this forge's per-file ceiling: the catalog lists it, quarry skips it, the row says "not carried (larger than this forge's limit)".

Detaching a mirror keeps its unversioned content, as it keeps its history. Divergence (diverged_at) is about check-ins and stays so; a local unversioned write on a tracking store is refused by the gate above, so it cannot happen silently.

Releases

quarry's version today is 0.1.0-beta+<check-in> stamped by scripts/build-linux.sh, nothing is tagged, and scripts/deploy.sh builds trunk on the box. The runner downloads for five platforms (internal/runnerdist/runnerdist.go, Platforms) are built by the same script, verified as a set by scripts/runner-check.sh, and served at /runner/<file> from the directory beside the binary.

Versions. A VERSION file at the tree's root holds the version the tree is working toward, 0.2.0. A release is the check-in carrying the tag version-0.2.0 (fossil's and zit's spelling; build-linux.sh already fetches zit by version-${ZIT_VERSION}). quarry version prints 0.2.0 when the built check-in carries the tag that matches VERSION, else 0.2.0-dev+<check-in>; the footer and the bug-report link carry the same string. Semantic versioning, with the beta suffix retired at the first tag.

Changelog. CHANGELOG.md is already the forge's news page. Cutting a release requires a ## 0.2.0 heading in it; the release job refuses otherwise, which is the one place a release is forced to say what it is.

Artifacts, under releases/0.2.0/ in the quarry repository's own unversioned store:

file holds
quarry-0.2.0-linux-amd64.tar.gz, -arm64 quarry, the zit it was built against (the pinned release, so the two cannot drift, RFC 0001 §2), CHANGELOG.md, LICENSE
quarry-runner-<os>-<arch>[.exe] the five runner builds, same names runnerdist serves
quarry-0.2.0-src.tar.gz the tagged tree
SHA256SUMS one line per file above
release-manifest.json zit's schema: per artifact target, sha256, size; release.version, source.checkin, release.date; omitted_targets named (macOS and Windows forge builds are omitted, the DuckDB bindings do not cross-link, as build-linux.sh records)

The release workflow, ci/release.yml in RFC 0019's form. RFC 0019's on: has push, landing, workflow_dispatch, schedule; this RFC adds tag with a tags: glob (version-*), evaluated by the quarry trigger from the tag artifact's check-in, and admitted by an environment's branch policy only when that check-in is on a protected branch (a tag on a branch nobody reviewed is not a release). The job:

on:
  tag:
    tags: [version-*]
jobs:
  release:
    environment: release
    steps:
      - run: scripts/release.sh
        env:
          RELEASE_SYNC_PASSWORD: ${{ secrets.RELEASE_SYNC_PASSWORD }}

The password reaches the script as an environment variable, the only place RFC 0019 lets a secret go, and never as an argument, which every user on the runner's machine can read.

scripts/release.sh checks VERSION against the tag and CHANGELOG.md for the heading, runs build-linux.sh for the two forge targets and the five runner targets, packages, writes SHA256SUMS and the manifest, verifies the sums from the files on disk, then zit uv add --as releases/0.2.0/<file> into a scratch clone and zit uv sync to the forge as the release account (y on the quarry repository, nothing else; the release environment holds its password, and protected branches only is the policy). The forge's ingest round shows the group on /r/quarry/releases within a minute. Retention is the job's too: it removes releases/<old>/ beyond the newest three minor versions and the newest patch of each, with zit uv rm, and the manifest of the newest release lists what was retired. This is the one y holder on the forge and it runs reviewed code only.

Deploys select a release. deploy.sh gains --release X.Y.Z: fetch releases/X.Y.Z/* from the forge's own /r/quarry/uv/ over localhost, verify SHA256SUMS, install quarry, zit and the runner set exactly as the build path installs them today (keep quarry.prev, restart, health-check, roll back on silence, restart the runner last and delayed), no docker, no build. --require-green and --expect keep their meaning for the trunk path and are refused with --release (a release was green when tagged, by the environment policy). quarry-deploy@<instance>.service reads its instance: a hash runs the trunk path with --require-green --expect <hash> as today; version-X.Y.Z runs --release X.Y.Z. The deploy-watch timer and the alert path (docs/running.md, "Deploys that say when they fail") cover both unchanged.

Rollback is deploy.sh --release <previous>, which the release page names beside each version, or deploy.sh --rollback for quarry.prev when the forge is down and nobody wants to think. OLTP migrations are additive columns (mirrorMigrations and its siblings), so an older binary runs against a newer database; the index is disposable and re-folds. A release that needs a non-additive migration says so in its changelog entry and the release page shows a "cannot roll back past" mark from the manifest.

/runner/ keeps serving the installed set from beside the binary: a forge that is not zeitforge has no quarry repository to redirect to, and the setup page's links keep working for every operator.

Does accepting a landing deploy?

The pieces are in place: quarry-deploy@<hash> exists so that "a CI deploy ships the check-in it tested, never a newer one" (deploy.sh's header), the runner restarts itself last and delayed so a deploy it started can report, and RFC 0019 can give a deploy job on trunk a production environment whose one secret is permission to run systemctl start --no-block quarry-deploy@<hash> through a sudo rule naming that unit pattern and nothing else. Every accepted landing would then deploy, through one narrow privileged unit, with --require-green asked of the forge rather than trusted from the job.

For: deploys stop being a thing a person remembers; what runs is always a check-in CI passed; the alert path already covers the failures; the 2026-09-19 drift between the installed deploy.sh and the tree's copy (the script's own comment) happened because deploys were hands-on.

Against: each deploy restarts the forge (17.6 s of refused syncs, measured in ci/quarry.json's comment, plus the 65 to 75 s register window deploy.sh waits through), and trunk takes several landings a day; a change nobody looked at running reaches production before a person has seen it run; and the CI runner that starts the deploy is restarted by it, so a second landing accepted inside that window waits on the lock, which is correct but surprising.

Recommendation: yes, with releases as the second track. Trunk deploys on accept through the unit; that is the forge's own development loop and the pieces were built for it. Releases are cut by tag, less often, for every other operator and as the named rollback points; zeitforge itself deploys --release only to roll back or when trunk is paused. Pausing is deleting the production environment's secret, which is a settings change with a trail. Left open for the owner because it changes who restarts production.

Build order

Rung Work Waits on
1 This RFC the owner
2 zit-A: store relation, uv add|list|cat|export|rm|hash, import-fossil and export carry the table a zit issue
3 zit-B: serve answers uv-hash, uvigot, uvgimme, accepts uvfile, chooses its pragma by X-Zit-UV; client sync -u, clone -u, uv sync [--pull-only] a zit issue; rung 2
4 zit-C: the stock-fossil round trip in zit's suite rung 3
5 quarry gate: scanRequest frames uvfile; y, the read gate and the limits decide; X-Zit-UV set; the trail table. Testable against fixtures before zit lands, since the gate refuses or forwards and never interprets nothing (framing); rung 3 to pass traffic through
6 quarry web: uv.list/get/put/rm, the listing, /uv/{name}, /uvlist and /uv/ in FossilRoute, upload and remove forms, index fold, uv.changed, the disposability snapshot rung 2
7 Releases page with the manifest schema and the changelog section rung 6
8 Mirrors: the uv switch, the credential, uv sync --pull-only after each pull, the status lines rung 3
9 RFC 0019 amendment: the tag event; VERSION, the stamp, scripts/release.sh, ci/release.yml, the release account and environment, and a runner for the quarry repository started with --job-users (a job holding a secret runs on no other; quarry's own runner runs jobs as itself today) rung 7; the trigger work is quarry's own
10 deploy.sh --release and --rollback, the unit's instance rule, docs/running.md rung 9
11 The first tagged release, version-0.2.0; zit's own release skill moves its publish step to quarry and the stock fossil server is retired rungs 4, 8, 10
12 The auto-deploy job, if the owner says yes rung 10, the answer

Rungs 5 and 6 begin beside the zit work; nothing in them interprets unversioned content, they frame, gate, list and serve what the store reports. Rung 8 is small once rung 3 exists. zit's rungs are filed as issues on zit and never worked around in quarry; until rung 3 lands the gate's new framing still ends in refusal, so no behaviour changes early.

What this is not

Open questions

  1. Deploy on accept. Laid out above; recommendation yes, trunk by unit, releases by tag.
  2. Tag spelling. version-X.Y.Z (fossil's and zit's, already in build-linux.sh) or vX.Y.Z (everyone else's). Recommendation: version-X.Y.Z, because the one tool that reads these tags already does, and a mirrored fossil repository's tags will look the same.
  3. The per-repository total. 2 GiB is a guess sized to zit's three retained releases and sqlite's download set. Recommendation: ship 2 GiB, show the use on the page, and let s raise it per repository; a quota nobody can see is a quota that breaks a release at midnight.

Testing