Skip to content
quarry Artifact [664f613077]
Login

Artifact [664f613077]

Check-ins Using Download Text

Artifact 664f613077013f1b014922cfd80c552b64429b5c36e5a860dbf940f9e27be47a:

Summary

Fossil keeps a second kind of content beside the artifact DAG: unversioned files, "files stored in a Fossil repository without history, meaning it retains the newest version of each such file, and that alone" (fossil www/unvers.wiki). They sync between repositories with fossil sync -u, the newest mtime wins, a deletion is a tombstone, a server accepts them from a user holding y and shows them to a user holding o, and the web serves them at /uv/<name>. Fossil and sqlite publish their own downloads this way, and zit's releases are published this way to a stock fossil server that quarry is meant to replace.

Today zit validates uvfile cards and does not act on them (zit#58), and quarry's sync gate fails closed on one (scanRequest in internal/xfer/container.go). A fossil repository mirrored onto quarry loses every download link in its own documentation.

This RFC puts the tier in:

Motivation

Three things want the same tier.

Mirrors break without it. zeitforge hosts fossil's and sqlite's own repositories as mirrors. Their documentation links /uv/download.html, /uv/fossil-linux-x64-2.25.tar.gz and the like; docs/fossil-paths.md already marks /uv/<name> as "the one with a clear path" and says what breaks: "the download links in that repository's own documentation".

zit's releases need a home that is not a stock fossil server. The release skill publishes with fossil uv add releases/X.Y.Z/<file> --as ..., serves download.html and download.js from the same store, and syncs with fossil sync -u. quarry exists to replace that server. A forge that cannot hold the release downloads of the tool it is a forge for has not replaced anything.

Releases that sync are not forge-only. quarry ticket #19 chose the opposite: "Release binaries are forge-hosted, not repository content ... upload through the UI, store the bytes the way attachments are already stored and served by hash, and serve them from a per-release download page", and called the unversioned tier "zit verb and protocol work quarry cannot do, and which is not wanted". This RFC supersedes that direction, decided by the owner, for the reasons above plus one: unversioned content travels with sync -u to every clone that asks, so a release published on quarry is also on every developer's machine and on every mirror, which a forge-only attachment never is. That is fossil's design and the reason fossil has the tier at all.

What #19 got right stands: checksums come from release-manifest.json and SHA256SUMS, there is a per-release page, the upload is a form on the forge for anyone holding the letter, the bytes live where attachments do, in the bucket the repository's admin supplies, and large binaries never enter the DAG. Unversioned content is outside the artifact graph by definition; the newest version replaces the last, so nothing accumulates.

Fossil's semantics, as relied on

Everything below is cited from fossil's www/sync.wiki §3.3.4, §3.6.1, §3.7.1, §3.11, §5.4, www/unvers.wiki, www/caps/ref.html, and the server's own source where the documents are silent (src/xfer.c, src/unversioned.c at trunk).

Fact Source
A file is name, mtime (seconds since 1970), content hash, size. Only the version with the largest mtime is kept. sync.wiki §5.4
uvfile NAME MTIME HASH SIZE FLAGS \n CONTENT, both directions. HASH is - for deleted content. Flag 0x0004: content omitted (too large, or only the mtime changed). The receiver accepts only if hash and size match the content and the mtime is newer than what it holds. sync.wiki §3.3.4
uvigot NAME MTIME HASH SIZE is the catalog, one per file the server holds; uvgimme NAME asks for one. §3.6.1, §3.7.1
The client sends pragma uv-hash H (a hash over every name, mtime and hash it holds). A server that holds something different answers pragma uv-push-ok when the user has y, else pragma uv-pull-only, then the catalog. Both replies need o (g.perm.Read); without it the server answers nothing and the client learns the server does not share. unvers.wiki says instead that "a user needs the 'y' capability on the remote before they can sync unversioned content"; the source is read as the authority for the read half, and the mirror test against a real fossil server (Testing) decides it before mirror import (rung 6) ships. §3.11; xfer.c, the uv-hash handler; unvers.wiki
A server accepts a uvfile card only from a user holding y ("Write permissions for unversioned files missing"). §3.3.4; xfer.c xfer_accept_unversioned_file
y, WrUnver: "Push unversioned content". o, Read. a does not grant y; s does (RFC 0004 keeps this). "Even the all-powerful setup user does not get unversioned file sync capability by default." caps/ref.html; unvers.wiki
A deletion keeps the row: hash=NULL, content=NULL, sz=0, with a new mtime. That tombstone is what syncs. unversioned.c, uv rm
/uvlist needs o; /uv/<name> serves the content. unversioned.c, doc.c
sync -u and clone -u carry unversioned content; push and pull have no -u. uv sync moves only unversioned content. The uv-sync setting makes a server send its catalog to every clone unasked. unvers.wiki; db.c
Names: no whitespace, no complex paths, at most 500 bytes; a directory prefix like releases/1.2.3/ is allowed and is only a name. A file larger than SQLite's blob limit is refused. unversioned.c
The content hash is the repository's hash policy (SHA1 or SHA3-256). §3.6.1

Where the bytes live

In the repository's bucket. RFC 0016's binding (internal/oltp/bucket.go: endpoint, bucket, region, a sealed key, public_base, limit_bytes) is the repository admin's storage, and unversioned content is stored there beside attachments, under its own prefix:

The bucket is the record. quarry keeps a catalog in OLTP (name, mtime, hash, size, deleted, who) so the listing and uv-hash never list the bucket on a request path, and rebuilds it from a bucket listing and the objects' metadata when it is missing or a different bucket is bound. That is RFC 0001 §11's test passed the way RFC 0016 passes it: the bytes are the steward's, outside the forge, and nothing the forge holds is the only copy.

No bucket, no unversioned files. The listing says "bind a bucket in Settings to publish unversioned files", the sync front answers a uv-hash the way a fossil server answers a user without o (nothing), and the mirror import switch is unavailable. The forge owns no storage for this, as RFC 0016 owns none for attachments; docs/fossil-paths.md's "forge bucket as the default" is withdrawn.

No quota. The bucket is the owner's and writing takes y, a letter granted by hand. The one bound is the binding's own limit_bytes, the per-file limit its admin already sets for attachments (default 100 MiB), and, for a uvfile arriving by sync, the sync front's request cap (maxRequest, internal/xfer/xfer.go, 256 MiB), since a card is one request. A file over either is refused by name, never partially applied.

Downloads are presigned GETs from the bucket (or its public_base), as attachments are (internal/s3/s3.go, PresignGet), so the forge never streams the bytes.

Who. Fossil records only the receive id. The catalog row carries who wrote it: the signed-in uploader, the sync login that pushed, or mirror:<host> for an import. A tombstone keeps its row, so a removal that arrived by sync still says who removed what. The uv.changed event joins RFC 0005's vocabulary, with the name and whether it was a deletion.

zit: verbs and wire

zit#58 chose "tolerate and report" for uvfile; this replaces it on the client side. quarry answers the server side itself, so zit's serve does not need it for a quarry-hosted repository. Two issues:

zit-A, the local store and the verbs. An unversioned relation in the store with fossil's columns (name, mtime, hash, sz, encoding, content), fossil's name rules, the hash policy's hash. Verbs, matching fossil's unversioned help page where quarry depends on them:

verb does
zit uv add FILE... [--as NAME] [--mtime T] add or replace; --mtime defaults to now
zit uv list [-l] [--glob G] names; with -l mtime, hash, size, and deleted for a tombstone, one record per line, machine-readable
zit uv cat NAME..., zit uv export NAME OUT content
zit uv rm NAME... the tombstone (null content, size 0, new mtime)
zit uv hash the uv-hash value

import-fossil carries the unversioned table; export writes it back.

zit-B, the client wire. zit sync -u, zit clone -u and zit uv sync URL [--pull-only]: send pragma uv-hash, read uv-push-ok or uv-pull-only and the uvigot catalog, send uvfile and uvgimme by fossil's rules (newer mtime wins, tombstones included). --pull-only never sends uvfile; it is what a mirror uses, and it has no fossil equivalent (fossil's nearest is uv revert, which overwrites local content). Against quarry and against a stock fossil server alike.

Whether zit's own serve answers unversioned content for a repository nobody fronts is zit's choice and outside this RFC.

quarry: the sync front

scanRequest (internal/xfer/container.go) frames a request's cards to gate pushes and today returns failClosed() on uvfile. It learns the unversioned cards: uvfile has five tokens, SIZE fourth, and SIZE payload bytes follow the line unless flag 0x0004 is set; uvgimme and pragma uv-hash are one line each. Fail-closed stays for a card it cannot frame.

quarry answers them itself. The front splits a request: the unversioned cards it handles from the catalog and the bucket, and everything else goes to the zit child as today. Its reply to the client is the child's reply with the unversioned cards appended, which fossil's message format allows (a reply is a sequence of cards). So:

request carries needs quarry answers
pragma uv-hash H the read gate (ReadGate): o on a private repository, anyone on a public one, and a bound bucket when H differs from the catalog's hash: pragma uv-push-ok if the login holds y, else uv-pull-only, then a uvigot per catalog entry; otherwise nothing
uvgimme NAME the read gate uvfile with the content from the bucket, or uvigot again when the reply is already large, as fossil does
uvfile ... a verified login holding y, the write guard (WriteGuard), the size bounds, a bound bucket accepted by fossil's rule (hash and size match, mtime newer): object written, catalog row updated, uv.changed; refused with fossil's error card and the name otherwise, and the DAG part of the request is not applied either (fossil rejects the whole message)
none of them unchanged unchanged

A tracking mirror (accept_writes off, internal/oltp/mirror.go) is uv-pull-only for everyone, as it takes no pushes: the forge's copy of someone else's repository takes no local writes, unversioned ones included. A mirror with accept_writes on takes them, and the next import applies fossil's rule: the newer mtime wins, whichever side wrote it.

Private repositories: MayMirror (internal/oltp/repo.go) already answers no for a private repository, and the read gate already admits a verified login that may read it. Unversioned content rides both.

Letters and roles

RFC 0004 §4 forbids a new letter and none is needed: y is in internal/oltp/capref.go already, listed as "Write-UV: Push unversioned content" and inert. It stops being inert. The op table gains:

op letter fossil surface
uv.list, uv.get o /uvlist, /uv/NAME, the catalog
uv.put, uv.rm y uvfile
the bucket binding (RFC 0016) a none; fossil keeps the content in the repository file

RFC 0008's roles do not change. Admin stays "everything except setup, private branches, unversioned writes", which is fossil's rule and the right one: an administrator of settings is not thereby a publisher of binaries. An account that publishes is Custom, read by 0008 as "Maintainer, plus push unversioned content", and that sentence is the whole interface. s holds y everywhere, as it holds everything.

The forge's own release account (Releases) is an ordinary account with y on the quarry repository and nothing else.

The web

Listing, /r/{repo}/uv, visible to whoever may read the repository: name, size, hash (prefix, full on hover), mtime, who (from the catalog), with tombstones in a collapsed "removed" section so a reader can see what used to be there and when it went. Sortable by name and by age, as fossil's /uvlist is. With no bucket bound, the page says so and, for a, links the bucket settings.

Download, /r/{repo}/uv/{name}, with name allowed to contain /: a redirect to a presigned GET on the bucket (or its public_base), as attachments are served (PresignGet in internal/s3/s3.go), so the forge never streams the bytes. The type comes from the extension by a fixed table and rides the presigned URL's response-content-type. HTML, SVG, JavaScript and anything else that would run in a browser is served with response-content-disposition: attachment, the rule RFC 0016 applies to attachments. This differs from fossil, which renders /uv/download.html as a page. Published HTML is one y grant from script in a reader's browser; the releases page below is the replacement for a hand-written download page.

Fossil's paths: /uv/<name> and /uvlist join FossilRoute (docs/fossil-paths.md), so a link in a mirrored repository's wiki resolves. Its line moves from "Future work" to "Mapped".

Upload and remove, for holders of y: the listing page's form, name defaulting to the file's, optional prefix, replaced on the same name; a remove button per row. The upload takes RFC 0016's path: the form through the forge up to the binding's limit, or straight to the bucket by a presigned PUT when the bucket allows the forge's origin (the binding's CORS probe). Both end in uv.put and uv.rm, which write or tombstone the object, update the catalog with the mtime and who, and emit uv.changed. Uploads are not held or screened: y is the trust, and fossil's own posture ("uncommonly dangerous ... no history to revert to") is why the letter is granted by hand.

Releases, /r/{repo}/releases: every name under releases/<name>/ grouped by that folder, newest first (by semantic order when the folder names are versions, else by date), with the newest marked. The folder name is the release's name, whatever the repository's tags look like (Releases). When the group holds release-manifest.json, the page reads it: each artifact's target, size and sha256, the source check-in (linked), the date, and the qualification notes if present, in the schema zit's releases already use (schema_version: 1, keys artifacts, omitted_targets, qualification, release, source). Without a manifest, the group is a list of files with SHA256SUMS beside them, and the page says checksums come from the file. A CHANGELOG.md section whose heading names the release is shown beside it, the way /changelog renders the file today (changelogPage, internal/web/web.go). The repository tabs show Releases only when a releases/ prefix exists.

Mirrors

A mirror pulls with zit pull URL (Client.Pull in internal/zitcli/zitcli.go, called from PullMirror in internal/api/register.go) on its own interval, and the mirror table in internal/oltp/mirror.go holds repo, url, every, enabled and its history. Two additions:

uv, a switch per mirror, off by default, on the fleet page's mirror form (/fleet?tab=mirrors) where the address and interval are set, and shown read-only in the repository's own Mirror section with its last result. It needs the mirror's repository to have a bucket bound; without one the switch is shown disabled with the reason. When on, each pull is followed by an import round that quarry runs itself, as the client half of the same cards its sync front answers: pragma uv-hash (with the login card when a credential is set), the uvigot catalog back, uvgimme for every name the upstream holds newer than the catalog, and each uvfile written to the bucket as an upload would be. The bytes go from the upstream to the bucket and nowhere else. Latest mtime wins; a tombstone upstream becomes a tombstone here; a local file the upstream never had stays (relevant only with accept_writes). The round is recorded in mirror_pull beside the DAG pull with its own uv_changed and uv_err, so "the code is current but the downloads are three days old" is visible.

A credential per mirror, user and password for the upstream, sealed the way a bucket secret is (internal/oltp/bucket.go, secret_enc), never returned by an op. Today a mirror pulls as whoever the URL names, usually nobody. Reading the unversioned catalog needs o on the upstream (xfer.c: the uv-hash reply is behind g.perm.Read); fossil's stock nobody holds o, so most public fossils answer anonymously, and the credential is for the ones that do not and for upstreams that gate the DAG too. y is never needed on the upstream: a mirror only reads.

When the upstream does not share. A uv-hash that draws no pragma is recorded as the round's uv_err; the Mirror section shows "upstream does not share its unversioned files with this account", and the DAG pull's result stands on its own. A mirror never fails because of its downloads. The same shape covers a file over the binding's per-file limit: the catalog lists it, quarry skips it, the row says "not carried (larger than this bucket's limit)".

Detaching a mirror keeps its unversioned content, as it keeps its history. Divergence (diverged_at) is about check-ins and stays so; a local unversioned write on a tracking mirror is refused by the gate above, so it cannot happen silently.

Releases

Releases and deploys are different things. A release is a version of a repository's code that its owner names, builds and publishes for whoever wants it. A deploy is one operator putting some build on one machine. Tagging a release can start the release's build and publish; nothing here makes a tag, a release or an accepted landing deploy anything. quarry's own deploys stay a person's act.

Tags have no imposed syntax. Fossil projects already disagree: fossil, sqlite, zit and pikchr tag version-X.Y.Z and move a release tag to the newest; tcl and tk write core-9-1-0, tcllib tcllib-2-0, lumosql v0.83, libfossil a bare 0.6.1 (counted on zeitforge's mirrors, 2026-10-01). A repository's release workflow says which tags are releases with a glob; quarry recognises nothing else about a tag. The Releases page groups by the folder a workflow publishes into, releases/<name>/, so it works whatever the tags look like.

The tag event, amending RFC 0019's on:: a tag added to a check-in starts the workflows whose on.tag.tags glob matches it (Actions' pattern language, globMatch in internal/ci/on.go; default version-* when tags: is absent). A propagating tag starts them once, on the check-in that carries it, not on every descendant. An environment's branch policy admits the tag's check-in only when it is on a protected branch, so a tag on a branch nobody reviewed is not a release with secrets.

quarry's own releases. quarry's version today is 0.1.0-beta+<check-in> stamped by scripts/build-linux.sh, nothing is tagged, and scripts/deploy.sh builds trunk on the box. The runner downloads for five platforms (internal/runnerdist/runnerdist.go, Platforms) are built by the same script, verified as a set by scripts/runner-check.sh, and served at /runner/<file> from the directory beside the binary.

file holds
quarry-0.2.0-linux-amd64.tar.gz, -arm64 quarry, the zit it was built against (the pinned release, so the two cannot drift, RFC 0001 §2), CHANGELOG.md, LICENSE
quarry-runner-<os>-<arch>[.exe] the five runner builds, same names runnerdist serves
quarry-0.2.0-src.tar.gz the tagged tree
SHA256SUMS one line per file above
release-manifest.json zit's schema: per artifact target, sha256, size; release.version, source.checkin, release.date; omitted_targets named (macOS and Windows forge builds are omitted, the DuckDB bindings do not cross-link, as build-linux.sh records)

The release workflow, ci/release.yml:

on:
  tag:
    tags: [version-*]
jobs:
  release:
    environment: release
    steps:
      - run: scripts/release.sh
        env:
          RELEASE_SYNC_PASSWORD: ${{ secrets.RELEASE_SYNC_PASSWORD }}

The password reaches the script as an environment variable, the only place RFC 0019 lets a secret go, never as an argument, which every user on the runner's machine can read. scripts/release.sh checks VERSION against the tag and CHANGELOG.md for the heading, runs build-linux.sh for the two forge targets and the five runner targets, packages, writes SHA256SUMS and the manifest, verifies the sums from the files on disk, then zit uv add --as releases/0.2.0/<file> into a scratch clone and zit sync -u to the forge as the release account (y on the quarry repository and nothing else; the release environment holds its password, protected branches only). The forge writes each file to the bucket and the group shows on /r/quarry/releases. Retention is the job's too: it removes releases/<old>/ beyond the newest three minor versions and the newest patch of each, with zit uv rm, and the newest manifest lists what was retired. The job runs on a runner started with --job-users, as every job holding a secret must.

Deploying a release is a person running deploy.sh --release X.Y.Z: fetch releases/X.Y.Z/* through the forge's own /r/quarry/uv/, verify SHA256SUMS, install quarry, zit and the runner set exactly as the build path installs them today (keep quarry.prev, restart, health-check, roll back on silence, restart the runner last and delayed), no build. The trunk path (deploy.sh with no flag, and --require-green --expect <hash>) stays as it is, for deploying what has not been released. Rollback is deploy.sh --release <previous>, which the releases page names beside each version, or deploy.sh --rollback for quarry.prev when the forge is down. OLTP migrations are additive columns, so an older binary runs against a newer database; the index is disposable and re-folds. A release that needs a non-additive migration says so in its changelog entry, and its manifest carries a "cannot roll back past" mark the page shows.

/runner/ keeps serving the installed set from beside the binary: a forge that is not zeitforge has no quarry repository to fetch from, and the setup page's links keep working for every operator.

Build order

Rung Work Waits on
1 This RFC the owner
2 quarry catalog: the unversioned OLTP table, objects under unversioned/ in the bound bucket, tombstones, uv.list/get/put/rm, rebuild from a bucket listing, uv.changed nothing
3 quarry sync front: scanRequest frames the cards; the front answers uv-hash, uvgimme, uvfile from the catalog and the bucket and forwards the rest; the stock-fossil round trip in quarry's suite rung 2
4 quarry web: the listing, /uv/{name}, /uvlist and /uv/ in FossilRoute, upload and remove, the no-bucket state rung 2
5 Releases page with the manifest schema and the changelog section rung 4
6 Mirrors: the uv switch, the credential, quarry's import round, the status lines rung 3
7 zit-A: the local store and uv add|list|cat|export|rm|hash; import-fossil and export carry the table a zit issue
8 zit-B: sync -u, clone -u, uv sync [--pull-only] against a fossil-speaking server a zit issue; rung 7
9 The tag event in RFC 0019's trigger; VERSION, the stamp, scripts/release.sh, ci/release.yml, the release account, environment and job-users runner rungs 5, 8
10 deploy.sh --release and --rollback, docs/running.md rung 9
11 The first tagged release, version-0.2.0; zit's release skill moves its publish step to quarry and the stock fossil server is retired rungs 6, 10

Rungs 2 to 6 are quarry's alone and wait on nothing upstream: the sync front is tested with a stock fossil binary as the client, so quarry's half does not need zit's. zit's rungs are filed as issues on zit and never worked around in quarry; only the release flow (rung 9) needs them, because the release job publishes with zit.

What this is not

Decisions

Settled by the owner on 2026-10-01, before this revision:

  1. Where the bytes live: the repository admin's S3-compatible bucket, RFC 0016's binding; no forge-held store and no forge quota.
  2. Who answers the protocol: quarry's sync front, from the bucket; zit carries only the client half.
  3. Without a bucket: no unversioned files until one is bound.
  4. Deploys: not on accepting a landing and not on a release; releases and deploys are separate, and a tag may start a release.
  5. Tags: no imposed syntax; each release workflow names its tags with a glob. quarry's own are version-X.Y.Z with a moving release tag.

Testing