Running quarry

Two ways: the compose stack, or by hand. The hand-rolled path is what this was developed against and is what the compose file encodes, so it is written down first — a run book that stays true even if the images are never built.

What the stack is

piece	role	state
zit	the store. quarry shells out to it for every read and write (0001 §2)	the .zit files themselves
quarry	the forge: indexes, UI, API, sync fronting	derived indexes, embedded NATS, OLTP database
notify	notifications: workflows, subscribers, topics, the inbox quarry embeds	SQLite
dagnats	the workflow engine notify runs on: embedded NATS + JetStream + an HTTP bridge	JetStream data

zit is not a runtime dependency in the ordinary sense — it is quarry's only store interface, so its version is part of quarry's behaviour. On 2026-08-16 three zit builds changed what quarry does: frames got 31% faster, a card-parsing fix changed which artifacts parse at all (and until it landed, sqlite's 794 tickets were invisible), and a new stat -R let quarry serve a bare store. Pin the pair; do not let a deploy resolve zit from whatever is on PATH.

By hand

dagnats

Runs under launchd on this machine as com.dmestas.dagnats, binding NATS on 4222 and its bridge on 8080, with data in ~/Library/Application Support/dagnats.

launchctl kickstart -k gui/$(id -u)/com.dmestas.dagnats

A freshly-built Go binary replacing it must be ad-hoc signed or launchd kills it with OS_REASON_CODESIGNING and the service silently fails to come back:

go build -o ~/go/bin/dagnats ./cmd/dagnats
codesign -s - -f ~/go/bin/dagnats
launchctl kickstart -k gui/$(id -u)/com.dmestas.dagnats

In standalone mode dagnats binds loopback, which is why anything talking to it shares its network namespace under compose rather than reaching it by service name.

notify

notify bootstrap -dev -db notify.db -org quarry -env dev   # prints apiKey once
notify serve -dev -db notify.db -addr 127.0.0.1:3000 \
  -nats-url nats://127.0.0.1:4222 -bridge-url http://127.0.0.1:8080
curl -s localhost:3000/readyz     # {"status":"ok","bridge":true,"db":true}

-dev derives a fixed, publicly-known master key. Outside development set NOTIFY_MASTER_KEY_HEX.

Provisioning is quarry's job, not an operator's (RFC 0011 §7). What used to be a page of curl here is now code: quarry creates a notify subscriber when an account is added, adds or removes the account from a repository's repo-<name> activity topic when it is granted, revoked, or subscribes to a thread, and creates its workflows (quarry-mention, quarry-forum-reply, quarry-activity, quarry-landing, quarry-changelog, quarry-error, quarry-slow, quarry-test — defined in internal/notify/workflows.go) at startup, leaving any workflow an operator already edited untouched. quarry-landing is provisioned ahead of need: nothing triggers it yet, that lands with the pull request review surface. All of it is best-effort: a notify outage never fails the account or grant op that raised it.

Scripts are compacted at start, not at build. quarry's own scripts are minified in memory when the process starts (the stylesheet has its comments and indentation stripped the same way), then served under content-hashed, immutable names; the sources in the repository stay readable and there is no build step to run. make wire URL=https://your-forge/ prints what a first visit costs per file. Set QUARRY_ASSETS_RAW=1 to serve the sources as written while editing them.

Email needs a provider, and quarry gives notify its own. A workflow with an email step and no mail provider behind it fails where only somebody reading notify's database would find it: store: not found, per message, while the settings page goes on offering an email column. So if quarry has a mailer — RESEND_API_KEY and QUARRY_MAIL_FROM, the same pair that sends invites — it hands notify the same credential at startup and logs notify: email now sends as …. One credential, in one place, and nothing to POST by hand.

Email is off until a person switches it on. The four family workflows carry an email step (plain text, one link; mention, reply and landing accept replies), but notify reads an absent preference as enabled, so a step alone would mail every account about everything. At every start, and when an account is created, quarry writes an explicit "no email" for each family the person has not answered, and never touches a cell they set themselves. The settings page is where they answer.

An email integration notify already holds is left exactly as it is, so configuring SMTP or SES in notify's own console is still the way to use a different provider from the one quarry sends invites with — a restart will not replace it. With no RESEND_API_KEY quarry asks the companion for nothing rather than creating an integration that cannot send.

Invitations are the exception that proves the split: they are sent by quarry's own mailer and are not a notify workflow, because an invitee has no account, no subscriber and no inbox for any other channel to reach — and a forge that sent them both ways delivered two invitations from one address.

Receiving email needs nothing but notify. A reply to a notification email comes back to reply+<token>@<inbound domain>, and notify can receive it itself (notify#340): set NOTIFY_SMTP_ADDR=0.0.0.0:25 and NOTIFY_INBOUND_MX_HOST=mail.example.org, let the service bind a privileged port, and point the domain's MX at the box. On a systemd host that is one drop-in:

# /etc/systemd/system/notify.service.d/smtp.conf
[Service]
Environment=NOTIFY_SMTP_ADDR=0.0.0.0:25
Environment=NOTIFY_INBOUND_MX_HOST=mail.example.org
AmbientCapabilities=CAP_NET_BIND_SERVICE

and two records — mail.example.org A <the box> (not proxied: a CDN does not carry SMTP) and mail.example.org MX 10 mail.example.org. Register the domain with POST /v1/inbound/domains {"name":…} and POST /v1/inbound/domains/{name}/verify; verification is a live MX lookup, so it fails until the records have propagated and can simply be repeated. This is not a mail server: an address without a valid reply token is refused at RCPT TO with 550 5.1.1, before any message body is read, so there is no inbox to spam and nothing to filter. It speaks plaintext today; a sending MTA that offers STARTTLS falls back cleanly.

A host that already runs an MTA can pipe instead of listening: notify inbound < message.eml reads one raw message on stdin and hands it to the same path, so an /etc/aliases line does the job without a second port.

What a reply can answer. Four notifications carry a binding quarry can read back, and hitting reply on any of them writes the same thing the web form would:

notification	the reply becomes
a forum thread you follow	a reply on that thread
an @mention in a forum post	a reply on that thread
an @mention in a repository's room	a line in that room, answering yours
an @mention in a team's room	a line in that room, answering yours

This is per ACCOUNT and needs no privilege: the gate is the room's own capability (C for a repository's chat, 3 for the forum, membership for a team's room), checked against the person the notification was addressed to. A reply from any other address is refused — the binding names one account and only that account can answer it.

The chat bindings were minted from the day chat had mentions and nothing read them back, so a reply to one was accepted, understood and dropped with "no thread behind transaction". They land now.

Two limits worth stating:

• A chat reply names the line it answers, when it still can. The window is finite and a mail can sit in a drafts folder for a week; a reference that has aged out is refused, and the reply is then posted without it. Losing the thread link beats losing the reply.
• A reply to a forum post held for moderation is refused, as the web form is. Nothing is silently queued.

Workflow definitions follow quarry's code. Provisioning upserts each of its workflows with notify#303's PUT /v1/workflows/{slug}?onConflict=update. notify compares the definition it holds with the one quarry sends: a missing workflow is created, an identical one is not written, and one quarry has changed (a new step, a new template such as an email's htmlBody) is rewritten in place on the next start, logged as notify: workflow <slug> updated in place. The workflow keeps its id, so the per-workflow preferences notify keys by that id survive. Never delete a workflow to pick up a new definition: recreating it loses every subscriber's preferences for it. A hand edit to one of these workflows in notify's console is undone by the next start; change internal/notify/workflows.go instead.

What a companion that does not answer costs. The preferences pair (GET|PUT /v1/subscribers/{id}/preferences) and the per-device credential DELETE have shipped, so both halves normally land. When notify is unreachable quarry says so rather than pretending:

• me.prefs.set writes its forge-local half first (the toast and desktop switches, the per-repository mute) and reports applied.notify: false when the notify half could not be stored. The channel matrix a person set is then not yet in force. A read that notify did not answer marks every family unknown and the matrix is left ALONE on the next save, rather than writing quarry's defaults over settings it could not read.
• me.push.unsubscribe keeps the device row marked pending_remote_delete when notify FAILED the delete, instead of forgetting a browser notify still pushes to, and retries on the next me.push.subscribe. A 404 is not a failure — a credential notify does not hold is the end state unsubscribe asked for — so the row is forgotten there and then. The push_device.pending_remote_delete column is now only ever a transient state between a failed delete and its retry.

Both clear themselves once notify answers — no migration, no reset.

quarry

quarry serve --addr 127.0.0.1:8950 --data ./data \
  --notify http://127.0.0.1:3000 --notify-key "$K" \
  --repo quarry=$HOME/quarry

--data is not optional in practice: it defaults to .quarry beside the first --repo, which writes derived state into whichever repository happens to be listed first. Always pass it.

Gates are open until they are closed. An account is the forge identity (argon2id); a push is signed with a separate token, because the sync protocol's RFC §6.3 secret is derived from whatever the client holds:

quarry account add --user you --admin --data ./data   # closes web/API writes

Pushes are gated by the letters: an account pushes with a push token it mints on its own account page (/me, Push credentials) or with POST /api/v1/me/sync-token, never with its password.

The account gate is global: the first account closes writes on every repository in the fleet, not just one.

--admin makes a site admin: the forge's accounts, invites, /fleet, runners, settings and quotas. It gives no repository. A site admin reads, pushes to or administers a repository only through a role on it (quarry account grant) or a team, like anyone; over the rest they keep only the operator acts in docs/hosting.md, each of which leaves the owner a receipt. Grant the repositories an admin or an automation account works on before relying on it there.

Behind a proxy, name the one header it sets that a client cannot, with --trusted-proxy-header (or QUARRY_TRUSTED_PROXY_HEADER). Only a request that carries that header AND arrives over loopback is believed for the client's address, which the login limiter keys on, and for X-Forwarded-Proto, which sends http readers to https and sets HSTS. From any other peer the header is ignored, so a directly reachable listener cannot be fooled by a client that writes it.

Behind cloudflared this flag must ship in the same deploy as the code that introduced it, set to CF-Connecting-IP. Unset, no forwarded header is believed, and every request arrives from cloudflared on 127.0.0.1. Every sign-in then keys on that one address: five wrong passwords from anyone lock sign-in for everyone, with backoff up to about two hours. The http-to-https redirect and the HSTS header also stop. Cookies stay Secure only because --origin is https.

quarry serve ... --origin https://forge.example --trusted-proxy-header CF-Connecting-IP

A proxy may also cap how large one request can be. Cloudflare refuses a request body over about 100 MB with a bare 413 before quarry sees it, so a fossil uv sync carrying a larger file fails there whatever the repository's bucket allows. Tell quarry the cap with --proxy-request-limit-mb (or QUARRY_PROXY_REQUEST_LIMIT_MB), in megabytes of 10^6 bytes: the Unversioned page and uv.list (syncLimit) then name the largest file a sync can publish. It changes what is shown, not what is accepted; the proxy is still what refuses.

quarry serve ... --proxy-request-limit-mb 100

The attention stream and HTTP/2

Signed-in pages open one Server-Sent Events connection for the attention stream — the inbox badge and the count in the tab title (RFC 0011 §5). In-page banners are off by default and are turned on per account on /me (the notifications section), which is also where desktop notifications are opted into; with both off the stream still moves the badge and the title, and nothing ever covers the page being read. On plain HTTP/1.1 a browser allows six connections per origin, and a repository page already spends one or two of them on the chat and timeline streams. So the default opens the attention stream on repository pages only: a stream on every page would compete with the page it sits on for the same six slots, and the symptom is a page that stalls halfway through loading rather than an error anybody can read.

With HTTP/2 in front there is no such limit — one connection carries every stream — so the whole forge can hold it open:

quarry serve ... --attention-everywhere

Use it when quarry is behind TLS termination that negotiates h2, or behind a proxy speaking h2c to a browser. quarry serves plain HTTP/1.1 itself, so the flag is a statement about what is IN FRONT of it and quarry cannot check it: turned on without h2, it is the six-connection problem on every page.

Nothing else changes. Without the stream a page still renders its own state and the inbox badge is fetched once after load, which is the Principle 3 behaviour every one of these surfaces keeps.

Push

/me's notifications section carries the whole of RFC 0011 §6: the family x channel matrix (mention, reply, landing, activity, by in-app, push and email), the two forge-local switches, the repositories this account muted, and Push on this device. /me/notifications still works as a bookmark — it redirects there.

Push is the one feature in quarry that needs JavaScript, recorded as an explicit exception in RFC 0011 §11. Enabling it registers /sw.js, asks the browser for notification permission from the click that asked for push, subscribes with notify's VAPID key, and posts the resulting PushSubscription to POST /me/push. With JavaScript off the control says so and nothing else on the page changes. On iOS a site must be added to the Home Screen before Safari will grant push at all — the control says that too.

Quarry never holds a VAPID key and never stores an endpoint: the subscription goes to notify as subscriber credentials, and quarry keeps a hash, a user-agent label and a timestamp so the page can list the browsers somebody may want to turn off. That is also why a browser other than the one being used is turned off from that browser.

What to expect until the upstream work lands:

• notify#302 (the webpush provider) — subscriptions are accepted and stored, but nothing is delivered by push yet. The in-app inbox, the badge and the attention stream are unaffected.

notify#305 (the preferences route, the per-device credential DELETE and the trigger identifier on every message row) has shipped. When the companion is unreachable the banner after saving preferences says the companion did not answer and the page reports applied.notify honestly; a device whose delete failed shows removal pending and is retried the next time push is enabled.

Neither needs a migration or a reset on this side.

The activity robot

A repository can have its timeline talk in its chat channel (RFC 0011 §10): settings → Activity robot, in the same event-pattern syntax webhooks take (*, a noun like checkin, or an exact kind like landing.accepted). Empty turns it off. Robot lines never count as unread and never mention anybody.

Your account

/me is where a person manages their own things: their identity (login, site-admin standing, when the account was created), their email, their password, their signed-in sessions, and the notifications matrix above. It absorbed the two pages that used to hold pieces of this alone — /me/notifications and /me/sessions both still work as bookmarks, redirecting to the matching section (#notifications, #sessions). The account menu's avatar popover links straight to it.

A small set of emails are transactional: they are about the account itself — an invite, being given or losing access to a repository, an address confirmation, a password change — and no preference switches them off. They never go through notify's matrix (internal/notify/workflows.go's four families, the matrix on /me): that matrix is for content a person chooses to hear about, and an account changing hands or losing its way to a repository silently is not something a preference should be able to produce. internal/api/transactional.go names the exact list and says why extending it is a decision, not a convenience.

A person sets their own address on /me's email section. Nothing is confirmed until a six-digit code — mailed to the NEW address, good for ten minutes — is typed back in on that same page; until then, quarry keeps mailing whatever address was confirmed before, if any. Five wrong codes burn the pending one outright, same as letting it expire, and either way the only way forward is asking for a fresh one. Losing the old address to a new confirmation sends the old one a notice, so a change nobody made does not go unnoticed just because the old inbox is still being read. An administrator can also propose an address for an existing account from /fleet/people. For an account that has a password it is only pending: the confirmation code goes to the new address, and only the account itself, signed in, can type it back. The confirmed address, if any, is told which admin proposed what and when, and a receipt is written. Otherwise an admin could point an account at their own inbox and send its reset link there. An invitee who has never set a password, and so cannot sign in, still gets the address at once, as the invite form sets it.

A person can also change their own password from /me, given their current one. Doing so ends every OTHER session that account holds — the browser making the change stays signed in — and mails the account's confirmed address to say so. An administrator changes their own password the same way; nobody changes anybody else's.

A forge started with no RESEND_API_KEY or no --mail-from ($QUARRY_MAIL_FROM) — the same two settings account.invite already needs — simply sends none of this. Every op still answers correctly: an invite link is returned either way, a caps change still takes effect. Only the mail itself does not go.

Agents

An account may be machine-operated. Settings is not where this lives — it is on /fleet/people, beside the account it describes: mark one an agent and name the person answerable for it, or take the mark off.

The mark is one fact, operator, and its non-emptiness is what makes the account an agent. Two questions with one field, because neither is worth much alone: "a machine wrote this" without "and X runs it" tells a reader to distrust the words without telling them who to ask.

Declaring is safe; hiding is not. Any account may mark ITSELF an agent, naming itself as the operator. Only a site admin may clear a mark, mark somebody else, or name another person as answerable — that last is a claim about a person, not about yourself.

The badge appears wherever a name is drawn: the timeline, tickets, the forum, chat, landings and attachments all render names through one function, so it reaches them from a single rule rather than a convention each surface has to remember.

The mark is on the ACCOUNT, not the write. A write-level mark is more honest and much larger — it would have to reach the artifact format — and this is what makes that worth building later: you cannot ask which of an account's writes were machine-made until you can ask whether the account is a machine.

A forgotten password

Press Send reset link on the person's row on /fleet/people (or quarry accounts reset --user NAME). It emails a one-time link to the account's own confirmed address, letting them choose a new password themselves. The email names you and the time you asked. You never see the link: it is not on the page, in the API's answer or in any log, and each reset leaves a receipt naming you and the account.

An account with no confirmed address cannot be reset from the forge at all, and its row offers no button. Neither can one on a forge with no mail or no configured origin. There is no op, page or networked command that sets somebody else's password: an admin who could choose one could sign in as anybody, including the owner of a repository they hold no grant on.

Three things about the link are deliberate:

• It is good for one hour, and once. An invite gets seven days because it is sent to somebody who is not waiting for it; a reset is asked for by somebody sitting there. A live link is a credential, so it lasts as long as the person needs and no longer.
• Their current password keeps working until they use it. A reset that blanked the password would lock them out harder the moment the link went astray, and somebody who ignores an email they did not expect loses nothing.
• Sending again supersedes. "Send it again" drops the unspent one, so there are never two live links where you believe there is one.

Redeeming ends every other session of the account, because a forgotten password is sometimes a lost device.

Recovering an account that has no confirmed address is the host operator's act, on the server, outside the forge: quarry account passwd --data DIR --user NAME against the database. It is journalled with the host user who ran it.

An account that has never been claimed has no password to reset: it wants the invite it is already waiting on, and the row offers Re-send invite instead.

Fleet admin

/fleet is the operator page — every store's ingest health, and the upstreams it mirrors. It is site admin only: admin on one repository is not enough, because the page reports operational detail across the whole fleet and can force writes into any of it.

Health is per-process. Total and Fallbacks count what THIS server has ingested since it started, so a restart with a warm index reports zero of both however much the store holds. The parse-fallback number worth looking at is the one a quarry reindex produces.

Mirroring

A mirrored store tracks an upstream: quarry pulls, and never pushes. The four postures a repository can be in, and what each one opens or refuses, are laid out in hosting.md.

Configuring a mirror needs admin on the repository being configured — which its creator holds from the moment they created it — not site admin. That includes mirror.writes: flipping a tracking store to accept local writes creates two write heads on it, but it is that repository's own fork to reconcile, not the forge operator's call.

curl -u you:pw -X POST http://localhost:8080/api/v1/mirrors \
  -H 'Content-Type: application/json' \
  -d '{"repo":"fossil","url":"https://fossil-scm.org/home","every":3600}'

every is seconds and is floored at 60 — these upstreams are other people's servers, and the floor is a courtesy the configuration surface cannot opt out of. --mirror-tick (default 1m) is how often due-ness is evaluated, not how often anything is pulled.

A repository also says, on its OWN side, whether other people may keep a copy that follows it — see hosting.md for what "mirrorable" means and why a private repository is never mirrorable regardless of the setting.

Pulls take the store's single-writer lease, so mirroring is safe with more than one node serving the same store. Only the node holding that lease runs the pull at all; the others skip it and record nothing, so a mirror's row never shows a failure because a node was not the holder. A pull that brings nothing new does not trigger a re-index and is not logged; almost every one is quiet, and a line per store per interval buries the ones that matter.

Two limits worth knowing before you rely on it:

• The store needs a working checkout. zit pull refuses -R and operates on a working directory, so a genuinely bare store cannot be mirrored. Stores created by zit clone or zit import-fossil carry one, so this bites only hand-built bare stores.
• The upstream must allow anonymous clone. Some Fossil servers serve a browsable timeline while refusing sync — sqlite.org/lemon is one. Note that fossil clone exits 0 on that refusal, so check for the repository, not the exit status.

Attachments: binding a bucket

Files uploaded through quarry go to a bucket the repository's steward brings (RFC 0016); the forge keeps a stub in the repository and never the bytes. A steward binds one bucket per repository from its settings page, under Attachments, or through the op:

curl -u you:pw -X POST http://localhost:8080/api/v1/r/fossil/bucket \
  -H 'Content-Type: application/json' \
  -d '{"endpoint":"https://<account>.r2.cloudflarestorage.com","bucket":"fossil-files",
       "region":"auto","accessKey":"...","secret":"...","publicBase":"https://files.example.org"}'

At R2, create the bucket, then an API token scoped to that bucket with object read and write; the region is auto. A public or custom domain on the bucket is optional — set it as the public base and downloads link there; without it every download is presigned. Any S3 endpoint works the same way with its own region.

The binding is probed before it is kept — a presigned PUT and DELETE of one small object under attachments/ — so a wrong key or a token without delete is refused on the form. Probe again from the page or with bucket.probe; Fleet shows each store's last probe beside its mirror's last pull. The secret is sealed under secret.key beside oltp.db (created 0600 on first use) and never returned by any op; a forge that loses the key file has the steward enter the secret again.

The probe also sends the browser's preflight — an OPTIONS naming the forge's origin (--origin / QUARRY_ORIGIN), PUT and content-type — and records the answer as corsOk. The page's scripted upload PUTs the file straight to the bucket from the browser, which the bucket refuses until its CORS rule allows the forge; the multipart form, which posts to quarry, needs no rule. A failed preflight therefore never refuses a binding: the settings section says browser uploads are blocked and shows the rule to paste, in R2's shape with the forge's own origin filled in:

[{"AllowedOrigins":["https://forge.example"],"AllowedMethods":["PUT","HEAD","GET"],
  "AllowedHeaders":["content-type","content-length"],"MaxAgeSeconds":3600}]

At R2 it goes under the bucket's Settings, CORS policy; any S3 takes the same rule as a PutBucketCors configuration. Probe again once it is in place. A forge started without --origin cannot ask on the browser's behalf and records corsOk false.

Attachments: uploads, letters and the lifecycle

With a bucket bound, a holder of the upload letter — b on tickets, wiki pages and technotes, B on forum posts (RFC 0008) — sees "Attach a file" in the Attachments section of those pages. With script the browser hashes the file, asks attachment.begin for a presigned PUT (fifteen minutes, the length and the type signed), sends the bytes to the bucket itself and calls attachment.commit, which HEADs the object, checks its length against what begin was told, and writes the stub and the A card through zit attach add. Without script the same form posts the file to quarry, which carries it to the bucket in memory — never to disk — and commits; that path is bounded at 32 MiB, since a presigned PUT signs a length the form cannot know until the file is read. API clients have the same floor at POST /api/v1/r/<repo>/attachments/<target> with a multipart file. Uploads are refused over the binding's limit and off the type policy: text, the common images, PDF, JSON, the common audio and video; archives only when "allow archives" is on in the section; executables never. Needs zit 0.3.13, which adds zit attach.

With script, a file can also be pasted or dropped, and nothing leaves the browser until something is posted. A paste or drop into a composer — a new thread, a reply, a new ticket, a ticket's comment or description box, the wiki editor — previews the file under the box (a thumbnail for a picture, the name and size otherwise; a clipboard screenshot arrives as image.png and is renamed pasted-YYYYMMDD-HHMMSS.png, in UTC), puts the reference the markup resolves at the caret — ![name](attach:name) for an image, [name](attach:name) otherwise — and offers a Remove that only forgets it. When the form is submitted the script uploads each pending file first: attachment.begin — with the target when the object exists, with the family alone when it does not — the PUT, the progress bar; then the form goes with the tickets as hidden attach inputs, and the op that writes the object (forum.post, forum.reply, ticket.new, ticket.set, wiki.save) commits each against it. A refusal on any file keeps the form from going and names the file on the error line; a refusal at commit comes back in attachErrors and never undoes the write. A paste or drop on an Attachments section — the one around the focused element, else the first on the page that offers an upload — fills its file picker, and its Upload button is the one control that sends at once. A post held for moderation leaves its files unattached. A new ticket takes files only when it is recorded in the fossil now: the local tier has no A card to write, so the control is offered to administrators alone.

An upload is held unless the uploader holds the family's moderation letter — q on tickets, l on the wiki and notes, 5 on the forum — or administers the repository. A held file is in the repository (a fossil client sees it as any attachment) but on quarry's pages it is a link with its name and size, never a picture, until a moderator vouches for it from the card or from the "Held uploads" list in the Attachments settings section; reject removes it. Removal — the card's Remove, or attachment.remove — writes fossil's sourceless A card and leaves the object in the bucket unless "delete the object when an attachment is removed" is on. Downloads of an upload redirect to the bucket: the public base when the repository is public and one is set, otherwise a presigned GET good for an hour.

Objects are keyed attachments/<sha256>, and an object no stub names is an orphan: an upload that began and never committed, or one whose attachment was removed with deletion off. quarry writes nothing else under the prefix. A lifecycle rule that deletes attachments/ objects a day old would take live files with it, so do not set one; sweep orphans instead by listing the prefix against attachment.list (or zit attach list, which marks stubs external:sha256:…) and deleting what neither names. Objects a day old that no stub names are safe to delete: an upload is complete or abandoned within the hour.

Unversioned files

A repository's unversioned files (RFC 0020) live in the same bucket under unversioned/<repo>/<name>, with removals as empty objects under unversioned/<repo>/.tombstones/<name>; each object's metadata carries the fossil mtime, hash and author. The repository's name is in the key because several repositories may bind one bucket: attachments are content-addressed and share safely, but two repositories' files of the same name would otherwise be one object, and each catalog rebuild would import the other's. A lifecycle rule must leave both prefixes alone. The forge's catalog is a copy, rebuilt from the bucket when it is missing or another bucket is bound, and on demand with uv.rebuild. Publishing takes y, which admin does not imply. A private repository whose binding has a public base cannot publish them: their keys are names anyone can guess, so remove the public base first. That refusal guards writes only. A repository made private, or given a public base, after it published files leaves those files reachable on the public base by anyone who guesses a name, until they are removed or the base is cleared.

Fossil clients sync unversioned files through the same URL as check-ins (fossil sync -u, clone -u, uv sync); the sync front answers those cards from the bucket and hands zit only the rest. Every refusal (no y, a tracking mirror, a file over the binding's limit, a hash that does not match) comes before zit sees the request, so the check-ins it carried are refused with it. A stock fossil client whose account lacks y is answered uv-pull-only, so it never sends its unversioned files and its check-ins land as usual; a hand-built request carrying a uvfile from such an account is refused whole, as a stock fossil server refuses it. A bucket or catalog failure after those checks is different: the check-ins are already applied, the client is told the unversioned part failed, and its next sync -u retries that part. A bucket transfer is allowed 30 seconds plus 2 per MiB, so a binding with a high limit on a slow link is the case to watch.

The catalog's write lock is held in the forge process, which is right for the way quarry runs: one process per OLTP database. Two forge processes sharing one oltp.db could interleave a write with a rebuild and leave the catalog missing a file until the next uv.rebuild.

Archive downloads

The tar.gz and zip links on a repository's pages (/r/<repo>/archive/<version>, ?format=zip; the API's archive.get) download any check-in's files. Building one costs seconds of compression, so with a bucket the forge builds each one once and keeps it there:

• Where. The repository's own bucket (its parent's for a companion store), else the forge's. Unlike uploads, archives do go to the forge's bucket: they are the forge's own output. That includes a private repository's archives, which is safe because the forge's bucket is private storage and a reader reaches an object only by a five-minute presigned link handed out after the read check. A binding whose last probe failed is not used. The key is archive/<repo>/<producer>/<check-in>.tar.gz (or .zip), where the producer is the zit version as zit version reports it (0.3.30): zit's container bytes have changed between releases, so a new zit builds new objects. A forge that could not learn its zit version does not use the bucket at all, since it cannot name what it built.
• Old producers. The first build into a bucket for a repository in each process starts a background sweep of that repository's prefix: every archive built by another zit version (or written with none) is deleted, at most 1,000 a run, with one log line saying how many. It never holds up the request; what a run leaves, the next process's sweep takes.
• First request. The forge builds into a temporary file, uploads it, and answers 302 to a presigned GET with the archive's file name as an attachment. Readers asking for the same archive while it builds wait for that one build, and the reader who started it leaving does not stop the upload the others are waiting on.
• Later requests. 302 straight away. The forge remembers which objects exist and asks again (one HEAD) only once that is an hour old, rebuilding if the object has gone. The link expires in five minutes and the redirect is no-store, so a link copied from a private repository stops working.
• A bucket that fails. A build the bucket refuses is not wasted: it is put in the disk cache and served by the forge, and the bucket is not asked again for a minute, so a broken binding costs one build per check-in rather than one per request.
• Reading first. A private repository refuses a reader who may not see it before anything is built or asked of the bucket, and only a check-in the repository's index holds is answered from the bucket.
• No bucket. Built per request, cached on disk under --archive-cache, and served by the forge, as before.

Builds run on three lanes. Signed-in readers have their own. Anonymous requests for an open branch's tip, a tagged check-in (by hash or by name: trunk, tag:v1.0) or an open landing's premerge pair use the normal lane. Every other anonymous request (an untagged check-in by hash, a closed branch's tip, any other pair: the requests a crawler makes and nobody reuses) is cold. A full lane answers 503 with Retry-After; an archive already in the bucket is a redirect and takes no lane. All three show on the ops page.

flag	default
--archive-reader-builds	2	concurrent builds for signed-in readers
--archive-reader-queue	8	readers' builds allowed to wait
--archive-builds	1	concurrent normal builds
--archive-queue	4	normal builds allowed to wait
--archive-cold-builds	1	concurrent cold builds
--archive-cold-queue	0	cold builds allowed to wait

Every build logs one line: archive: built repo= checkin= format= size= duration= queue= into= (bucket, disk or temp). repo.delete removes archive/<repo>/ in the background from the repository's own bucket and the forge's, whatever their last probe said, so a repository created later under the same name starts with none. A build still running when the repository is deleted is not uploaded.

Fossil chat bridge

An opt-in connector between one repository's chat channel and one Fossil server's chat (RFC 0011 §9). It is configured per store, from /fleet or through the op:

curl -u you:pw -X POST http://localhost:8080/api/v1/r/fossil/mirror/chatbridge \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://fossil-scm.org/home","user":"quarrybot","password":"..."}'

Detach with {"detach":true}. The password is stored and is never returned by any op or rendered on any page; the only reader is the poller.

The upstream needs two things, neither of them a default:

1. A login holding fossil's C capability. Chat is gated on it.
2. http_authentication_ok in the repository's config, or fossil refuses HTTP Basic and every request is anonymous. It is not a fossil setting, so it goes in directly:

   fossil sql -R /path/to/repo.fossil \
     "INSERT OR REPLACE INTO config(name,value,mtime) \
      VALUES('http_authentication_ok','1',now());"

Quarry polls /chat-poll with a nine-minute client timeout (fossil hangs the request for chat-poll-timeout, 420 s by default) and backs off from 30 s to 5 minutes while an upstream is failing. Every request carries a same-origin Referer: without one fossil treats the poll as CSRF-unsafe and refuses its own table setup, which surfaces as a database error rather than as an authentication failure.

What it does. Messages from the upstream arrive in the channel as fossil:<user> with Origin: fossil. A fossil deletion becomes a tombstone on the message it removed. An attachment becomes a link to the upstream's /chat-download — fossil keeps the file. Quarry's own messages go out to /chat-send prefixed **name:**, because fossil attributes everything to the login the bridge signs in as. The bridge never forwards robot lines or lines that came from fossil, and never re-imports its own login's messages.

What it does not do.

• No edits. A fossil message cannot be edited from quarry, and a quarry edit is not carried upstream. Only deletions cross, and only in the fossil-to-quarry direction.
• No mentions. Fossil chat has none, so a bridged line raises nobody's badge and triggers no notification.
• No account mapping. fossil:drh is a name from another server's user table, not a quarry account, and the prefix says so.
• Two Fossil servers are two chats. Fossil's chat is a server-local table that never syncs, so bridging a mirror connects you to that ONE server's conversation, not to the project's.
• The window is still quarry's. A bridged channel holds seven days like every other; the bridge is a connector, not an import.

When this repository's activity robot is on, the upstream's chat-timeline-user announcements are skipped — otherwise every check-in is announced twice, once by each robot.

The bridge runs on one node. Like a mirror pull, it runs only on the node holding the repository's single-writer lease; the others do not start a poller, and a node that loses the lease stops its bridge at the next mirror tick. Two nodes bridging the same repository would import each upstream message twice and send each quarry message upstream twice, which is why the lease decides. Restarting a node is safe: the msgid map makes a re-read of the same message a no-op, so a cursor that has not caught up does not double anything.

Private addresses are refused by default. The bridge is a server-side fetcher holding a credential, so it will not connect to a loopback, private or link-local address — that last range is where cloud metadata services live. The check runs on the resolved address at connect time, so a public name pointing inward is refused too. Bridging a fossil on the same host is an opt-in:

quarry serve --allow-private-bridges ...

Redirects are refused, not followed. Go re-sends the Authorization header across a same-host redirect, and here that header is the fossil password. Configure the final URL. A URL carrying credentials (https://user:pass@host/) is rejected outright: the user and password are separate fields, and a URL is copied into log lines and status rows.

CI: workflows, runners and secrets

RFC 0019 is the design; this is how to use it.

A workflow is a YAML file in ci/, in GitHub Actions' shape. The repository's CI page offers two starters in the pipeline editor, a test workflow and a deploy after a landing:

on: [push, landing]          # push: a check-in on a branch; landing: a landing's merge
jobs:
  test:
    timeout-minutes: 30
    steps:
      - run: make test

on: takes push and landing (with branches or branches-ignore), tag (with tags or tags-ignore; version-* when neither is given), workflow_dispatch (a Re-run) and schedule. A tag workflow starts once when a tag is added to a check-in (zit tag add version-0.2.0 CHECKIN), on that check-in; a branch's own name is not a tag, and a propagating tag starts it on the check-in that carries it, never on the descendants, and a tag pushed long after it was made still starts it. Its run receives secrets only when the tagged check-in is on a branch the environment admits. A Re-run of the tagged check-in re-runs its push workflows, not its tag workflows; to repeat a release, give the release workflow workflow_dispatch too and Re-run, or cancel the tag and add it again. A tag run's steps read the tag from QUARRY_TAG (unset in any other run), so a check-in carrying two version tags runs each release with its own name; the next push to the branch does not cancel a tag run. The tags a repository already holds when CI first sees it with a check-in start nothing.

A schedule workflow names cron lines, GitHub's five fields in UTC:

on:
  schedule:
    - cron: "17 3 * * *"   # every day at 03:17 UTC

The forge starts it, not the engine: once a minute it reads the default branch tip's workflows and starts each one due, on that tip, so a schedule always runs the default branch's file whatever a branch says. Each scheduled minute starts at most once, across restarts and fleet nodes: the minute is claimed before the run is launched, so a launch that fails loses that minute, and the CI page shows it on the schedule's row. A minute missed while the forge was down is not run later, as GitHub does not. A schedule whose previous run is still going (within its jobs' timeout, six hours at most) is skipped for that minute, and the forge log says so. A workflow may have at most ten cron lines, and together they may not fire more often than every 5 minutes: the times of day all its lines name (hour and minute) are judged as one set, wrapping at midnight, whatever the day fields say, so two lines for different weekdays at 03:00 and 03:02 are refused too; a workflow that breaks it is refused where workflow errors show and listed on the CI page as not firing. Repositories whose CI is declined, or that no runner serves, are not scheduled. A scheduled run is on the default branch, so it receives the secrets of its environment and the repository's own secrets as any default-branch run does (as on GitHub); it has no starter, so it receives a run token only from an environment that requires no starter letter. The CI page lists each scheduled workflow with its cron and next fire, and marks scheduled runs. needs: orders jobs; a job's steps run in order and stop at the first failure. There is no if: and no expression but ${{ secrets.NAME }}, and an Actions key quarry does not implement (uses:, strategy:, container: and the rest) is refused by name. Engine JSON in ci/*.json still works.

A runner runs a repository's jobs on a machine you choose. Mint a worker token in the repository's Settings (Runners), then:

QUARRY_RUNNER_TOKEN=... quarry runner https://forge.example/r/REPO

With no --tasks it polls the repository's workflow jobs (<group>.job) and whatever ci/runner.json on trunk names.

Secrets live in the repository's Settings (Environments). A job names an environment and takes a secret as an env value:

  deploy:
    needs: [test]
    environment: production
    steps:
      - run: ./deploy.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

An environment gives its secrets only to check-ins on protected branches (RFC 0007) unless it names other branch patterns, and never to a landing's merge; protect the branch you deploy from. The values are sealed with QUARRY_CI_SECRETS_KEY (32 random bytes, base64) from the forge's environment; without it the forge stores and delivers nothing. A backup restored without the key keeps names and policies, and every value must be set again.

A runner that receives secrets runs every job as one of a pool of users, so no job can read another's environment or the runner's token:

useradd --system --user-group --no-create-home --shell /usr/sbin/nologin qjob1   # and qjob2...
useradd --system --user-group --create-home --home-dir /srv/ci-REPO --shell /usr/sbin/nologin ci-REPO
usermod -aG qjob1,qjob2 ci-REPO
chmod 0711 /srv/ci-REPO
echo 'ci-REPO ALL=(qjob1,qjob2) NOPASSWD: ALL' > /etc/sudoers.d/quarry-job-users   # then visudo -c
quarry runner --job-users qjob1,qjob2 --work /srv/ci-REPO/runner https://forge.example/r/REPO   # as ci-REPO

Run it as a user of its own (ci-REPO above), not one another runner shares: a second runner's jobs under the same user could read its token. A job's TMPDIR is inside its checkout; tests that write to /tmp by name collide between pool users.

A task on a branch knows what the branch changed. For a branch check-in or a landing's merge, QUARRY_CHANGED names a file listing the paths the branch changed since its fork point, one per line, so a slow step can test only those (quarry's own race step does, scripts/race.sh). A trunk check-in, or a branch the forge could not answer for, gets QUARRY_CHANGED set empty, and the step should run whole.

After every job, and once per job user at start, the runner ends whatever that user left running (job-reclaim, run as the job user through the same sudo rule; it needs no pkill or other tools on the image). A runner that cannot do it refuses to start, and a job user whose processes cannot be ended stays out of the pool until the runner restarts, so no job runs beside another's leftovers.

Roles, and what a role change tells

A person's standing on a repository is a capability string (RFC 0004), and there are two doors onto it:

• account.grant / account.revoke take one of the four preset roles — reader, developer, maintainer, admin.
• account.caps takes the whole string, which is what the settings editor writes and what no preset covers. Empty removes the account's explicit string; the repository's categories still apply.

All three raise RFC 0005's account.added, account.caps and account.removed — through one helper, because three copies would drift and the drift would be a role change that reaches a webhook when it is made one way and vanishes when it is made another. A change that changed nothing raises nothing: these are operational events, so they must name real transitions rather than every save of the same role.

The event is raised at the write, and only there. Telling the person their access changed (their inbox line and the mail that goes with it) is a separate job and does not raise one: when both did, a webhook heard a grant twice, and heard account.caps for an arrival that is account.added.

The payload carries a summary — the same sentence the settings page shows ("added as maintainer", "reader → maintainer", "removed from the repository") — so a webhook body, the chat robot's line and the settings trail all say what happened instead of naming the kind.

What the forge retires on its own

RFC 0001 §8 keeps the OLTP database small by construction: it holds the forge's own bookkeeping, not the repository's history, and it is litestream-replicated. Two tables broke that by only ever growing, and both are now swept on the same six-hourly beat (and once at startup):

table	kept	what goes
exception	30 days since the last occurrence	a signature nobody has seen again, and its recent hits
ci_run	90 days	a finished run that is not the newest for its subject

Neither sweep needs configuring and neither loses anything a page still reads. Two rules bound the CI sweep:

• An unfinished run is never pruned, whatever its age. That row is the poller's work list, and deleting it strands a run whose verdict nobody will ever record. An ancient unfinished row is a bug to find, not a row to hide — look for one before assuming the sweep is broken.
• The newest finished run per subject survives the window. quarry stores only the mapping from run to check-in, and that mapping is what makes a verdict explicable: a ci-fail badge that cannot name the run behind it is a verdict with no evidence. So the badge stays traceable for as long as the check-in does, and what the window removes is the retries — the several runs a check-in accumulates from re-runs and supersessions, which is where the growth was. Subject is (repo, check-in, pair), so a premerge run (RFC 0009) keeps its own newest alongside the check-in's.

Each sweep logs only when it removed something, so a quiet forge stays quiet in the journal.

Building on Linux

scripts/build-linux.sh is the build for a Linux host, and the one the VPS deploy uses. It installs the pinned zit release for the host's architecture (checksum verified against the release's SHA256SUMS), then builds quarry from the tree with the version stamped from the checkout, and leaves both binaries in ./bin:

ZIT_VERSION=0.3.7 scripts/build-linux.sh
bin/quarry version   # quarry 0.2.0-dev+<checkin>, or quarry 0.2.0 on the check-in tagged version-0.2.0
bin/zit version

It builds ON Linux rather than cross-compiling for a reason worth knowing: the DuckDB bindings ship a static C++ library per platform that links against the GNU C++ runtime, and a cross-link from macOS with zig as the C compiler fails on missing std::string symbols. Proving the recipe without a Linux box takes one container:

docker run --rm -v "$PWD":/src -w /src "golang:$(sed -n 's/^go //p' go.mod)" scripts/build-linux.sh

zit comes from its GitHub release rather than a source build so the host needs no Zig toolchain, and the pin keeps quarry and zit from drifting (0001 §2: the zit CLI is the whole store interface).

With compose

export NOTIFY_SRC=../notify           # a notify checkout; it builds dagnats too
export ZIT_REF=<commit>               # pin the store implementation
docker compose up --build

Then bootstrap notify once (docker compose run --rm notify bootstrap …), put the key in QUARRY_NOTIFY_KEY, and create the workflows and topics as above.

docker run <image> cat /etc/quarry-build reports which quarry and which zit an image carries, without starting it.

Unverified. Dockerfile and compose.yml have not been built — no container runtime is installed on the development machine. The two build recipes inside them are verified natively: zit compiles from a clean clone of the GitHub mirror with zig build --release=fast, and quarry builds fully static with CGO_ENABLED=0 (its SQLite is WASM-based and Pikchr runs on wazero, so nothing needs a C toolchain). What remains untested is the image assembly and the compose wiring.

Ports

port	what
4222	dagnats NATS
8080	dagnats HTTP bridge (and its console — unauthenticated, keep it on loopback)
3000	notify API
8950	quarry

quarry's own embedded NATS binds nothing (DontListen), so it never collides.

Working with several agents on one store

Several people — or several agents — committing to one .zit is the ordinary shape here, and 2026-08-29 taught its failure modes in one afternoon. The rules that keep it sane:

rule	why
Land through the forge — tools/quarry-land, the landing page, or the API; never zit landing accept	required checks, override-with-reason, auto-land and supersede exist only in the forge's write path. A store-level accept walks past all of them, and the store cannot refuse it — twice in one day, red-checked and unformatted work reached trunk that way
Arm auto-land at open (quarry-land <repo> <id> --when-green)	nothing merges red out of impatience, and nothing waits on a human who is actually a busy agent
Prefix branches per effort (attention-*, ci-*, resilience-*)	supersede-cancel, filters and the eye can all scope to whose work is whose
Commit or stash before leaving a shared checkout	an uncommitted edit left in a serving tree turned a routine zit update into a merge conflict mid-deploy
Deploy only via the sanctioned script, and never build from a tree another agent may be editing	the deploy script's job is to be the one honest path; a second, casual path is where the conflicts and truncated logs came from. The script now opens trunk into a scratch tree from the STORE and builds there, so no shared checkout is touched and nobody's uncommitted work can collide with a deploy
Announce a deploy window; hold restarts while another session's CI is in flight	every restart kills the clones its own workers are mid-way through. A quiet restart cost five runs across three sessions in one night
Only the session that opened a landing settles it	an armed landing whose branch someone else closes as "abandoned" reverts work the owner is still driving, and a rejection is unattributable — every session commits as the same user
Treat the zit binary as part of quarry's behaviour (see above)	two zit releases in two weeks each silently broke shell-out seams. quarry's CI tests each check-in with the zit it pins (scripts/ci-zit.sh reads ZIT_VERSION from scripts/build-linux.sh and unpacks that release from ZIT_RELEASE_DIR on the runner); the TestZitContract canary names the next drift

The forge holds the gates; the store holds the truth; the agreement is that nobody asks the store to do the gate's job.

What must survive a reboot

Three long-running things back this forge, and on 2026-08-30 a reboot proved which of them were actually supervised. /tmp does not survive, so nothing whose life depends on it does either.

process	supervision	notes
the dagnats engine	launchd, com.dmestas.dagnats	came back on its own
the CI worker	launchd, com.dmestas.quarry-ci-worker	did NOT, before this: it was hand-run with a /tmp work directory, so the wipe took it and CI was dead six hours. A dead worker is invisible from the outside — every run simply reads running forever — which is why supervision matters more here than anywhere else. Its work directory now lives beside the binary, not in /tmp
the forge itself	the deploy script, by hand	the one still unsupervised; a reboot leaves it down until somebody looks

The worker's log is quarry-demo/quarry-ci-worker.log; the forge's is quarry-demo/logs/quarry-<date>.log, appended rather than truncated so a deploy can no longer destroy the evidence of the outage before it.

Deploys that say when they fail

scripts/deploy.sh (installed as /srv/quarry/deploy.sh) is the one way the forge on zeitforge.app is deployed. Run by hand it still does what it always did — open trunk from the store, build, install, restart, roll back if the new binary does not answer, wait for ready — and adds:

flag	what it does
(none)	a deploy, as before
--dry-run	build and check the binary runs; install nothing, restart nothing, pull nothing
--require-green	refuse unless every check trunk requires is green on the check-in about to ship — asked of the forge (checkin.get's required and unmet), never written into the script. A trunk that requires no checks is refused too
--expect HASH	refuse unless trunk's tip is still HASH: the full hash, or a prefix of at least 12 hex characters
--release X.Y.Z	deploy a published release instead of building trunk: fetch every file releases/X.Y.Z/SHA256SUMS lists from the forge's own unversioned files (http://127.0.0.1:8950/r/quarry/uv/releases/X.Y.Z/<file>, following the redirect to the bucket), verify every line before anything is unpacked, then install quarry, its zit and the runner set as a build does. With --dry-run, fetch and verify only. Refused beside --require-green and --expect, which check a trunk build
--rollback	put quarry.prev, zit.prev and runner.prev back and restart, for when the forge cannot serve a release; alone, never with another flag

Two deploys never overlap: the script takes flock on /run/quarry-deploy.lock, and a second one exits 1 saying since when the first has run. Every failure — build, health check and rollback, rollback that did not work, a forge never ready, or the script dying anywhere else — exits non-zero with one line naming the stage, and writes the same line to the journal. The runner downloads (/runner/) are published only as a complete set whose SHA256SUMS verifies (scripts/runner-check.sh); an incomplete one is not installed, the forge still deploys, and the deploy ends FAILED stage=runner. A delayed runner restart that cannot be scheduled ends it FAILED stage=runner-restart. sh scripts/deploy-test.sh runs every one of these paths against temp directories and stubs, off the host:

journalctl -t quarry-deploy          # START / OK / FAILED stage=<stage> tip=… id=… via=unit|manual

The CI runner restarts last, five seconds after the script ends (systemd-run --on-active=5), so a deploy the runner started can report before its runner goes away.

Releases and rollback (RFC 0020). A trunk build is stamped by scripts/version-stamp.sh from the newest ## X.Y.Z heading in CHANGELOG.md: 0.2.0 when trunk's tip carries the tag version-0.2.0 itself (added to it, propagating or not, but not inherited from an ancestor: the same rule that starts a tag workflow). Any other tip is on its way to a release: 0.2.0-dev+<tip> while version-0.2.0 is not yet a tag anywhere, and 0.2.1-dev+<tip>, the next patch, once it is. Nothing is bumped after a release; quarry version, the footer and the bug-report link show the stamp.

The release layout --release reads, under releases/X.Y.Z/ in the quarry repository's unversioned files:

file	holds
SHA256SUMS	one line per other file, <sha256> <name> as sha256sum prints it; names are bare file names
quarry-X.Y.Z-linux-amd64.tar.gz, -arm64	exactly one directory, quarry-X.Y.Z-linux-<arch>/, holding quarry, zit, CHANGELOG.md and LICENSE; any other shape is refused; quarry version must print exactly quarry X.Y.Z
quarry-runner-<os>-<arch>[.exe]	the runner downloads, named as /runner/ serves them; their lines of SHA256SUMS become /runner/SHA256SUMS. Every platform scripts/build-linux.sh at the tag version-X.Y.Z builds must be there (scripts/runner-check.sh, read from the store at that tag and run as quarry, as a build's set is checked); a set short of it is refused whole

Any other file listed (the source tarball, the manifest) is fetched and verified, and not installed. From there --release is the build path: quarry.prev, zit.prev and runner.prev kept, restart, health check, roll all three back on silence, runner last. Every install, build or release, keeps the three that ran together just before it, a runner set it did not replace included, so the .prev files are always one deploy old together. To go back a release, deploy the older one with --release; --rollback reinstates quarry.prev, zit.prev and runner.prev without the forge, once (they are moved back, not copied), and has nothing to fall back to if that binary does not answer either.

Before fetching anything, --release X.Y.Z asks the deploy box's store which check-in carries version-X.Y.Z: exactly one must, labelled trunk and in trunk's history (trunk's tip or an ancestor of it, as zit open trunk resolves trunk), or the release is refused (a tag the store does not have yet means the store needs a pull, or the release is not tagged). The tag is not a gate of its own: anyone who can push can tag a check-in on their own branch. Trunk's review is the gate, and the check ties the release to it. The scripts the deploy then reads from that check-in run as quarry, never as root, on a copy of the runner set: the verified files root installs stay in a root-owned directory they cannot change.

What --release trusts. SHA256SUMS proves the files arrived as they were published, not who published them: it comes from the same place as the files. Whoever can write the quarry repository's unversioned files (the y capability on it) or its bucket can publish a releases/X.Y.Z/ whose binaries the deploy, run as root, installs and runs, as long as a reviewed trunk check-in carries the tag. Grant y on quarry, and the bucket's keys, as you would root on the host.

Cutting a release

During the beta every release is a patch bump: 0.2.0, 0.2.1, 0.2.2.

1. On a branch, add a ## X.Y.Z section at the top of CHANGELOG.md saying what changed; land it. That heading is the version: write ## 0.3.0 for a minor release. A section started early, ahead of the release, makes trunk builds read X.Y.Z-dev until it is tagged.
2. Tag the landed trunk check-in: zit tag add version-X.Y.Z <check-in>, then push. The tag must be on that check-in itself, not an ancestor.
3. The tag starts ci/release.yml, which runs scripts/release.sh X.Y.Z in the release environment. It refuses unless the changelog's newest section and the tag agree and the checkout is clean, then builds the linux-amd64 tarball, the five runner downloads and the source tarball, writes SHA256SUMS and release-manifest.json (zit's schema 1), checks the sums, and pushes each file to releases/X.Y.Z/ with zit uv push, SHA256SUMS first and the manifest last. The release then shows at /r/quarry/releases. linux-arm64 is opt-in (RELEASE_ARM64=1 in the job's env) until its cross build is qualified; without it the manifest lists linux-arm64 as omitted.
4. Deploy it: sudo /srv/quarry/deploy.sh --release X.Y.Z.

A publish that failed part-way is finished by cancelling the tag and adding it again: files the forge already holds are skipped. Once the manifest is published the release is done: a re-run whose check-in the manifest names publishes nothing, and one from any other check-in is refused. sh scripts/release.sh --dry-run X.Y.Z in a clean checkout of the tagged check-in builds everything into $TMPDIR/quarry-release-X.Y.Z (or --out DIR, which must be new or empty) and publishes nothing. That directory stays after a local dry run or a failed publish, and the next run refuses it until it is removed: rm -rf "$TMPDIR/quarry-release-X.Y.Z" before retrying by hand. CI gets a fresh TMPDIR for every job, so a retried tag never meets it. The run token never reaches the build: release.sh takes QUARRY_TOKEN out of the environment first and hands it only to zit uv as ZIT_PASSWORD.

The job publishes with a run token (RFC 0021), not an account. Its permissions: asks for uv: write (y); the release environment allows only that, takes trunk only, and requires the person who pushed the tag to hold a on quarry. Someone without a can tag; the build runs and cannot publish. An admin sets the environment once:

curl -fsS -u "$QUARRY_ADMIN:$TOKEN" -H 'Content-Type: application/json' \
  -X POST https://zeitforge.app/api/v1/r/quarry/ci/environments \
  -d '{"name":"release","branches":["trunk"],"allowed":"y","starter":"a"}'

That is op ci.environment.set (a on quarry), the same as the Environments rows in Settings.

The token reaches a job only on a runner started with --job-users. quarry-runner.service runs the test pipelines as itself, so releases get their own runner, scripts/systemd/quarry-runner-release.service: the only runner polling quarry.job (every workflow job of quarry has that task type), as the pool user qrel1, for trunk check-ins only, with a zit checkout so the job can read its tags. Its header lists the one-time setup: the two system users, the sudoers line, a new worker token, and, for linux-arm64, the gcc-aarch64-linux-gnu and g++-aarch64-linux-gnu packages. Any other ci/*.yml workflow added to quarry would also run on this runner, under these limits.

Installing the units

The units live in scripts/systemd/, the scripts beside deploy.sh. Alerts go through the notify companion quarry already uses, as the quarry-ops workflow (inbox and push, to the admins topic; quarry provisions it at startup). The notify key is QUARRY_NOTIFY_KEY in doppler quarry/prd, the same one quarry.service reads, reached with the service token in /srv/quarry/env. From a checkout of trunk:

sudo install -m 0755 scripts/deploy.sh scripts/quarry-alert.sh scripts/quarry-deploy-watch.sh /srv/quarry/
sudo install -m 0644 scripts/systemd/[email protected] scripts/systemd/[email protected] \
  scripts/systemd/quarry-deploy-watch.service scripts/systemd/quarry-deploy-watch.timer /etc/systemd/system/
sudo install -d /etc/systemd/system/quarry.service.d
sudo install -m 0644 scripts/systemd/quarry.service.d/on-failure.conf /etc/systemd/system/quarry.service.d/
sudo systemctl daemon-reload
sudo systemctl enable --now quarry-deploy-watch.timer

Then prove the alert path once, without breaking anything:

sudo sh -c 'set -a; . /srv/quarry/env; doppler run --project quarry --config prd -- \
  /srv/quarry/quarry-alert.sh --key test:$(date +%s) --message "alert path test"'

That proves the script from a shell, which has a $HOME; a unit does not get one unless it names a User=, and doppler fails without it. Prove the units themselves the way systemd runs them:

sudo systemctl start quarry-deploy-watch.service
systemctl show quarry-deploy-watch.service -p Result      # Result=success
sudo systemctl start [email protected]   # sends one "install-check failed" alert
systemctl show [email protected] -p Result

unit	what it is for
quarry-deploy@<hash>.service	a detached deploy: systemctl start --no-block quarry-deploy@<hash> runs deploy.sh --require-green --expect <hash>. Fails into an alert (OnFailure=)
quarry-alert@<unit>.service	sends "<unit> failed" with that run's FAILED line, or its last log lines. At most one per unit per ten minutes, so a crash loop is one alert
quarry.service.d/on-failure.conf	the forge crashing alerts the same way — on every crash systemd restarts (RestartMode=normal passes through failed), rate-limited to one per ten minutes
quarry-deploy-watch.timer	every five minutes: a deploy holding the lock over ten minutes; a deploy that ended mid-way (killed, terminal gone, a rollback begun) with no result line; a FAILED line from a deploy run by hand. Each alerts once