Artifact [a01fc0c381]

Artifact a01fc0c38108a97de353229fd489b4b55e7497f6c23e9341eea0f19067b8c905:

• File docs/running.md — part of check-in [497c301ba4] at 2026-10-02 23:46:33 on branch trunk — land release-workflow

  Checks tested the merge with trunk at a107f408edc9; trunk has moved since. (user: dmestas, size: 64589) [annotate] [blame] [check-ins using] [more...]

# Running quarry

Two ways: the compose stack, or by hand. The hand-rolled path is what
this was developed against and is what the compose file encodes, so it
is written down first — a run book that stays true even if the images
are never built.

## What the stack is

| piece | role | state |
|---|---|---|
| **zit** | the store. quarry shells out to it for every read and write (0001 §2) | the `.zit` files themselves |
| **quarry** | the forge: indexes, UI, API, sync fronting | derived indexes, embedded NATS, OLTP database |
| **notify** | notifications: workflows, subscribers, topics, the inbox quarry embeds | SQLite |
| **dagnats** | the workflow engine notify runs on: embedded NATS + JetStream + an HTTP bridge | JetStream data |

`zit` is not a runtime dependency in the ordinary sense — it is quarry's
only store interface, so **its version is part of quarry's behaviour**.
On 2026-08-16 three zit builds changed what quarry does: `frames` got
31% faster, a card-parsing fix changed which artifacts parse at all (and
until it landed, sqlite's 794 tickets were invisible), and a new
`stat -R` let quarry serve a bare store. Pin the pair; do not let a
deploy resolve zit from whatever is on PATH.

## By hand

### dagnats

Runs under launchd on this machine as `com.dmestas.dagnats`, binding
NATS on 4222 and its bridge on 8080, with data in
`~/Library/Application Support/dagnats`.

```sh
launchctl kickstart -k gui/$(id -u)/com.dmestas.dagnats
```

A freshly-built Go binary replacing it must be **ad-hoc signed** or
launchd kills it with `OS_REASON_CODESIGNING` and the service silently
fails to come back:

```sh
go build -o ~/go/bin/dagnats ./cmd/dagnats
codesign -s - -f ~/go/bin/dagnats
launchctl kickstart -k gui/$(id -u)/com.dmestas.dagnats
```

In standalone mode dagnats binds **loopback**, which is why anything
talking to it shares its network namespace under compose rather than
reaching it by service name.

### notify

```sh
notify bootstrap -dev -db notify.db -org quarry -env dev   # prints apiKey once
notify serve -dev -db notify.db -addr 127.0.0.1:3000 \
  -nats-url nats://127.0.0.1:4222 -bridge-url http://127.0.0.1:8080
curl -s localhost:3000/readyz     # {"status":"ok","bridge":true,"db":true}
```

`-dev` derives a fixed, publicly-known master key. Outside development
set `NOTIFY_MASTER_KEY_HEX`.

**Provisioning is quarry's job, not an operator's** (RFC 0011 §7). What
used to be a page of `curl` here is now code: quarry creates a notify
subscriber when an account is added, adds or removes the account from
a repository's `repo-<name>` activity topic when it is granted,
revoked, or subscribes to a thread, and creates its workflows
(`quarry-mention`, `quarry-forum-reply`, `quarry-activity`,
`quarry-landing`, `quarry-changelog`, `quarry-error`, `quarry-slow`,
`quarry-test` — defined in `internal/notify/workflows.go`) at startup, leaving any workflow an
operator already edited untouched. `quarry-landing` is provisioned
ahead of need: nothing triggers it yet, that lands with the pull
request review surface. All of it is best-effort: a notify outage
never fails the account or grant op that raised it.

**Scripts are compacted at start, not at build.** quarry's own scripts
are minified in memory when the process starts (the stylesheet has its
comments and indentation stripped the same way), then served under
content-hashed, immutable names; the sources in the repository
stay readable and there is no build step to run. `make wire
URL=https://your-forge/` prints what a first visit costs per file. Set
`QUARRY_ASSETS_RAW=1` to serve the sources as written while editing them.

**Email needs a provider, and quarry gives notify its own.** A workflow
with an email step and no mail provider behind it fails where only
somebody reading notify's database would find it: `store: not found`,
per message, while the settings page goes on offering an email column.
So if quarry has a mailer — `RESEND_API_KEY` and `QUARRY_MAIL_FROM`,
the same pair that sends invites — it hands notify the same credential
at startup and logs `notify: email now sends as …`. One credential, in
one place, and nothing to POST by hand.

**Email is off until a person switches it on.** The four family
workflows carry an email step (plain text, one link; mention, reply
and landing accept replies), but notify reads an absent preference as
enabled, so a step alone would mail every account about everything.
At every start, and when an account is created, quarry writes an
explicit "no email" for each family the person has not answered, and
never touches a cell they set themselves. The settings page is where
they answer.

An email integration notify already holds is left exactly as it is, so
configuring SMTP or SES in notify's own console is still the way to use
a different provider from the one quarry sends invites with — a restart
will not replace it. With no `RESEND_API_KEY` quarry asks the companion
for nothing rather than creating an integration that cannot send.

Invitations are the exception that proves the split: they are sent by
quarry's own mailer and are *not* a notify workflow, because an invitee
has no account, no subscriber and no inbox for any other channel to
reach — and a forge that sent them both ways delivered two invitations
from one address.

**Receiving email needs nothing but notify.** A reply to a notification
email comes back to `reply+<token>@<inbound domain>`, and notify can
receive it itself (notify#340): set `NOTIFY_SMTP_ADDR=0.0.0.0:25` and
`NOTIFY_INBOUND_MX_HOST=mail.example.org`, let the service bind a
privileged port, and point the domain's MX at the box. On a systemd
host that is one drop-in:

```ini
# /etc/systemd/system/notify.service.d/smtp.conf
[Service]
Environment=NOTIFY_SMTP_ADDR=0.0.0.0:25
Environment=NOTIFY_INBOUND_MX_HOST=mail.example.org
AmbientCapabilities=CAP_NET_BIND_SERVICE
```

and two records — `mail.example.org A <the box>` (not proxied: a CDN
does not carry SMTP) and `mail.example.org MX 10 mail.example.org`.
Register the domain with `POST /v1/inbound/domains {"name":…}` and
`POST /v1/inbound/domains/{name}/verify`; verification is a live MX
lookup, so it fails until the records have propagated and can simply be
repeated. This is not a mail server: an address without a valid reply
token is refused at `RCPT TO` with `550 5.1.1`, before any message body
is read, so there is no inbox to spam and nothing to filter. It speaks
plaintext today; a sending MTA that offers STARTTLS falls back cleanly.

A host that already runs an MTA can pipe instead of listening:
`notify inbound < message.eml` reads one raw message on stdin and hands
it to the same path, so an `/etc/aliases` line does the job without a
second port.

**What a reply can answer.** Four notifications carry a binding quarry
can read back, and hitting reply on any of them writes the same thing
the web form would:

| notification | the reply becomes |
| --- | --- |
| a forum thread you follow | a reply on that thread |
| an @mention in a forum post | a reply on that thread |
| an @mention in a repository's room | a line in that room, answering yours |
| an @mention in a team's room | a line in that room, answering yours |

This is per ACCOUNT and needs no privilege: the gate is the room's own
capability (`C` for a repository's chat, `3` for the forum, membership
for a team's room), checked against the person the notification was
addressed to. A reply from any other address is refused — the binding
names one account and only that account can answer it.

The chat bindings were minted from the day chat had mentions and
nothing read them back, so a reply to one was accepted, understood and
dropped with "no thread behind transaction". They land now.

Two limits worth stating:

- **A chat reply names the line it answers, when it still can.** The
  window is finite and a mail can sit in a drafts folder for a week; a
  reference that has aged out is refused, and the reply is then posted
  without it. Losing the thread link beats losing the reply.
- **A reply to a forum post held for moderation is refused**, as the
  web form is. Nothing is silently queued.

**Workflow definitions follow quarry's code.** Provisioning upserts
each of its workflows with notify#303's
`PUT /v1/workflows/{slug}?onConflict=update`. notify compares the
definition it holds with the one quarry sends: a missing workflow is
created, an identical one is not written, and one quarry has changed
(a new step, a new template such as an email's `htmlBody`) is rewritten
in place on the next start, logged as `notify: workflow <slug> updated
in place`. The workflow keeps its id, so the per-workflow preferences
notify keys by that id survive. Never delete a workflow to pick up a
new definition: recreating it loses every subscriber's preferences for
it. A hand edit to one of these workflows in notify's console is undone
by the next start; change `internal/notify/workflows.go` instead.

**What a companion that does not answer costs.** The preferences pair
(`GET|PUT /v1/subscribers/{id}/preferences`) and the per-device
credential `DELETE` have shipped, so both halves normally land. When
notify is unreachable quarry says so rather than pretending:

- `me.prefs.set` writes its forge-local half first (the toast and
  desktop switches, the per-repository mute) and reports
  `applied.notify: false` when the notify half could not be stored. The
  channel matrix a person set is then not yet in force. A read that
  notify did not answer marks every family `unknown` and the matrix is
  left ALONE on the next save, rather than writing quarry's defaults
  over settings it could not read.
- `me.push.unsubscribe` keeps the device row marked
  `pending_remote_delete` when notify FAILED the delete, instead of
  forgetting a browser notify still pushes to, and retries on the next
  `me.push.subscribe`. A 404 is not a failure — a credential notify
  does not hold is the end state unsubscribe asked for — so the row is
  forgotten there and then. The `push_device.pending_remote_delete`
  column is now only ever a transient state between a failed delete and
  its retry.

Both clear themselves once notify answers — no migration, no reset.

### quarry

```sh
quarry serve --addr 127.0.0.1:8950 --data ./data \
  --notify http://127.0.0.1:3000 --notify-key "$K" \
  --repo quarry=$HOME/quarry
```

`--data` is not optional in practice: it defaults to `.quarry` beside the
**first** `--repo`, which writes derived state into whichever repository
happens to be listed first. Always pass it.

Gates are open until they are closed. An account is the forge identity
(argon2id); a push is signed with a separate token, because the sync
protocol's RFC §6.3 secret is derived from whatever the client holds:

```sh
quarry account add --user you --admin --data ./data   # closes web/API writes
```

Pushes are gated by the letters: an account pushes with a push token it
mints on its own account page (`/me`, Push credentials) or with
`POST /api/v1/me/sync-token`, never with its password.

The account gate is **global**: the first account closes writes on every
repository in the fleet, not just one.

`--admin` makes a **site admin**: the forge's accounts, invites, `/fleet`,
runners, settings and quotas. It gives no repository. A site admin reads,
pushes to or administers a repository only through a role on it
(`quarry account grant`) or a team, like anyone; over the rest they keep
only the operator acts in docs/hosting.md, each of which leaves the owner a
receipt. Grant the repositories an admin or an automation account works on
before relying on it there.

Behind a proxy, name the one header it sets that a client cannot, with
`--trusted-proxy-header` (or `QUARRY_TRUSTED_PROXY_HEADER`). Only a
request that carries that header AND arrives over loopback is believed
for the client's address, which the login limiter keys on, and for
`X-Forwarded-Proto`, which sends http readers to https and sets HSTS.
From any other peer the header is ignored, so a directly reachable
listener cannot be fooled by a client that writes it.

**Behind cloudflared this flag must ship in the same deploy as the code
that introduced it, set to `CF-Connecting-IP`.** Unset, no forwarded
header is believed, and every request arrives from cloudflared on
127.0.0.1. Every sign-in then keys on that one address: five wrong
passwords from anyone lock sign-in for everyone, with backoff up to
about two hours. The http-to-https redirect and the HSTS header also
stop. Cookies stay Secure only because `--origin` is https.

```sh
quarry serve ... --origin https://forge.example --trusted-proxy-header CF-Connecting-IP
```

### The attention stream and HTTP/2

Signed-in pages open one Server-Sent Events connection for the attention
stream — the inbox badge and the count in the tab title (RFC 0011 §5).
In-page banners are off by default and are turned on per account on
`/me` (the notifications section), which is also where desktop
notifications are opted into; with both off the stream still moves the badge and the
title, and nothing ever covers the page being read. On plain HTTP/1.1 a browser allows **six connections per
origin**, and a repository page already spends one or two of them on the
chat and timeline streams. So the default opens the attention stream on
**repository pages only**: a stream on every page would compete with the
page it sits on for the same six slots, and the symptom is a page that
stalls halfway through loading rather than an error anybody can read.

With HTTP/2 in front there is no such limit — one connection carries
every stream — so the whole forge can hold it open:

```sh
quarry serve ... --attention-everywhere
```

Use it when quarry is behind TLS termination that negotiates h2, or
behind a proxy speaking h2c to a browser. quarry serves plain HTTP/1.1
itself, so the flag is a statement about what is IN FRONT of it and
quarry cannot check it: turned on without h2, it is the six-connection
problem on every page.

Nothing else changes. Without the stream a page still renders its own
state and the inbox badge is fetched once after load, which is the
Principle 3 behaviour every one of these surfaces keeps.

### Push

`/me`'s notifications section carries the whole of RFC 0011 §6: the
family x channel matrix (mention, reply, landing, activity, by in-app,
push and email), the two forge-local switches, the repositories this
account muted, and **Push on this device**. `/me/notifications` still
works as a bookmark — it redirects there.

Push is the one feature in quarry that needs JavaScript, recorded as an
explicit exception in RFC 0011 §11. Enabling it registers `/sw.js`, asks
the browser for notification permission from the click that asked for
push, subscribes with notify's VAPID key, and posts the resulting
`PushSubscription` to `POST /me/push`. With JavaScript off the control
says so and nothing else on the page changes. On iOS a site must be
added to the Home Screen before Safari will grant push at all — the
control says that too.

Quarry never holds a VAPID key and never stores an endpoint: the
subscription goes to notify as subscriber credentials, and quarry keeps
a hash, a user-agent label and a timestamp so the page can list the
browsers somebody may want to turn off. That is also why a browser other
than the one being used is turned off **from that browser**.

What to expect until the upstream work lands:

- **notify#302** (the `webpush` provider) — subscriptions are accepted
  and stored, but nothing is delivered by push yet. The in-app inbox,
  the badge and the attention stream are unaffected.

notify#305 (the preferences route, the per-device credential `DELETE`
and the trigger identifier on every message row) has shipped. When the
companion is unreachable the banner after saving preferences says the
companion did not answer and the page reports `applied.notify` honestly;
a device whose delete failed shows `removal pending` and is retried the
next time push is enabled.

Neither needs a migration or a reset on this side.

### The activity robot

A repository can have its timeline talk in its chat channel
(RFC 0011 §10): **settings → Activity robot**, in the same event-pattern
syntax webhooks take (`*`, a noun like `checkin`, or an exact kind like
`landing.accepted`). Empty turns it off. Robot lines never count as
unread and never mention anybody.

### Your account

`/me` is where a person manages their own things: their identity (login,
site-admin standing, when the account was created), their email, their
password, their signed-in sessions, and the notifications matrix above.
It absorbed the two pages that used to hold pieces of this alone —
`/me/notifications` and `/me/sessions` both still work as bookmarks,
redirecting to the matching section (`#notifications`, `#sessions`).
The account menu's avatar popover links straight to it.

A small set of emails are **transactional**: they are about the account
itself — an invite, being given or losing access to a repository, an
address confirmation, a password change — and no preference switches
them off. They never go through notify's matrix
(`internal/notify/workflows.go`'s four families, the matrix on `/me`):
that matrix is for content a person chooses to hear about, and an
account changing hands or losing its way to a repository silently is
not something a preference should be able to produce.
`internal/api/transactional.go` names the exact list and says why
extending it is a decision, not a convenience.

A person sets their own address on `/me`'s email section. Nothing is
confirmed until a six-digit code — mailed to the NEW address, good for
ten minutes — is typed back in on that same page; until then, quarry
keeps mailing whatever address was confirmed before, if any. Five wrong
codes burn the pending one outright, same as letting it expire, and
either way the only way forward is asking for a fresh one. Losing the
old address to a new confirmation sends the old one a notice, so a
change nobody made does not go unnoticed just because the old inbox is
still being read. An administrator can also set or correct an existing
account's address from `/fleet/people`, stored confirmed at once — the
same trust as typing an invite's address — and the account is told,
there, that an admin did it.

A person can also change their own password from `/me`, given their
current one. Doing so ends every OTHER session that account holds —
the browser making the change stays signed in — and mails the account's
confirmed address to say so, the same as an admin-driven
`account.passwd` ends every session of the account it targets.

A forge started with no `RESEND_API_KEY` or no `--mail-from`
(`$QUARRY_MAIL_FROM`) — the same two settings account.invite already
needs — simply sends none of this. Every op still answers correctly:
an invite link is returned either way, a caps change still takes
effect. Only the mail itself does not go.

### Agents

An account may be **machine-operated**. Settings is not where this
lives — it is on `/fleet/people`, beside the account it describes: mark
one an agent and name the person answerable for it, or take the mark
off.

The mark is one fact, `operator`, and its non-emptiness is what makes
the account an agent. Two questions with one field, because neither is
worth much alone: "a machine wrote this" without "and X runs it" tells
a reader to distrust the words without telling them who to ask.

**Declaring is safe; hiding is not.** Any account may mark ITSELF an
agent, naming itself as the operator. Only a site admin may clear a
mark, mark somebody else, or name another person as answerable — that
last is a claim about a person, not about yourself.

The badge appears wherever a name is drawn: the timeline, tickets, the
forum, chat, landings and attachments all render names through one
function, so it reaches them from a single rule rather than a
convention each surface has to remember.

The mark is on the ACCOUNT, not the write. A write-level mark is more
honest and much larger — it would have to reach the artifact format —
and this is what makes that worth building later: you cannot ask which
of an account's writes were machine-made until you can ask whether the
account is a machine.

### A forgotten password

Press **Reset password** on the person's row on `/fleet/people`. It mints
a one-time link that lets them choose a new password themselves. The link
goes to the address on the account when it has one, and is shown to you
either way so you can pass it on by hand — which is the usual case on a
forge where most accounts have no address.

Three things about the link are deliberate:

- **It is good for one hour, and once.** An invite gets seven days
  because it is sent to somebody who is not waiting for it; a reset is
  asked for by somebody sitting there. A live link is a credential, so
  it lasts as long as the person needs and no longer.
- **Their current password keeps working until they use it.** A reset
  that blanked the password would lock them out harder the moment the
  link went astray — the failure this exists to end, not to reproduce.
- **Minting again supersedes.** "Send it again" drops the unspent one,
  so there are never two live links where you believe there is one.

Redeeming ends every other session of the account, because a forgotten
password is sometimes a lost device.

Never set a password on somebody's behalf. `account.passwd` still exists
for scripts and for an account whose owner cannot be reached at all, but
using it means a password that you chose travels to them down some
channel, and both of you hold it afterwards. The link means nobody but
its owner ever sees it.

An account that has never been claimed has no password to reset: it
wants the invite it is already waiting on, and the row offers **Re-send
invite** instead.

## Fleet admin

`/fleet` is the operator page — every store's ingest health, and the
upstreams it mirrors. It is **site admin** only: admin on one repository
is not enough, because the page reports operational detail across the
whole fleet and can force writes into any of it.

Health is per-process. `Total` and `Fallbacks` count what THIS server
has ingested since it started, so a restart with a warm index reports
zero of both however much the store holds. The parse-fallback number
worth looking at is the one a `quarry reindex` produces.

### Mirroring

A mirrored store tracks an upstream: quarry pulls, and never pushes.
The four postures a repository can be in, and what each one opens or
refuses, are laid out in [hosting.md](hosting.md).

Configuring a mirror needs admin on the repository being configured —
which its creator holds from the moment they created it — not site
admin. That includes `mirror.writes`: flipping a tracking store to
accept local writes creates two write heads on it, but it is that
repository's own fork to reconcile, not the forge operator's call.

```sh
curl -u you:pw -X POST http://localhost:8080/api/v1/mirrors \
  -H 'Content-Type: application/json' \
  -d '{"repo":"fossil","url":"https://fossil-scm.org/home","every":3600}'
```

`every` is seconds and is floored at 60 — these upstreams are other
people's servers, and the floor is a courtesy the configuration surface
cannot opt out of. `--mirror-tick` (default 1m) is how often due-ness is
evaluated, not how often anything is pulled.

A repository also says, on its OWN side, whether other people may keep
a copy that follows it — see [hosting.md](hosting.md#open-to-being-mirrored)
for what "mirrorable" means and why a private repository is never
mirrorable regardless of the setting.

Pulls take the store's single-writer lease, so mirroring is safe with
more than one node serving the same store. Only the node holding that
lease runs the pull at all; the others skip it and record nothing, so a
mirror's row never shows a failure because a node was not the holder.
A pull that brings nothing
new does not trigger a re-index and is not logged; almost every one is
quiet, and a line per store per interval buries the ones that matter.

Two limits worth knowing before you rely on it:

- **The store needs a working checkout.** `zit pull` refuses `-R` and
  operates on a working directory, so a genuinely bare store cannot be
  mirrored. Stores created by `zit clone` or `zit import-fossil` carry
  one, so this bites only hand-built bare stores.
- **The upstream must allow anonymous clone.** Some Fossil servers serve
  a browsable timeline while refusing sync — `sqlite.org/lemon` is one.
  Note that `fossil clone` exits 0 on that refusal, so check for the
  repository, not the exit status.

### Attachments: binding a bucket

Files uploaded through quarry go to a bucket the repository's steward
brings (RFC 0016); the forge keeps a stub in the repository and never
the bytes. A steward binds one bucket per repository from its settings
page, under Attachments, or through the op:

```sh
curl -u you:pw -X POST http://localhost:8080/api/v1/r/fossil/bucket \
  -H 'Content-Type: application/json' \
  -d '{"endpoint":"https://<account>.r2.cloudflarestorage.com","bucket":"fossil-files",
       "region":"auto","accessKey":"...","secret":"...","publicBase":"https://files.example.org"}'
```

At R2, create the bucket, then an API token scoped to that bucket
with object read and write; the region is `auto`. A public or custom
domain on the bucket is optional — set it as the public base and
downloads link there; without it every download is presigned. Any S3
endpoint works the same way with its own region.

The binding is probed before it is kept — a presigned PUT and DELETE
of one small object under `attachments/` — so a wrong key or a token
without delete is refused on the form. Probe again from the page or
with `bucket.probe`; Fleet shows each store's last probe beside its
mirror's last pull. The secret is sealed under `secret.key` beside
`oltp.db` (created 0600 on first use) and never returned by any op; a
forge that loses the key file has the steward enter the secret again.

The probe also sends the browser's preflight — an `OPTIONS` naming the
forge's origin (`--origin` / `QUARRY_ORIGIN`), `PUT` and `content-type`
— and records the answer as `corsOk`. The page's scripted upload PUTs
the file straight to the bucket from the browser, which the bucket
refuses until its CORS rule allows the forge; the multipart form, which
posts to quarry, needs no rule. A failed preflight therefore never
refuses a binding: the settings section says browser uploads are
blocked and shows the rule to paste, in R2's shape with the forge's own
origin filled in:

```json
[{"AllowedOrigins":["https://forge.example"],"AllowedMethods":["PUT","HEAD","GET"],
  "AllowedHeaders":["content-type","content-length"],"MaxAgeSeconds":3600}]
```

At R2 it goes under the bucket's Settings, CORS policy; any S3 takes
the same rule as a `PutBucketCors` configuration. Probe again once it
is in place. A forge started without `--origin` cannot ask on the
browser's behalf and records `corsOk` false.

### Attachments: uploads, letters and the lifecycle

With a bucket bound, a holder of the upload letter — `b` on tickets,
wiki pages and technotes, `B` on forum posts (RFC 0008) — sees "Attach
a file" in the Attachments section of those pages. With script the
browser hashes the file, asks `attachment.begin` for a presigned PUT
(fifteen minutes, the length and the type signed), sends the bytes to
the bucket itself and calls `attachment.commit`, which HEADs the object,
checks its length against what begin was told, and writes the stub and
the A card through `zit attach add`. Without script the same form posts
the file to quarry, which carries it to the bucket in memory — never to
disk — and commits; that path is bounded at 32 MiB, since a presigned
PUT signs a length the form cannot know until the file is read. API
clients have the same floor at `POST /api/v1/r/<repo>/attachments/<target>`
with a multipart `file`. Uploads are refused over the binding's limit
and off the type policy: text, the common images, PDF, JSON, the common
audio and video; archives only when "allow archives" is on in the
section; executables never. Needs zit 0.3.13, which adds `zit attach`.

With script, a file can also be **pasted or dropped**, and nothing
leaves the browser until something is posted. A paste or drop into a
composer — a new thread, a reply, a new ticket, a ticket's comment or
description box, the wiki editor — previews the file under the box (a
thumbnail for a picture, the name and size otherwise; a clipboard
screenshot arrives as `image.png` and is renamed
`pasted-YYYYMMDD-HHMMSS.png`, in UTC), puts the reference the markup
resolves at the caret — `![name](attach:name)` for an image,
`[name](attach:name)` otherwise — and offers a Remove that only
forgets it. When the form is submitted the script uploads each pending
file first: `attachment.begin` — with the target when the object
exists, with the family alone when it does not — the PUT, the progress
bar; then the form goes with the tickets as hidden
`attach` inputs, and the op that writes the object (`forum.post`,
`forum.reply`, `ticket.new`, `ticket.set`, `wiki.save`) commits each
against it. A refusal on any file keeps the form from going and names
the file on the error line; a refusal at commit comes back in
`attachErrors` and never undoes the write. A paste or drop on an
Attachments section — the one around the focused element, else the
first on the page that offers an upload — fills its file picker, and
its Upload button is the one control that sends at once. A post held
for moderation leaves its files unattached. A new ticket takes files
only when it is recorded in the fossil now: the local tier has no A
card to write, so the control is offered to administrators alone.

An upload is **held** unless the uploader holds the family's moderation
letter — `q` on tickets, `l` on the wiki and notes, `5` on the forum —
or administers the repository. A held file is in the repository (a
fossil client sees it as any attachment) but on quarry's pages it is a
link with its name and size, never a picture, until a moderator vouches
for it from the card or from the "Held uploads" list in the Attachments
settings section; reject removes it. Removal — the card's Remove, or
`attachment.remove` — writes fossil's sourceless A card and leaves the
object in the bucket unless "delete the object when an attachment is
removed" is on. Downloads of an upload redirect to the bucket: the
public base when the repository is public and one is set, otherwise a
presigned GET good for an hour.

Objects are keyed `attachments/<sha256>`, and an object no stub names
is an orphan: an upload that began and never committed, or one whose
attachment was removed with deletion off. quarry writes nothing else
under the prefix. A lifecycle rule that deletes `attachments/` objects
a day old would take live files with it, so do not set one; sweep
orphans instead by listing the prefix against `attachment.list` (or
`zit attach list`, which marks stubs `external:sha256:…`) and deleting
what neither names. Objects a day old that no stub names are safe to
delete: an upload is complete or abandoned within the hour.

### Unversioned files

A repository's unversioned files (RFC 0020) live in the same bucket
under `unversioned/<repo>/<name>`, with removals as empty objects under
`unversioned/<repo>/.tombstones/<name>`; each object's metadata carries
the fossil mtime, hash and author. The repository's name is in the key
because several repositories may bind one bucket: attachments are
content-addressed and share safely, but two repositories' files of the
same name would otherwise be one object, and each catalog rebuild would
import the other's. A lifecycle rule must leave both
prefixes alone. The forge's catalog is a copy, rebuilt from the bucket
when it is missing or another bucket is bound, and on demand with
`uv.rebuild`. Publishing takes `y`, which admin does not imply. A
private repository whose binding has a public base cannot publish
them: their keys are names anyone can guess, so remove the public base
first. That refusal guards writes only. A repository made private, or
given a public base, after it published files leaves those files
reachable on the public base by anyone who guesses a name, until they
are removed or the base is cleared.

Fossil clients sync unversioned files through the same URL as
check-ins (`fossil sync -u`, `clone -u`, `uv sync`); the sync front
answers those cards from the bucket and hands zit only the rest. Every
refusal (no `y`, a tracking mirror, a file over the binding's limit, a
hash that does not match) comes before zit sees the request, so the
check-ins it carried are refused with it. A stock fossil client whose
account lacks `y` is answered uv-pull-only, so it never sends its
unversioned files and its check-ins land as usual; a hand-built request
carrying a `uvfile` from such an account is refused whole, as a stock
fossil server refuses it. A bucket or catalog failure
after those checks is different: the check-ins are already applied, the
client is told the unversioned part failed, and its next `sync -u`
retries that part. A bucket transfer is allowed 30 seconds plus 2 per
MiB, so a binding with a high limit on a slow link is the case to
watch.

The catalog's write lock is held in the forge process, which is right
for the way quarry runs: one process per OLTP database. Two forge
processes sharing one `oltp.db` could interleave a write with a rebuild
and leave the catalog missing a file until the next `uv.rebuild`.

### Fossil chat bridge

An opt-in connector between one repository's chat channel and one
Fossil server's chat (RFC 0011 §9). It is configured per store, from
`/fleet` or through the op:

```sh
curl -u you:pw -X POST http://localhost:8080/api/v1/r/fossil/mirror/chatbridge \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://fossil-scm.org/home","user":"quarrybot","password":"..."}'
```

Detach with `{"detach":true}`. The password is stored and is never
returned by any op or rendered on any page; the only reader is the
poller.

**The upstream needs two things**, neither of them a default:

1. A login holding fossil's `C` capability. Chat is gated on it.
2. `http_authentication_ok` in the repository's config, or fossil
   refuses HTTP Basic and every request is anonymous. It is not a
   `fossil setting`, so it goes in directly:

   ```sh
   fossil sql -R /path/to/repo.fossil \
     "INSERT OR REPLACE INTO config(name,value,mtime) \
      VALUES('http_authentication_ok','1',now());"
   ```

Quarry polls `/chat-poll` with a nine-minute client timeout (fossil
hangs the request for `chat-poll-timeout`, 420 s by default) and backs
off from 30 s to 5 minutes while an upstream is failing. Every request
carries a same-origin `Referer`: without one fossil treats the poll as
CSRF-unsafe and refuses its own table setup, which surfaces as a
database error rather than as an authentication failure.

**What it does.** Messages from the upstream arrive in the channel as
`fossil:<user>` with `Origin: fossil`. A fossil deletion becomes a
tombstone on the message it removed. An attachment becomes a link to
the upstream's `/chat-download` — fossil keeps the file. Quarry's own
messages go out to `/chat-send` prefixed `**name:** `, because fossil
attributes everything to the login the bridge signs in as. The bridge
never forwards robot lines or lines that came from fossil, and never
re-imports its own login's messages.

**What it does not do.**

- **No edits.** A fossil message cannot be edited from quarry, and a
  quarry edit is not carried upstream. Only deletions cross, and only
  in the fossil-to-quarry direction.
- **No mentions.** Fossil chat has none, so a bridged line raises
  nobody's badge and triggers no notification.
- **No account mapping.** `fossil:drh` is a name from another server's
  user table, not a quarry account, and the prefix says so.
- **Two Fossil servers are two chats.** Fossil's chat is a
  server-local table that never syncs, so bridging a mirror connects
  you to that ONE server's conversation, not to the project's.
- **The window is still quarry's.** A bridged channel holds seven
  days like every other; the bridge is a connector, not an import.

When this repository's activity robot is on, the upstream's
`chat-timeline-user` announcements are skipped — otherwise every
check-in is announced twice, once by each robot.

**The bridge runs on one node.** Like a mirror pull, it runs only on
the node holding the repository's single-writer lease; the others do
not start a poller, and a node that loses the lease stops its bridge
at the next mirror tick. Two nodes bridging the same repository would
import each upstream message twice and send each quarry message
upstream twice, which is why the lease decides.
Restarting a node is safe: the msgid map makes a re-read of the same
message a no-op, so a cursor that has not caught up does not double
anything.

**Private addresses are refused by default.** The bridge is a
server-side fetcher holding a credential, so it will not connect to a
loopback, private or link-local address — that last range is where
cloud metadata services live. The check runs on the resolved address
at connect time, so a public name pointing inward is refused too.
Bridging a fossil on the same host is an opt-in:

```sh
quarry serve --allow-private-bridges ...
```

**Redirects are refused, not followed.** Go re-sends the
`Authorization` header across a same-host redirect, and here that
header is the fossil password. Configure the final URL. A URL carrying
credentials (`https://user:pass@host/`) is rejected outright: the user
and password are separate fields, and a URL is copied into log lines
and status rows.

## CI: workflows, runners and secrets

RFC 0019 is the design; this is how to use it.

**A workflow** is a YAML file in `ci/`, in GitHub Actions' shape. The
repository's CI page offers two starters in the pipeline editor, a test
workflow and a deploy after a landing:

```yaml
on: [push, landing]          # push: a check-in on a branch; landing: a landing's merge
jobs:
  test:
    timeout-minutes: 30
    steps:
      - run: make test
```

`on:` takes `push` and `landing` (with `branches` or `branches-ignore`),
`tag` (with `tags` or `tags-ignore`; `version-*` when neither is given),
`workflow_dispatch` (a Re-run) and `schedule`. A `tag` workflow starts
once when a tag is added to a check-in (`zit tag add version-0.2.0
CHECKIN`), on that check-in; a branch's own name is not a tag, and a
propagating tag starts it on the check-in that carries it, never on the
descendants, and a tag pushed long after it was made still starts it.
Its run receives secrets only when the tagged check-in is on a branch
the environment admits. A Re-run of the tagged check-in re-runs its push
workflows, not its tag workflows; to repeat a release, give the release
workflow `workflow_dispatch` too and Re-run, or cancel the tag and add
it again. A tag run's steps read the tag from `QUARRY_TAG` (unset in any
other run), so a check-in carrying two version tags runs each release
with its own name; the next push to the branch does not cancel a tag
run. The tags a repository already holds when CI first sees it with a
check-in start nothing.

A `schedule` workflow names cron lines, GitHub's five fields in UTC:

```yaml
on:
  schedule:
    - cron: "17 3 * * *"   # every day at 03:17 UTC
```

The forge starts it, not the engine: once a minute it reads the default
branch tip's workflows and starts each one due, on that tip, so a
schedule always runs the default branch's file whatever a branch says.
Each scheduled minute starts at most once, across restarts and fleet
nodes: the minute is claimed before the run is launched, so a launch
that fails loses that minute, and the CI page shows it on the
schedule's row. A minute missed while the forge was down is not run
later, as GitHub does not. A schedule whose previous run is still going
(within its jobs' timeout, six hours at most) is skipped for that
minute, and the forge log says so. A workflow may have at most ten
cron lines, and together they may not fire more often than every 5
minutes: the times of day all its lines name (hour and minute) are
judged as one set, wrapping at midnight, whatever the day fields say,
so two lines for different weekdays at 03:00 and 03:02 are refused too;
a workflow that breaks it is refused where workflow errors
show and listed on the CI page as not firing. Repositories whose CI is
declined, or that no runner serves, are not scheduled. A scheduled run
is on the default branch, so it receives the secrets of its environment
and the repository's own secrets as any default-branch run does (as on
GitHub); it has no starter, so it receives a run token only from an
environment that requires no starter letter. The CI page lists each
scheduled workflow with its cron and next fire, and marks scheduled
runs. `needs:` orders jobs; a
job's steps run in order and stop at the first failure. There is no `if:`
and no expression but `${{ secrets.NAME }}`, and an Actions key quarry
does not implement (`uses:`, `strategy:`, `container:` and the rest) is
refused by name. Engine JSON in `ci/*.json` still works.

**A runner** runs a repository's jobs on a machine you choose. Mint a
worker token in the repository's Settings (Runners), then:

```sh
QUARRY_RUNNER_TOKEN=... quarry runner https://forge.example/r/REPO
```

With no `--tasks` it polls the repository's workflow jobs
(`<group>.job`) and whatever `ci/runner.json` on trunk names.

**Secrets** live in the repository's Settings (Environments). A job names
an environment and takes a secret as an env value:

```yaml
  deploy:
    needs: [test]
    environment: production
    steps:
      - run: ./deploy.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
```

An environment gives its secrets only to check-ins on protected branches
(RFC 0007) unless it names other branch patterns, and never to a
landing's merge; protect the branch you deploy from. The values are
sealed with `QUARRY_CI_SECRETS_KEY` (32 random bytes, base64) from the
forge's environment; without it the forge stores and delivers nothing.
A backup restored without the key keeps names and policies, and every
value must be set again.

**A runner that receives secrets** runs every job as one of a pool of
users, so no job can read another's environment or the runner's token:

```sh
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin qjob1   # and qjob2...
useradd --system --user-group --create-home --home-dir /srv/ci-REPO --shell /usr/sbin/nologin ci-REPO
usermod -aG qjob1,qjob2 ci-REPO
chmod 0711 /srv/ci-REPO
echo 'ci-REPO ALL=(qjob1,qjob2) NOPASSWD: ALL' > /etc/sudoers.d/quarry-job-users   # then visudo -c
quarry runner --job-users qjob1,qjob2 --work /srv/ci-REPO/runner https://forge.example/r/REPO   # as ci-REPO
```

Run it as a user of its own (`ci-REPO` above), not one another runner
shares: a second runner's jobs under the same user could read its token.
A job's `TMPDIR` is inside its checkout; tests that write to `/tmp` by
name collide between pool users.

After every job, and once per job user at start, the runner ends
whatever that user left running (`job-reclaim`, run as the job user
through the same sudo rule; it needs no `pkill` or other tools on the
image). A runner that cannot do it refuses to start, and a job user
whose processes cannot be ended stays out of the pool until the runner
restarts, so no job runs beside another's leftovers.

## Roles, and what a role change tells

A person's standing on a repository is a capability string (RFC 0004),
and there are two doors onto it:

- `account.grant` / `account.revoke` take one of the four preset roles —
  reader, developer, maintainer, admin.
- `account.caps` takes the whole **string**, which is what the settings
  editor writes and what no preset covers. Empty removes the account's
  explicit string; the repository's categories still apply.

All three raise RFC 0005's `account.added`, `account.caps` and
`account.removed` — through one helper, because three copies would drift
and the drift would be a role change that reaches a webhook when it is
made one way and vanishes when it is made another. A change that changed
nothing raises nothing: these are operational events, so they must name
real transitions rather than every save of the same role.

The event is raised at the **write**, and only there. Telling the person
their access changed (their inbox line and the mail that goes with it)
is a separate job and does not raise one: when both did, a webhook heard
a grant twice, and heard `account.caps` for an arrival that is
`account.added`.

The payload carries a `summary` — the same sentence the settings page
shows ("added as maintainer", "reader → maintainer", "removed from the
repository") — so a webhook body, the chat robot's line and the settings
trail all say what happened instead of naming the kind.

## What the forge retires on its own

RFC 0001 §8 keeps the OLTP database *small by construction*: it holds
the forge's own bookkeeping, not the repository's history, and it is
litestream-replicated. Two tables broke that by only ever growing, and
both are now swept on the same six-hourly beat (and once at startup):

| table | kept | what goes |
|-------|------|-----------|
| `exception` | 30 days since the last occurrence | a signature nobody has seen again, and its recent hits |
| `ci_run` | 90 days | a finished run that is **not** the newest for its subject |

Neither sweep needs configuring and neither loses anything a page still
reads. Two rules bound the CI sweep:

- An **unfinished** run is never pruned, whatever its age. That row *is*
  the poller's work list, and deleting it strands a run whose verdict
  nobody will ever record. An ancient unfinished row is a bug to find,
  not a row to hide — look for one before assuming the sweep is broken.
- The **newest** finished run per subject survives the window. quarry
  stores only the mapping from run to check-in, and that mapping is what
  makes a verdict explicable: a `ci-fail` badge that cannot name the run
  behind it is a verdict with no evidence. So the badge stays traceable
  for as long as the check-in does, and what the window removes is the
  *retries* — the several runs a check-in accumulates from re-runs and
  supersessions, which is where the growth was. Subject is
  (repo, check-in, pair), so a premerge run (RFC 0009) keeps its own
  newest alongside the check-in's.

Each sweep logs only when it removed something, so a quiet forge stays
quiet in the journal.

## Building on Linux

`scripts/build-linux.sh` is the build for a Linux host, and the one the
VPS deploy uses. It installs the pinned zit release for the host's
architecture (checksum verified against the release's `SHA256SUMS`),
then builds quarry from the tree with the version stamped from the
checkout, and leaves both binaries in `./bin`:

```
ZIT_VERSION=0.3.7 scripts/build-linux.sh
bin/quarry version   # quarry 0.2.0-dev+<checkin>, or quarry 0.2.0 on the check-in tagged version-0.2.0
bin/zit version
```

It builds ON Linux rather than cross-compiling for a reason worth
knowing: the DuckDB bindings ship a static C++ library per platform that
links against the GNU C++ runtime, and a cross-link from macOS with zig
as the C compiler fails on missing `std::string` symbols. Proving the
recipe without a Linux box takes one container:

```
docker run --rm -v "$PWD":/src -w /src "golang:$(sed -n 's/^go //p' go.mod)" scripts/build-linux.sh
```

zit comes from its GitHub release rather than a source build so the
host needs no Zig toolchain, and the pin keeps quarry and zit from
drifting (0001 §2: the zit CLI is the whole store interface).

## With compose

```sh
export NOTIFY_SRC=../notify           # a notify checkout; it builds dagnats too
export ZIT_REF=<commit>               # pin the store implementation
docker compose up --build
```

Then bootstrap notify once (`docker compose run --rm notify bootstrap …`),
put the key in `QUARRY_NOTIFY_KEY`, and create the workflows and topics
as above.

`docker run <image> cat /etc/quarry-build` reports which quarry and which
zit an image carries, without starting it.

> **Unverified.** `Dockerfile` and `compose.yml` have not been built —
> no container runtime is installed on the development machine. The two
> build recipes inside them *are* verified natively: zit compiles from a
> clean clone of the GitHub mirror with `zig build --release=fast`, and
> quarry builds fully static with `CGO_ENABLED=0` (its SQLite is
> WASM-based and Pikchr runs on wazero, so nothing needs a C toolchain).
> What remains untested is the image assembly and the compose wiring.

## Ports

| port | what |
|---|---|
| 4222 | dagnats NATS |
| 8080 | dagnats HTTP bridge (and its console — unauthenticated, keep it on loopback) |
| 3000 | notify API |
| 8950 | quarry |

quarry's own embedded NATS binds nothing (`DontListen`), so it never
collides.

## Working with several agents on one store

Several people — or several agents — committing to one `.zit` is the
ordinary shape here, and 2026-08-29 taught its failure modes in one
afternoon. The rules that keep it sane:

| rule | why |
|---|---|
| **Land through the forge** — `tools/quarry-land`, the landing page, or the API; never `zit landing accept` | required checks, override-with-reason, auto-land and supersede exist only in the forge's write path. A store-level accept walks past all of them, and the store cannot refuse it — twice in one day, red-checked and unformatted work reached trunk that way |
| **Arm auto-land at open** (`quarry-land <repo> <id> --when-green`) | nothing merges red out of impatience, and nothing waits on a human who is actually a busy agent |
| **Prefix branches per effort** (`attention-*`, `ci-*`, `resilience-*`) | supersede-cancel, filters and the eye can all scope to whose work is whose |
| **Commit or stash before leaving a shared checkout** | an uncommitted edit left in a serving tree turned a routine `zit update` into a merge conflict mid-deploy |
| **Deploy only via the sanctioned script**, and never build from a tree another agent may be editing | the deploy script's job is to be the one honest path; a second, casual path is where the conflicts and truncated logs came from. The script now opens trunk into a scratch tree from the STORE and builds there, so no shared checkout is touched and nobody's uncommitted work can collide with a deploy |
| **Announce a deploy window; hold restarts while another session's CI is in flight** | every restart kills the clones its own workers are mid-way through. A quiet restart cost five runs across three sessions in one night |
| **Only the session that opened a landing settles it** | an armed landing whose branch someone else closes as "abandoned" reverts work the owner is still driving, and a rejection is unattributable — every session commits as the same user |
| **Treat the zit binary as part of quarry's behaviour** (see above) | two zit releases in two weeks each silently broke shell-out seams. quarry's CI tests each check-in with the zit it pins (`scripts/ci-zit.sh` reads `ZIT_VERSION` from `scripts/build-linux.sh` and unpacks that release from `ZIT_RELEASE_DIR` on the runner); the `TestZitContract` canary names the next drift |

The forge holds the gates; the store holds the truth; the agreement is
that nobody asks the store to do the gate's job.

## What must survive a reboot

Three long-running things back this forge, and on 2026-08-30 a reboot
proved which of them were actually supervised. `/tmp` does not survive,
so nothing whose life depends on it does either.

| process | supervision | notes |
|---|---|---|
| the dagnats engine | launchd, `com.dmestas.dagnats` | came back on its own |
| the CI worker | launchd, `com.dmestas.quarry-ci-worker` | did NOT, before this: it was hand-run with a `/tmp` work directory, so the wipe took it and CI was dead six hours. A dead worker is invisible from the outside — every run simply reads `running` forever — which is why supervision matters more here than anywhere else. Its work directory now lives beside the binary, not in `/tmp` |
| the forge itself | the deploy script, by hand | the one still unsupervised; a reboot leaves it down until somebody looks |

The worker's log is `quarry-demo/quarry-ci-worker.log`; the forge's is
`quarry-demo/logs/quarry-<date>.log`, appended rather than truncated so
a deploy can no longer destroy the evidence of the outage before it.

## Deploys that say when they fail

`scripts/deploy.sh` (installed as `/srv/quarry/deploy.sh`) is the one
way the forge on zeitforge.app is deployed. Run by hand it still does
what it always did — open trunk from the store, build, install, restart,
roll back if the new binary does not answer, wait for ready — and adds:

| flag | what it does |
|---|---|
| (none) | a deploy, as before |
| `--dry-run` | build and check the binary runs; install nothing, restart nothing, pull nothing |
| `--require-green` | refuse unless every check trunk requires is green on the check-in about to ship — asked of the forge (`checkin.get`'s `required` and `unmet`), never written into the script. A trunk that requires no checks is refused too |
| `--expect HASH` | refuse unless trunk's tip is still `HASH`: the full hash, or a prefix of at least 12 hex characters |
| `--release X.Y.Z` | deploy a published release instead of building trunk: fetch every file `releases/X.Y.Z/SHA256SUMS` lists from the forge's own unversioned files (`http://127.0.0.1:8950/r/quarry/uv/releases/X.Y.Z/<file>`, following the redirect to the bucket), verify every line before anything is unpacked, then install `quarry`, its `zit` and the runner set as a build does. With `--dry-run`, fetch and verify only. Refused beside `--require-green` and `--expect`, which check a trunk build |
| `--rollback` | put `quarry.prev`, `zit.prev` and `runner.prev` back and restart, for when the forge cannot serve a release; alone, never with another flag |

Two deploys never overlap: the script takes `flock` on
`/run/quarry-deploy.lock`, and a second one exits 1 saying since when
the first has run. Every failure — build, health check and rollback,
rollback that did not work, a forge never ready, or the script dying
anywhere else — exits non-zero with one line naming the stage, and
writes the same line to the journal. The runner downloads (`/runner/`)
are published only as a complete set whose `SHA256SUMS` verifies
(`scripts/runner-check.sh`); an incomplete one is not installed, the
forge still deploys, and the deploy ends `FAILED stage=runner`. A
delayed runner restart that cannot be scheduled ends it
`FAILED stage=runner-restart`. `sh scripts/deploy-test.sh` runs every
one of these paths against temp directories and stubs, off the host:

```
journalctl -t quarry-deploy          # START / OK / FAILED stage=<stage> tip=… id=… via=unit|manual
```

The CI runner restarts last, five seconds after the script ends
(`systemd-run --on-active=5`), so a deploy the runner started can
report before its runner goes away.

**Releases and rollback** (RFC 0020). A trunk build is stamped from the
tree's `VERSION` file by `scripts/version-stamp.sh`: `0.2.0` when trunk's
tip carries the tag `version-0.2.0` itself (added to it, propagating or
not, but not inherited from an ancestor: the same rule that starts a
`tag` workflow), else `0.2.0-dev+<tip>`; `quarry version`, the footer
and the bug-report link show that string.

The release layout `--release` reads, under `releases/X.Y.Z/` in the
quarry repository's unversioned files:

| file | holds |
|---|---|
| `SHA256SUMS` | one line per other file, `<sha256>  <name>` as `sha256sum` prints it; names are bare file names |
| `quarry-X.Y.Z-linux-amd64.tar.gz`, `-arm64` | exactly one directory, `quarry-X.Y.Z-linux-<arch>/`, holding `quarry`, `zit`, `CHANGELOG.md` and `LICENSE`; any other shape is refused; `quarry version` must print exactly `quarry X.Y.Z` |
| `quarry-runner-<os>-<arch>[.exe]` | the runner downloads, named as `/runner/` serves them; their lines of `SHA256SUMS` become `/runner/SHA256SUMS`. Every platform `scripts/build-linux.sh` at the tag `version-X.Y.Z` builds must be there (`scripts/runner-check.sh`, read from the store at that tag and run as `quarry`, as a build's set is checked); a set short of it is refused whole |

Any other file listed (the source tarball, the manifest) is fetched and
verified, and not installed. From there `--release` is the build path:
`quarry.prev`, `zit.prev` and `runner.prev` kept, restart, health check,
roll all three back on silence, runner last. Every install, build or
release, keeps the three that ran together just before it, a runner
set it did not replace included, so the `.prev` files are always one
deploy old together. To go back a release, deploy the older one with
`--release`; `--rollback` reinstates `quarry.prev`, `zit.prev` and
`runner.prev` without the forge, once (they are moved back, not copied),
and has nothing to fall back to if that binary does not answer either.

Before fetching anything, `--release X.Y.Z` asks the deploy box's store
which check-in carries `version-X.Y.Z`: exactly one must, labelled
trunk and in trunk's history (trunk's tip or an ancestor of it, as
`zit open trunk` resolves trunk), or the release is refused (a tag the store does not have yet
means the store needs a pull, or the release is not tagged). The tag is
not a gate of its own: anyone who can push can tag a check-in on their
own branch. Trunk's review is the gate, and the check ties the release
to it. The scripts the deploy then reads from that check-in run as
`quarry`, never as root, on a copy of the runner set: the verified
files root installs stay in a root-owned directory they cannot change.

What `--release` trusts. `SHA256SUMS` proves the files arrived as they
were published, not who published them: it comes from the same place
as the files. Whoever can write the quarry repository's unversioned
files (the `y` capability on it) or its bucket can publish a
`releases/X.Y.Z/` whose binaries the deploy, run as root, installs and
runs, as long as a reviewed trunk check-in carries the tag. Grant `y` on
quarry, and the bucket's keys, as you would root on the host.

### Cutting a release

During the beta every release is a patch bump: 0.2.0, 0.2.1, 0.2.2.

1. On a branch, set `VERSION` to `X.Y.Z` and add a `## X.Y.Z` section
   to `CHANGELOG.md` saying what changed; land it.
2. Tag the landed trunk check-in: `zit tag add version-X.Y.Z <check-in>`,
   then push. The tag must be on that check-in itself, not an ancestor.
3. The tag starts `ci/release.yml`, which runs `scripts/release.sh X.Y.Z`
   in the `release` environment. It refuses unless `VERSION`, the
   changelog section and the tag agree and the checkout is clean, then
   builds the linux-amd64 tarball, the five runner downloads and the
   source tarball, writes `SHA256SUMS` and `release-manifest.json` (zit's
   schema 1), checks the sums, and pushes each file to `releases/X.Y.Z/`
   with `zit uv push`, `SHA256SUMS` first and the manifest last. The
   release then shows at `/r/quarry/releases`. linux-arm64 is opt-in
   (`RELEASE_ARM64=1` in the job's env) until its cross build is
   qualified; without it the manifest lists linux-arm64 as omitted.
4. Deploy it: `sudo /srv/quarry/deploy.sh --release X.Y.Z`.

A publish that failed part-way is finished by cancelling the tag and
adding it again: files the forge already holds are skipped. Once the
manifest is published the release is done: a re-run whose check-in the
manifest names publishes nothing, and one from any other check-in is
refused. `sh scripts/release.sh --dry-run X.Y.Z` in a clean checkout of
the tagged check-in builds everything into `$TMPDIR/quarry-release-X.Y.Z`
(or `--out DIR`, which must be new or empty) and publishes nothing.
That directory stays after a local dry run or a failed publish, and the
next run refuses it until it is removed: `rm -rf
"$TMPDIR/quarry-release-X.Y.Z"` before retrying by hand. CI gets a fresh
`TMPDIR` for every job, so a retried tag never meets it. The
run token never reaches the build: release.sh takes `QUARRY_TOKEN` out of
the environment first and hands it only to `zit uv` as `ZIT_PASSWORD`.

The job publishes with a run token (RFC 0021), not an account. Its
`permissions:` asks for `uv: write` (`y`); the `release` environment
allows only that, takes trunk only, and requires the person who pushed
the tag to hold `a` on quarry. Someone without `a` can tag; the build
runs and cannot publish. An admin sets the environment once:

```
curl -fsS -u "$QUARRY_ADMIN:$TOKEN" -H 'Content-Type: application/json' \
  -X POST https://zeitforge.app/api/v1/r/quarry/ci/environments \
  -d '{"name":"release","branches":["trunk"],"allowed":"y","starter":"a"}'
```

That is op `ci.environment.set` (`a` on quarry), the same as the
Environments rows in Settings.

The token reaches a job only on a runner started with `--job-users`.
`quarry-runner.service` runs the test pipelines as itself, so releases
get their own runner, `scripts/systemd/quarry-runner-release.service`:
the only runner polling `quarry.job` (every workflow job of quarry has
that task type), as the pool user `qrel1`, for trunk check-ins only,
with a zit checkout so the job can read its tags. Its header lists the
one-time setup: the two system users, the sudoers line, a new worker
token, and, for linux-arm64, the `gcc-aarch64-linux-gnu` and
`g++-aarch64-linux-gnu` packages. Any other `ci/*.yml` workflow added to quarry would also
run on this runner, under these limits.

### Installing the units

The units live in `scripts/systemd/`, the scripts beside `deploy.sh`.
Alerts go through the notify companion quarry already uses, as the
`quarry-ops` workflow (inbox and push, to the `admins` topic; quarry
provisions it at startup). The notify key is `QUARRY_NOTIFY_KEY` in
doppler `quarry/prd`, the same one quarry.service reads, reached with
the service token in `/srv/quarry/env`. From a checkout of trunk:

```
sudo install -m 0755 scripts/deploy.sh scripts/quarry-alert.sh scripts/quarry-deploy-watch.sh /srv/quarry/
sudo install -m 0644 scripts/systemd/[email protected] scripts/systemd/[email protected] \
  scripts/systemd/quarry-deploy-watch.service scripts/systemd/quarry-deploy-watch.timer /etc/systemd/system/
sudo install -d /etc/systemd/system/quarry.service.d
sudo install -m 0644 scripts/systemd/quarry.service.d/on-failure.conf /etc/systemd/system/quarry.service.d/
sudo systemctl daemon-reload
sudo systemctl enable --now quarry-deploy-watch.timer
```

Then prove the alert path once, without breaking anything:

```
sudo sh -c 'set -a; . /srv/quarry/env; doppler run --project quarry --config prd -- \
  /srv/quarry/quarry-alert.sh --key test:$(date +%s) --message "alert path test"'
```

That proves the script from a shell, which has a `$HOME`; a unit does
not get one unless it names a `User=`, and doppler fails without it.
Prove the units themselves the way systemd runs them:

```
sudo systemctl start quarry-deploy-watch.service
systemctl show quarry-deploy-watch.service -p Result      # Result=success
sudo systemctl start [email protected]   # sends one "install-check failed" alert
systemctl show [email protected] -p Result
```

| unit | what it is for |
|---|---|
| `quarry-deploy@<hash>.service` | a detached deploy: `systemctl start --no-block quarry-deploy@<hash>` runs `deploy.sh --require-green --expect <hash>`. Fails into an alert (`OnFailure=`) |
| `quarry-alert@<unit>.service` | sends "<unit> failed" with that run's FAILED line, or its last log lines. At most one per unit per ten minutes, so a crash loop is one alert |
| `quarry.service.d/on-failure.conf` | the forge crashing alerts the same way — on every crash systemd restarts (RestartMode=normal passes through failed), rate-limited to one per ten minutes |
| `quarry-deploy-watch.timer` | every five minutes: a deploy holding the lock over ten minutes; a deploy that ended mid-way (killed, terminal gone, a rollback begun) with no result line; a FAILED line from a deploy run by hand. Each alerts once |