Running quarry

Two ways: the compose stack, or by hand. The hand-rolled path is what this was developed against and is what the compose file encodes, so it is written down first — a run book that stays true even if the images are never built.

What the stack is

piece	role	state
zit	the store. quarry shells out to it for every read and write (0001 §2)	the .zit files themselves
quarry	the forge: indexes, UI, API, sync fronting	derived indexes, embedded NATS, OLTP database
notify	notifications: workflows, subscribers, topics, the inbox quarry embeds	SQLite
dagnats	the workflow engine notify runs on: embedded NATS + JetStream + an HTTP bridge	JetStream data

zit is not a runtime dependency in the ordinary sense — it is quarry's only store interface, so its version is part of quarry's behaviour. On 2026-08-16 three zit builds changed what quarry does: frames got 31% faster, a card-parsing fix changed which artifacts parse at all (and until it landed, sqlite's 794 tickets were invisible), and a new stat -R let quarry serve a bare store. Pin the pair; do not let a deploy resolve zit from whatever is on PATH.

By hand

dagnats

Runs under launchd on this machine as com.dmestas.dagnats, binding NATS on 4222 and its bridge on 8080, with data in ~/Library/Application Support/dagnats.

launchctl kickstart -k gui/$(id -u)/com.dmestas.dagnats

A freshly-built Go binary replacing it must be ad-hoc signed or launchd kills it with OS_REASON_CODESIGNING and the service silently fails to come back:

go build -o ~/go/bin/dagnats ./cmd/dagnats
codesign -s - -f ~/go/bin/dagnats
launchctl kickstart -k gui/$(id -u)/com.dmestas.dagnats

In standalone mode dagnats binds loopback, which is why anything talking to it shares its network namespace under compose rather than reaching it by service name.

notify

notify bootstrap -dev -db notify.db -org quarry -env dev   # prints apiKey once
notify serve -dev -db notify.db -addr 127.0.0.1:3000 \
  -nats-url nats://127.0.0.1:4222 -bridge-url http://127.0.0.1:8080
curl -s localhost:3000/readyz     # {"status":"ok","bridge":true,"db":true}

-dev derives a fixed, publicly-known master key. Outside development set NOTIFY_MASTER_KEY_HEX.

Provisioning is quarry's job, not an operator's (RFC 0011 §7). What used to be a page of curl here is now code: quarry creates a notify subscriber when an account is added, adds or removes the account from a repository's repo-<name> activity topic when it is granted, revoked, or subscribes to a thread, and creates its workflows (quarry-mention, quarry-forum-reply, quarry-activity, quarry-landing, quarry-changelog, quarry-error, quarry-slow, quarry-test — defined in internal/notify/workflows.go) at startup, leaving any workflow an operator already edited untouched. quarry-landing is provisioned ahead of need: nothing triggers it yet, that lands with the pull request review surface. All of it is best-effort: a notify outage never fails the account or grant op that raised it.

Scripts are compacted at start, not at build. quarry's own scripts are minified in memory when the process starts (the stylesheet has its comments and indentation stripped the same way), then served under content-hashed, immutable names; the sources in the repository stay readable and there is no build step to run. make wire URL=https://your-forge/ prints what a first visit costs per file. Set QUARRY_ASSETS_RAW=1 to serve the sources as written while editing them.

Email needs a provider, and quarry gives notify its own. A workflow with an email step and no mail provider behind it fails where only somebody reading notify's database would find it: store: not found, per message, while the settings page goes on offering an email column. So if quarry has a mailer — RESEND_API_KEY and QUARRY_MAIL_FROM, the same pair that sends invites — it hands notify the same credential at startup and logs notify: email now sends as …. One credential, in one place, and nothing to POST by hand.

Email is off until a person switches it on. The four family workflows carry an email step (plain text, one link; mention, reply and landing accept replies), but notify reads an absent preference as enabled, so a step alone would mail every account about everything. At every start, and when an account is created, quarry writes an explicit "no email" for each family the person has not answered, and never touches a cell they set themselves. The settings page is where they answer.

An email integration notify already holds is left exactly as it is, so configuring SMTP or SES in notify's own console is still the way to use a different provider from the one quarry sends invites with — a restart will not replace it. With no RESEND_API_KEY quarry asks the companion for nothing rather than creating an integration that cannot send.

Invitations are the exception that proves the split: they are sent by quarry's own mailer and are not a notify workflow, because an invitee has no account, no subscriber and no inbox for any other channel to reach — and a forge that sent them both ways delivered two invitations from one address.

Receiving email needs nothing but notify. A reply to a notification email comes back to reply+<token>@<inbound domain>, and notify can receive it itself (notify#340): set NOTIFY_SMTP_ADDR=0.0.0.0:25 and NOTIFY_INBOUND_MX_HOST=mail.example.org, let the service bind a privileged port, and point the domain's MX at the box. On a systemd host that is one drop-in:

# /etc/systemd/system/notify.service.d/smtp.conf
[Service]
Environment=NOTIFY_SMTP_ADDR=0.0.0.0:25
Environment=NOTIFY_INBOUND_MX_HOST=mail.example.org
AmbientCapabilities=CAP_NET_BIND_SERVICE

and two records — mail.example.org A <the box> (not proxied: a CDN does not carry SMTP) and mail.example.org MX 10 mail.example.org. Register the domain with POST /v1/inbound/domains {"name":…} and POST /v1/inbound/domains/{name}/verify; verification is a live MX lookup, so it fails until the records have propagated and can simply be repeated. This is not a mail server: an address without a valid reply token is refused at RCPT TO with 550 5.1.1, before any message body is read, so there is no inbox to spam and nothing to filter. It speaks plaintext today; a sending MTA that offers STARTTLS falls back cleanly.

A host that already runs an MTA can pipe instead of listening: notify inbound < message.eml reads one raw message on stdin and hands it to the same path, so an /etc/aliases line does the job without a second port.

Upgrading a forge provisioned before RFC 0011 Phase 3. The workflows now carry more than one step each: quarry-mention, quarry-forum-reply and quarry-landing gained a push step after their in-app one, and quarry-activity gained a 15-minute digest before its in-app step (RFC 0011 §6). Provisioning upserts with notify#303's PUT /v1/workflows/{slug}?onConflict=keep, which creates a missing workflow and leaves an existing one exactly as it is — an operator may have edited its copy, and keep is what protects that edit from the next restart. So an existing install keeps the single-step definitions it was provisioned with. A forge provisioned for the first time needs nothing.

To pick the new definitions up, delete those four workflows once and restart quarry; the next startup recreates them from internal/notify/workflows.go. The {slug} is the trigger identifier, not the workflow's ULID:

K=...   # the notify apiKey from bootstrap
for w in quarry-mention quarry-forum-reply quarry-landing quarry-activity; do
  curl -fsS -X DELETE -H "Authorization: ApiKey $K" \
    http://127.0.0.1:3000/v1/workflows/$w
done
# then restart quarry; startup recreates all six.

Deleting a workflow deletes nothing a person can see — messages already delivered stay in their inbox. Do it during a quiet moment anyway: a trigger fired between the delete and the restart reaches nobody.

What a companion that does not answer costs. The preferences pair (GET|PUT /v1/subscribers/{id}/preferences) and the per-device credential DELETE have shipped, so both halves normally land. When notify is unreachable quarry says so rather than pretending:

• me.prefs.set writes its forge-local half first (the toast and desktop switches, the per-repository mute) and reports applied.notify: false when the notify half could not be stored. The channel matrix a person set is then not yet in force. A read that notify did not answer marks every family unknown and the matrix is left ALONE on the next save, rather than writing quarry's defaults over settings it could not read.
• me.push.unsubscribe keeps the device row marked pending_remote_delete when notify FAILED the delete, instead of forgetting a browser notify still pushes to, and retries on the next me.push.subscribe. A 404 is not a failure — a credential notify does not hold is the end state unsubscribe asked for — so the row is forgotten there and then. The push_device.pending_remote_delete column is now only ever a transient state between a failed delete and its retry.

Both clear themselves once notify answers — no migration, no reset.

quarry

quarry serve --addr 127.0.0.1:8950 --data ./data \
  --notify http://127.0.0.1:3000 --notify-key "$K" \
  --repo quarry=$HOME/quarry

--data is not optional in practice: it defaults to .quarry beside the first --repo, which writes derived state into whichever repository happens to be listed first. Always pass it.

Gates are open until they are closed, and closing them is two separate credentials on purpose — accounts are the forge identity (argon2id), the sync secret is the RFC §6.3 value the protocol fixes:

quarry account add --user you --admin --data ./data   # closes web/API writes
quarry user add --repo name=$HOME/quarry --user you --data ./data   # closes push

The account gate is global: the first account closes writes on every repository in the fleet, not just one.

The attention stream and HTTP/2

Signed-in pages open one Server-Sent Events connection for the attention stream — the inbox badge and the count in the tab title (RFC 0011 §5). In-page banners are off by default and are turned on per account on /me/notifications, which is also where desktop notifications are opted into; with both off the stream still moves the badge and the title, and nothing ever covers the page being read. On plain HTTP/1.1 a browser allows six connections per origin, and a repository page already spends one or two of them on the chat and timeline streams. So the default opens the attention stream on repository pages only: a stream on every page would compete with the page it sits on for the same six slots, and the symptom is a page that stalls halfway through loading rather than an error anybody can read.

With HTTP/2 in front there is no such limit — one connection carries every stream — so the whole forge can hold it open:

quarry serve ... --attention-everywhere

Use it when quarry is behind TLS termination that negotiates h2, or behind a proxy speaking h2c to a browser. quarry serves plain HTTP/1.1 itself, so the flag is a statement about what is IN FRONT of it and quarry cannot check it: turned on without h2, it is the six-connection problem on every page.

Nothing else changes. Without the stream a page still renders its own state and the inbox badge is fetched once after load, which is the Principle 3 behaviour every one of these surfaces keeps.

Push

/me/notifications carries the whole of RFC 0011 §6: the family x channel matrix (mention, reply, landing, activity, by in-app, push and email), the two forge-local switches, the repositories this account muted, and Push on this device.

Push is the one feature in quarry that needs JavaScript, recorded as an explicit exception in RFC 0011 §11. Enabling it registers /sw.js, asks the browser for notification permission from the click that asked for push, subscribes with notify's VAPID key, and posts the resulting PushSubscription to POST /me/push. With JavaScript off the control says so and nothing else on the page changes. On iOS a site must be added to the Home Screen before Safari will grant push at all — the control says that too.

Quarry never holds a VAPID key and never stores an endpoint: the subscription goes to notify as subscriber credentials, and quarry keeps a hash, a user-agent label and a timestamp so the page can list the browsers somebody may want to turn off. That is also why a browser other than the one being used is turned off from that browser.

What to expect until the upstream work lands:

• notify#302 (the webpush provider) — subscriptions are accepted and stored, but nothing is delivered by push yet. The in-app inbox, the badge and the attention stream are unaffected.

notify#305 (the preferences route, the per-device credential DELETE and the trigger identifier on every message row) has shipped. When the companion is unreachable the banner after saving preferences says the companion did not answer and the page reports applied.notify honestly; a device whose delete failed shows removal pending and is retried the next time push is enabled.

Neither needs a migration or a reset on this side.

The activity robot

A repository can have its timeline talk in its chat channel (RFC 0011 §10): settings → Activity robot, in the same event-pattern syntax webhooks take (*, a noun like checkin, or an exact kind like landing.accepted). Empty turns it off. Robot lines never count as unread and never mention anybody.

Fleet admin

/fleet is the operator page — every store's ingest health, and the upstreams it mirrors. It is site admin only: admin on one repository is not enough, because the page reports operational detail across the whole fleet and can force writes into any of it.

Health is per-process. Total and Fallbacks count what THIS server has ingested since it started, so a restart with a warm index reports zero of both however much the store holds. The parse-fallback number worth looking at is the one a quarry reindex produces.

Mirroring

A mirrored store tracks an upstream: quarry pulls, and never pushes. The four postures a repository can be in, and what each one opens or refuses, are laid out in hosting.md.

curl -u you:pw -X POST http://localhost:8080/api/v1/mirrors \
  -H 'Content-Type: application/json' \
  -d '{"repo":"fossil","url":"https://fossil-scm.org/home","every":3600}'

every is seconds and is floored at 60 — these upstreams are other people's servers, and the floor is a courtesy the configuration surface cannot opt out of. --mirror-tick (default 1m) is how often due-ness is evaluated, not how often anything is pulled.

Pulls take the store's single-writer lease, so mirroring is safe with more than one node serving the same store. A pull that brings nothing new does not trigger a re-index and is not logged; almost every one is quiet, and a line per store per interval buries the ones that matter.

Two limits worth knowing before you rely on it:

• The store needs a working checkout. zit pull refuses -R and operates on a working directory, so a genuinely bare store cannot be mirrored. Stores created by zit clone or zit import-fossil carry one, so this bites only hand-built bare stores.
• The upstream must allow anonymous clone. Some Fossil servers serve a browsable timeline while refusing sync — sqlite.org/lemon is one. Note that fossil clone exits 0 on that refusal, so check for the repository, not the exit status.

Attachments: binding a bucket

Files uploaded through quarry go to a bucket the repository's steward brings (RFC 0016); the forge keeps a stub in the repository and never the bytes. A steward binds one bucket per repository from its settings page, under Attachments, or through the op:

curl -u you:pw -X POST http://localhost:8080/api/v1/r/fossil/bucket \
  -H 'Content-Type: application/json' \
  -d '{"endpoint":"https://<account>.r2.cloudflarestorage.com","bucket":"fossil-files",
       "region":"auto","accessKey":"...","secret":"...","publicBase":"https://files.example.org"}'

At R2, create the bucket, then an API token scoped to that bucket with object read and write; the region is auto. A public or custom domain on the bucket is optional — set it as the public base and downloads link there; without it every download is presigned. Any S3 endpoint works the same way with its own region.

The binding is probed before it is kept — a presigned PUT and DELETE of one small object under attachments/ — so a wrong key or a token without delete is refused on the form. Probe again from the page or with bucket.probe; Fleet shows each store's last probe beside its mirror's last pull. The secret is sealed under secret.key beside oltp.db (created 0600 on first use) and never returned by any op; a forge that loses the key file has the steward enter the secret again.

Phase 3 will PUT objects that may never be committed, so plan a lifecycle rule on the bucket that deletes attachments/ objects a day old — quarry's own writes are complete or absent within the hour.

Fossil chat bridge

An opt-in connector between one repository's chat channel and one Fossil server's chat (RFC 0011 §9). It is configured per store, from /fleet or through the op:

curl -u you:pw -X POST http://localhost:8080/api/v1/r/fossil/mirror/chatbridge \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://fossil-scm.org/home","user":"quarrybot","password":"..."}'

Detach with {"detach":true}. The password is stored and is never returned by any op or rendered on any page; the only reader is the poller.

The upstream needs two things, neither of them a default:

1. A login holding fossil's C capability. Chat is gated on it.
2. http_authentication_ok in the repository's config, or fossil refuses HTTP Basic and every request is anonymous. It is not a fossil setting, so it goes in directly:

   fossil sql -R /path/to/repo.fossil \
     "INSERT OR REPLACE INTO config(name,value,mtime) \
      VALUES('http_authentication_ok','1',now());"

Quarry polls /chat-poll with a nine-minute client timeout (fossil hangs the request for chat-poll-timeout, 420 s by default) and backs off from 30 s to 5 minutes while an upstream is failing. Every request carries a same-origin Referer: without one fossil treats the poll as CSRF-unsafe and refuses its own table setup, which surfaces as a database error rather than as an authentication failure.

What it does. Messages from the upstream arrive in the channel as fossil:<user> with Origin: fossil. A fossil deletion becomes a tombstone on the message it removed. An attachment becomes a link to the upstream's /chat-download — fossil keeps the file. Quarry's own messages go out to /chat-send prefixed **name:**, because fossil attributes everything to the login the bridge signs in as. The bridge never forwards robot lines or lines that came from fossil, and never re-imports its own login's messages.

What it does not do.

• No edits. A fossil message cannot be edited from quarry, and a quarry edit is not carried upstream. Only deletions cross, and only in the fossil-to-quarry direction.
• No mentions. Fossil chat has none, so a bridged line raises nobody's badge and triggers no notification.
• No account mapping. fossil:drh is a name from another server's user table, not a quarry account, and the prefix says so.
• Two Fossil servers are two chats. Fossil's chat is a server-local table that never syncs, so bridging a mirror connects you to that ONE server's conversation, not to the project's.
• The window is still quarry's. A bridged channel holds seven days like every other; the bridge is a connector, not an import.

When this repository's activity robot is on, the upstream's chat-timeline-user announcements are skipped — otherwise every check-in is announced twice, once by each robot.

Run the bridge on one node. Unlike a mirror pull, which takes the store's single-writer lease, the bridge holds no fleet-wide lock: every node that serves the repository and sees the row starts its own poller. Two nodes bridging the same repository means each upstream message is imported twice and each quarry message is sent upstream twice. Until there is an advisory lock for it, bridge from one node — in a multi-node deployment, configure the bridge and run --allow-private-bridges-style node-specific configuration on exactly one of them, or keep the row out of the shared OLTP on the others. Restarting a node is safe: the msgid map makes a re-read of the same message a no-op, so a cursor that has not caught up does not double anything.

Private addresses are refused by default. The bridge is a server-side fetcher holding a credential, so it will not connect to a loopback, private or link-local address — that last range is where cloud metadata services live. The check runs on the resolved address at connect time, so a public name pointing inward is refused too. Bridging a fossil on the same host is an opt-in:

quarry serve --allow-private-bridges ...

Redirects are refused, not followed. Go re-sends the Authorization header across a same-host redirect, and here that header is the fossil password. Configure the final URL. A URL carrying credentials (https://user:pass@host/) is rejected outright: the user and password are separate fields, and a URL is copied into log lines and status rows.

Building on Linux

scripts/build-linux.sh is the build for a Linux host, and the one the VPS deploy uses. It installs the pinned zit release for the host's architecture (checksum verified against the release's SHA256SUMS), then builds quarry from the tree with the version stamped from the checkout, and leaves both binaries in ./bin:

ZIT_VERSION=0.3.7 scripts/build-linux.sh
bin/quarry version   # quarry 0.1.0-beta+<checkin>
bin/zit version

It builds ON Linux rather than cross-compiling for a reason worth knowing: the DuckDB bindings ship a static C++ library per platform that links against the GNU C++ runtime, and a cross-link from macOS with zig as the C compiler fails on missing std::string symbols. Proving the recipe without a Linux box takes one container:

docker run --rm -v "$PWD":/src -w /src golang:1.26 scripts/build-linux.sh

zit comes from its GitHub release rather than a source build so the host needs no Zig toolchain, and the pin keeps quarry and zit from drifting (0001 §2: the zit CLI is the whole store interface).

With compose

export NOTIFY_SRC=../notify           # a notify checkout; it builds dagnats too
export ZIT_REF=<commit>               # pin the store implementation
docker compose up --build

Then bootstrap notify once (docker compose run --rm notify bootstrap …), put the key in QUARRY_NOTIFY_KEY, and create the workflows and topics as above.

docker run <image> cat /etc/quarry-build reports which quarry and which zit an image carries, without starting it.

Unverified. Dockerfile and compose.yml have not been built — no container runtime is installed on the development machine. The two build recipes inside them are verified natively: zit compiles from a clean clone of the GitHub mirror with zig build --release=fast, and quarry builds fully static with CGO_ENABLED=0 (its SQLite is WASM-based and Pikchr runs on wazero, so nothing needs a C toolchain). What remains untested is the image assembly and the compose wiring.

Ports

port	what
4222	dagnats NATS
8080	dagnats HTTP bridge (and its console — unauthenticated, keep it on loopback)
3000	notify API
8950	quarry

quarry's own embedded NATS binds nothing (DontListen), so it never collides.

Working with several agents on one store

Several people — or several agents — committing to one .zit is the ordinary shape here, and 2026-08-29 taught its failure modes in one afternoon. The rules that keep it sane:

rule	why
Land through the forge — tools/quarry-land, the landing page, or the API; never zit landing accept	required checks, override-with-reason, auto-land and supersede exist only in the forge's write path. A store-level accept walks past all of them, and the store cannot refuse it — twice in one day, red-checked and unformatted work reached trunk that way
Arm auto-land at open (quarry-land <repo> <id> --when-green)	nothing merges red out of impatience, and nothing waits on a human who is actually a busy agent
Prefix branches per effort (attention-*, ci-*, resilience-*)	supersede-cancel, filters and the eye can all scope to whose work is whose
Commit or stash before leaving a shared checkout	an uncommitted edit left in a serving tree turned a routine zit update into a merge conflict mid-deploy
Deploy only via the sanctioned script, and never build from a tree another agent may be editing	the deploy script's job is to be the one honest path; a second, casual path is where the conflicts and truncated logs came from. The script now opens trunk into a scratch tree from the STORE and builds there, so no shared checkout is touched and nobody's uncommitted work can collide with a deploy
Announce a deploy window; hold restarts while another session's CI is in flight	every restart kills the clones its own workers are mid-way through. A quiet restart cost five runs across three sessions in one night
Only the session that opened a landing settles it	an armed landing whose branch someone else closes as "abandoned" reverts work the owner is still driving, and a rejection is unattributable — every session commits as the same user
Treat the zit binary as part of quarry's behaviour (see above)	two zit releases in two weeks each silently broke shell-out seams. The CI worker runs a pinned zit (-zit); the TestZitContract canary names the next drift

The forge holds the gates; the store holds the truth; the agreement is that nobody asks the store to do the gate's job.

What must survive a reboot

Three long-running things back this forge, and on 2026-08-30 a reboot proved which of them were actually supervised. /tmp does not survive, so nothing whose life depends on it does either.

process	supervision	notes
the dagnats engine	launchd, com.dmestas.dagnats	came back on its own
the CI worker	launchd, com.dmestas.quarry-ci-worker	did NOT, before this: it was hand-run with a /tmp work directory, so the wipe took it and CI was dead six hours. A dead worker is invisible from the outside — every run simply reads running forever — which is why supervision matters more here than anywhere else. Its work directory now lives beside the binary, not in /tmp
the forge itself	the deploy script, by hand	the one still unsupervised; a reboot leaves it down until somebody looks

The worker's log is quarry-demo/quarry-ci-worker.log; the forge's is quarry-demo/logs/quarry-<date>.log, appended rather than truncated so a deploy can no longer destroy the evidence of the outage before it.