Skip to content
quarry Documentation
Login

Documentation

The working name is provisional and appears nowhere in a wire format, storage layout, or config key that would make renaming expensive; a rename is a find-replace on this document and the repo name.

Summary

quarry is a separate binary that serves zit repositories the way Fossil's built-in server serves Fossil repositories — timelines, file browsing, diffs, rendered wiki and documentation, full-text search, per-repository chat, CI/CD status, and multi-repository management — without putting any of that inside zit. zit stays the store and the verbs; quarry owns presentation, indexing, events, and fleet management, and holds no hostage bytes: every piece of durable content lives in zit artifact DAGs (spec'd by the Fossil RFC suite) or in S3, and every quarry index is derived, disposable, and rebuildable.

It follows the dagnats/notify operational model exactly: quarry serve is one process — embedded NATS JetStream, embedded SQLite indexes, the ingest workers, the HTTP surface — with a defined path to horizontal scale by pointing the same binary at external NATS and shared S3-backed storage. The scale posture in one sentence: NATS moves things, derived indexes answer things, S3 is the only thing that is true.

Motivation

Fossil's 101 reasons, minus Fossil's coupling

Roughly 60 of Fossil's "101 reasons" are the forge: web timeline, wiki, tickets, forum, chat, multi-repo serving, skins. Fossil buys them with coupling — rendering, TH1 scripting, skins, user tables, and server config live inside the repository file and the single binary. That coupling is the lock-in: you cannot scale the forge independently of the store, cannot put a different UI on the same data, cannot share accounts across repositories without login groups, and the repository carries presentation state it never needed to carry.

The factoring quarry asserts: Fossil's one genuinely great forge idea is that content is synced artifacts — wiki pages, tickets, technotes, and forum posts are DAG artifacts that clone with the repository. zit already parses every one of these types (ArtifactType: wiki, ticket, event, attachment, forum). quarry keeps that idea and evicts everything else — rendering, search, auth, fleet state — to a replaceable layer outside the store.

The property that makes this cheap

zit's store is an append-only log with a validated-prefix discipline. That makes the store its own event stream: a watcher that remembers its last offset validates the prefix and folds only the new tail — the same pattern repocache extension and the checkpoint already use inside zit. quarry's entire ingest layer is that pattern, one level up, fanned out over NATS. No hook framework, no polling, no post-receive machinery, and nothing added to zit core. Fossil (SQLite pages) and git (unordered object arrival) both lack this property; it is the single biggest architectural gift the log design gives a forge.

Why a separate binary and not zit verbs

The smol contract: zit is a program that can be finished. A forge is a program that is never finished. Coupling them dooms the first to the second's lifecycle. Separate binaries also mean quarry can carry heavyweight dependencies (NATS, embedded indexes, web assets) that would be absurd inside zit, and can be replaced wholesale — the anti-lock-in guarantee is only credible if the forge is killable.

Guide-level explanation

Single node

quarry serve --data /srv/quarry --repos /srv/zits

One process. Embedded NATS with JetStream, embedded SQLite indexes, a watcher per registered store, the web UI, and sync fronting on one port. Registering a repository is pointing quarry at a directory containing .zit (or letting it discover a tree of them). Everything quarry computes lives under --data and may be deleted at any time; the next start re-ingests from the stores.

What a user sees

The one rule that keeps quarry honest

Delete quarry's data directory, re-ingest every store, and the rendered output must be identical.

This is enforced by a standing e2e test (§reference), not by intention. Any feature whose data would not survive that test either writes artifacts into a zit DAG, writes to S3 through a component with an S3 durability story, or does not ship.

Scale-out

The same binary, three flags different: external NATS URL, shared store access, S3 lake bucket. N quarry nodes serve reads statelessly, each from its own SQLite indexes folded off the shared event stream; each store has exactly one writer at a time (a NATS-leased role, not a node identity). Scale-out is also where the Parquet interchange lake enters: a worker on the same event stream writes append-only Parquet to S3 so new nodes hydrate their indexes from the lake instead of replaying every store, and analytics engines attach to it zero-ETL. Losing any node loses nothing; losing NATS pauses events but corrupts nothing; only S3 holds truth.

Principles

Normative. Every feature is reviewed against these; a PR that violates one either changes or carries an explicit, recorded exception.

  1. Standards-compliant HTTP, self-identifying pages. Every view is valid HTML served with plain HTTP semantics: real URLs, GET is safe and cacheable where true, POST mutates, status codes mean what they say. Every page footer shows the quarry version, the zit version in use, and render timing — the Fossil/Forgejo convention.
  2. Server-owned state, hypermedia wire. As much state as possible lives on the server; the client receives hypermedia, not application state. Datastar is the only framework allowed by default; where custom elements are warranted, web components over framework components, and they enhance server-rendered light DOM — no shadow-DOM-only content, no client-side templating. Exceptions for genuinely great UX are possible but are made explicitly, case by case, never by drift.
  3. JavaScript is enhancement, not foundation. Every view remains usable with JavaScript disabled: it renders its current state on load, and core reads and writes work through plain forms and links. Live features (chat, CI status) degrade to load-time state plus refresh — ugly is acceptable, broken is not. Datastar makes pages live; it never makes them possible.
  4. Performance over beauty. Server-rendered, small pages, fast first byte. Visual polish is welcome only when it costs nothing measurable.
  5. API-complete features. A feature is not done until its full capability is exposed through a documented API — sufficient that a complete quarry client could be built from the API spec alone, with zero screen-scraping. Parity is enforced structurally, not by serialization: the HTML UI and the API share one internal query/command layer, and per-feature definition of done includes the API surface (Fossil's /json is the precedent shape; the ops layer, §9, is the mechanism).
  6. Standard async surfaces; internal machinery stays internal. Every asynchronous behavior (events, chat, CI status, ingest progress) is consumable by third-party programs through open standards — webhooks, SSE, websockets — such that a third-party actions runner, chat client, or diff tool is buildable without forking quarry. NATS and dagnats are quarry's internal implementation; no user or integrator is ever required to speak them. (Integrators who want the NATS subjects may use them, but the supported, versioned contract is the standard surface.)
  7. Exit is always possible. Data leaves via zit clone (no hostage bytes, §11); clients live via the API spec (principle 5); events flow via standard surfaces (principle 6). Anything that would close an exit does not ship.

Reference-level explanation

1. Planes and their truth status

Plane Component Holds Truth?
Storage S3 via ZeroFS (9P) or JuiceFS; local disk for dev zit stores (.zit logs) YES — the only truth
Storage S3 directly litestream replicas; Parquet interchange lake (scale-out) Derived / replicated
Event NATS JetStream (embedded or external) streams, KV (leases, cursors), chat transport Never — everything rederivable from stores
Index SQLite per node: structured tables + FTS5 timeline, file, ticket, forum tables; full-text search Derived, disposable
Execution dagnats CI/CD DAG runs, ingest pipelines Event-sourced in JetStream; verdicts that matter are written back as artifacts
Notification notify channels, digests, inbox Its own RFC's story (SQLite + litestream)
Forge OLTP SQLite + litestream → S3 accounts, sessions, repo registry, ACLs (v2) Authoritative but tiny; replicated to S3 continuously

Every guarantee quarry makes traces to the right column: user data → zit DAG → S3; index → rebuildable from stores; coordination → refusable (see leases).

Two framing notes on this table. First, the event and execution planes are one dependency, not three: notify runs on dagnats, and dagnats runs on NATS — adopting NATS gets the whole stack, and there is no configuration in which one appears without its base. Second, the table's pattern — not its product names — is the commitment: storage separated from compute, the way MotherDuck separates DuckDB from its lake and turbopuffer separates its index from S3. S3 holds truth; every compute layer attaches to it, is derived from it, and is swappable without a data migration. Named products in the derived rows are current best choices, not architectural bets.

2. Store access and the storage matrix

quarry accesses stores only through the zit binary — a subprocess boundary, never linked code and never direct file reads. zit is Zig and quarry is not; more importantly, the process boundary makes the versioned zit CLI the whole contract between the two, so releases decouple and the .zit container format stays private to zit. That format is deliberately not specified by this RFC and never will be: portability lives at the artifact-format and sync-protocol layer (already RFC'd), and the escape hatch from any forge is zit clone, not file copying. Speccing the file would freeze what repocache-class work needs free.

Reads work through zit read verbs (tail frames since a cursor for the watcher; file-at-checkin, diff, annotate on demand). Most page data is served from quarry's own indexes, so on-demand subprocess reads sit off the hot path and behind caching; if a profiled hot path ever demands it, the named escape hatch is a C-ABI read library exported from zit — a decision to revisit with evidence, not a default.

The read-verb contract needs no invented schema for artifact-bearing reads: the watcher verb emits raw artifacts in the already-RFC'd artifact format, plus a cursor and a prefix-validation answer. zit's job at this boundary is transport out of the private container (resolving delta chains to whole artifacts); interpretation is quarry's, against the public artifact-format RFC — which quarry's ingest must implement anyway to index manifests, wiki, and tickets. The container layout stays private; the artifact format stays the one machine contract; the CLI is merely how bytes cross the process line.

Writes to a store happen only by invoking zit operations (commit, sync receive, artifact mint) — quarry never writes store bytes itself.

Backend Mount Verdict
Local disk — Dev and small deployments. Full guarantees.
ZeroFS over 9P one ZeroFS server per bucket; nodes mount over network 9P Supported, primary S3 path. 9P is required: over ZeroFS's NFS mount, fsync may return before data reaches S3, which silently voids zit's log.commit() durability contract. Over 9P, fsync returns after object storage acknowledges — the contract holds. One ZeroFS process owns a bucket; quarry treats that process as part of the storage plane (supervised, restart = remount). Never point two ZeroFS instances at one bucket.
JuiceFS shared POSIX mount, metadata engine Supported alternative. Atomic rename via transactional metadata; flock and traditional POSIX record locks supported; open-after-unlink holds on the same mount — the pack-swap discipline works. Brings a metadata service (Redis/SQL) as an extra failure domain, which is why ZeroFS-9P is primary.
ZeroFS NFS — Refused for stores (fsync hazard above). Acceptable for the Parquet lake, which is derived.
ZeroFS NBD block device Single-client by nature; fine for a one-node deployment that wants a real local filesystem on S3.

Two lessons inherited from zit issues apply to every quarry reader: zit#17 — pack/shun swaps the store inode; ingest watchers stat the path and reopen between rounds, never trusting a held handle across rounds; and zit#59 — the append-append flock is required before any two processes write one store. On network mounts, file-lock semantics are the weakest primitive in the system, so quarry does not rest on them:

3. Single-writer leases

Per store, a NATS KV entry lease.<project_code> with TTL held by the node currently allowed to invoke write operations on that store. Writes (sync push received at any node, web-minted artifacts, CI verdict tags) are routed to the lease holder over NATS request/reply. Lease expiry → another node acquires. NATS unreachable → writes refuse, reads continue: the lease is coordination, not truth, and the failure posture is correctness over availability. The zit#59 flock remains the second line of defense on the holder itself (it also protects against out-of-band CLI writers on the same mount, which the lease cannot see).

Expiry-during-write. A TTL lease has one inherent race: the holder's lease expires mid-write (a slow fsync over 9P is the realistic trigger) while a new holder acquires and starts writing. True fencing would require the storage layer to reject stale writers, which POSIX filesystems cannot do, so quarry bounds the window instead of pretending to close it:

  1. Commit-point re-check — the holder records the KV revision at acquisition and re-reads the lease immediately before the final log.commit() fsync; if it no longer holds at that revision, it aborts. The danger window shrinks from the whole write to one fsync.
  2. Acquisition grace — a new holder waits one full TTL after acquiring before its first write, so any old holder that passed its commit-point check has finished or died before the new one starts.
  3. Append-only backstop — this is why residual overlap is safe rather than probably-safe: zit's log is append-only with validated-prefix reads, and the #59 flock serializes the physical append. The worst interleaving yields two serialized appends — an ordering wart under an expired lease, not corruption.

The KV revision doubles as the fencing epoch for free; the cost is one KV read per write and a startup delay on failover.

Shipped. internal/lease implements this over the embedded NATS KV: a bucket with TTL = lease lifetime, Create for atomic acquire, CAS Update for renewal, the commit-point re-check, and a failover grace applied only after displacing a foreign holder (a clean single-node start pays no grace). Both write-op families route through it — the web/API Guard on wiki.save / ticket.new / ticket.set, and the sync fronter's push guard — so a node that does not hold the lease answers a push with a read-only error card and a web write with a refusal, never an uncoordinated write.

4. Ingest pipeline

watcher (per store, on any node)
  → validatePrefix(cursor); read tail frames via zit subprocess (§2)
  → publish: zit.evt.<project_code>.artifact  (one msg per artifact: type, ids, refs)
             zit.evt.<project_code>.checkin   (derived check-in events)
  → per-node indexer (durable JetStream consumer):
       fold events into local SQLite — structured tables + FTS5,
       one transaction per batch

Cursors live in NATS KV but are advisory — a lost cursor re-validates from an earlier offset or from zero. Full rebuild (quarry reindex [repo]) drops the affected tables and replays stores; the standing disposability test runs exactly this and diffs rendered output.

The ZITEVT stream uses work-queue retention: an event is dropped once its per-repo durable indexer acks it, so the stream's disk footprint tracks the in-flight backlog, not store history. This matters at real scale — an early limits-retention stream reached 5.6 GB beside sqlite.org's 218 MB store. The store is the source of truth, so the stream is safe to shed: cursor loss (or a retention migration) just republishes the census and idempotent folds reconverge. Bulk-ingest measurements and the fixes that made real corpora (sqlite.org, tcl) tractable — WAL on the index, one branch recompute per fold round, consumer-pending-count drain detection — are recorded with the implementation plan.

The index engine is SQLite (driver choice and rationale live in the implementation plan, 0003). Each node owns its index files exclusively, so SQLite's single-writer constraint never binds, and the whole plane is derived and disposable. Structured queries and search (FTS5) live in one engine; a forge's read shapes (point lookups, one repo's timeline page) are B-tree shapes, which is also why a columnar engine buys v1 nothing.

The Parquet interchange lake is deferred to scale-out (see Guide): when N nodes exist, a lake-writer worker (a dagnats DAG — durable, DLQ'd) joins the same event stream and writes append-only Parquet to S3, partitioned by repo/type/date, with compaction as a dagnats maintenance DAG. New nodes hydrate their SQLite from the lake instead of replaying every store, and the engine that reads the lake is chosen when the lake exists — the event stream, not any engine, is the interface, so every swap is additive. The same logic covers FTS escape hatches (a Tantivy sidecar, a libsql/turbopuffer-shaped index-over-S3) if per-node indexes ever pinch.

Graph-shaped analytics (ancestry across merges, cross-repo dependency graphs, contributor/file affinity) are deliberately not specified: the interchange lake's open Parquet format means any zero-ETL graph engine can attach later without a migration, which is the point of the storage/compute separation — see Future possibilities.

5. Chat: one channel per repository, and the durability decision

Every repository gets chat.<project_code> (plus chat.quarry fleet-wide). The decision this RFC takes — the one Fossil got half right:

Archival cadence, batching (one artifact per N messages or T minutes), and the author-identity mapping (quarry account → U card name) are v1 constants in config, defaulting to 15-minute batches.

6. CI/CD: dagnats is the engine, quarry is the trigger and the display

7. Sync fronting

quarry terminates HTTP(S) for everything, including /r/<repo>/xfer:

8. Forge OLTP and auth

Accounts, sessions, org/ACLs, tokens, repo registry: one SQLite database, litestream-replicated to S3 continuously (the firestorm-proven pattern). This is the only authoritative non-DAG state in quarry, it is small by construction, and its S3 replica is the recovery story. v1 ships single-org, flat repo namespace, admin + per-repo tokens; role granularity is v2 and adds tables, not architecture.

Shipped (v1). The OLTP database currently holds one table, sync_user (repo, user, secret) — the §7 shared secrets. quarry user add derives a secret (learning the project code via a clone handshake against a transient zit serve, never by reading .zit bytes) and stores it; the write ops (wiki.save, ticket.new, ticket.set — all Kind=Command, Auth=Token) and the HTML forms verify HTTP Basic against it, carrying the authenticated user into the op through context so zit records the right author. Write ops run one ingest round before returning (read-your-writes). litestream supervision is deferred with the rest of scale-out; the table is trivially rebuildable meanwhile.

9. The ops layer: one seam for HTML, API, and events

Every capability in quarry is an op — one named, typed unit in a single registry:

type Op[In, Out any] struct {
    Name  string        // "timeline.list", "checkin.get", "chat.post"
    Kind  Kind          // Query | Command
    Auth  Requirement   // anonymous | token | admin (v1's whole auth model)
    Route string        // "GET /api/v1/r/{repo}/timeline"
    Run   func(ctx, In) (Out, error)
}

In and Out are plain structs, JSON-taggable and expressible as form/URL values — a deliberate constraint that forbids exotic nested inputs and keeps the API curl-able. The registry has exactly three consumers, and the design's one inversion is that the registry generates the API spec, not the other way around:

  1. The API router mounts every registered op and generates the OpenAPI document from the In/Out structs by reflection. The spec is a build artifact; Principle 5 becomes structural because a capability cannot exist without an endpoint — registering the op is creating it. Parity cannot drift because there is nothing to keep in sync.
  2. HTML handlers are thin translators: parse form/URL → In struct → call op → hand Out struct to a template. No JSON round-trip, so HTML pays zero serialization tax (Principle 4).
  3. The test harnesses enumerate the registry: the disposability test (§11) snapshots every query op's response against fixture stores, and API golden tests are generated rather than hand-listed.

Enforcement is package layout, not discipline: only internal/ops may import the data planes (SQLite indexes, forge OLTP, the zit subprocess boundary). HTML handlers and the API router physically cannot reach around the seam; a CI import-graph assertion makes it permanent.

Consequences taken deliberately:

10. Staged delivery

Prerequisites in zit, all already filed: #59 (append-append lock — required before any multi-process writes), #17 (inode-swap reopen — required for every long-running reader), #53 (clone batching — quarry inherits the win). Nothing else is asked of zit core; a docs/ note describing the watch-and-extend recipe is nice-to-have, not blocking.

11. The disposability test (normative)

CI for quarry itself runs, on a corpus of real stores (sqlite, tcl, a wiki/ticket/forum-heavy fixture):

  1. Ingest fully; snapshot rendered output (HTML of every page class, search results, chat archives, API responses).
  2. rm -rf the data directory (indexes, and lake partitions where present); re-ingest from stores.
  3. Semantically diff the snapshots: HTML compared as canonicalized DOM, JSON with sorted keys, timestamps normalized to one representation, unordered collections sorted before comparison. Any semantic diff fails the build.

Byte-diff was the original formulation and remains the aspiration, but it conflates two failure classes: hostage bytes (what the test exists to catch) and rendering nondeterminism (map iteration order, parallel-ingest row order, timestamp formatting), which byte comparison reports identically. Semantic equality catches every hostage-bytes escape — data that exists only in the old snapshot is a semantic diff by definition — without turning every harmless ordering change into a red build.

A feature that cannot pass this test is holding hostage bytes and does not merge.

Drawbacks

Rationale and alternatives

Prior art

Fossil (the forge feature set and the artifact content model, both studied in the zit RFC suite); Gitea/soft-serve (single-binary git forges); sourcehut (component forges); chiselapp (multi-repo Fossil hosting, the closest ancestor to the fleet page); notify 0001 (the operational model and this document's format); dagnats (the execution substrate); litestream + firestorm-dataworks production practice (SQLite-on-S3 replication); ZeroFS/JuiceFS documentation for the storage matrix.

Unresolved questions

None deliberately left. Decisions this document takes that most want an owner's veto, listed for cheap redirect rather than reopened by default: the working name; chat archival default-on; CI verdicts as tag artifacts default-on; ZeroFS-9P as primary over JuiceFS; the SQLite-only v1 index plane with the interchange lake deferred to scale-out; single-org v1.

Future possibilities