0003 — quarry implementation plan

Companion to 0001-quarry-forge.md (the spec) and 0002-forgejo-salvage.md (the UI salvage map). This document owns what the spec deliberately does not: concrete technology choices, build order, testing strategy, and observability wiring. Decisions here are revisable without touching the spec; decisions that would change the spec's architecture go back through 0001.

0. Guiding constraints (inherited, not restated)

• 0001 Principles 1–7 govern every phase; the ops layer (0001 §9) is the seam everything routes through; the disposability test (0001 §11) is the merge gate from phase 0 onward.
• The smol contract: each package is a program that can be finished.
• 0002 §4 mechanics govern all Forgejo salvage (ledger first, cleanroom sanitizer, resolver inversion).

1. Build order

The organizing rule: the spec's riskiest claims get proven by a tracer bullet before any breadth work, and the three enforcement mechanisms (disposability harness, ops registry, provenance ledger) are built first, because retrofitting a merge gate onto an existing codebase never works.

Phase 0 — rails before features

• Ops registry core + OpenAPI generator + the import-graph CI assertion. Empty registry, one dummy op, spec generation working end to end.
• Disposability harness scaffold: fixture corpus of real converted stores (sqlite, tcl, plus a synthetic wiki/ticket/forum/chat-heavy fixture — the synthetic fixture is worth real effort; it is the corpus's coverage), ingest → snapshot → wipe → re-ingest → semantic-diff loop, in CI from day one even while it covers a single page.
• Provenance ledger + CI check (0002 §4 item 1).
• Go module skeleton with the boundaries drawn: internal/ops (sole importer of data planes), internal/ingest, internal/markup and sibling salvage packages, internal/web, internal/api.

Exit: CI runs all three gates; a dummy op renders a dummy page and appears in a generated OpenAPI document.

Phase 1 — the tracer bullet

One store → watcher (validated-prefix fold via the zit subprocess, zit#17 stat-and-reopen discipline from the start) → embedded NATS event → per-node SQLite fold → one op (timeline.list) → one HTML page and its generated API endpoint, footer with versions and render timing.

Proves the four load-bearing claims at minimum cost: log-as-event-stream, embedded-index-as-derived-state, registry-generates-spec, disposability-as-gate.

Phase 2 — read plane breadth

0002 §4's extraction order: markup first (cleanroom sanitizer + conformance corpus before anything renders) → base layout → fleet → repo home/files → timeline/check-in → diff stack → graph → wiki render → search (FTS5) → activity/stats. Each page lands against the per-page definition of done (0002 §4 item 5) and grows the disposability snapshot set incrementally.

Phase 3 — liveness

Datastar SSE wiring, chat transport + forum archival, notify integration. First real test of Principle 3's refresh-fallback discipline.

Phase 4 — sync fronting + leases

/xfer fronting, then the 0001 §3 lease machinery. Deliberately last in v1: it is the only distributed-systems-hard part, and everything before it works single-node.

v2/v3 then follow 0001 §10's staging. Each phase ends with the harness green and a docs/ page generated from the spec — documentation is a phase exit criterion, not a v-final activity.

Status (as built)

Phases 0–4's single-node scope is shipped and dogfooded from commit zero in a zit store. Delivered beyond the original slice boundaries:

• Reads: fossil-grade DAG timeline (geometry verified by a browser lint), wiki + wiki edit history/revisions, tickets with folded state and cursor pagination, FTS5 search, check-in/diff/file surfaces — all served by the same ops the API exposes.
• Writes (needs zit's authoring verbs, zit#71): wiki.save, ticket.new, ticket.set, forum.post/forum.reply, technote.add plus HTML forms, read-your-writes. Every authoring op writes the cards the format allows: the technote C card (a title zit refuses to attach later, so it is write-once at save) and the forum N card — an omitted N card is text/x-fossil-wiki by fossil's rule, so quarry's markdown forms were producing posts that rendered as wiki markup (zit#77).
• Accounts and per-repo roles (0001 §8's v2 tables): argon2id verifiers, a site-admin flag, and read/write/admin per repository, managed by quarry account. ops.Auth is the required role, which gives the Admin level declared in v1 its meaning. Two credentials on purpose: accounts are the forge identity, while the RFC §6.3 shared secret stays the SYNC credential because the protocol fixes its derivation and the server must recompute it. The gate stays open until the first account exists, mirroring the sync-user policy, so local single-user serving is unaffected.
• Attachments: the A card folds to an attachment table and the content is served by a download route. Fossil models attachments as a log, not a set — newest card per filename wins and a card with no source is a deletion — so a naive resolve resurrects deleted files. Counts match fossil exactly (sqlite 11, tcl 3,333, deletions included). The route is gated by the index so it cannot serve arbitrary artifacts by hash, and is hostile to its own payload: download-only, octet-stream, nosniff, deny-all CSP.
• Sync fronting: /r/{repo} (+ /xfer alias) proxied to per-repo zit serve; push gating verifies §6 login per request. Cross-implementation interop proven both directions with real Fossil 2.28: fossil clones a zit-converted store through quarry (17,535 artifacts, integrity-clean), and a fossil-authored push lands in the zit store and folds into the index.
• Liveness: Datastar SSE on the timeline and the fleet page, plus an indexing badge — a first index runs for minutes, and without one a large repo reads as empty rather than as catching up. The badge exposed a latent bug worth recording: data-init sat on the element the stream patches, so the first patch morphed Datastar's own connection attribute away and the stream went silent. The timeline had it too, masked because it only sends on change and so appeared to work exactly once. The attribute now sits on a wrapper outside the patch target.

• Docs: quarry docs generates docs/api.md from the registry with a staleness gate, and /docs serves it.

• Single-writer leases (0001 §3, shipped): a NATS KV TTL lease per store, keyed by project code, gates both web/API write ops and sync pushes. Guard renews-or-acquires, applies a failover grace after displacing a holder, re-checks the lease at its commit point, and refuses writes when the coordinator is unreachable (correctness over availability). One embedded NATS is one node, so on a single node it holds its own lease with no grace delay — the machinery and its failure posture are identical at N nodes, which is what makes the rest of scale-out a deployment change, not a code change.

Phase 4 is complete. The remaining v1 horizon is the scale-out trio:

1. External NATS URL (shipped): --nats <url> joins an external cluster so N nodes share one event plane; the embedded server stays the single-node default. This forced the retention correction above (work-queue → limits+cap) so every node folds the full stream, and per-node durable names so nodes don't collide on one consumer. Proven live: two nodes on one external NATS over a shared store — each folds the whole store, the lease-holding node writes while the other returns 503 and still folds that write cross-node off the shared stream.

The cap has a corollary that cost real data before it was found: ZITEVT discards OLD messages, and watch() published the whole census in one burst, so a repository larger than the cap deleted its own head while the indexer was still folding it. The index then drained what remained, saw NumPending == 0, and reported itself done. sqlite indexed 41,478 of 131,558 artifacts that way and lost every ticket, wiki page and forum post with no error surfaced. Publishing now yields to folding every 64 MiB, so the stream applies backpressure instead of dropping; the regression test reproduces the original loss against a deliberately small stream.

1. Shared S3-backed store access (ZeroFS/9P per §storage): no quarry code — --repo already takes a path, and a 9P mount is just a path. The work is deployment (one ZeroFS process per bucket, supervised; 9P not NFS for the fsync durability contract zit's log.commit() depends on) and validation on a Linux+9P host, which this dev Mac can't exercise.
2. Parquet interchange lake: deferred by the tech-choices table ("no Parquet dependency in v1"; when the lake lands, write via a query engine's COPY TO, not a standalone library). It optimizes new-node hydration (read the lake instead of replaying every store) — an efficiency layer over a system that is already correct without it.

The trio turns the single-process build into N stateless read nodes plus one leased writer. Part 1 is the code; parts 2 and 3 are deployment and a v-next efficiency layer.

An adversarial audit of the write surface (5 dimensions, per-finding verification) hardened it: it caught and we fixed a whitespace-card sync-gate bypass (the gate now tokenizes exactly as zit does), ungated edit-form GETs, and two input faults that 500'd instead of 400ing.

Interop and scale findings surfaced upstream zit work, all landed: authoring verbs (zit#71), the sync client's per-round credential gap vs. the protocol's own §6.5 cookie-login MUST (zit#74 — authenticated zit push authenticates through quarry's gate end-to-end), a bare-store -R gap in zit diff (zit#69), whole-second D-card resolution causing same-second writes to fold in hash order rather than time order (zit#75), the technote/forum cards the verbs could not write (zit#77), and zit frames reconstruction cost (zit#84 — a plaintext memo plus parallel batch verification, ~31% off a cold sqlite read).

Two remain open and shape what quarry can do next:

• zit#85 — no way to read a project code. quarry must learn it to derive the §6.3 shared secret, and the only route today is standing up a transient zit serve and syncing against it: a subprocess, a socket and a 5s worst case per repository at startup. Because serve is checkout-only, this is also the single thing keeping quarry from serving a BARE store — the read plane is already -R-only, with no cmd.Dir anywhere in zitcli.
• zit#86 — the artifact format's §4.2 token grammar. token = 1*tchar forbids the empty trailing token that fossil emits for an empty value, so a reader implementing the text literally rejects whole artifacts. quarry did: sqlite's 2,100 ticket changes and 1,009 empty-comment check-ins across the two corpora vanished into the content fallback. zit's reader is corrected; the RFC text still needs the erratum.

The lesson quarry keeps re-learning from these is in the failure mode, not the bugs: an unparsable artifact falling back to opaque content is silent by construction. Parity against fossil's own tables — event, ticket, forumpost, attachment — is the only check that catches it, and it belongs in CI against a real corpus rather than in a person's memory.

2. Technology choices

Each entry: choice, rationale, escape hatch. These are ADR-grade decisions recorded compactly; split any into a standalone ADR when it accretes real debate.

Area	Choice	Rationale	Escape hatch
SQLite driver	ncruces/go-sqlite3	Upstream SQLite compiled to Wasm (wazero): real SQLite behavior, fast tracking of releases, no CGO (cross-compilation stays trivial), FTS5 enabled, and a Go-implementable VFS surface — the scale-out hook if node indexes ever want S3-backed hydration. Preferred over modernc (transpiler artifacts, historic concurrency edges) and mattn (CGO). Known costs: Wasm boundary tax on write-heavy paths (fits our read-dominated profile), amd64/arm64-only compiler (interpreter elsewhere), ~tens-of-ms startup compile, narrow bus factor (mitigated: unmodified upstream SQLite makes a forced move driver-swap-shaped, not data-shaped).	mattn (CGO) or modernc; schema is plain SQLite either way
HTTP router	stdlib http.ServeMux (1.22+ patterns)	The ops layer already does what chi would add; a router dependency is a shallow module here.	chi, mechanically
Ops registry	Hand-rolled on stdlib + generics (~1.5k lines total)	The registry is quarry's spine; you don't build the central abstraction on someone else's framework. The flat, form-expressible struct constraint (0001 §9) makes the reflector genuinely small. Pieces: generic Register[In,Out] over a descriptor map (~150 lines, explicit RegisterAll() wiring); JSON codec (stdlib); form/URL-values codec (~150 lines, needed for HTML regardless of framework); path-param binding onto ServeMux wildcards; OpenAPI reflector (~300–400 lines); tag-driven validator shared by both codecs; instrumentation wrapper around Run.	Huma (danielgtaylor/huma) — our Run signature is already its handler shape, so migration is mechanical
OpenAPI document types	Reuse dagnats' OpenAPI 3.1 synthesis types (currently internal/openapi in dagnats — promotion to a public package is a prerequisite, see §5)	One shared, tested document-assembly layer; quarry writes only the struct-reflection front half; both projects emit consistent 3.1 documents.	Hand-roll the back half too (~200 more lines)
Spec correctness gate	Off-the-shelf OpenAPI validator lints the generated spec in CI (dev-time tool, not a runtime dep); every registered op's fixtures round-trip through both codecs	Closes "our generator emits a subtly invalid spec" without a runtime framework.	—
Markup	goldmark + bluemonday + chroma	Per 0002 §2; sanitizer policy is cleanroom with the two-lineage conformance corpus + fuzz target.	—
Embedded NATS	nats-server as a library	The dagnats/notify pattern, already proven in this stack.	External NATS URL (already the scale-out flag)
litestream	Supervised sidecar	firestorm-proven shape; embedding buys nothing.	—
Parquet writing	Deferred with the interchange lake (0001: scale-out)	No Parquet dependency in v1 at all. When the lake lands, prefer writing via a query engine's COPY TO over a standalone parquet library — decide then.	—
Frontend	Datastar (only default-allowed framework, 0001 Principle 2) + html/template	Per 0001/0002.	Explicit per-case exception process
zit access	Subprocess boundary — the zit binary is the only store interface, reads and writes (0001 §2)	zit is Zig, quarry is Go; the versioned CLI is the whole contract, so releases decouple and quarry stays CGO-free. Most page data serves from quarry's indexes; on-demand reads (file-at-checkin, diff, annotate) pay ~1–5ms spawn, off the hot path and behind caching. Quarry's CI tests against pinned zit versions and each release declares a minimum zit version.	C-ABI read library exported from zit — revisit only with profiling evidence

3. dagnats integration

dagnats provides durable HTTP endpoints (HTTP trigger → event-sourced run → engine-resolved respond step) with OpenAPI 3.1 synthesis from trigger defs + declared schemas. Division of labor:

• Reads never route through dagnats. Two reasons that outrank latency: durable execution persists every request as run history (page-view-rate event-log garbage, retention becomes a problem), and 0001 §3 commits to "NATS unreachable → reads continue" — reads through dagnats would make a NATS outage a forge outage. Query ops are in-process function calls over the local index.
• Command ops that want durability get it inside Run. Chat archival, webhook dispatch, CI verdict minting — anything with retry/DLQ semantics — launches a dagnats workflow behind the seam. The registry stays the single OpenAPI source; no split-brain between two generators.
• CI/CD (0001 §6) uses dagnats natively, as spec'd.

4. Testability — one strategy per risk class

• Derived-state correctness: the disposability harness (0001 §11) is the keystone; every page and API response feeds it.
• Ops: query ops are pure functions over index state → golden tests are generated from the registry (enumerate ops, run fixtures, snapshot). New op = automatic coverage.
• The lease: deterministic simulation. Abstract the clock and the KV behind interfaces so the 0001 §3 expiry-during-write interleavings can be exhaustively driven in simulated schedules — no sleeps, no flakes. This is the one place in the system where a rare race matters.
• Sanitizer: the two-lineage conformance corpus + fuzz target (0002 §2).
• No-JS baseline: nearly free — pages are server-rendered, so the snapshot harness is the no-JS test. Plus one template lint: every data-on has a form/anchor sibling.
• Watcher: property tests on cursor loss / prefix re-validation (a lost cursor must converge to identical index state — a mini-disposability property).

5. Observability

Lean on dagnats: quarry imports dagnats' observe package for OTel exporter wiring, resource attributes, and conventions rather than building its own setup. Command ops propagate W3C trace context into the workflows they launch (dagnats' bridge already speaks tracecontext), so one stitched trace runs from HTTP request through the durable run.

Quarry adds only what dagnats cannot see:

• The ops-layer wrapper: span + latency histogram + counter per op call, labeled by op name (in-process reads never touch dagnats).
• Three gauges: ingest lag per store (cursor vs. log head — the one number that says quarry is healthy), lease state per store, and JetStream consumer lag.
• Prometheus /metrics; structured slog throughout; the Principle-1 footer render timing is the same measurement plumbed to metrics — one clock, two consumers.

6. Prerequisites and upstream work

• Fossil interop is a stated major priority — it is the adoption front door (Fossil users are the first expected adopters). Current state, verified in the zit tree 2026-08-14: wire-level interop already works — zit clones from a live Fossil server over the RFC'd sync protocol with hand-verified identical leaves, and the sync path handles all artifact types (wiki, ticket, forum, event, attachment). The open gap is consuming a bare .fossil file without a running server (zit's docs/plans/FOSSIL-IMPORT-PLAN.md has the route analysis; zit#1 is the adjacent issue; not yet built). Consequence for quarry: a Fossil user's first contact is zit clone of their existing served repo — no converter — and quarry can even front Fossil repos as continuously-synced mirrors. The .fossil-file reader is upstream zit work, prioritized accordingly; quarry's fixture tooling should use the serve-and-clone path meanwhile.
• zit: #59 (append-append flock), #17 (inode-swap reopen), #53 (clone batching) — already filed, per 0001 §10.
• zit, new (falls out of the subprocess boundary): a machine-consumable read-verb surface — at minimum "emit artifacts since cursor N with prefix validation" for the watcher, plus stable, parseable output for file-at-checkin / diff / annotate. The watcher verb's payload is raw artifacts in the RFC'd artifact format (0001 §2): zit resolves delta chains and streams whole artifacts; no new output schema is invented, and the container layout stays private. Audited 2026-08-14: stat, ls, cat, artifact --raw, diff --from/--to, and timeline are already machine-consumable; the gaps were filed as zit#66 (the watcher verb) and zit#67 (annotate at a VERSION, full IDs). Both shipped in zit 0.1.3 (verified 2026-08-15: append/no-new/invalid exit codes 0/3/4, bytes identical to artifact --raw, shun-reclaim kills the cursor while a no-op pack rightly does not). internal/zitcli wraps frames; the phase-1 blocker is cleared.
• dagnats: promote internal/openapi to a public package so quarry can import the 3.1 document types (§2). Small PR, no behavior change.
• notify: wiring per its own RFC; no changes requested.
• zit#70 (checkout registry) alignment, noted 2026-08-15: quarry's fleet registration stays ephemeral --repo name=path flags until zit's global config db (zit#70 phase 2, fossil's ~/.fossil shape) exists — then --discover consumes zit's enumeration rather than quarry growing a persistent store-of-stores. Boundary: zit's registry owns "what stores exist and where" (host fact); quarry's v2 OLTP repo registry owns forge metadata only (serve-name, description, ACLs), seeded from zit's answer — never a second source of truth for locations. zit#59+#70 together also unblock bare-store serving, which retires the checkout-only constraint zit#69 documents.

7. Risks

• ncruces bus factor — mitigated by unmodified-upstream-SQLite (see §2).
• Hand-rolled reflector correctness — mitigated by the CI validator lint
  • codec round-trip tests (§2).
• Synthetic fixture coverage — the disposability harness is only as good as its corpus; the wiki/ticket/forum/chat-heavy fixture needs deliberate investment in phase 0, not incidental accretion.
• Datastar no-JS discipline — enforced by the template lint and per-page definition of done, but the first genuinely live page (chat) will test whether refresh-fallback stays honest under feature pressure.