# 0003 — quarry implementation plan
- Status: Historical — as built (§Status); §2 (technology choices) and §8 (salvage and provenance rules, formerly RFC 0002) remain in force
Companion to `0001-quarry-forge.md` (the spec) and `0002-forgejo-salvage.md`
(the UI salvage map). This document owns what the spec deliberately does not:
concrete technology choices, build order, testing strategy, and observability
wiring. Decisions here are revisable without touching the spec; decisions
that would change the spec's architecture go back through 0001.
## 0. Guiding constraints (inherited, not restated)
- 0001 Principles 1–7 govern every phase; the ops layer (0001 §9) is the seam
everything routes through; the disposability test (0001 §11) is the merge
gate from phase 0 onward.
- The smol contract: each package is a program that can be finished.
- 0002 §4 mechanics govern all Forgejo salvage (ledger first, cleanroom
sanitizer, resolver inversion).
## 1. Build order
The organizing rule: **the spec's riskiest claims get proven by a tracer
bullet before any breadth work**, and the three enforcement mechanisms
(disposability harness, ops registry, provenance ledger) are built first,
because retrofitting a merge gate onto an existing codebase never works.
### Phase 0 — rails before features
- Ops registry core + OpenAPI generator + the import-graph CI assertion.
Empty registry, one dummy op, spec generation working end to end.
- Disposability harness scaffold: fixture corpus of real converted stores
(sqlite, tcl, plus a synthetic wiki/ticket/forum/chat-heavy fixture — the
synthetic fixture is worth real effort; it is the corpus's coverage),
ingest → snapshot → wipe → re-ingest → semantic-diff loop, in CI from day
one even while it covers a single page.
- Provenance ledger + CI check (0002 §4 item 1).
- Go module skeleton with the boundaries drawn: `internal/ops` (sole importer
of data planes), `internal/ingest`, `internal/markup` and sibling salvage
packages, `internal/web`, `internal/api`.
Exit: CI runs all three gates; a dummy op renders a dummy page and appears in
a generated OpenAPI document.
### Phase 1 — the tracer bullet
One store → watcher (validated-prefix fold via the zit subprocess, zit#17
stat-and-reopen discipline from the start) → embedded NATS event → per-node
SQLite fold → one op (`timeline.list`) → one HTML page *and* its generated
API endpoint, footer with versions and render timing.
Proves the four load-bearing claims at minimum cost: log-as-event-stream,
embedded-index-as-derived-state, registry-generates-spec,
disposability-as-gate.
### Phase 2 — read plane breadth
0002 §4's extraction order: markup first (cleanroom sanitizer + conformance
corpus before anything renders) → base layout → fleet → repo home/files →
timeline/check-in → diff stack → graph → wiki render → search (FTS5) →
activity/stats. Each page lands against the per-page definition of done
(0002 §4 item 5) and grows the disposability snapshot set incrementally.
### Phase 3 — liveness
Datastar SSE wiring, chat transport + forum archival, notify integration.
First real test of Principle 3's refresh-fallback discipline.
### Phase 4 — sync fronting + leases
`/xfer` fronting, then the 0001 §3 lease machinery. Deliberately last in v1:
it is the only distributed-systems-hard part, and everything before it works
single-node.
v2/v3 then follow 0001 §10's staging. Each phase ends with the harness green
and a `docs/` page generated from the spec — documentation is a phase exit
criterion, not a v-final activity.
### Status (as built)
Phases 0–4's single-node scope is shipped and dogfooded from commit zero in a
zit store. Delivered beyond the original slice boundaries:
- **Reads**: fossil-grade DAG timeline (geometry verified by a browser
lint), wiki + wiki edit history/revisions, tickets with folded state and
cursor pagination, FTS5 search, check-in/diff/file surfaces — all served by
the same ops the API exposes.
- **Writes** (needs zit's authoring verbs, zit#71): `wiki.save`,
`ticket.new`, `ticket.set`, `forum.post`/`forum.reply`, `technote.add` plus
HTML forms, read-your-writes. Every authoring op writes the cards the
format allows: the technote C card (a title zit refuses to attach later, so
it is write-once at save) and the forum N card — an omitted N card is
`text/x-fossil-wiki` by fossil's rule, so quarry's markdown forms were
producing posts that rendered as wiki markup (zit#77).
- **Accounts and per-repo roles** (0001 §8's v2 tables): argon2id verifiers,
a site-admin flag, and read/write/admin per repository, managed by
`quarry account`. `ops.Auth` is the required role, which gives the `Admin`
level declared in v1 its meaning. Two credentials on purpose: accounts are
the forge identity, while the RFC §6.3 shared secret stays the SYNC
credential because the protocol fixes its derivation and the server must
recompute it. The gate stays open until the first account exists, mirroring
the sync-user policy, so local single-user serving is unaffected.
- **Attachments**: the A card folds to an `attachment` table and the content
is served by a download route. Fossil models attachments as a log, not a
set — newest card per filename wins and a card with no source is a deletion
— so a naive resolve resurrects deleted files. Counts match fossil exactly
(sqlite 11, tcl 3,333, deletions included). The route is gated by the index
so it cannot serve arbitrary artifacts by hash, and is hostile to its own
payload: download-only, octet-stream, nosniff, deny-all CSP.
- **Sync fronting**: `/r/{repo}` (+ `/xfer` alias) proxied to per-repo
`zit serve`; push gating verifies §6 login per request. **Cross-implementation
interop proven both directions with real Fossil 2.28**: fossil clones a
zit-converted store through quarry (17,535 artifacts, integrity-clean), and
a fossil-authored push lands in the zit store and folds into the index.
- **Liveness**: Datastar SSE on the timeline and the fleet page, plus an
indexing badge — a first index runs for minutes, and without one a large
repo reads as empty rather than as catching up. The badge exposed a latent
bug worth recording: `data-init` sat on the element the stream patches, so
the first patch morphed Datastar's own connection attribute away and the
stream went silent. The timeline had it too, masked because it only sends
on change and so appeared to work exactly once. The attribute now sits on a
wrapper outside the patch target.
- **Docs**: `quarry docs` generates `docs/api.md` from the registry with a
staleness gate, and `/docs` serves it.
- **Single-writer leases** (0001 §3, shipped): a NATS KV TTL lease per store,
keyed by project code, gates both web/API write ops and sync pushes. Guard
renews-or-acquires, applies a failover grace after displacing a holder,
re-checks the lease at its commit point, and refuses writes when the
coordinator is unreachable (correctness over availability). One embedded
NATS is one node, so on a single node it holds its own lease with no grace
delay — the machinery and its failure posture are identical at N nodes, which
is what makes the rest of scale-out a deployment change, not a code change.
**Phase 4 is complete.** The remaining v1 horizon is the scale-out *trio*:
1. **External NATS URL** (shipped): `--nats <url>` joins an external cluster so
N nodes share one event plane; the embedded server stays the single-node
default. This forced the retention correction above (work-queue → limits+cap)
so every node folds the full stream, and per-node durable names so nodes
don't collide on one consumer. Proven live: two nodes on one external NATS
over a shared store — each folds the whole store, the lease-holding node
writes while the other returns 503 and still folds that write cross-node
off the shared stream.
The cap has a corollary that cost real data before it was found: ZITEVT
discards OLD messages, and `watch()` published the whole census in one
burst, so a repository larger than the cap deleted its own head while the
indexer was still folding it. The index then drained what remained, saw
`NumPending == 0`, and reported itself done. sqlite indexed 41,478 of
131,558 artifacts that way and lost every ticket, wiki page and forum post
with no error surfaced. Publishing now yields to folding every 64 MiB, so
the stream applies backpressure instead of dropping; the regression test
reproduces the original loss against a deliberately small stream.
2. **Shared S3-backed store access** (ZeroFS/9P per §storage): no quarry code —
`--repo` already takes a path, and a 9P mount is just a path. The work is
deployment (one ZeroFS process per bucket, supervised; 9P not NFS for the
fsync durability contract zit's `log.commit()` depends on) and validation on
a Linux+9P host, which this dev Mac can't exercise.
3. **Parquet interchange lake**: deferred by the tech-choices table ("no Parquet
dependency in v1"; when the lake lands, write via a query engine's `COPY TO`,
not a standalone library). It optimizes new-node hydration (read the lake
instead of replaying every store) — an efficiency layer over a system that
is already correct without it.
The trio turns the single-process build into N stateless read nodes plus one
leased writer. Part 1 is the code; parts 2 and 3 are deployment and a
v-next efficiency layer.
An adversarial audit of the write surface (5 dimensions, per-finding
verification) hardened it: it caught and we fixed a whitespace-card sync-gate
bypass (the gate now tokenizes exactly as zit does), ungated edit-form GETs,
and two input faults that 500'd instead of 400ing.
Interop and scale findings surfaced upstream zit work, all landed: authoring
verbs (zit#71), the sync client's per-round credential gap vs. the protocol's
own §6.5 cookie-login MUST (zit#74 — authenticated `zit push` authenticates
through quarry's gate end-to-end), a bare-store `-R` gap in `zit diff`
(zit#69), whole-second `D`-card resolution causing same-second writes to fold
in hash order rather than time order (zit#75), the technote/forum cards the
verbs could not write (zit#77), and `zit frames` reconstruction cost
(zit#84 — a plaintext memo plus parallel batch verification, ~31% off a cold
sqlite read).
Two remain open and shape what quarry can do next:
- **zit#85 — no way to read a project code.** quarry must learn it to derive
the §6.3 shared secret, and the only route today is standing up a transient
`zit serve` and syncing against it: a subprocess, a socket and a 5s worst
case per repository at startup. Because `serve` is checkout-only, this is
also the single thing keeping quarry from serving a BARE store — the read
plane is already `-R`-only, with no `cmd.Dir` anywhere in `zitcli`.
- **zit#86 — the artifact format's §4.2 token grammar.** `token = 1*tchar`
forbids the empty trailing token that fossil emits for an empty value, so a
reader implementing the text literally rejects whole artifacts. quarry did:
sqlite's 2,100 ticket changes and 1,009 empty-comment check-ins across the
two corpora vanished into the content fallback. zit's reader is corrected;
the RFC text still needs the erratum.
The lesson quarry keeps re-learning from these is in the failure mode, not the
bugs: an unparsable artifact falling back to opaque content is silent by
construction. Parity against fossil's own tables — `event`, `ticket`,
`forumpost`, `attachment` — is the only check that catches it, and it belongs
in CI against a real corpus rather than in a person's memory.
## 2. Technology choices
Each entry: choice, rationale, escape hatch. These are ADR-grade decisions
recorded compactly; split any into a standalone ADR when it accretes real
debate.
| Area | Choice | Rationale | Escape hatch |
|---|---|---|---|
| SQLite driver | `ncruces/go-sqlite3` | Upstream SQLite compiled to Wasm (wazero): real SQLite behavior, fast tracking of releases, no CGO (cross-compilation stays trivial), FTS5 enabled, and a **Go-implementable VFS surface** — the scale-out hook if node indexes ever want S3-backed hydration. Preferred over modernc (transpiler artifacts, historic concurrency edges) and mattn (CGO). Known costs: Wasm boundary tax on write-heavy paths (fits our read-dominated profile), amd64/arm64-only compiler (interpreter elsewhere), ~tens-of-ms startup compile, narrow bus factor (mitigated: unmodified upstream SQLite makes a forced move driver-swap-shaped, not data-shaped). | mattn (CGO) or modernc; schema is plain SQLite either way |
| HTTP router | stdlib `http.ServeMux` (1.22+ patterns) | The ops layer already does what chi would add; a router dependency is a shallow module here. | chi, mechanically |
| Ops registry | Hand-rolled on stdlib + generics (~1.5k lines total) | The registry is quarry's spine; you don't build the central abstraction on someone else's framework. The flat, form-expressible struct constraint (0001 §9) makes the reflector genuinely small. Pieces: generic `Register[In,Out]` over a descriptor map (~150 lines, explicit `RegisterAll()` wiring); JSON codec (stdlib); form/URL-values codec (~150 lines, needed for HTML regardless of framework); path-param binding onto ServeMux wildcards; OpenAPI reflector (~300–400 lines); tag-driven validator shared by both codecs; instrumentation wrapper around `Run`. | Huma (`danielgtaylor/huma`) — our `Run` signature is already its handler shape, so migration is mechanical |
| OpenAPI document types | Reuse dagnats' OpenAPI 3.1 synthesis types (currently `internal/openapi` in dagnats — promotion to a public package is a prerequisite, see §5) | One shared, tested document-assembly layer; quarry writes only the struct-reflection front half; both projects emit consistent 3.1 documents. | Hand-roll the back half too (~200 more lines) |
| Spec correctness gate | Off-the-shelf OpenAPI validator lints the generated spec in CI (dev-time tool, not a runtime dep); every registered op's fixtures round-trip through both codecs | Closes "our generator emits a subtly invalid spec" without a runtime framework. | — |
| Markup | goldmark + bluemonday + chroma | Per §8; sanitizer policy is cleanroom with the two-lineage conformance corpus + fuzz target. | — |
| Embedded NATS | `nats-server` as a library | The dagnats/notify pattern, already proven in this stack. | External NATS URL (already the scale-out flag) |
| litestream | Supervised sidecar | firestorm-proven shape; embedding buys nothing. | — |
| Parquet writing | Deferred with the interchange lake (0001: scale-out) | No Parquet dependency in v1 at all. When the lake lands, prefer writing via a query engine's `COPY TO` over a standalone parquet library — decide then. | — |
| Frontend | Datastar (only default-allowed framework, 0001 Principle 2) + `html/template` | Per 0001/0002. | Explicit per-case exception process |
| zit access | Subprocess boundary — the zit binary is the only store interface, reads and writes (0001 §2) | zit is Zig, quarry is Go; the versioned CLI is the whole contract, so releases decouple and quarry stays CGO-free. Most page data serves from quarry's indexes; on-demand reads (file-at-checkin, diff, annotate) pay ~1–5ms spawn, off the hot path and behind caching. Quarry's CI tests against pinned zit versions and each release declares a minimum zit version. | C-ABI read library exported from zit — revisit only with profiling evidence |
## 3. dagnats integration
dagnats provides durable HTTP endpoints (HTTP trigger → event-sourced run →
engine-resolved respond step) with OpenAPI 3.1 synthesis from trigger defs +
declared schemas. Division of labor:
- **Reads never route through dagnats.** Two reasons that outrank latency:
durable execution *persists* every request as run history (page-view-rate
event-log garbage, retention becomes a problem), and 0001 §3 commits to
"NATS unreachable → reads continue" — reads through dagnats would make a
NATS outage a forge outage. Query ops are in-process function calls over
the local index.
- **Command ops that want durability get it inside `Run`.** Chat archival,
webhook dispatch, CI verdict minting — anything with retry/DLQ semantics —
launches a dagnats workflow behind the seam. The registry stays the single
OpenAPI source; no split-brain between two generators.
- CI/CD (0001 §6) uses dagnats natively, as spec'd.
## 4. Testability — one strategy per risk class
- **Derived-state correctness**: the disposability harness (0001 §11) is the
keystone; every page and API response feeds it.
- **Ops**: query ops are pure functions over index state → golden tests are
*generated from the registry* (enumerate ops, run fixtures, snapshot). New
op = automatic coverage.
- **The lease**: deterministic simulation. Abstract the clock and the KV
behind interfaces so the 0001 §3 expiry-during-write interleavings can be
exhaustively driven in simulated schedules — no sleeps, no flakes. This is
the one place in the system where a rare race matters.
- **Sanitizer**: the two-lineage conformance corpus + fuzz target (0002 §2).
- **No-JS baseline**: nearly free — pages are server-rendered, so the
snapshot harness *is* the no-JS test. Plus one template lint: every
`data-on` has a form/anchor sibling.
- **Watcher**: property tests on cursor loss / prefix re-validation (a lost
cursor must converge to identical index state — a mini-disposability
property).
## 5. Observability
Lean on dagnats: quarry **imports dagnats' `observe` package** for OTel
exporter wiring, resource attributes, and conventions rather than building
its own setup. Command ops propagate W3C trace context into the workflows
they launch (dagnats' bridge already speaks tracecontext), so one stitched
trace runs from HTTP request through the durable run.
Quarry adds only what dagnats cannot see:
- The ops-layer wrapper: span + latency histogram + counter per op call,
labeled by op name (in-process reads never touch dagnats).
- Three gauges: **ingest lag per store** (cursor vs. log head — the one
number that says quarry is healthy), **lease state per store**, and
**JetStream consumer lag**.
- Prometheus `/metrics`; structured `slog` throughout; the Principle-1 footer
render timing is the same measurement plumbed to metrics — one clock, two
consumers.
## 6. Prerequisites and upstream work
- **Fossil interop is a stated major priority — it is the adoption front
door** (Fossil users are the first expected adopters). Current state,
verified in the zit tree 2026-08-14: **wire-level interop already works** —
zit clones from a live Fossil server over the RFC'd sync protocol with
hand-verified identical leaves, and the sync path handles all artifact
types (wiki, ticket, forum, event, attachment). The open gap is consuming
a **bare `.fossil` file** without a running server (zit's
`docs/plans/FOSSIL-IMPORT-PLAN.md` has the route analysis; zit#1 is the
adjacent issue; not yet built). Consequence for quarry: a Fossil user's
first contact is `zit clone` of their existing served repo — no converter
— and quarry can even front Fossil repos as continuously-synced mirrors.
The `.fossil`-file reader is upstream zit work, prioritized accordingly;
quarry's fixture tooling should use the serve-and-clone path meanwhile.
- zit: #59 (append-append flock), #17 (inode-swap reopen), #53 (clone
batching) — already filed, per 0001 §10.
- zit, new (falls out of the subprocess boundary): a **machine-consumable
read-verb surface** — at minimum "emit artifacts since cursor N with
prefix validation" for the watcher, plus stable, parseable output for
file-at-checkin / diff / annotate. The watcher verb's payload is **raw
artifacts in the RFC'd artifact format** (0001 §2): zit resolves delta
chains and streams whole artifacts; no new output schema is invented, and
the container layout stays private. Audited 2026-08-14: `stat`, `ls`,
`cat`, `artifact --raw`, `diff --from/--to`, and `timeline` are already
machine-consumable; the gaps were filed as zit#66 (the watcher verb) and
zit#67 (annotate at a VERSION, full IDs). **Both shipped in zit 0.1.3**
(verified 2026-08-15: append/no-new/invalid exit codes 0/3/4, bytes
identical to `artifact --raw`, shun-reclaim kills the cursor while a
no-op pack rightly does not). `internal/zitcli` wraps `frames`; the
phase-1 blocker is cleared.
- dagnats: promote `internal/openapi` to a public package so quarry can
import the 3.1 document types (§2). Small PR, no behavior change.
- notify: wiring per its own RFC; no changes requested.
- zit#70 (checkout registry) alignment, noted 2026-08-15: quarry's fleet
registration stays ephemeral `--repo name=path` flags until zit's
global config db (zit#70 phase 2, fossil's `~/.fossil` shape) exists —
then `--discover` consumes zit's enumeration rather than quarry
growing a persistent store-of-stores. Boundary: zit's registry owns
"what stores exist and where" (host fact); quarry's v2 OLTP repo
registry owns forge metadata only (serve-name, description, ACLs),
seeded from zit's answer — never a second source of truth for
locations. zit#59+#70 together also unblock bare-store serving, which
retires the checkout-only constraint zit#69 documents.
## 7. Risks
- **ncruces bus factor** — mitigated by unmodified-upstream-SQLite (see §2).
- **Hand-rolled reflector correctness** — mitigated by the CI validator lint
+ codec round-trip tests (§2).
- **Synthetic fixture coverage** — the disposability harness is only as good
as its corpus; the wiki/ticket/forum/chat-heavy fixture needs deliberate
investment in phase 0, not incidental accretion.
- **Datastar no-JS discipline** — enforced by the template lint and per-page
definition of done, but the first genuinely live page (chat) will test
whether refresh-fallback stays honest under feature pressure.
## 8. Salvage and provenance (formerly RFC 0002)
RFC 0002 was the Forgejo salvage map: which of their pages and modules
quarry would copy, import, or reimplement, in what order. The inventory
is done and the document was retired on 2026-09-10; its rules are the
part that outlives it, and they live here.
**Three modes of "take", every item tagged.** Forgejo's post-fork code is
GPLv3; Gitea-inherited code is MIT.
- **[COPY]** lifts code, only from files whose lineage is MIT (present in
`go-gitea/gitea` before the fork) or where the GPL consequence is
accepted knowingly. Take the Gitea upstream version, not the Forgejo
one, and record it in `third_party/PROVENANCE.md` — path, source repo,
commit, license. No exceptions to the ledger: the arch test refuses an
unledgered salvage and a ledger row pointing at nothing.
- **[DEP]** imports a library, after checking the module's own license.
- **[MAP]** reads for structure and reimplements fresh. Layouts, route
taxonomy, UX decisions, empty-state wording. No code crosses.
**What crossed.** Only the sanitizer test corpora, as conformance
vectors (`third_party/sanitizer-corpus/`, one MIT, one GPL accepted
knowingly as test data). The sanitizer policy itself is cleanroom. Markup
is goldmark + bluemonday + chroma, assembled here, not copied.
**Rules that still bind new work.**
- CSS is [MAP] only: take the information — spacing scale, diff colour
semantics, text hierarchy — never the stylesheet.
- All reference resolution in markup goes through one `Resolver` seam
(hash, ticket, wiki name, `branch@date`), so markup stays diffable
against upstream for future security re-syncs.
- Per-page definition of done: renders from derived-index data only
(0001 §11), no git vocabulary in user-facing strings ("check-in", not
"commit"), both themes, keyboard focus, usable with JS disabled (0001
Principle 3), the page's capability exposed through the API (0001
Principle 5).
- What Forgejo does not have was built fresh and owes it nothing: the
cross-repo timeline, chat, the ingest and index plane, sync fronting,
lease routing, the disposability harness, timestamp-resolution links,
amend-provenance display, submanifest-aware tree browsing.