Skip to content
quarry Artifact [5ffa5c2d3d]
Login

Artifact [5ffa5c2d3d]

Check-ins Using Download Hex Wiki

Artifact 5ffa5c2d3d7bf31ea5f1813721bc6533f468506864679ac9a31717ebd24e82e5:

# 0003 — quarry implementation plan

Companion to `0001-quarry-forge.md` (the spec) and `0002-forgejo-salvage.md`
(the UI salvage map). This document owns what the spec deliberately does not:
concrete technology choices, build order, testing strategy, and observability
wiring. Decisions here are revisable without touching the spec; decisions
that would change the spec's architecture go back through 0001.

## 0. Guiding constraints (inherited, not restated)

- 0001 Principles 1–7 govern every phase; the ops layer (0001 §9) is the seam
  everything routes through; the disposability test (0001 §11) is the merge
  gate from phase 0 onward.
- The smol contract: each package is a program that can be finished.
- 0002 §4 mechanics govern all Forgejo salvage (ledger first, cleanroom
  sanitizer, resolver inversion).

## 1. Build order

The organizing rule: **the spec's riskiest claims get proven by a tracer
bullet before any breadth work**, and the three enforcement mechanisms
(disposability harness, ops registry, provenance ledger) are built first,
because retrofitting a merge gate onto an existing codebase never works.

### Phase 0 — rails before features

- Ops registry core + OpenAPI generator + the import-graph CI assertion.
  Empty registry, one dummy op, spec generation working end to end.
- Disposability harness scaffold: fixture corpus of real converted stores
  (sqlite, tcl, plus a synthetic wiki/ticket/forum/chat-heavy fixture — the
  synthetic fixture is worth real effort; it is the corpus's coverage),
  ingest → snapshot → wipe → re-ingest → semantic-diff loop, in CI from day
  one even while it covers a single page.
- Provenance ledger + CI check (0002 §4 item 1).
- Go module skeleton with the boundaries drawn: `internal/ops` (sole importer
  of data planes), `internal/ingest`, `internal/markup` and sibling salvage
  packages, `internal/web`, `internal/api`.

Exit: CI runs all three gates; a dummy op renders a dummy page and appears in
a generated OpenAPI document.

### Phase 1 — the tracer bullet

One store → watcher (validated-prefix fold via the zit subprocess, zit#17
stat-and-reopen discipline from the start) → embedded NATS event → per-node
SQLite fold → one op (`timeline.list`) → one HTML page *and* its generated
API endpoint, footer with versions and render timing.

Proves the four load-bearing claims at minimum cost: log-as-event-stream,
embedded-index-as-derived-state, registry-generates-spec,
disposability-as-gate.

### Phase 2 — read plane breadth

0002 §4's extraction order: markup first (cleanroom sanitizer + conformance
corpus before anything renders) → base layout → fleet → repo home/files →
timeline/check-in → diff stack → graph → wiki render → search (FTS5) →
activity/stats. Each page lands against the per-page definition of done
(0002 §4 item 5) and grows the disposability snapshot set incrementally.

### Phase 3 — liveness

Datastar SSE wiring, chat transport + forum archival, notify integration.
First real test of Principle 3's refresh-fallback discipline.

### Phase 4 — sync fronting + leases

`/xfer` fronting, then the 0001 §3 lease machinery. Deliberately last in v1:
it is the only distributed-systems-hard part, and everything before it works
single-node.

v2/v3 then follow 0001 §10's staging. Each phase ends with the harness green
and a `docs/` page generated from the spec — documentation is a phase exit
criterion, not a v-final activity.

### Status (as built)

Phases 0–4's single-node scope is shipped and dogfooded from commit zero in a
zit store. Delivered beyond the original slice boundaries:

- **Reads**: fossil-grade DAG timeline (geometry verified by a browser
  lint), wiki + wiki edit history/revisions, tickets with folded state and
  cursor pagination, FTS5 search, check-in/diff/file surfaces — all served by
  the same ops the API exposes.
- **Writes** (needs zit's authoring verbs, zit#71): `wiki.save`,
  `ticket.new`, `ticket.set`, `forum.post`/`forum.reply`, `technote.add` plus
  HTML forms, read-your-writes. Every authoring op writes the cards the
  format allows: the technote C card (a title zit refuses to attach later, so
  it is write-once at save) and the forum N card — an omitted N card is
  `text/x-fossil-wiki` by fossil's rule, so quarry's markdown forms were
  producing posts that rendered as wiki markup (zit#77).
- **Accounts and per-repo roles** (0001 §8's v2 tables): argon2id verifiers,
  a site-admin flag, and read/write/admin per repository, managed by
  `quarry account`. `ops.Auth` is the required role, which gives the `Admin`
  level declared in v1 its meaning. Two credentials on purpose: accounts are
  the forge identity, while the RFC §6.3 shared secret stays the SYNC
  credential because the protocol fixes its derivation and the server must
  recompute it. The gate stays open until the first account exists, mirroring
  the sync-user policy, so local single-user serving is unaffected.
- **Attachments**: the A card folds to an `attachment` table and the content
  is served by a download route. Fossil models attachments as a log, not a
  set — newest card per filename wins and a card with no source is a deletion
  — so a naive resolve resurrects deleted files. Counts match fossil exactly
  (sqlite 11, tcl 3,333, deletions included). The route is gated by the index
  so it cannot serve arbitrary artifacts by hash, and is hostile to its own
  payload: download-only, octet-stream, nosniff, deny-all CSP.
- **Sync fronting**: `/r/{repo}` (+ `/xfer` alias) proxied to per-repo
  `zit serve`; push gating verifies §6 login per request. **Cross-implementation
  interop proven both directions with real Fossil 2.28**: fossil clones a
  zit-converted store through quarry (17,535 artifacts, integrity-clean), and
  a fossil-authored push lands in the zit store and folds into the index.
- **Liveness**: Datastar SSE on the timeline and the fleet page, plus an
  indexing badge — a first index runs for minutes, and without one a large
  repo reads as empty rather than as catching up. The badge exposed a latent
  bug worth recording: `data-init` sat on the element the stream patches, so
  the first patch morphed Datastar's own connection attribute away and the
  stream went silent. The timeline had it too, masked because it only sends
  on change and so appeared to work exactly once. The attribute now sits on a
  wrapper outside the patch target.
- **Docs**: `quarry docs` generates `docs/api.md` from the registry with a
  staleness gate, and `/docs` serves it.

- **Single-writer leases** (0001 §3, shipped): a NATS KV TTL lease per store,
  keyed by project code, gates both web/API write ops and sync pushes. Guard
  renews-or-acquires, applies a failover grace after displacing a holder,
  re-checks the lease at its commit point, and refuses writes when the
  coordinator is unreachable (correctness over availability). One embedded
  NATS is one node, so on a single node it holds its own lease with no grace
  delay — the machinery and its failure posture are identical at N nodes, which
  is what makes the rest of scale-out a deployment change, not a code change.

**Phase 4 is complete.** The remaining v1 horizon is the scale-out *trio*:

1. **External NATS URL** (shipped): `--nats <url>` joins an external cluster so
   N nodes share one event plane; the embedded server stays the single-node
   default. This forced the retention correction above (work-queue → limits+cap)
   so every node folds the full stream, and per-node durable names so nodes
   don't collide on one consumer. Proven live: two nodes on one external NATS
   over a shared store — each folds the whole store, the lease-holding node
   writes while the other returns 503 and still folds that write cross-node
   off the shared stream.

   The cap has a corollary that cost real data before it was found: ZITEVT
   discards OLD messages, and `watch()` published the whole census in one
   burst, so a repository larger than the cap deleted its own head while the
   indexer was still folding it. The index then drained what remained, saw
   `NumPending == 0`, and reported itself done. sqlite indexed 41,478 of
   131,558 artifacts that way and lost every ticket, wiki page and forum post
   with no error surfaced. Publishing now yields to folding every 64 MiB, so
   the stream applies backpressure instead of dropping; the regression test
   reproduces the original loss against a deliberately small stream.
2. **Shared S3-backed store access** (ZeroFS/9P per §storage): no quarry code —
   `--repo` already takes a path, and a 9P mount is just a path. The work is
   deployment (one ZeroFS process per bucket, supervised; 9P not NFS for the
   fsync durability contract zit's `log.commit()` depends on) and validation on
   a Linux+9P host, which this dev Mac can't exercise.
3. **Parquet interchange lake**: deferred by the tech-choices table ("no Parquet
   dependency in v1"; when the lake lands, write via a query engine's `COPY TO`,
   not a standalone library). It optimizes new-node hydration (read the lake
   instead of replaying every store) — an efficiency layer over a system that
   is already correct without it.

The trio turns the single-process build into N stateless read nodes plus one
leased writer. Part 1 is the code; parts 2 and 3 are deployment and a
v-next efficiency layer.

An adversarial audit of the write surface (5 dimensions, per-finding
verification) hardened it: it caught and we fixed a whitespace-card sync-gate
bypass (the gate now tokenizes exactly as zit does), ungated edit-form GETs,
and two input faults that 500'd instead of 400ing.

Interop and scale findings surfaced upstream zit work, all landed: authoring
verbs (zit#71), the sync client's per-round credential gap vs. the protocol's
own §6.5 cookie-login MUST (zit#74 — authenticated `zit push` authenticates
through quarry's gate end-to-end), a bare-store `-R` gap in `zit diff`
(zit#69), whole-second `D`-card resolution causing same-second writes to fold
in hash order rather than time order (zit#75), the technote/forum cards the
verbs could not write (zit#77), and `zit frames` reconstruction cost
(zit#84 — a plaintext memo plus parallel batch verification, ~31% off a cold
sqlite read).

Two remain open and shape what quarry can do next:

- **zit#85 — no way to read a project code.** quarry must learn it to derive
  the §6.3 shared secret, and the only route today is standing up a transient
  `zit serve` and syncing against it: a subprocess, a socket and a 5s worst
  case per repository at startup. Because `serve` is checkout-only, this is
  also the single thing keeping quarry from serving a BARE store — the read
  plane is already `-R`-only, with no `cmd.Dir` anywhere in `zitcli`.
- **zit#86 — the artifact format's §4.2 token grammar.** `token = 1*tchar`
  forbids the empty trailing token that fossil emits for an empty value, so a
  reader implementing the text literally rejects whole artifacts. quarry did:
  sqlite's 2,100 ticket changes and 1,009 empty-comment check-ins across the
  two corpora vanished into the content fallback. zit's reader is corrected;
  the RFC text still needs the erratum.

The lesson quarry keeps re-learning from these is in the failure mode, not the
bugs: an unparsable artifact falling back to opaque content is silent by
construction. Parity against fossil's own tables — `event`, `ticket`,
`forumpost`, `attachment` — is the only check that catches it, and it belongs
in CI against a real corpus rather than in a person's memory.

## 2. Technology choices

Each entry: choice, rationale, escape hatch. These are ADR-grade decisions
recorded compactly; split any into a standalone ADR when it accretes real
debate.

| Area | Choice | Rationale | Escape hatch |
|---|---|---|---|
| SQLite driver | `ncruces/go-sqlite3` | Upstream SQLite compiled to Wasm (wazero): real SQLite behavior, fast tracking of releases, no CGO (cross-compilation stays trivial), FTS5 enabled, and a **Go-implementable VFS surface** — the scale-out hook if node indexes ever want S3-backed hydration. Preferred over modernc (transpiler artifacts, historic concurrency edges) and mattn (CGO). Known costs: Wasm boundary tax on write-heavy paths (fits our read-dominated profile), amd64/arm64-only compiler (interpreter elsewhere), ~tens-of-ms startup compile, narrow bus factor (mitigated: unmodified upstream SQLite makes a forced move driver-swap-shaped, not data-shaped). | mattn (CGO) or modernc; schema is plain SQLite either way |
| HTTP router | stdlib `http.ServeMux` (1.22+ patterns) | The ops layer already does what chi would add; a router dependency is a shallow module here. | chi, mechanically |
| Ops registry | Hand-rolled on stdlib + generics (~1.5k lines total) | The registry is quarry's spine; you don't build the central abstraction on someone else's framework. The flat, form-expressible struct constraint (0001 §9) makes the reflector genuinely small. Pieces: generic `Register[In,Out]` over a descriptor map (~150 lines, explicit `RegisterAll()` wiring); JSON codec (stdlib); form/URL-values codec (~150 lines, needed for HTML regardless of framework); path-param binding onto ServeMux wildcards; OpenAPI reflector (~300–400 lines); tag-driven validator shared by both codecs; instrumentation wrapper around `Run`. | Huma (`danielgtaylor/huma`) — our `Run` signature is already its handler shape, so migration is mechanical |
| OpenAPI document types | Reuse dagnats' OpenAPI 3.1 synthesis types (currently `internal/openapi` in dagnats — promotion to a public package is a prerequisite, see §5) | One shared, tested document-assembly layer; quarry writes only the struct-reflection front half; both projects emit consistent 3.1 documents. | Hand-roll the back half too (~200 more lines) |
| Spec correctness gate | Off-the-shelf OpenAPI validator lints the generated spec in CI (dev-time tool, not a runtime dep); every registered op's fixtures round-trip through both codecs | Closes "our generator emits a subtly invalid spec" without a runtime framework. | — |
| Markup | goldmark + bluemonday + chroma | Per 0002 §2; sanitizer policy is cleanroom with the two-lineage conformance corpus + fuzz target. | — |
| Embedded NATS | `nats-server` as a library | The dagnats/notify pattern, already proven in this stack. | External NATS URL (already the scale-out flag) |
| litestream | Supervised sidecar | firestorm-proven shape; embedding buys nothing. | — |
| Parquet writing | Deferred with the interchange lake (0001: scale-out) | No Parquet dependency in v1 at all. When the lake lands, prefer writing via a query engine's `COPY TO` over a standalone parquet library — decide then. | — |
| Frontend | Datastar (only default-allowed framework, 0001 Principle 2) + `html/template` | Per 0001/0002. | Explicit per-case exception process |
| zit access | Subprocess boundary — the zit binary is the only store interface, reads and writes (0001 §2) | zit is Zig, quarry is Go; the versioned CLI is the whole contract, so releases decouple and quarry stays CGO-free. Most page data serves from quarry's indexes; on-demand reads (file-at-checkin, diff, annotate) pay ~1–5ms spawn, off the hot path and behind caching. Quarry's CI tests against pinned zit versions and each release declares a minimum zit version. | C-ABI read library exported from zit — revisit only with profiling evidence |

## 3. dagnats integration

dagnats provides durable HTTP endpoints (HTTP trigger → event-sourced run →
engine-resolved respond step) with OpenAPI 3.1 synthesis from trigger defs +
declared schemas. Division of labor:

- **Reads never route through dagnats.** Two reasons that outrank latency:
  durable execution *persists* every request as run history (page-view-rate
  event-log garbage, retention becomes a problem), and 0001 §3 commits to
  "NATS unreachable → reads continue" — reads through dagnats would make a
  NATS outage a forge outage. Query ops are in-process function calls over
  the local index.
- **Command ops that want durability get it inside `Run`.** Chat archival,
  webhook dispatch, CI verdict minting — anything with retry/DLQ semantics —
  launches a dagnats workflow behind the seam. The registry stays the single
  OpenAPI source; no split-brain between two generators.
- CI/CD (0001 §6) uses dagnats natively, as spec'd.

## 4. Testability — one strategy per risk class

- **Derived-state correctness**: the disposability harness (0001 §11) is the
  keystone; every page and API response feeds it.
- **Ops**: query ops are pure functions over index state → golden tests are
  *generated from the registry* (enumerate ops, run fixtures, snapshot). New
  op = automatic coverage.
- **The lease**: deterministic simulation. Abstract the clock and the KV
  behind interfaces so the 0001 §3 expiry-during-write interleavings can be
  exhaustively driven in simulated schedules — no sleeps, no flakes. This is
  the one place in the system where a rare race matters.
- **Sanitizer**: the two-lineage conformance corpus + fuzz target (0002 §2).
- **No-JS baseline**: nearly free — pages are server-rendered, so the
  snapshot harness *is* the no-JS test. Plus one template lint: every
  `data-on` has a form/anchor sibling.
- **Watcher**: property tests on cursor loss / prefix re-validation (a lost
  cursor must converge to identical index state — a mini-disposability
  property).

## 5. Observability

Lean on dagnats: quarry **imports dagnats' `observe` package** for OTel
exporter wiring, resource attributes, and conventions rather than building
its own setup. Command ops propagate W3C trace context into the workflows
they launch (dagnats' bridge already speaks tracecontext), so one stitched
trace runs from HTTP request through the durable run.

Quarry adds only what dagnats cannot see:

- The ops-layer wrapper: span + latency histogram + counter per op call,
  labeled by op name (in-process reads never touch dagnats).
- Three gauges: **ingest lag per store** (cursor vs. log head — the one
  number that says quarry is healthy), **lease state per store**, and
  **JetStream consumer lag**.
- Prometheus `/metrics`; structured `slog` throughout; the Principle-1 footer
  render timing is the same measurement plumbed to metrics — one clock, two
  consumers.

## 6. Prerequisites and upstream work

- **Fossil interop is a stated major priority — it is the adoption front
  door** (Fossil users are the first expected adopters). Current state,
  verified in the zit tree 2026-08-14: **wire-level interop already works** —
  zit clones from a live Fossil server over the RFC'd sync protocol with
  hand-verified identical leaves, and the sync path handles all artifact
  types (wiki, ticket, forum, event, attachment). The open gap is consuming
  a **bare `.fossil` file** without a running server (zit's
  `docs/plans/FOSSIL-IMPORT-PLAN.md` has the route analysis; zit#1 is the
  adjacent issue; not yet built). Consequence for quarry: a Fossil user's
  first contact is `zit clone` of their existing served repo — no converter
  — and quarry can even front Fossil repos as continuously-synced mirrors.
  The `.fossil`-file reader is upstream zit work, prioritized accordingly;
  quarry's fixture tooling should use the serve-and-clone path meanwhile.
- zit: #59 (append-append flock), #17 (inode-swap reopen), #53 (clone
  batching) — already filed, per 0001 §10.
- zit, new (falls out of the subprocess boundary): a **machine-consumable
  read-verb surface** — at minimum "emit artifacts since cursor N with
  prefix validation" for the watcher, plus stable, parseable output for
  file-at-checkin / diff / annotate. The watcher verb's payload is **raw
  artifacts in the RFC'd artifact format** (0001 §2): zit resolves delta
  chains and streams whole artifacts; no new output schema is invented, and
  the container layout stays private. Audited 2026-08-14: `stat`, `ls`,
  `cat`, `artifact --raw`, `diff --from/--to`, and `timeline` are already
  machine-consumable; the gaps were filed as zit#66 (the watcher verb) and
  zit#67 (annotate at a VERSION, full IDs). **Both shipped in zit 0.1.3**
  (verified 2026-08-15: append/no-new/invalid exit codes 0/3/4, bytes
  identical to `artifact --raw`, shun-reclaim kills the cursor while a
  no-op pack rightly does not). `internal/zitcli` wraps `frames`; the
  phase-1 blocker is cleared.
- dagnats: promote `internal/openapi` to a public package so quarry can
  import the 3.1 document types (§2). Small PR, no behavior change.
- notify: wiring per its own RFC; no changes requested.
- zit#70 (checkout registry) alignment, noted 2026-08-15: quarry's fleet
  registration stays ephemeral `--repo name=path` flags until zit's
  global config db (zit#70 phase 2, fossil's `~/.fossil` shape) exists —
  then `--discover` consumes zit's enumeration rather than quarry
  growing a persistent store-of-stores. Boundary: zit's registry owns
  "what stores exist and where" (host fact); quarry's v2 OLTP repo
  registry owns forge metadata only (serve-name, description, ACLs),
  seeded from zit's answer — never a second source of truth for
  locations. zit#59+#70 together also unblock bare-store serving, which
  retires the checkout-only constraint zit#69 documents.

## 7. Risks

- **ncruces bus factor** — mitigated by unmodified-upstream-SQLite (see §2).
- **Hand-rolled reflector correctness** — mitigated by the CI validator lint
  + codec round-trip tests (§2).
- **Synthetic fixture coverage** — the disposability harness is only as good
  as its corpus; the wiki/ticket/forum/chat-heavy fixture needs deliberate
  investment in phase 0, not incidental accretion.
- **Datastar no-JS discipline** — enforced by the template lint and per-page
  definition of done, but the first genuinely live page (chat) will test
  whether refresh-fallback stays honest under feature pressure.