Skip to content
quarry Artifact [e8679e1aa7]
Login

Artifact [e8679e1aa7]

Check-ins Using Download Hex Wiki

Artifact e8679e1aa7d4c18d99177cd15589ddce72263023ae3fa3c1647b3994e181064b:

# 0003 — quarry implementation plan

Companion to `0001-quarry-forge.md` (the spec) and `0002-forgejo-salvage.md`
(the UI salvage map). This document owns what the spec deliberately does not:
concrete technology choices, build order, testing strategy, and observability
wiring. Decisions here are revisable without touching the spec; decisions
that would change the spec's architecture go back through 0001.

## 0. Guiding constraints (inherited, not restated)

- 0001 Principles 1–7 govern every phase; the ops layer (0001 §9) is the seam
  everything routes through; the disposability test (0001 §11) is the merge
  gate from phase 0 onward.
- The smol contract: each package is a program that can be finished.
- 0002 §4 mechanics govern all Forgejo salvage (ledger first, cleanroom
  sanitizer, resolver inversion).

## 1. Build order

The organizing rule: **the spec's riskiest claims get proven by a tracer
bullet before any breadth work**, and the three enforcement mechanisms
(disposability harness, ops registry, provenance ledger) are built first,
because retrofitting a merge gate onto an existing codebase never works.

### Phase 0 — rails before features

- Ops registry core + OpenAPI generator + the import-graph CI assertion.
  Empty registry, one dummy op, spec generation working end to end.
- Disposability harness scaffold: fixture corpus of real converted stores
  (sqlite, tcl, plus a synthetic wiki/ticket/forum/chat-heavy fixture — the
  synthetic fixture is worth real effort; it is the corpus's coverage),
  ingest → snapshot → wipe → re-ingest → semantic-diff loop, in CI from day
  one even while it covers a single page.
- Provenance ledger + CI check (0002 §4 item 1).
- Go module skeleton with the boundaries drawn: `internal/ops` (sole importer
  of data planes), `internal/ingest`, `internal/markup` and sibling salvage
  packages, `internal/web`, `internal/api`.

Exit: CI runs all three gates; a dummy op renders a dummy page and appears in
a generated OpenAPI document.

### Phase 1 — the tracer bullet

One store → watcher (validated-prefix fold via the zit subprocess, zit#17
stat-and-reopen discipline from the start) → embedded NATS event → per-node
SQLite fold → one op (`timeline.list`) → one HTML page *and* its generated
API endpoint, footer with versions and render timing.

Proves the four load-bearing claims at minimum cost: log-as-event-stream,
embedded-index-as-derived-state, registry-generates-spec,
disposability-as-gate.

### Phase 2 — read plane breadth

0002 §4's extraction order: markup first (cleanroom sanitizer + conformance
corpus before anything renders) → base layout → fleet → repo home/files →
timeline/check-in → diff stack → graph → wiki render → search (FTS5) →
activity/stats. Each page lands against the per-page definition of done
(0002 §4 item 5) and grows the disposability snapshot set incrementally.

### Phase 3 — liveness

Datastar SSE wiring, chat transport + forum archival, notify integration.
First real test of Principle 3's refresh-fallback discipline.

### Phase 4 — sync fronting + leases

`/xfer` fronting, then the 0001 §3 lease machinery. Deliberately last in v1:
it is the only distributed-systems-hard part, and everything before it works
single-node.

v2/v3 then follow 0001 §10's staging. Each phase ends with the harness green
and a `docs/` page generated from the spec — documentation is a phase exit
criterion, not a v-final activity.

## 2. Technology choices

Each entry: choice, rationale, escape hatch. These are ADR-grade decisions
recorded compactly; split any into a standalone ADR when it accretes real
debate.

| Area | Choice | Rationale | Escape hatch |
|---|---|---|---|
| SQLite driver | `ncruces/go-sqlite3` | Upstream SQLite compiled to Wasm (wazero): real SQLite behavior, fast tracking of releases, no CGO (cross-compilation stays trivial), FTS5 enabled, and a **Go-implementable VFS surface** — the scale-out hook if node indexes ever want S3-backed hydration. Preferred over modernc (transpiler artifacts, historic concurrency edges) and mattn (CGO). Known costs: Wasm boundary tax on write-heavy paths (fits our read-dominated profile), amd64/arm64-only compiler (interpreter elsewhere), ~tens-of-ms startup compile, narrow bus factor (mitigated: unmodified upstream SQLite makes a forced move driver-swap-shaped, not data-shaped). | mattn (CGO) or modernc; schema is plain SQLite either way |
| HTTP router | stdlib `http.ServeMux` (1.22+ patterns) | The ops layer already does what chi would add; a router dependency is a shallow module here. | chi, mechanically |
| Ops registry | Hand-rolled on stdlib + generics (~1.5k lines total) | The registry is quarry's spine; you don't build the central abstraction on someone else's framework. The flat, form-expressible struct constraint (0001 §9) makes the reflector genuinely small. Pieces: generic `Register[In,Out]` over a descriptor map (~150 lines, explicit `RegisterAll()` wiring); JSON codec (stdlib); form/URL-values codec (~150 lines, needed for HTML regardless of framework); path-param binding onto ServeMux wildcards; OpenAPI reflector (~300–400 lines); tag-driven validator shared by both codecs; instrumentation wrapper around `Run`. | Huma (`danielgtaylor/huma`) — our `Run` signature is already its handler shape, so migration is mechanical |
| OpenAPI document types | Reuse dagnats' OpenAPI 3.1 synthesis types (currently `internal/openapi` in dagnats — promotion to a public package is a prerequisite, see §5) | One shared, tested document-assembly layer; quarry writes only the struct-reflection front half; both projects emit consistent 3.1 documents. | Hand-roll the back half too (~200 more lines) |
| Spec correctness gate | Off-the-shelf OpenAPI validator lints the generated spec in CI (dev-time tool, not a runtime dep); every registered op's fixtures round-trip through both codecs | Closes "our generator emits a subtly invalid spec" without a runtime framework. | — |
| Markup | goldmark + bluemonday + chroma | Per 0002 §2; sanitizer policy is cleanroom with the two-lineage conformance corpus + fuzz target. | — |
| Embedded NATS | `nats-server` as a library | The dagnats/notify pattern, already proven in this stack. | External NATS URL (already the scale-out flag) |
| litestream | Supervised sidecar | firestorm-proven shape; embedding buys nothing. | — |
| Parquet writing | Deferred with the interchange lake (0001: scale-out) | No Parquet dependency in v1 at all. When the lake lands, prefer writing via a query engine's `COPY TO` over a standalone parquet library — decide then. | — |
| Frontend | Datastar (only default-allowed framework, 0001 Principle 2) + `html/template` | Per 0001/0002. | Explicit per-case exception process |
| zit access | Subprocess boundary — the zit binary is the only store interface, reads and writes (0001 §2) | zit is Zig, quarry is Go; the versioned CLI is the whole contract, so releases decouple and quarry stays CGO-free. Most page data serves from quarry's indexes; on-demand reads (file-at-checkin, diff, annotate) pay ~1–5ms spawn, off the hot path and behind caching. Quarry's CI tests against pinned zit versions and each release declares a minimum zit version. | C-ABI read library exported from zit — revisit only with profiling evidence |

## 3. dagnats integration

dagnats provides durable HTTP endpoints (HTTP trigger → event-sourced run →
engine-resolved respond step) with OpenAPI 3.1 synthesis from trigger defs +
declared schemas. Division of labor:

- **Reads never route through dagnats.** Two reasons that outrank latency:
  durable execution *persists* every request as run history (page-view-rate
  event-log garbage, retention becomes a problem), and 0001 §3 commits to
  "NATS unreachable → reads continue" — reads through dagnats would make a
  NATS outage a forge outage. Query ops are in-process function calls over
  the local index.
- **Command ops that want durability get it inside `Run`.** Chat archival,
  webhook dispatch, CI verdict minting — anything with retry/DLQ semantics —
  launches a dagnats workflow behind the seam. The registry stays the single
  OpenAPI source; no split-brain between two generators.
- CI/CD (0001 §6) uses dagnats natively, as spec'd.

## 4. Testability — one strategy per risk class

- **Derived-state correctness**: the disposability harness (0001 §11) is the
  keystone; every page and API response feeds it.
- **Ops**: query ops are pure functions over index state → golden tests are
  *generated from the registry* (enumerate ops, run fixtures, snapshot). New
  op = automatic coverage.
- **The lease**: deterministic simulation. Abstract the clock and the KV
  behind interfaces so the 0001 §3 expiry-during-write interleavings can be
  exhaustively driven in simulated schedules — no sleeps, no flakes. This is
  the one place in the system where a rare race matters.
- **Sanitizer**: the two-lineage conformance corpus + fuzz target (0002 §2).
- **No-JS baseline**: nearly free — pages are server-rendered, so the
  snapshot harness *is* the no-JS test. Plus one template lint: every
  `data-on` has a form/anchor sibling.
- **Watcher**: property tests on cursor loss / prefix re-validation (a lost
  cursor must converge to identical index state — a mini-disposability
  property).

## 5. Observability

Lean on dagnats: quarry **imports dagnats' `observe` package** for OTel
exporter wiring, resource attributes, and conventions rather than building
its own setup. Command ops propagate W3C trace context into the workflows
they launch (dagnats' bridge already speaks tracecontext), so one stitched
trace runs from HTTP request through the durable run.

Quarry adds only what dagnats cannot see:

- The ops-layer wrapper: span + latency histogram + counter per op call,
  labeled by op name (in-process reads never touch dagnats).
- Three gauges: **ingest lag per store** (cursor vs. log head — the one
  number that says quarry is healthy), **lease state per store**, and
  **JetStream consumer lag**.
- Prometheus `/metrics`; structured `slog` throughout; the Principle-1 footer
  render timing is the same measurement plumbed to metrics — one clock, two
  consumers.

## 6. Prerequisites and upstream work

- **Fossil interop is a stated major priority — it is the adoption front
  door** (Fossil users are the first expected adopters). Current state,
  verified in the zit tree 2026-08-14: **wire-level interop already works** —
  zit clones from a live Fossil server over the RFC'd sync protocol with
  hand-verified identical leaves, and the sync path handles all artifact
  types (wiki, ticket, forum, event, attachment). The open gap is consuming
  a **bare `.fossil` file** without a running server (zit's
  `docs/plans/FOSSIL-IMPORT-PLAN.md` has the route analysis; zit#1 is the
  adjacent issue; not yet built). Consequence for quarry: a Fossil user's
  first contact is `zit clone` of their existing served repo — no converter
  — and quarry can even front Fossil repos as continuously-synced mirrors.
  The `.fossil`-file reader is upstream zit work, prioritized accordingly;
  quarry's fixture tooling should use the serve-and-clone path meanwhile.
- zit: #59 (append-append flock), #17 (inode-swap reopen), #53 (clone
  batching) — already filed, per 0001 §10.
- zit, new (falls out of the subprocess boundary): a **machine-consumable
  read-verb surface** — at minimum "emit artifacts since cursor N with
  prefix validation" for the watcher, plus stable, parseable output for
  file-at-checkin / diff / annotate. The watcher verb's payload is **raw
  artifacts in the RFC'd artifact format** (0001 §2): zit resolves delta
  chains and streams whole artifacts; no new output schema is invented, and
  the container layout stays private. Audited 2026-08-14: `stat`, `ls`,
  `cat`, `artifact --raw`, `diff --from/--to`, and `timeline` are already
  machine-consumable; the gaps were filed as zit#66 (the watcher verb) and
  zit#67 (annotate at a VERSION, full IDs). **Both shipped in zit 0.1.3**
  (verified 2026-08-15: append/no-new/invalid exit codes 0/3/4, bytes
  identical to `artifact --raw`, shun-reclaim kills the cursor while a
  no-op pack rightly does not). `internal/zitcli` wraps `frames`; the
  phase-1 blocker is cleared.
- dagnats: promote `internal/openapi` to a public package so quarry can
  import the 3.1 document types (§2). Small PR, no behavior change.
- notify: wiring per its own RFC; no changes requested.
- zit#70 (checkout registry) alignment, noted 2026-08-15: quarry's fleet
  registration stays ephemeral `--repo name=path` flags until zit's
  global config db (zit#70 phase 2, fossil's `~/.fossil` shape) exists —
  then `--discover` consumes zit's enumeration rather than quarry
  growing a persistent store-of-stores. Boundary: zit's registry owns
  "what stores exist and where" (host fact); quarry's v2 OLTP repo
  registry owns forge metadata only (serve-name, description, ACLs),
  seeded from zit's answer — never a second source of truth for
  locations. zit#59+#70 together also unblock bare-store serving, which
  retires the checkout-only constraint zit#69 documents.

## 7. Risks

- **ncruces bus factor** — mitigated by unmodified-upstream-SQLite (see §2).
- **Hand-rolled reflector correctness** — mitigated by the CI validator lint
  + codec round-trip tests (§2).
- **Synthetic fixture coverage** — the disposability harness is only as good
  as its corpus; the wiki/ticket/forum/chat-heavy fixture needs deliberate
  investment in phase 0, not incidental accretion.
- **Datastar no-JS discipline** — enforced by the template lint and per-page
  definition of done, but the first genuinely live page (chat) will test
  whether refresh-fallback stays honest under feature pressure.